{"id":6736,"date":"2017-10-09T07:34:29","date_gmt":"2017-10-09T11:34:29","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=6736"},"modified":"2018-04-02T10:21:30","modified_gmt":"2018-04-02T14:21:30","slug":"quilt-data-package-manager","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/quilt-data-package-manager\/","title":{"rendered":"Quilt – a Data Package Manager"},"content":{"rendered":"

We have been testing data-rich applications for a long time. And like any experienced tester, we realize how difficult it is to create, maintain and update data every time the data model changes. So we were excited to come across Quilt<\/a>, a data package manager, via Hacker News<\/a>. We were thrilled that it integrated well with our favorite programming language – Python. So we set about exploring Quilt to see if it could be ‘GitHub for data’. In this post, we will talk about basic operations that we performed on data and how we could access the older data versions using Quilt. We will write a follow up soon with a step by step guide on how to integrate Quilt with your test data.<\/p>\n

Note:<\/strong> Quilt talks about storing binary data and therefore not duplicating too much data. As testers, we don’t (yet) care too much about this cool feature. But we are sure if you are a developer or data analyst you will care about this feature.<\/p>\n\n

The need for data versioning<\/h3>\n

We like creating our own data for testing data-heavy applications. This presents a few problems<\/p>\n

1. It is hard to maintain test data when the underlying data model changes. Over time, in any data-rich application, the data model undergoes changes. For example, new fields are added, existing fields get deleted, split or combined. All this means that the test data needs to evolve. We end up writing separate scripts to create data and then to maintain the data. <\/p>\n

2 It is very difficult to intelligently name files. As a tester, our instinct is not to design and use databases for our data. Instead, we try to rely on flat files. So if you are like us, you end up a few dozen (or hundred) files named intelligently. To us, this approach has always felt like the poor man’s version control.<\/p>\n

Data versioning can solve both these problems. In many cases, simply using git (or your version control tool) is not an option when you have large amounts of data. So we are glad to find version control that is specifically designed for data. <\/p>\n

What we especially liked about Quilt, was that it integrates so well with Python. Quilt uses Python pandas’ DataFrame as the default data structure. This makes reading and (especially) editing data so easy! If you have ever tried editing specific columns of a large csv (yeah, we know about fileinput<\/a>) you will know why we are so happy.<\/p>\n\n

Quilt<\/h3>\n

Quilt is a data package manager which is a versioned bundle of serialized data wrapped in a Python module. Phew! That was a mouthful! We are new to Quilt, so our understanding may be wrong. But here is our mental model for Quilt. Quilt takes your data and:
\na) converts it into a special format (serialized binary)
\nb) magically transforms your data into a Python module (wrapped in a Python module)
\nc) lets you commit\/store the data in a version control system (versioned)<\/p>\n

Once you run your dataset (or a portion of your dataset) through Quilt, you get what is called a ‘data package’. A data package is an abstraction that encapsulates and automates data preparation. By packaging the dataset, you can easily reuse it and manage the different versions. Quilt comes with a command-line utility that builds, pushes, and installs data packages. In this post, we will be covering below areas and show different operations which we have tried using Quilt.<\/p>\n