{"id":9530,"date":"2018-08-03T09:41:18","date_gmt":"2018-08-03T13:41:18","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=9530"},"modified":"2018-08-03T09:41:18","modified_gmt":"2018-08-03T13:41:18","slug":"quilt-local-and-production-packages","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/quilt-local-and-production-packages\/","title":{"rendered":"Dynamically import Quilt packages"},"content":{"rendered":"

This post is primarily aimed at Quilt<\/a> users. Read-only section 2 if you were looking for examples of dynamically importing Python modules. In this post, we will show you how to use the same code to interact with two similarly structured but different Quilt Packages. This is particularly useful when you use one Quilt package on your development environment and another (similarly structured) Quilt package in your production environment.<\/p>\n

 <\/p>\n

1. The problem<\/h4>\n

Qxf2<\/a> is developing a simple survey application for internal use only. We decided (reasons in section 5 below) to use Quilt to store our data instead of SQLite. We need to use one Quilt data package for our development and test environments and one Quilt data package for our production. The two packages would be similar in structure (same variables, dataframe columns, etc.) but have different data in them. With databases, you would store the database configuration in a settings file and read that into your code. However, a similar approach with Quilt poses an additional problem – Quilt scripts have an import statement that uses the name of the package explicitly. <\/p>\n

 <\/p>\n

2. Dynamic imports with Python<\/h4>\n

To solve the above problem of an import statement referencing a package name, we need to be able to dynamically import the module. This will help us use the same application code in both production and the development environments. There are two popular ways to do dynamic import in Python: __import__<\/code> and the importlib<\/code> module. We decided to go with the importlib<\/code> module since we found its syntax intuitive. For example,<\/p>\n

a. import foo<\/code> is simply foo = importlib.import_module('foo')<\/code>
\nb. import foo as blah<\/code> is simply blah = importlib.import_module('foo')<\/code>
\nc. from foo.bar import blah<\/code> is app = importlib.import_module('foo.bar.blah')<\/code>. This particular statement is what we will use to dynamically import the Quilt package.<\/p>\n

 <\/p>\n

3. The Quilt script without dynamic imports<\/h4>\n

In this example, we will write a Python script to create a new package (called jose), store a simple table and then push the package to Quilt. Without dynamic imports, the script would look something like the code below.<\/p>\n

\r\n\"\"\"\r\nThis file builds a Quilt package, inserts a 2-row table and pushes it to the Quilt repository\r\nWe are calling the new package 'jose' after my favourite chess player - Jose Raoul Capablanca\r\n\"\"\"\r\n\r\nimport quilt\r\n\r\ndef create_package():\r\n    \"Create a new quilt package\"\r\n    quilt.build(\"qxf2\/jose\")\r\n    from quilt.data.qxf2 import jose #This line is the limitation we are solving!\r\n    import pandas as pd\r\n\r\n    #Let us store a table\r\n    \"\"\"\r\n    Name, ID\r\n    Katerina, 001\r\n    Kady, 002\r\n    \"\"\"\r\n    columns = ['Name','ID']\r\n    employees = [[\"Katerina\",\"001\"],['Kady','002']]\r\n    employees_df = pd.DataFrame(data=employees, columns=columns)\r\n    jose._set(['employees'],employees_df)\r\n    quilt.build(\"qxf2\/jose\", jose)\r\n\r\n    #For non-teams, run 'quilt login' in your terminal before executing this script\r\n    # quilt.login(team=\"qxf2\") \r\n    quilt.push(\"qxf2\/jose\", is_public=True)\r\n\r\n\r\n#----START OF SCRIPT\r\nif __name__==\"__main__\":\r\n    create_package()\r\n<\/pre>\n
You will notice the problem is the line from quilt.data.qxf2 import jose<\/code> which hard codes both ‘qxf2’ (the repo name) and ‘jose’ (package name). If I wanted to use the package ‘jose’ in my development environment and the package ‘capa’ in production, I would need two different scripts. <\/p>\n
 <\/p>\n
4. The Quilt script with dynamic imports<\/h4>\n
To make the above code handle multiple environments (i.e., different packages), we would store the package name in a file that had different configuration values on your development and production environment. In our example, the file (let’s say we call it quilt_settings.py<\/code>) would have only one line PACKAGE_NAME=\"capa\"<\/code>. Then, change the import statement to look like: <\/p>\n
\r\n    app = importlib.import_module('{}.{}'.format(QUILT_DATA_PATH,package_name)) \r\n    #Now you can use app just like you would use the module jose in the previous example\r\n<\/pre>\n
Putting it all together, the code would look like this:<\/p>\n
\r\n\"\"\"\r\nThis file builds a Quilt package, inserts a 2-row table and pushes it to the Quilt repository\r\nWe are calling the new package 'capa' after my favourite chess player - Capablanca\r\n\"\"\"\r\n\r\nimport importlib\r\nimport quilt\r\nimport quilt_settings as conf\r\n\r\nQUILT_REPO= \"qxf2\"\r\n\r\ndef create_package(package_name):\r\n    \"Create a new quilt package\"\r\n    quilt.build(\"{}\/{}\".format(QUILT_REPO,package_name))\r\n    QUILT_DATA_PATH = \"quilt.data.{}\".format(QUILT_REPO)\r\n    app = importlib.import_module('{}.{}'.format(QUILT_DATA_PATH,package_name))\r\n    import pandas as pd\r\n\r\n    #Let us store a table\r\n    \"\"\"\r\n    Name, ID\r\n    Katerina, 001\r\n    Kady, 002\r\n    \"\"\"\r\n    columns = ['Name','ID']\r\n    employees = [[\"Katerina\",\"001\"],['Kady','002']]\r\n    employees_df = pd.DataFrame(data=employees, columns=columns)\r\n    app._set(['employees'],employees_df)\r\n    quilt.build(\"{}\/{}\".format(QUILT_REPO,package_name),app)\r\n\r\n    #or non-teams, run 'quilt login' in your terminal before executing this script\r\n    # quilt.login(team=\"qxf2\") \r\n    quilt.push(\"{}\/{}\".format(QUILT_REPO,package_name), is_public=True)\r\n\r\n#----START OF SCRIPT\r\nif __name__==\"__main__\":\r\n    #Read the package name from your quilt settings file\r\n    create_package(conf.PACKAGE_NAME)\r\n<\/pre>\n
That’s it! Now you have the exact same code working on both production and development environments while still using two different (but similarly structured) data packages on each environment.<\/p>\n
 <\/p>\n
5. Why we are using Quilt instead of a database <\/h4>\n
Because we really like using Python. I know, I know – “if all you have is a hammer, everything looks like a nail”. But in this case, we feel good about simply using Python to manage our data instead of designing tables, designing an ORM, managing configurations & deploys, thinking about different platforms and then adding extra steps to our setup documents, etc.  It helps that the application we are building will be small, not have too many users and the data is ideally suited to storing data in dataframes. <\/p>\n
 <\/p>\n
Disclaimer:<\/strong> We are using Quilt as a database. We aren’t sure that the creators of Quilt built Quilt for this use case. This is simply Qxf2 creatively using the data package manager to write small web applications. <\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
This post is primarily aimed at Quilt users. Read-only section 2 if you were looking for examples of dynamically importing Python modules. In this post, we will show you how to use the same code to interact with two similarly structured but different Quilt Packages. This is particularly useful when you use one Quilt package on your development environment and […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18,147],"tags":[],"class_list":["post-9530","post","type-post","status-publish","format-standard","hentry","category-python","category-quilt"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/9530","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=9530"}],"version-history":[{"count":14,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/9530\/revisions"}],"predecessor-version":[{"id":9551,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/9530\/revisions\/9551"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=9530"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=9530"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=9530"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}