{"id":4109,"date":"2016-07-07T04:04:12","date_gmt":"2016-07-07T08:04:12","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=4109"},"modified":"2018-04-03T10:36:23","modified_gmt":"2018-04-03T14:36:23","slug":"sqlite-getting-started-guide","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/sqlite-getting-started-guide\/","title":{"rendered":"Get started with SQLite and Python"},"content":{"rendered":"<p>There are times when we repurpose our automated checks to scrape information off the pages it visits. This can happen when you are using automation to explore a product, looking for patterns in client data, analyzing production data to spot patterns, etc. <\/p>\n<p>Storing the scraped data can be tricky. Approaches to storing the scraped data range between two extremes:<br \/>\na) [lightweight] writing out log files<br \/>\nb) [heavyweight] using a full-fledged relational database<\/p>\n<p>The data that you end up scraping usually has some structure. E.g.: a table. Log files have the drawback of being purely string based and unable to capture the structure within the data. Full-fledged relational databases have the drawback of needing setup, design and maintenance. Using SQLite to store your data, lies somewhere between the two extreme options. SQLite is a flat file database that needs no extra setup. But it still affords us the convenience of querying the data with SQL commands. In this post, we show you an example of scraping data from an HTML table and storing it in a SQLite table.<\/p>\n<p><strong>NOTE:<\/strong> We want to emphasize that using SQLite is just one option. Whether it suits you or not depends on your specific situation. <\/p>\n<hr>\n<h3>What is SQLite?<\/h3>\n<p>SQLite is a server-less, lightweight, disk-based relational database management system that does not require any installation. The absence of any intermediate server allows it to be embedded in any application or, in our case, a test. The different data type values stored in the database can be accessed through a variant of the SQL query language. Click <a href=\"https:\/\/www.sqlite.org\/datatype3.html\"> here<\/a> to know more about SQL data types  <\/p>\n<hr>\n<h3>Setup<\/h3>\n<p>Getting setup with SQLite is easy. Python versions 2.5 and above ship a module called <strong>sqlite3<\/strong> by default. So you need to do nothing extra. To test if you have the sqlite3 module, try this statement in a Python interpreter <code>import sqlite3<\/code>. If you get no import errors, you are all set.<\/p>\n<hr>\n<h3>Working with SQLite<\/h3>\n<p>Any module that lets you connect to a database exposes a few standard operations like creating and connecting to a database, adding a table, inserting rows, updating rows, fetching\/selecting data, closing a connection, etc. In the remainder of this post, I&#8217;ll show you how to perform these operations with sqlite3.<\/p>\n<p>To use the sqlite3 module a connection object has to be created. You can do that like this:<\/p>\n<pre lang=\"python\">\r\n  connection_obj = sqlite3.connect(db_name)\r\n<\/pre>\n<p><strong>db_name<\/strong> can be any name you like. In case there is no database, one is created. <\/p>\n<p>Once we have the connection object, a cursor object has to be created. The cursor object is the one that executes the SQL commands.<\/p>\n<pre lang=\"python\">\r\n  cursor_obj = connection_obj.cursor()\r\n<\/pre>\n<p>Now that we have a cursor object, we can begin executing SQL commands. Let&#8217;s try creating a table in a SQLite database,  insert values from the <a href=\"https:\/\/qxf2.com\/selenium-tutorial-main\">Tutorial Page<\/a> and later fetch a value from the table.<br \/>\n<b>Create a table:<\/b><\/p>\n<pre lang='python'>\r\nimport sqlite3\r\ndef create_table(db_name,table_name,table_header):\r\n\t\"Create a table\"\r\n\ttry:\r\n\t\tconnection_obj = sqlite3.connect(db_name)\r\n\t\tcursor_obj = connection_obj.cursor()\r\n\t\tcursor_obj.execute(\"CREATE TABLE %s%s\"%(table_name,tuple(table_header)))\r\n\t\tconnection_obj.commit()\r\n\t\tconnection_obj.close()\r\n\t\tprint \"-Created Table %s\"%table_name\r\n\texcept Exception,e:\r\n\t\tprint \"Python says:\",str(e)\r\n<\/pre>\n<p>The <code>cursor_obj.execute(\"CREATE TABLE %s%s\"%(table_name,tuple(table_header)))<\/code> command creates a new table. The <code>connection_obj.commit()<\/code> and <code>connection_obj.close()<\/code>saves and closes the database.<\/p>\n<p><b>Insert values into the table:<\/b><\/p>\n<pre lang='python'>\r\ndef insert_values(db_name,table_name,values):\r\n\t\"Insert values into the table\"\r\n\ttry:\r\n\t\tconnection_obj = sqlite3.connect(db_name)\r\n\t\tcursor_obj = connection_obj.cursor()\r\n\t\tcursor_obj.execute(\"INSERT INTO %s VALUES (?,?,?,?)\"%table_name,tuple(values))\r\n\t\tconnection_obj.commit()\r\n\t\tconnection_obj.close()\r\n\texcept Exception,e:\r\n\t\tprint \"Python says:\",str(e)\r\n<\/pre>\n<p>The <code>cursor_obj.execute(\"INSERT INTO %s VALUES (?,?,?,?)\"%table_name,tuple(values))<\/code> is used to insert values into the table. A placeholder<code>(?)<\/code> is used in the execute statements.Use <code>?<\/code> as a placeholder wherever you want to use a value, and then provide a tuple of values.It is preffered over the usual python string formatters. It helps you avoid <a href=\"https:\/\/xkcd.com\/327\/\">SQL injection attacks<\/a>. Once the table is created and the values are inserted into the table the <code>connection_obj.commit()<\/code> command is used to save the changes and <code>connection_obj.close()<\/code> is used to close the connection.<br \/>\n<b>Fetch value form the table:<\/b><\/p>\n<pre lang='python'>\r\ndef fetch_value(db_name,table_name,required_key,key,value):\r\n\t\"Fetch a value from the table\"\r\n\ttry:\r\n\t\tconnection_obj = sqlite3.connect(db_name)\r\n\t\tcursor_obj = connection_obj.cursor()\r\n\t\tcursor_obj.execute(\"SELECT %s FROM %s WHERE %s=? \"%(required_key,table_name,key),(value,)) #Note: The string formatting is done differently(<code>(value,)<\/code>) to prevent SQL injection attack.\r\n\t\tname = cursor_obj.fetchone()[0] #To get the first element in the resulting list\r\n\t\tprint \"-The %s from the Table:'%s' where %s='%s' is: %s\"%(required_key,table_name,key,value,name)\r\n\texcept Exception,e:\r\n\t\tprint \"Python says:\",str(e) \r\n\t\r\n<\/pre>\n<p>The execute statement <code>cursor_obj.execute(\"SELECT %s FROM %s WHERE %s=? \"%(required_key,table_name,key),(value,))<\/code> searches through the table to find a value corresponding to the condition. The <code>fetchone()<\/code> command is used to fetch a single value from the table whereas the <code>fetchall()<\/code> can be used to fetch all the values from the table.<\/p>\n<p><b>Putting it all together:<\/b><\/p>\n<pre lang='python'>\r\n\"\"\"\r\nQxf2 Services: Utility script to create a SQLite table\r\nNOTE: This was written up for a blog post\/tutorial \r\nWe use a cleaned up and more robust version at clients\r\n\"\"\"\r\n\r\nfrom selenium import webdriver\r\nimport sqlite3\r\n\r\n\r\ndef test_create_table():\r\n\t\"Create a table, insertable and fetch values from the table\"\r\n\tdriver = webdriver.Firefox()\r\n\tdriver.get(\"https:\/\/qxf2.com\/selenium-tutorial-main\")\r\n\tdriver.maximize_window()\r\n\tdb_name = \"example.db\"\r\n\ttable_name = \"test_table\"\r\n\ttable_header = ['Name','Email','Phone','Gender']\r\n\r\n\t#Create a table\r\n\tcreate_table(db_name=db_name,table_name=table_name,table_header=table_header) \r\n\tfor i in range(1,4):\r\n\t\trow = driver.find_elements_by_xpath(\"\/\/tbody\/descendant::tr[%d]\/td\"%i)\r\n\t\ttable_values = []\r\n\t\tfor data in row:\r\n\t\t\tvalue = data.text #To get the text property of the element\r\n\t\t\ttable_values.append(value.encode('utf-8'))\r\n\t\t\r\n\t\t#Insert values into the table\t\r\n\t\tinsert_values(db_name=db_name,table_name=table_name,values=table_values)\r\n\t\r\n\t#Fetch value from the table\t\r\n\tfetch_value(db_name=db_name,table_name=table_name,required_key='Name',key='Email',value='williams@example.com')  \r\n\tdriver.quit()\r\n\r\n\r\ndef create_table(db_name,table_name,table_header):\r\n\t\"Create a table\"\r\n\ttry:\r\n\t\tconnection_obj = sqlite3.connect(db_name)\r\n\t\tcursor_obj = connection_obj.cursor()\r\n\t\tcursor_obj.execute(\"CREATE TABLE %s%s\"%(table_name,tuple(table_header)))\r\n\t\tconnection_obj.commit()\r\n\t\tconnection_obj.close()\r\n\t\tprint \"-Created Table %s\"%table_name\r\n\texcept Exception,e:\r\n\t\tprint \"Python says:\",str(e)\r\n\r\n\r\ndef insert_values(db_name,table_name,values):\r\n\t\"Insert values into the table\"\r\n\ttry:\r\n\t\tconnection_obj = sqlite3.connect(db_name)\r\n\t\tcursor_obj = connection_obj.cursor()\r\n\t\tcursor_obj.execute(\"INSERT INTO %s VALUES (?,?,?,?)\"%table_name,tuple(values))\r\n\t\tconnection_obj.commit()\r\n\t\tconnection_obj.close()\r\n\texcept Exception,e:\r\n\t\tprint \"Python says:\",str(e)\r\n\r\n\r\ndef fetch_value(db_name,table_name,required_key,key,value):\r\n\t\"Fetch a value from the table\"\r\n\ttry:\r\n\t\tconnection_obj = sqlite3.connect(db_name)\r\n\t\tcursor_obj = connection_obj.cursor()\r\n\t\tcursor_obj.execute(\"SELECT %s FROM %s WHERE %s=? \"%(required_key,table_name,key),(value,))\r\n\t\tname = cursor_obj.fetchone()[0]\r\n\t\tprint \"-The %s from the Table:'%s' where %s='%s' is:\\n %s\"%(required_key,table_name,key,value,name)\r\n\texcept Exception,e:\r\n\t\tprint \"Python says:\",str(e) \r\n\r\n\r\nif __name__ == '__main__':\r\n\ttest_create_table()\r\n<\/pre>\n<p><strong>NOTE:<\/strong> To make this a class and reduce the repeated code, you can take a look at the example we have in our <a href=\"https:\/\/qxf2.com\/blog\/getting-started-with-mongodb-and-python\/\">MongoDB tutorial<\/a><\/p>\n<p>When you run this file your output would look like this,<br \/>\n<a href=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2016\/07\/test_run2.png\" data-rel=\"lightbox-image-0\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2016\/07\/test_run2.png\" alt=\"test_run\" width=\"545\" height=\"57\" class=\"aligncenter size-full wp-image-4229\" srcset=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2016\/07\/test_run2.png 545w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2016\/07\/test_run2-300x31.png 300w\" sizes=\"auto, (max-width: 545px) 100vw, 545px\" \/><\/a><\/p>\n<hr>\n<p>There you have it, a getting started guide to the sqlite3 module in Python and a detailed example on how to implement it with tests.<\/p>\n<p><strong>If you liked this article, learn more <a href=\"https:\/\/qxf2.com\/blog\/about-qxf2\/\">about Qxf2&#8217;s<\/a> testing services for startups.<\/strong><\/p>\n<hr>\n<h3>References:<\/h3>\n<p>1. SQLite3- Official documentation : <a href=\"https:\/\/docs.python.org\/2\/library\/sqlite3.html\">SQLite3<\/a><br \/>\n2. TutorialsPoint : <a href=\"http:\/\/www.tutorialspoint.com\/sqlite\/sqlite_python.htm\">SQLite Python Tutorial<\/a><br \/>\n3. Python Central: <a href=\"http:\/\/pythoncentral.io\/introduction-to-sqlite-in-python\/\">introduction-to-sqlite-in-python<\/a><\/p>\n<hr>\n<script>(function() {\n\twindow.mc4wp = window.mc4wp || {\n\t\tlisteners: [],\n\t\tforms: {\n\t\t\ton: function(evt, cb) {\n\t\t\t\twindow.mc4wp.listeners.push(\n\t\t\t\t\t{\n\t\t\t\t\t\tevent   : evt,\n\t\t\t\t\t\tcallback: cb\n\t\t\t\t\t}\n\t\t\t\t);\n\t\t\t}\n\t\t}\n\t}\n})();\n<\/script><!-- Mailchimp for WordPress v4.10.1 - https:\/\/wordpress.org\/plugins\/mailchimp-for-wp\/ --><form id=\"mc4wp-form-1\" class=\"mc4wp-form mc4wp-form-6165 mc4wp-form-theme mc4wp-form-theme-blue\" method=\"post\" data-id=\"6165\" data-name=\"Newsletter\" ><div class=\"mc4wp-form-fields\"><div style=\"border:3px; border-style:dashed;border-color:#56d1e1;padding:1.2em;\">\r\n  <h1 style=\"text-align: center; padding-top: 20px; padding-bottom: 20px; color: #592b1b;\">Subscribe to our weekly Newsletter<\/h1>\r\n  <input style=\"margin: auto;\" type=\"email\" name=\"EMAIL\" placeholder=\"Your email address\" required \/>\r\n  <br>\r\n  <p style=\"text-align: center;\">\r\n    <input style=\"background-color: #890c06 !important; border-color: #890c06;\" type=\"submit\" value=\"Sign up\" \/>\r\n    \r\n  <\/p>\r\n  <p style=\"text-align: center;\">\r\n    <a href=\"http:\/\/mailchi.mp\/c9c7b81ddf13\/the-informed-testers-newsletter-20-oct-2017\"><small>View a sample<\/small><\/a>\r\n  <\/p>\r\n  <br>\r\n<\/div><\/div><label style=\"display: none !important;\">Leave this field empty if you're human: <input type=\"text\" name=\"_mc4wp_honeypot\" value=\"\" tabindex=\"-1\" autocomplete=\"off\" \/><\/label><input type=\"hidden\" name=\"_mc4wp_timestamp\" value=\"1776317363\" \/><input type=\"hidden\" name=\"_mc4wp_form_id\" value=\"6165\" \/><input type=\"hidden\" name=\"_mc4wp_form_element_id\" value=\"mc4wp-form-1\" \/><div class=\"mc4wp-response\"><\/div><\/form><!-- \/ Mailchimp for WordPress Plugin -->\n<hr>\n","protected":false},"excerpt":{"rendered":"<p>There are times when we repurpose our automated checks to scrape information off the pages it visits. This can happen when you are using automation to explore a product, looking for patterns in client data, analyzing production data to spot patterns, etc. Storing the scraped data can be tricky. Approaches to storing the scraped data range between two extremes: a) [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[38,18,30],"tags":[],"class_list":["post-4109","post","type-post","status-publish","format-standard","hentry","category-automation","category-python","category-selenium"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/4109","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=4109"}],"version-history":[{"count":93,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/4109\/revisions"}],"predecessor-version":[{"id":6232,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/4109\/revisions\/6232"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=4109"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=4109"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=4109"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}