{"id":3230,"date":"2015-10-28T05:22:06","date_gmt":"2015-10-28T09:22:06","guid":{"rendered":"http:\/\/qxf2.com\/blog\/?p=3230"},"modified":"2015-10-28T05:22:06","modified_gmt":"2015-10-28T09:22:06","slug":"getting-started-with-mongodb-and-python","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/getting-started-with-mongodb-and-python\/","title":{"rendered":"Getting started with MongoDB and Python"},"content":{"rendered":"<p>I was recently playing around with <a href=\"https:\/\/www.mongodb.org\/\">MongoDB<\/a>. It took me a little while to get setup and write some simple Python scripts to access it. I thought I would write up a summary and share my learning.<br \/>\n <\/p>\n<hr>\n<h3>Why this post?<\/h3>\n<p>Companies have begun <a href=\"https:\/\/www.mongodb.com\/leading-nosql-database\">adopting MongoDB<\/a> at a good clip. I think, interacting with a NoSQL database will soon become a regular action for most testers. I am summarizing my learning so testers who are new to MongoDB can save time by having one place to get setup and understand how to interact with MongoDB.<\/p>\n<hr>\n<h3>What is MongoDB?<\/h3>\n<p>MongoDB is a <a href=\"https:\/\/en.wikipedia.org\/wiki\/NoSQL\">NoSQL<\/a> database. MongoDB does not have the concept of tables static columns. Instead MongoDB is a set of &#8216;collections&#8217;. Each &#8216;collection&#8217; is made of &#8216;documents&#8217;. The documents are very similar to JSON. MongoDB is useful when you don&#8217;t know the exact structure of your data upfront. It is also useful when you don&#8217;t want to specify your schema upfront. MongoDB makes it extremely easy to modify new fields (have you tried adding new columns in RDBMS land?). While I did do a lot more reading online to get comfortable with the idea, at this stage of my learning, the key takeaways for me were:<\/p>\n<ul>\n<li style=\"text-align: left;\">the database is organized into documents and collections<\/li>\n<li style=\"text-align: left;\">no strict enforcement of table constraints<\/li>\n<\/ul>\n<hr>\n<h3>Tutorial overview<\/h3>\n<p>In this tutorial, we&#8217;ll cover installation, setup and basic CRUD operations. You need access to an Ubuntu machine to follow along.<\/p>\n<p><strong>1. Install MongoDB on Ubuntu<\/strong><br \/>\nTo install Mongo on Ubuntu, you need to first let the Aptitude package manager know where to look for the MongoDB installer. You can do that by running these commands:<\/p>\n<pre lang=\"bash\">\r\nsudo apt-key adv --keyserver hkp:\/\/keyserver.ubuntu.com:80 --recv 7F0CEB10\r\necho 'deb http:\/\/downloads-distro.mongodb.org\/repo\/ubuntu-upstart dist 10gen' | sudo tee \/etc\/apt\/sources.list.d\/mongodb.list\r\nsudo apt-get update\r\n<\/pre>\n<p>You can then install MongoDB with:<\/p>\n<pre lang=\"bash\">\r\nsudo apt-get install -y mongodb-org\r\n<\/pre>\n<p>By default, MongoDB binds itself to 127.0.0.1 and port 27017. It stores its data files in <em>\/var\/lib\/mongodb<\/em> and its configuration file is in <em>\/etc\/mongod.config<\/em>.<\/p>\n<p>You can test your installation by simply typing in <em>mongo<\/em> (or &#8216;sudo mongo&#8217;) in your terminal. You should see a something like the image below if things went well.<\/p>\n<p><a href=\"http:\/\/qxf2.com\/blog\/wp-content\/uploads\/2015\/10\/mongo_1.png\" data-rel=\"lightbox-image-0\" data-rl_title=\"\" data-rl_caption=\"\" title=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/qxf2.com\/blog\/wp-content\/uploads\/2015\/10\/mongo_1.png\" alt=\"mongodb install\" width=\"550\" height=\"79\" class=\"aligncenter size-full wp-image-3335\" srcset=\"https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2015\/10\/mongo_1.png 550w, https:\/\/qxf2.com\/blog\/wp-content\/uploads\/2015\/10\/mongo_1-300x43.png 300w\" sizes=\"auto, (max-width: 550px) 100vw, 550px\" \/><\/a> <\/p>\n<p><strong>Reference:<\/strong> <a href=\"http:\/\/www.liquidweb.com\/kb\/how-to-install-mongodb-on-ubuntu-14-04\/\">Liquid Web: How to install MongoDB<\/a><\/p>\n<hr>\n<p><strong>2. Start and stop MongoDB<\/strong><br \/>\nYou can start|stop|restart MongoDB like this:<\/p>\n<pre lang=\"bash\">\r\nsudo service mongod start\r\nsudo service mongod stop\r\nsudo service mongod restart\r\n<\/pre>\n<hr>\n<p><strong>3. Create a new database<\/strong><br \/>\nMongo uses the <strong>use<\/strong> command to create a database if it is not already present. Sigh &#8211; I found this pretty confusing. Pull up your <a href=\"https:\/\/docs.mongodb.org\/getting-started\/shell\/client\/\">MongoDB shell<\/a> by typing <em>mongo<\/em> in your terminal. Let us create a database called <em>qxf2<\/em>:<\/p>\n<pre lang=\"bash\">\r\nuse qxf2\r\n<\/pre>\n<p>If things went well, you will see the message &#8216;switched to db qxf2&#8217;<\/p>\n<hr>\n<p><strong>4. Create a new collection<\/strong><br \/>\nA collection is something similar to a table in SQL. Well, not exactly &#8211; but it is a useful mental model to have when you start. Let us create a collection called <em>Employees<\/em> by typing the command below in your Mongo shell:<\/p>\n<pre lang=\"bash\">\r\ndb.createCollection('Employees')\r\n<\/pre>\n<p>If things went well, you will see the message &#8216;{ &#8220;ok&#8221; : 1 }&#8217;<\/p>\n<hr>\n<p><strong>5. Add a new user<\/strong><br \/>\nIts easy to add a user to MongoDB through the Mongo shell. Let us add me as a user to the qxf2 database.You simply need something similar to the following command:<\/p>\n<pre lang=\"bash\">\r\ndb.createUser({\"user\":\"raji\", \"pwd\":\"Testdb123\", \"roles\":[{\"role\":\"dbAdmin\",\"db\":\"qxf2\"}]})\r\n<\/pre>\n<p>If things went well, you will see the message &#8216;Successfully added user:&#8217;<\/p>\n<hr>\n<p>Now you are setup to learn how to connect to your MongoDB database using Python. But I strongly encourage you to <a href=\"https:\/\/docs.mongodb.org\/manual\/reference\/mongo-shell\/\">revisit the Mongo shell<\/a> and try other commands too!<\/p>\n<hr>\n<p><strong>6. Install PyMongo<\/strong><br \/>\nPyMongo is the recommended Python module to work with MongoDB. You can install it by:<\/p>\n<pre lang=\"bash\">\r\npip install \u2013U pymongo\r\n<\/pre>\n<hr>\n<p><strong>7. Connect to MongoDB<\/strong><br \/>\nThere are a couple of ways to connect to the MongoDB instance using Python. We&#8217;ll show you both. <\/p>\n<pre lang=\"python\">\r\nimport pymongo\r\n\r\nhost = \"127.0.0.1\"\r\nport = \"27017\"\r\nuser = \"raji\"\r\npassword = \"Testdb123\"\r\ndb_name = \"qxf2\"\r\nauth_mechanism = \"MONGODB-CR\"\r\nuri = \"mongodb:\/\/\"+user+\":\"+password+\"@\"+host+\":\"+port+\"\/\"+db_name+\"?\"+\"authMechanism=\"+auth_mechanism\r\nclient = pymongo.MongoClient(uri) #client object to connect\r\n\r\n#Logic to obtain the db object\r\ndb = None\r\nif db_name in client.database_names():\r\n    db = client[db_name] \r\n#And now you have a db object to interact with\r\n<\/pre>\n<p>The other way to go about connecting is:<\/p>\n<pre lang=\"python\">\r\nimport pymongo\r\n\r\nhost = \"127.0.0.1\"\r\nport = \"27017\"\r\nuser = \"raji\"\r\npassword = \"Testdb123\"\r\ndb_name = \"qxf2\"\r\n\r\nuri = host+\":\"+port\r\nclient = pymongo.MongoClient(uri)\r\ndb = client[db_name]\r\ndb.authenticate(user,password)\r\n#And now you have a db object to interact with\r\n<\/pre>\n<hr>\n<p><strong>8. Insert a new document<\/strong><br \/>\nWe have created a collection called &#8216;Employees&#8217; already. Let us insert a document into a collection with <\/p>\n<pre lang=\"python\">\r\ndb = client[database] \r\ncollection = db[collection_name] \r\ncollection.insert(document)\r\n<\/pre>\n<hr>\n<p><strong>9. Update an existing document<\/strong><br \/>\nYou can update an existing document like this:<\/p>\n<pre lang=\"python\">\r\ndb = client[database] \r\ncollection = db[collection_name] \r\ncollection.update(doc_key,doc_update)\r\n<\/pre>\n<hr>\n<p><strong>10. Delete a document<\/strong><br \/>\nYou probably guessed the call to delete a document looks like this:<\/p>\n<pre lang=\"python\">\r\ndb = client[database] \r\ncollection = db[collection_name] \r\ncollection.remove(doc_key)\r\n<\/pre>\n<hr>\n<p><strong>11. Putting it all together <\/strong><br \/>\nLet&#8217;s put it all together into a useful MongoDriver class.<\/p>\n<pre lang=\"python\">\r\n\"\"\"\r\nQxf2 Services: Utility script to connect to MongoDB\r\nNOTE: This was written up for a blog post\/tutorial \r\nWe use a cleaned up and more robust version at clients\r\n\"\"\"\r\n\r\nimport pymongo\r\n\r\nclass Mongo_Driver:\r\n    \"Class to connect to MongoDB\"\r\n    def __init__(self,host,port):\r\n        \"Constructor\"\r\n        self.host = host\r\n        self.port = port\r\n\r\n\r\n    def connect(self,database,username,password,auth_mechanism=None):\r\n        \"Connect to MongoDB database\"\r\n        #src: https:\/\/docs.mongodb.org\/manual\/core\/authentication\/#security-authentication-mechanisms\r\n        if auth_mechanism is None:\r\n            self.auth_mechanism = \"MONGODB-CR\"#by default authMechanism=MONGODB-CR , change if required\r\n        else:\r\n            self.auth_mechanism = auth_mechanism\r\n\r\n        #Construct the connection URI \"mongodb:\/\/user:password@hostnameOrIP:port\/database?authMechanism=MONGODB-CR\"\r\n        self.uri = \"mongodb:\/\/\"+username+\":\"+password+\"@\"+self.host+\":\"+self.port+\"\/\"+database+\"?\"+\"authMechanism=\"+self.auth_mechanism\r\n        self.client = pymongo.MongoClient(self.uri)\r\n        self.db = None\r\n        result_flag = False\r\n        try:\r\n            if database in self.client.database_names():\r\n                self.db = self.client[database] \r\n            if self.db is not None:\r\n                result_flag = True\r\n        except Exception,e:\r\n            print '\\nException when connecting to Mongo instance'\r\n            print 'PYTHON SAYS:'\r\n            print e\r\n            print '\\n'\r\n\r\n        return result_flag\r\n\r\n\r\n    def connect2(self,database,username,password):\r\n        \"Another way to connect to the database\"\r\n        self.uri = self.host+\":\"+self.port\r\n        self.client = pymongo.MongoClient(self.uri)\r\n        self.db = self.client[database]\r\n        result_flag = self.db.authenticate(username,password)\r\n\r\n        return result_flag\r\n\r\n        \r\n    def insert_document(self,document,collection_name):\r\n        \"Insert a document into a collection\"\r\n        result_flag = False\r\n        try:\r\n            self.collection = self.get_collection(collection_name)\r\n            self.collection.insert(document)\r\n        except Exception,e:\r\n            print '\\nException in insert_document'\r\n            print 'PYTHON SAYS:'\r\n            print e\r\n            print '\\n'\r\n        else:\r\n            result_flag = True\r\n\r\n        return result_flag\r\n\r\n\r\n    def get_collection(self,collection_name):\r\n        \"Return the collection object with name collection_name\"\r\n        self.collection = None\r\n        try:\r\n            self.collection = self.db[collection_name] \r\n        except Exception,e:\r\n            print \"Collection %s not found\"%collection_name\r\n\r\n        return self.collection\r\n\r\n        \r\n    def update_document(self,doc_key,doc_update,collection_name):   \r\n        \"Update a document based on its key\"\r\n        result_flag = False\r\n        try:\r\n            self.collection = self.get_collection(collection_name)\r\n            result_obj = self.collection.update(doc_key,doc_update)\r\n            if result_obj['updatedExisting'] is True:\r\n                result_flag = True\r\n        except Exception,e:\r\n            print '\\nException in update_document'\r\n            print 'PYTHON SAYS:'\r\n            print e\r\n            print '\\n'\r\n\r\n        return result_flag\r\n\r\n            \r\n    def delete_document(self,doc_key,collection_name):\r\n        \"Delete a given document\"\r\n        result_flag = False\r\n        try:\r\n            self.collection = self.get_collection(collection_name)            \r\n            result_obj = self.collection.remove(doc_key)\r\n            if result_obj['n'] is 1:\r\n                result_flag = True\r\n        except Exception,e:\r\n            print '\\nException in delete_document'\r\n            print 'PYTHON SAYS:'\r\n            print e\r\n            print '\\n'\r\n        else:\r\n            return result_flag\r\n\r\n\r\n#---EXAMPLE USAGE---\r\nif __name__=='__main__':    \r\n    'user created with role:readWrite - Example: db.createUser({\"user\":\"raji1\",\"pwd\":\"Testdb123\",\"roles\":[{\"role\":\"readWrite\" OR \"dbAdmin\",\"db\":\"qxf2\"}]}) '\r\n    #mongo connection string parameters (read these from config file)\r\n    host = \"127.0.0.1\"\r\n    port = \"27017\"\r\n    user = \"raji\"\r\n    password = \"Testdb123\"\r\n    db_name = \"qxf2\"\r\n    collection = \"Employees\"\r\n    document = {\"name\":\"Jack\",\"address\":[{\"area\":\"NRI Layout\",\"city\":\"blr\",\"pin\":\"560016\",\"state\":\"KA\"},{\"area\":\"NRI Layout\",\"city\":\"blr\",\"pin\":\"560016\",\"state\":\"KA\"}],\"contact\":\"90000000006\",\"emp_id\":\"0007\"}\r\n    document_key = {\"name\":\"Jack\"}\r\n    \r\n    #initialize mongo driver object\r\n    mongo_obj = Mongo_Driver(host,port)\r\n\r\n    #connect to a db\r\n    if mongo_obj.connect(db_name,user,password) is True:\r\n        print \"DB authorization successful\"\r\n        #insert document\r\n        if mongo_obj.insert_document(document,collection) is True:\r\n            print \"insert document successful\"\r\n        else:\r\n            print \"insert document unsuccessful\"\r\n\r\n        #update document\r\n        if mongo_obj.update_document(document_key,document,collection) is True:\r\n            print \"update document successful\"\r\n        else:\r\n           print \"update document unsuccessful\"\r\n        \r\n        #delete document\r\n        if mongo_obj.delete_document(document_key,collection) is True:\r\n            print \"delete document successful\"\r\n        else:\r\n            print \"delete document unsuccessful\"\r\n    else:\r\n        print \"DB authorization unsuccessful\"\r\n\r\n<\/pre>\n<hr>\n<p>I hope this helps you get started with MongoDB. Questions? Comments? Post them in the comments section below and I will get back to you soon.<\/p>\n<hr>\n","protected":false},"excerpt":{"rendered":"<p>I was recently playing around with MongoDB. It took me a little while to get setup and write some simple Python scripts to access it. I thought I would write up a summary and share my learning. Why this post? Companies have begun adopting MongoDB at a good clip. I think, interacting with a NoSQL database will soon become a [&hellip;]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[100,18],"tags":[95,96,24,98,97],"class_list":["post-3230","post","type-post","status-publish","format-standard","hentry","category-mongodb","category-python","tag-mongodb","tag-pymongo","tag-python-2","tag-python-mongo","tag-pythonmongodb"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/3230","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=3230"}],"version-history":[{"count":80,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/3230\/revisions"}],"predecessor-version":[{"id":3353,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/3230\/revisions\/3353"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=3230"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=3230"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=3230"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}