Python Mechanize – the missing manual

Problem: The online documentation for Mechanize in Python is lacking. Case in point, this question on stackoverflow remained unanswered until we added the answer. This post hopes to provide you with the key missing pieces.

We love Python at Qxf2 Services. We chose the Mechanize module to test REST services and automate a lot of our test setup tasks by using REST end points that are used in the products. One of the BIG drawbacks of using the Python module Mechanize is the sparse online support. Let’s hope this post begins to change that.


1. How to encode parameters in Python

#Parameters can be URL encoded like this:
my_params={key1:'value1',key2:'value2'} 
my_params_encoded = urllib.urlencode(my_params)
#For JSON parameters do this:
my_params={key1:'value1',key2:'value2'} 
my_params_encoded = json.dumps(my_params)

2. How to do a HTTP GET with Mechanize and Python
Mechanize supports HTTP GET and POST methods out of the box.

url = "your desired url"
#NOTE: browser is the instance of mechanize.Browser()
browser.open(mechanize.Request(url,data=my_params_encoded,headers={"Content-type":"application/json"}))

Bonus: We just showed you how to make a request with an application/json header

3. How to do a HTTP PUT with Mechanize and Python
Mechanize (the Python module) supports only HTTP GET and POST methods out of the box. We arrived at this solution by digging into the mechanize code packages to find out where mechanize was setting the HTTP method. We noticed that when we call mechanize.Request, we are using the Request class in _request.py which in turn is extending the Request class in _urllib2_fork.py. The http method is actually set in get_method of the Request class in _urllib2_fork.py. Turns out get_method in _urllib2_fork.py was allowing only GET and POST methods. To get past this limitation, we ended up writing my own put and delete classes that extended mechanize. Request but over-rode get_method() only.

class PutRequest(mechanize.Request):
    "Extend the mechanize Request class to allow a http PUT"
    def get_method(self):
        return "PUT"
 
 
#NOTE: browser is the instance of mechanize.Browser()
browser.open(PutRequest(url,data=sub_level_params_encoded,headers={"Content-type":"application/json"}))

4. How to do a HTTP DELETE with Mechanize and Python

class DeleteRequest(mechanize.Request):
    "Extend the mechanize Request class to allow a http DELETE"
    def get_method(self):
        return "DELETE"
 
 
#NOTE: browser is the instance of mechanize.Browser()
browser.open(DeleteRequest(url))

Other useful links to learn Mechanize+Python:
1. If you are new to Mechanize, use this link
2. Using Mechanize to login to CAS

If you have questions regarding any of these solutions, drop a comment below. Qxf2 like helping testers!


WARNING: Mechanize has its limitations as an automated testing tool. The biggest drawback for web testers is that Mechanize does not mimic rendering of executed JavaScript in the browser. So DOM elements that are rendered by executing JavaScript cannot be accessed by Mechanize.


Sign up for our monthly newsletter




Arunkumar Muralidharan
I want to find out what conditions produce remarkable software. A few years ago, I chose to work as the first professional tester at a startup. I successfully won credibility for testers and established a world class team. I have lead the testing for early versions of multiple products. Today, I run Qxf2 Services. Qxf2 provides software testing services for startups. If you are interested in what Qxf2 offers or simply want to talk about testing, you can contact me at: mak@qxf2.com. I like testing, math, chess and dogs.

© 2013-2017, Arunkumar Muralidharan. All rights reserved.

2 Comments

  1. Anonymous said:

    Very helpful.Thank you.

    February 26, 2015
    Reply

Leave a Reply

Your email address will not be published.