Recently, Qxf2 has been transitioning to Rust and has been writing many blogs on various topics related to Rust. The main aim of these blogs is to document our progress. In this post, I am writing about how I used Py03/maturin tool to call a Rust library from Python. We have initiated a Rust-based project called “meme-generator” to generate attention-grabbing test results. This project began by defining a post-endpoint method using FastAPI and Uvicorn to run the app. The API endpoint of the FastAPI app converts JSON data to a string and passes it to a Rust function. The Rust function returns a result as a string. To achieve this, people usually go down the Foreign Function Interface (FFI) route. But we came across the Py03 crate and maturin which made FFI really easy.
Introduction to Py03:
PyO3 can be used to generate a native Python module for code written in Rust or to embed Python in a Rust binary. This allows developers to write Python code that can interface with Rust code. Py03 will take care of the interaction between Python and Rust and you will not have to worry about how it will translate Python String to something in C and then in Rust. This applies to the other data types as well like Integers, floats, lists, dictionaries, etc.
The maturin is a tool that simplifies the process of creating and distributing Rust-based Python extensions. The maturin tool compiles the Rust code and installs it as a Python module in your Virtual environment. After installation, you can import the module in the Python code. In this post, we will interface with maturin’s ‘develop’ command only.
Solution Overview:
Our solution spans Python and Rust. We will first outline what to do in Python land and then talk about changes to make with Rust code.
1) Python setup
In our project, we have a virtual environment setup. We have installed FastAPI, Uvicorn, and maturin.
2) FastAPI code
Created an “endpoint” directory under the project directory and create a Python file “GetMemes.py”. We imported all required packages, created a FastAPI instance, and defined a simple home endpoint. At this stage, our code looks something like this:
from json import JSONDecodeError from fastapi import FastAPI, Request,HTTPException import meme_generator app = FastAPI() v1 = FastAPI() @app.get("/") def home(): "Return the home page" return "Meme generator!" |
We call the Rust library via a Python module. You will notice we are importing a module called “meme_generator” which is yet to be created. We will be doing this by writing Rust code and using maturin. However, on the FastAPI side of things, we just need to add code like below. Here we have defined a Post method from which we call the Rust Function by passing test data in JSON as a String parameter.
@v1.post("/get-memes") async def get_meme_data(request : Request): """ Defined the Post method, from which we call the Rust function by passing test data in JSON as a String Parameter """ try: testreport = await request.json() total = testreport.get("total") passcnt = testreport.get("pass") # "get meme data" if total is not None and passcnt is not None: # Calling Rust function by passing testreport in JSON return {"url": meme_generator.get_meme(str(testreport))} raise HTTPException(405, detail="Provide total and pass count") except JSONDecodeError as exc: raise HTTPException(400, detail="Invalid data") from exc app.mount("/api/v1", v1) |
At this point, we are ready to start working on the Rust code.
3) Update Cargo.toml
Add Py03 crate to the “Cargo.toml” Rust configuration file under the dependencies section. Note: If you did not yet create a Rust project using Cargo new, you can use the command maturin init and avoid this step.
[package] name = "meme-generator" version = "0.1.0" edition = "2021" [dependencies] pyo3 = { version = "0.18.0", features = ["extension-module"] } |
4) Define the Python module meme_generator
Add a “lib.rs” file under your src directory. In lib.rs, we will define a Python module. We need py03 for that. To create a module, we use the #[pymodule] attribute. Similarly, we use the #[pyfunction] attribute to mark some Rust code as a Python function and then use the add_function() method to register the Python function with the Python module.
use pyo3::prelude::*; mod pipeline; use crate::pipeline::generate_meme; //the crate that contains the function we want to call #[pyfunction] fn get_meme(test_data: &str) -> String { let meme_url = generate_meme(test_data); return meme_url; } #[pymodule] fn meme_generator(_py: Python, m: &PyModule) -> PyResult<()> { m.add_function(wrap_pyfunction!(get_meme,m)?)?; Ok(()) } |
For the purposes of this blog, we will pretend that the only thing within crate::pipeline is this function.
///Function to generate meme url pub fn generate_meme(test_data: &str) -> String { return format!("https://imgflip.com/i/79o98d"); } |
5) Update Cargo.toml with the library name
Ensure the name of the library in the “Cargo.toml” matches the name of the function that was defined under #[pymodule]. In this example, I have mentioned below. “cdylib” is necessary to produce a shared library for Python to import from.
[lib] name = "meme_generator" crate-type = ["cdylib"] |
6) Finally run the maturin develop
Make sure you have your virtual environment running. Run the maturin develop command to build the package and install it into the Python virtual environment and the package is ready to be used from Python.
7) Run the FastAPI app
Start the app with uvicorn GetMemes:app –reload and the post method endpoint to request a meme.
Note:
a) Any changes to FastAPI/Python file, uvicorn will automatically reload with newly added changes.
b) Any changes to Rust source code will need you to re-run the maturin develop command.
8) Complete Source Code
We modified four files. Here is the complete source code for them.
Cargo.toml
=======
[package] name = "meme-generator" version = "0.1.0" edition = "2021" # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [lib] name = "meme_generator" crate-type = ["cdylib"] [dependencies] pyo3 = { version = "0.18.0", features = ["extension-module"] } |
GetMemes.py
=========
"""this is the post endpoint that takes testreport in JSON as String Parameter and returns the corresponding meme URL as a String""" from json import JSONDecodeError from fastapi import FastAPI, Request, HTTPException import meme_generator app = FastAPI() v1 = FastAPI() @app.get("/") def home(): "Return the home page" return "Meme generator!" @v1.post("/get-memes") async def get_meme_data(request : Request): """ Defined the Post method, from which we call Rust function by passing test data in JSON as a String Parameter """ try: testreport = await request.json() total = testreport.get("total") passcnt = testreport.get("pass") # "get meme data" if total is not None and passcnt is not None: return {"url": meme_generator.get_meme(str(testreport))} raise HTTPException(405, detail="Provide total and pass count") except JSONDecodeError as exc: raise HTTPException(400, detail="Invalid data") from exc app.mount("/api/v1", v1) |
lib.rs
===
use pyo3::prelude::*; mod pipeline; use crate::pipeline::generate_meme; #[pyfunction] fn get_meme(test_data: &str) -> String { let meme_url = generate_meme(test_data); return meme_url; } #[pymodule] fn meme_generator(_py: Python, m: &PyModule) -> PyResult<()> { m.add_function(wrap_pyfunction!(get_meme,m)?)?; Ok(()) } |
pipline.rs
=====
///Function to generate meme url pub fn generate_meme(test_data: &str) -> String { return format!("https://imgflip.com/i/79o98d"); } |
References:
1. Calling Rust from Python using PyO3
2. The PyO3 user guide
3. Py03
Hire Qxf2 for your testing needs
Qxf2 is the home of the technical tester. Our testers go well beyond traditional test automation. Contact us to hire software testers for your Rust project.
I have total of 18+ years of industry experience and worked in all the phases of project lifecycle. My career started as a manual tester, Test Lead and Program Test Manager. Recently I shifted to new company Qxf2, which provides testing services to startups. From Qxf2 joined one of the Qxf2 client where I got to know about Infrastructure as code. My hobbies are reading Telugu books, watching cricket and movies.
