Once a machine learning model is developed and its accuracy and related metrics have been thoroughly examined, it might seem like the model is ready for real-world deployment. However in reality this is hardly the case. Major part of testing begins when the model is integrated into the application it was designed for. We at Qxf2 Services feel most of the ML projects don’t pay enough attention to this crucial phase. This is where testing plays an important role.

In this blog post, we’ll delve into the crucial aspect of testing an ML model by sharing our insights on successfully testing some ML Projects. For some practical references we will also be using a comment classification model discussed in one of our previous blog post. We’ll cover different testing approaches, from treating the model like an application to planning better regression tests for ML projects. Additionally, we’ll also discuss on different types of data suitable for testing ML models.

Why this post?

When testers are tasked with testing ML models, it’s common to focus solely on evaluation metrics and scores during the planning phase. We want to encourage testers and developers to broaden their perspective. We should closely pay attention to how it integrates into the larger application system and also think about what kind of testing need to be performed at different phases.

Testing approaches for Machine Learning Models:

The approaches that we are listing are not definitive or exhaustive. We are in the process of learning how to enhance our ML model testing skills. Based on our knowledge and experience of testing ML models, we are sharing some ideas we have developed. Also these approaches can vary depending on the context. When we initially began testing ML models, our focus was on evaluation metrics like Accuracy, Precision, Recall, and F1 score. While these metrics are crucial, a comprehensive testing approach requires a more thorough examination. Testing an ML project isn’t vastly different from testing regular projects. Along with your regular practices its also good to consider some of these additional approaches which we are listing below.

1) Testing the Model as an Application:

Approaching a machine learning model as an application involves assessing it similarly to any other software application. This approach helps ensure that the model not only performs well in terms of accuracy but also seamlessly integrates into the larger software ecosystem. Consider the following aspects when treating your model like an application during testing

1.1) Input Validation:

Just like in any application, we need to validate the inputs to our model. Ensure that the data you’re providing to the model during testing is representative of what it will encounter in real-world scenarios. This includes handling different data types, data ranges, and any potential outliers. In case of our comment classification model other than the regular inputs we can also consider below inputs.

1.2) Edge Cases:

Test edge cases rigorously as edge cases can often lead to unexpected behaviour. For a classification model, this could include testing comments with unusual structures or uncommon language patterns.

1.3) Integration:

Ensure that the model integrates seamlessly with other part of application. If the model interacts with other components or databases, make sure those interactions work correctly during testing. We won’t delve deeply into this aspect, assuming that most of you are already familiar with it.

1.4) Scalability:

Check how the model scales. If it’s part of an application that will receive a large number of requests simultaneously, test its performance under load. How does it handle concurrent requests, and does it maintain its accuracy and response time?
Maintaining a record and utilising a dashboard that consolidates metrics and logs of Model artefacts such as the frequency of model calls and the corresponding results, can prove beneficial. In one of our previous assignments we used Prometheus and Grafana to monitor and track the model performance and prediction results. This turned out to a great tool for monitoring our Model and application performance metrics.

1.5) Error Handling:

Examine the model’s error-handling capabilities. Evaluate its response when confronted with data it can’t classify, data processing errors, or component failures. In an application we tested for monitoring and predicting the next data signal, the model encountered issues with all-zero data points or when it stopped receiving signals. Incorporate similar tests to ensure the model is capable of handling such errors.

1.6) User Experience:

Consider the end-user perspective. If the model’s output is displayed to users, verify that it is presented in a user-friendly manner. Assess how users engage with the model’s predictions and whether it aligns with their expectations. From our experience, there were instances where the user interface (UI) wasn’t equipped to handle certain missing prediction scenarios, leading to the display of incorrect information to users. In some cases, we overlooked scenarios to effectively communicate the model’s failures.

1.7) Monitoring and Logging:

Set up monitoring and logging, just as you would do for any other application. Track the model’s performance, gather data on its predictions, and log any issues or errors for post-deployment analysis.

2) Testing for Model Refinement

One of the challenging aspects of ML model testing is assessing whether the model’s performance improves with newer versions. As a tester, determining this can be complex. The most intuitive approach would be to design a ‘good’ labelled dataset and then calculate the evaluation score like the F1 score for the model under test. While a score would be useful it’s essential to design a convenient method for quickly running these tests with diverse data and evaluating results. So some things which are important to think over here are

2.1) Designing a good testing dataset

This dataset should ideally differ from the training and testing datasets used during model development. In a previous project, we constructed this dataset over time based on customer feedback and our analysis of production data. It can be derived from production data where the model performs sub optimally, or synthetic data representing ideal cases to identify model failures.

2.2) Explore an Efficient Method for Result Evaluation

Represent the test results in a format that facilitates easy comparison with previous test runs, enabling quick assessments. Displaying the outcomes on a graph proves beneficial, allowing anyone reviewing the graphs to evaluate whether the model’s performance has improved or declined. Additionally, having an overall score provides a helpful metric for making informed judgements.

2.3) Snapshot Tests

Snapshot tests ensure the consistency of a system’s output over time. To illustrate, consider our snapshot test example in a practice testing AIML project.

snapshot.assert_match(f"{accuracy},{precision},{recall},{f1_score}","overall_score.txt")

Using the assert_match function, we validate performance metrics such as accuracy, precision, recall, and F1 score against a reference score saved in ‘overall_score.txt’. The reference file is generated during the initial snapshot test run. Subsequent runs compare the model’s current output with the saved reference output to identify unexpected changes. If the assert fails, a decision is made to update the snapshot file if the new snapshot has an improved score.

Conclusion:

In a journey of exploring different ways to test Machine Learning model we are continuously learning new things. As we navigate through different challenges like looking beyond evaluation scores, assessing performance across iterations, the importance of a thoughtfully designed testing dataset becomes evident. We hope some of the strategies listed above will enhance your perspectives on testing ML models. Embrace the testing journey!

Hire QA from Qxf2 Services

Qxf2 comprises adept technical testers well-versed in traditional testing methodologies and equipped to tackle the unique challenges of testing modern software systems. Our proficiency extends to testing microservices, data pipelines, and AI/ML-driven applications. Qxf2 engineers excel in working independently and thrive within compact engineering teams. Feel free to contact us here.

Avinash Shetty

I am a dedicated quality assurance professional with a true passion for ensuring product quality and driving efficient testing processes. Throughout my career, I have gained extensive expertise in various testing domains, showcasing my versatility in testing diverse applications such as CRM, Web, Mobile, Database, and Machine Learning-based applications. What sets me apart is my ability to develop robust test scripts, ensure comprehensive test coverage, and efficiently report defects. With experience in managing teams and leading testing-related activities, I foster collaboration and drive efficiency within projects. Proficient in tools like Selenium, Appium, Mechanize, Requests, Postman, Runscope, Gatling, Locust, Jenkins, CircleCI, Docker, and Grafana, I stay up-to-date with the latest advancements in the field to deliver exceptional software products. Outside of work, I find joy and inspiration in sports, maintaining a balanced lifestyle.

Related posts:

Robustness Testing of Machine Learning Models Testers, practice testing a machine learning algorithm! Qxf2’s Data analytics, Machine Learning and AI roadmap