{"id":19490,"date":"2023-09-05T08:22:40","date_gmt":"2023-09-05T12:22:40","guid":{"rendered":"https:\/\/qxf2.com\/blog\/?p=19490"},"modified":"2023-09-05T08:22:40","modified_gmt":"2023-09-05T12:22:40","slug":"dataset-and-model-evaluation-using-deepchecks","status":"publish","type":"post","link":"https:\/\/qxf2.com\/blog\/dataset-and-model-evaluation-using-deepchecks\/","title":{"rendered":"Dataset and Model Evaluation using Deepchecks"},"content":{"rendered":"

At Qxf2<\/a>, we have always been curious on how to test and validate the datasets and models. A good machine learning team should continuously monitor the model to identify any changes in model performance. You need to be confident that your models are accurate, reliable, and fair. Deepchecks<\/a> can help you achieve this by providing a comprehensive set of tools for validating your models and datasets. Deepchecks is a Python open-source tool used for testing and validation of machine learning models. This post outlines few steps required for thorough testing and validation of both machine learning models and datasets using Deepchecks.<\/p>\n

Setup and Installation<\/h3>\n

Installing Deepchecks For NLP is pretty straight forward. <\/p>\n

\r\npip install -U deepchecks[nlp]<\/pre>\n
About our Dataset and Model<\/h3>\n
Our dataset PTO_messages.csv<\/em> is a collection of text messages for Paid Time Off (PTO) and non-PTO. The labels denote binary classification, with 0 representing not PTO and 1 representing PTO-related messages. You can find our dataset here<\/a><\/p>\n
\r\ndata.head()\r\n<\/pre>\n
\"PTO<\/a>
PTO detector dataset<\/figcaption><\/figure>\n
For the purpose of this blog post, I have done few modifications to our Python script PTO_detector.py<\/em> to train on RandomForestClassifier<\/a> model to generate outputs for various checks.<\/p>\n
Note: This script PTO_detector.py<\/em> was originally trained on LinearSVC<\/a> model to classify the PTO or non-PTO.<\/p>\n
Deepchecks Suites<\/h3>\n
Deepchecks is bundled with a set of pre-built suites that can be used to run a set of checks on your data. There are 3 suites, the data integrity suite, the train test validation suite and the model evaluation suite. You can also run all the checks at once using the full_suite. These check sets help you make sure your data and models are reliable and accurate.<\/p>\n
Data Integrity<\/h4>\n
This is a check for data integrity that examines the correctness of text formatting. Its purpose is to verify if the dataset is both accurate and complete.<\/p>\n
Train Test Validation<\/h4>\n
The train-test validation suite is used to compare two datasets: the training dataset and the testing dataset. It ensures that the division between these two sets of data is accurate.<\/p>\n
Model Evaluation<\/h4>\n
The model evaluation suite runs after a training the model and requires model predictions. This check is to evaluate the performance of a model.<\/p>\n
Deepchecks data types<\/h3>\n
Deepcheck supports different data types commonly used in Machine Learning. <\/p>\n
  • Tabular: Handles data stored in tabular like the pandas DataFrame<\/li>\n
  • NLP: Handles Textual data<\/li>\n
  • Vision: Handles image datasets<\/li>\n
    In this post, I will be testing our Text classification model, focusing on NLP checks and suites that are relevant for this particular scenario. Please note that our dataset is designed for binary classification data with “Text” and “Labels”. <\/p>\n
    Implementing Deepchecks NLP<\/h3>\n
    Deepchecks NLP offers various data checks, such as data integrity and drift checks, which can work on any NLP task.
    \nTo execute deepchecks for NLP, you’ll need to create a     TextData<\/a> Object for both your training and testing data. The TextData Object serves as a container for your textual data, associated labels, and relevant metadata for NLP tasks.<\/p>\n
    Firstly, import the TextData from Deepchecks<\/p>\n
    \r\nfrom deepchecks.nlp import TextData<\/pre>\n
    You’ll need to create a TextData object for the train and test data as shown below<\/p>\n
    \r\ntrain = TextData(X_train, label=y_train, task_type='text_classification')\r\ntest = TextData(X_test, label=y_test, task_type='text_classification')<\/pre>\n
    The arguments required by the TextData is the train data, label, task_type and metadata(Optional). In the above code, the train object is created using training data (X_train) and their corresponding labels (y_train), the test object is created using testing data (X_test) and their corresponding labels (y_test) with the specified task type.<\/p>\n
    Data Integrity Checks<\/h3>\n
    Next, now you can run integrity suite on this TextData object:<\/p>\n
    \r\nfrom deepchecks.nlp.suites import data_integrity\r\ndata_integrity_suite = data_integrity()\r\ndata_integrity_suite.run(train, test)\r\n<\/pre>\n
    Deepchecks conducts data checks and outputs pass or fail based on default conditions. Now, let’s deep dive into the summary for more understanding. The summary displays few passed and failed results as shown below. Our focus will be on analysing the failed conditions and figure out the issues. Note<\/strong>: For this blog purpose, the process of cleaning data has been disabled in order to showcase the presence of unknown tokens within the output.<\/p>\n
    \"Data<\/a>
    Data Integrity checks<\/figcaption><\/figure>\n
    Unknown Tokens<\/h5>\n
    Unknown tokens are words, special characters, or emojis in the text that are not recognized by model’s tokenizer. The initial check (marked with the x in Didn’t Pass) tells us that the samples contains unexpected tokens in the dataset. Since the actual ratio is 0.03% and the check expects it to be 0%, the check has failed. This low ratio indicates that most of the text in the training dataset is already recognized and processed effectively by the tokenizer. In the instances of high ratio, you can simply pre-process the data or update your tokenizer to minimize the occurrence of unknown tokens.<\/p>\n
    \"Unknown<\/a>
    Unknown Tokens check<\/figcaption><\/figure>\n
    Conflicting Labels – Train Dataset and Text Dataset<\/h5>\n
    Train Dataset and Text Dataset<\/em><\/strong> – The second check identifies identical or nearly identical samples in the dataset that have different labels. This failed case indicates the presence of conflicting labels in the dataset. This insight is valuable as it points out inconsistencies in labelling to ensure accurate model training and evaluation. The check has not passed because the percentage of samples with inconsistent labels in train dataset is 5.06%, while the Test data contains 2.56%. This exceeds the defined threshold of 0%, indicating that there are cases where the same sample possesses different labels. <\/p>\n
    \"Data<\/a>
    Data Integrity checks – Conflicting Train labels<\/figcaption><\/figure>\n
    Text Duplicates – Train Dataset and Text Dataset<\/h5>\n
    This check examines the presence of duplicate sample in both Train and Test sets. Our output shows that the Train data contains 24.95% duplicate data<\/em>, while the Test data contains 7.69% duplicate data<\/em>. These percentages exceed the threshold of 5%, alarming potential data redundancy that needs to be fixed.<\/p>\n
    \"Data<\/a>
    Data Integrity checks – Text Duplicates<\/figcaption><\/figure>\n
    Similarly you can conduct checks for Text Property Outliers, Property Label Correlation, Special Characters, Under Annotated Metadata Segments etc checks. Please refer Deepchecks NLP<\/a> documentation for more details. By rectifying these issues, you enhance the quality of the data.<\/p>\n
    Train Test Evaluation<\/h3>\n
    Once you are confident that your data is all good to be trained, next step is to validate the split and compare train and test dataset.<\/p>\n
    \r\nfrom deepchecks.nlp.suites import train_test_validation\r\ntrain_test_validation().run(train, test)<\/pre>\n
    Train Test Samples Mix<\/h5>\n
    The output shows one failed and one passed condition in our case. This check tells us about the percentage of test data samples that also appear in the training data. The goal here is to maintain a balance and avoiding an overlap between the two sets. In our case, about 29.49% of the test data samples also appear in our training data. The desired percentage is lower than 5% and we are above that threshold. This information is very valuable for fine tuning our dataset and model’s evaluation. <\/p>\n
    \"Train<\/a>
    Train Test Evaluation<\/figcaption><\/figure>\n
    Upon closer examination specific test samples were found to have corresponding instances within the training set(46 out of 156). To address this you can apply different split techniques suitable for your requirements to reduce the data overlap. <\/p>\n
    Label Drift<\/h5>\n
    We also have a passed status for the “Label Drift” check. The Label Drift check acts as a measure to gauge the dissimilarity between the distributions of labels in the train and test datasets. It calculates the difference in label distributions between these datasets. The “Passed” status signifies that the label drift is within an acceptable range, with a minimal score of 0.15.<\/p>\n
    \"Train<\/a>
    Train Test Evaluation – Label Drift<\/figcaption><\/figure>\n
    This check also provides insights into the distributions, showing the top 10 categories with the most significant differences between the two datasets. Similarly, you an conduct Embedding Drift, NLP Property Drift checks. Please refer Deepchecks NLP<\/a> documentation for more details<\/p>\n
    Model Evaluation<\/h3>\n
    The model evaluation suite, is designed to be run after a model has been trained and requires model predictions and probabilities which can be supplied via the arguments in the run function.<\/p>\n
    \r\nfrom deepchecks.nlp.suites import model_evaluation\r\nmodel_evaluation().run(train, test,    \r\n                                train_predictions=train_preds,\r\n                                test_predictions=test_preds,\r\n                                train_probabilities=train_probs,\r\n                                test_probabilities=test_probs)\r\n<\/pre>\n
    Train Test Performance<\/h5>\n
    The condition “Train Test Performance” failed in the output. The train test degradation score is a measure of how much a model’s performance drops when moving from the training dataset to the testing dataset. It’s an important metric to evaluate how well a model performs on the unseen data.<\/p>\n
    \"Model<\/a>
    Model Evaluation – Train Test Performance<\/figcaption><\/figure>\n
    The output also highlights that there are 2 scores that failed this degradation test. The most significant degradation observed is for the “Recall” metric, specifically for class 1 with a maximum degradation of 12.28%. It signifies a notable drop in the model’s ability to correctly identify instances belonging to class 1 when applied to new or unseen data. This could imply that the model is not maintaining consistent performance levels across training and testing data.<\/p>\n
    Prediction Drift<\/h5>\n
    The condition for “Prediction Drift” passed in the output. It is indicating the outcome of monitoring the model for prediction drift using the Kolmogorov-Smirnov (KS)<\/a> drift score. The KS drift score reported is 0.14. Even though the KS drift score of 0.14 is slightly lower than the specified threshold 0.15, it is important to analyze the source of this drift. Compare the distribution of predictions at various time points to pinpoint when the drift started occurring. Consider retraining or fine-tuning the model to ensure that the model remains effective and reliable in its predictions over time.
    \n
    \"Model<\/a>
    Model Evaluation \u2013 Prediction Drift<\/figcaption><\/figure><\/p>\n
    Full Suite<\/h3>\n
    Deepchecks gives a quick overview of your model and data with Full Suite that includes many of the implemented checks. full_suite is a collection of the prebuilt checks.
    \nTo import the full_suite method, use the code below<\/p>\n
    \r\nfrom deepchecks.nlp.suites import full_suite\r\nsuite = full_suite()\r\nsuite.run(train_dataset=train,\r\n    test_dataset=test,\r\n    with_display=True,\r\n    train_predictions=train_preds,\r\n    test_predictions=test_preds,\r\n    train_probabilities=train_probs,\r\n    test_probabilities=test_probs)\r\n<\/pre>\n
    This will perform full checks on the dataset objects and the model and generates a consolidated output with Passed\/Didn’t Pass.<\/p>\n
    You can find the final code here<\/a>. <\/p>\n
    Using Deepchecks, helps us check and fine tune our models and data. It’s user-friendly and offers different tests to catch issues early, making our models work better. Happy Testing!!<\/p>\n\n
    References<\/h4>\n
    1. Deepchecks Documentation<\/a>
    \n2.     Deepchecks NLP: ML Validation for Text Made Easy<\/a>
    \n3.     High-standard ML validation with Deepchecks<\/a><\/p>\n
    \n
    Hire technical testers from Qxf2<\/h4>\n
    Qxf2 offers valuable expertise in testing Machine Learning projects. Our team goes beyond traditional QA to ensure data quality, model accuracy, and system robustness. With us, you’ll have skilled testers who understand the nuances of ML and can enhance the reliability of your projects. Get in touch<\/a> with us!<\/p>\n
    \n","protected":false},"excerpt":{"rendered":"
    At Qxf2, we have always been curious on how to test and validate the datasets and models. A good machine learning team should continuously monitor the model to identify any changes in model performance. You need to be confident that your models are accurate, reliable, and fair. Deepchecks can help you achieve this by providing a comprehensive set of tools […]<\/p>\n","protected":false},"author":16,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[379,387,385,386],"tags":[],"class_list":["post-19490","post","type-post","status-publish","format-standard","hentry","category-deepchecks","category-model-evaluation-deepchecks","category-nlp","category-train-test-split"],"_links":{"self":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/19490","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/comments?post=19490"}],"version-history":[{"count":143,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/19490\/revisions"}],"predecessor-version":[{"id":20040,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/posts\/19490\/revisions\/20040"}],"wp:attachment":[{"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/media?parent=19490"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/categories?post=19490"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/qxf2.com\/blog\/wp-json\/wp\/v2\/tags?post=19490"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}