I recently was greeted with a ValueError when using evaluation parameters in Great Expectations. If you hit a similar error, read on.

At the time of writing of this blog post, I was working on implementing data tests using Great Expectations. While trying out Evaluation Parameters in Expectations for the first time, I faced a cryptic ValueError message related to the return type of the parameter. Actually, I missed declaring it altogether. Though I resolved the issue looking at their documentation, I felt it is useful to note it in a blog post. Because most of us (especially me) turn to search for a solution online. Which most of the time saves the day. Imagine a software world without Stack Overflow! But, sometimes it helps to revisit the documentation and read through the lines well to understand if we did do things the right way. Let me tell you how missing doing that cost me.

---

What was the problem

Evaluation Parameters help in utilizing dynamic results in Great Expectations tests. My storage backend to store such results was Postgres SQL database. My select query to retrieve a particular value which worked perfectly well in the database refused to give the desired output when I used it in the Expectation. The “value” column of ge_metrics table stored the result of Expectation run as {“value”:55}. I want the number alone which I want to use it for further comparison. I setup a query as:

SELECT (value::json->>'value') as value FROM public.ge_metrics where metric_name = 'expect_table_row_count_to_equal.result.observed_value' order by run_time desc limit 1

I used this in the Expectation and ran it, but faced an error – ‘ValueError: Provided row count must be an integer’. This puzzled me as when I tested the same query in database, it showed a return type of integer. My first instinct was to reason that probably Great Expectations treats this as string and so I should type cast it. I quickly searched online, and modified the query as:
(value::json->>'value')::int
This did not work and nor did a few other such attempts.

---

What helped

Their documentation! First, I put a few debug statements to their library to check out what is happening. It told me that the query result is returning a list. I was confused and didn’t understand where the list was coming from. I looked back at the tutorial example I was referring to and found this line ‘By default, query results will be returned as a list. If instead you need a scalar for your expectation, you can specify the return_type’. And then it hit me, I did not declare a return_type for my query. Unfortunately, it did not throw any error but was just returning a list! To make me feel worse, I found that I had provided return_type for two other queries which I used but not for this.

Well, my miss was an obvious and minor one (or probably not) but the thing that struck me was why I did not go back to look at the tutorial example first instead of hurrying to Google. Guess this habit is my “default”!

---

You can read more about Qxf2’s adventures with Great Expectations here.

---