Result Analytics provide insights into the results and outcome metrics of individual QueryPair executions at the row-, cell-, and column-level. This allows you to gain a deeper understanding of your data, and identify specifics points of failures within each test.
Viewing Results Analytics
Results Analytics are available in both Design-Time Runs and Scenario executions.
From a Design-Time Run, open a QueryPair and navigate to the Design-Time Run tab. Click the View Query Results button to open the QueryPair results modal and navigate to the Results Analytics tab.
Similarly, you can view these metrics in the Run Dashboard by selecting a Scenario from the Scenario Queue, and either double-clicking a Test Suite from the Test Suite Details grid or double-clicking a QueryPair from the Activity grid.
Understanding the Analytics
There's a lot going on in the Results Analytics tab, so let's dive into each section to get a better understanding of what it all means.
Row Metrics
As the name implies, the Row Metrics table deals specifically with failures at the row-level. Any data failure in any column in a given row, as well as instances where rows are missing from either the Source and/or Target dataset, are considered failures for the purposes of this table.
Let's start with the table itself.
We get a snapshot of the number of passing and failing rows, the failure rate, as well as the failure tolerance (if configured). In this execution, we have 2,000 passing rows and 2,000 failing rows, yielding a failure rate of exactly 50%.
The bar below the table gives a visual representation of the pass-fail metric. The section of the bar shaded in red represents the number (and percentage) of failing rows, while the green section represents the number (and percentage) of passing rows.
Lastly, we have the failure tolerance. This, too, is a new feature available in QuerySurge 12.1+ (check out our Failure Tolerance in Analysis article to learn more about what this is and how to set it up). Essentially, a row failure tolerance allows either a certain number or a certain percentage of rows to contain failures without causing the entire QueryPair to fail.
For this QueryPair, we've configured a 10% row failure tolerance, meaning that up to 10% of the rows may contain failures without causing the entire QueryPair to fail. Since 50% of the rows contain failures, that tolerance has been exceeded. The icon in the table indicates that the tolerance was exceeded. Similarly, the vertical line in the bar represents where on that sliding scale the tolerance is set.
Note, the Failure Tolerance metrics here are only available if that property was configured on the QueryPair prior to execution.
Cell Metrics
Similar to the Row Metrics table, the Cell Metrics table focuses on failures at the cell-level. It aggregates all of your rows and columns together, including cells that might be missing from the Source and/or Target dataset.
It looks a lot like the Row Metrics table. And it is, except it's dealing with cell-level data.
Here we have a glimpse at all of the cells involved in this QueryPair execution, of which 14,000 passed and 6,000 failed. Based on this figured, we have a 30% failure rate, which is within our 35% failure tolerance. This time, we have a icon indicating that although there were failures, the number or rate was within the acceptable tolerance configured for this QueryPair.
And again, the bar below the table offers a visual representation of this same data. The vertical line represents where on the sliding scale the failure tolerance was set, and in this case the failing segment of the bar doesn't exceed the tolerance setting.
Note, the Failure Tolerance metrics here are only available if that property was configured on the QueryPair prior to execution.
Cell Failures per Column Graphs
To drill down a little deeper into the cell failures, we have a couple graphs that can help you visualize those failures within the context of each column in your dataset.
By default, we show a bar chart where each bar represents a column in your dataset with at least one failing cell. In this simple example, we have three columns — address_type, region, and tier — all of which contain the same number of failing cells. You can hover over any of the bars for a summary of the metric, as well as it how it factors into your overall test.
The region column contains 2,000 failing cells, which represents 50% of the cells in that column. Furthermore, a third of the overall failures in this QueryPair can be attributed specifically to the region column, which is helpful when trying to identify problematic columns in your dataset.
Use the toggle button in the upper-right to view this same data in the form of a pie chart.
Here, each slice represents a column with at least one failing cell, with the same available hover-over explanation of each column.
Column Metrics
Lastly, we have the Column Metrics table. This gives a column-by-column inspection into the number of failures, failure rate, and failure tolerance of each column in your dataset. It also includes the name and datatype of the column from both the Source and Target side of your QueryPair.
We have five columns in our dataset. The first two — source_id and city — didn't contain any failures, and as noted by the N/A in the Failure Tolerance column, weren't configured with a Failure Tolerance setting.
The last three, though — address_type, region, and tier — do contain failures and have Failure Tolerance settings enabled. Just like in the Row and Cell Metrics tables, the icons indicate that the failures (either the number or percentage of them) exceeded the tolerance setup for those columns.