Reading production telemetry for quality signals
Have the model read production telemetry — errors, latency, drop-offs in flows, feature use, repeated actions — to find where real users struggle, and set it against what users say.
How it works
Feedback tells you what users noticed and chose to report; telemetry tells you what they did. The model reads what the product already records: errors and slow requests, where users leave a flow, actions repeated in quick succession, features nobody uses, and paths nobody designed for. It compares against a baseline — before a release, on one platform, for one customer group — describes each change in plain terms, and attaches the query behind it so anyone can rerun it. The two sources check each other. A complaint that matches a drop in completed purchases is urgent; one from a customer who keeps buying is less so; and a flow where users quietly give up, with no complaints at all, is something feedback alone would never show. The output is a short list of places where testing should look, each backed by data.
When to use it
After releases, alongside the feedback review; when deciding where to spend testing effort on a product with real traffic; to check whether a loud complaint reflects a wide problem; and to find failures nobody reports, such as users who give up without saying why.
Limitations
Telemetry shows only what was instrumented, and a missing event looks the same as nothing happening. It shows what users did, not why: an abandoned checkout may be a bug, the price, or someone comparing options. Steady behaviour despite complaints isn't proof the problem is small — customers on a contract, or with no alternative, keep going until renewal and then leave at once. The model will offer a causal story for any dip, and with many metrics something always moves by chance. Usage data about individuals needs the same care as any personal data.
Watch for
Checkout errors drop to zero after a release because the event was renamed, and the dashboard reads it as a fix. When a metric improves suddenly, check whether what is recorded, or how, changed in the same release.
Before you reach for it
Agree a few signals for each critical flow — completion, errors, latency — and alert on them, so the obvious problems surface without anyone reading the data.
What you still own
Deciding what a pattern means for testing, and checking the model's explanation against the data before anyone acts on it.
Cite this
Qlarify Labs. (2026). Reading production telemetry for quality signals. Retrieved from https://labs.qlarify.fi/ai-in-testing/reading-production-telemetry