Ask a typical engineering leader how often their test suite actually runs locally before code gets pushed, and the honest answer is usually a shrug. Ask which environment a specific test executed in last Tuesday, or whether last quarter’s promise to “test more thoroughly” actually happened, and the shrug gets bigger.
This isn’t a discipline problem. It’s a data problem. A test suite that spans a dozen developer laptops and several CI runners produces a huge amount of information about who ran what, where, and when, and almost none of it is captured anywhere durable.
The shadow-testing problem
Every test run happens somewhere: a specific machine, under a specific user or CI context, with specific environment variables set. But unless that context is captured at the moment of execution, it evaporates the second the run finishes. CI logs might retain some of it for a while, scattered across build systems and retention windows, but local runs typically leave no trace at all.
The practical effect is that “testing” becomes something people describe rather than something anyone can verify. A manager asking “did QA run the full suite against the release candidate?” is relying on someone remembering and reporting back accurately. A engineer wondering “does this test even get exercised outside of CI?” often can’t find out without adding print statements and asking around.
That blindness has real consequences. Resourcing and risk decisions (where to invest in more testing, which areas are under-covered, whether a stated testing process is actually happening) all get made on assumption rather than record.
How Obvyr ties every execution to an agent, user, and environment
Obvyr’s collectors (a CLI tool that wraps any test command, and a Gradle plugin for JVM projects) capture execution data at the moment tests run, wherever they run. Every execution is registered to the specific agent (a token tied to a machine or CI job) that ran it, along with a user or context identifier and a set of environment tags, all the way down to the individual test inside that run.
That data isn’t just stored: it’s aggregated into daily metrics that answer the questions that used to require guesswork: how many unique machines ran tests today, how many unique people ran the suite this week, and how those numbers trend over time. The full execution history is filterable by agent, by tag, and by date range, so “who ran what, where, and how often” stops being a question you ask a colleague and becomes a query you run yourself.
Tag-based filtering supports matching on any tag, all tags, or an exact set, which means you can slice the data by environment, team, or whatever labelling scheme already makes sense for your projects, without having to restructure how you organise tests to get the visibility.
What gets captured, and how
The CLI wraps your existing test command: prefix it, and nothing else about your scripts changes. Under the hood it captures the command itself, the exit code, duration, output, and a timestamp, alongside two fields that make the rest of this possible: a user/context identifier (set via OBVYR_CLI_USER, e.g. local-dev or github-ci, rather than just whatever your OS username happens to be) and a set of environment tags. CI is auto-detected from the environment variables every major provider already sets (CI, GITHUB_ACTIONS, GITLAB_CI, JENKINS_URL, BUILDKITE, CIRCLECI), so nobody has to manually flag “this run was in CI”. It already knows.
For JVM projects, the Gradle plugin does the same job without wrapping a command at all: it hooks directly into every Test task and submits execution data automatically. Its user/tags configuration is deliberately framed as describing what is running the build, not who (github-ci with tags ci,github-actions, say, rather than a person’s name), which is the right unit for the “which environment did this run in” question this guide is about.
Tagging in practice tends to be simpler than the API’s own naming suggestion might imply. A category:value convention like env:ci is documented as an option, but most real setups just use flat tags instead: development, local, staging, production, ci, github-actions, set once per profile and inherited by every run through it. And because the pipeline that turns a raw execution into a queryable record is event-driven, a run typically shows up in your dashboards within seconds of finishing, not after some overnight batch job.
The outcome is a record, not an impression: a durable answer to whether a test suite is actually being exercised the way everyone assumes it is.
The same tags that answer “where did this run” also carry across projects, which is what makes comparing test health across your whole portfolio possible in the first place.