Case Study: User-Centric Feature Coverage

This paper was written in collaboration with UC Davis' Computer Science Department and accepted at COLM 2026.

01

User-centric feature coverage

LLM agents are now capable at black-box mobile testing.^1,2,3 Given an app binary, without source code, it will install the app, navigate, and produce test cases. The metric in the literature is Activity Coverage: the fraction of the app's activities, its distinct screens, that a session reaches. For native Android Apps, it is easy to compute from the APK and easy to reproduce. It was used for various methods like random input generation,⁴ model-based exploration,⁵ deep learning and reinforcement learning,^6,7,8 and even current LLM agents.^1,2,3

A feature is a capability for accomplishing something, like adding an item to a cart or resuming playback where it stopped, and it usually runs across a sequence of activities. A session can visit each of those activities and never exercise the feature. Developer surveys report the same preference: use-case and functional tests over structural tests are preferred.⁹

We built two things: a benchmark that measures whether a generated test suite covers the features users depend on, and ECS, a testing pipeline that raises that coverage.

Our benchmark:

Our results:

02

Measuring feature coverage

There is no public ground truth for a commercial app's features, since internal specifications are not released. The closest prior work measured feature coverage by manual counting,² which doesn't scale across apps, and lacks the insight into which features were missed.

We used two public sources as validated proxies.

1. 1App description: lists the features a developer built and chose to advertise.^10,11
2. 2Reviews: records what users experienced, and because reviewers explain why they kept or dropped an app, they name minor features and missing ones.^12,13,14

We pulled the description and 500 reviews (250 most recent, 250 most relevant) from 10 top Play Store apps. This yields 5,779 raw feature candidates. A large share is not observable software behavior, so we filtered by hand against a fixed taxonomy, removing pricing and billing, shipping and logistics, account policy, customer support, and vague complaints. 3,012 target features remain, most of them (2,548) from negative reviews, which is where users are most specific about app features.

Automated scoring is difficult since LLM agent outputs and target feature descriptions are likely worded differently. A user review might say "The add-to-cart button in search results is a coin flip," while the target feature is "add-to-cart functionality".

LLM-as-a-judge is the standard setup for this kind of comparison.^15,16 To make the judging as accurate as possible, both agent output and target features are first translated into short, objective descriptions (using Gemini 3 Flash, temperature 0), stripping the opinion and bug language. Against 100 manually labeled pairs, the judge agrees 93% of the time (Cohen's kappa 0.83). The pipeline runs on any tool that outputs natural language test cases, which makes it usable and useful as a metric.

03

Designing the CodeDroid (ECS) pipeline with Deterministic AI

We turn an APK into a suite of reproducible test cases in three stages.

Exploration. We use static analysis to extract Manifest declarations, resource identifier strings, and intent filter definitions that describe the app's activities and how control passes between them. All of it is loaded into a knowledge base to build the agent's prior knowledge. The agent then works through the knowledge base one activity at a time. Each session writes the paths and elements it observed back into the knowledge base for the next session. Exploration is forced into every known activity.

Critical User Journeys. For each trace, the LLM writes an inferred user intent for every interactable element on the screen. Intents must be anchored to elements present in the accessibility tree. A CUJ is the intent plus the exact interaction steps plus the activity path which allows reliable replay.

Aggregation. Many features only exist as a combination of journeys. A purchase is a search CUJ, a cart CUJ, and a payment CUJ in sequence, and no single trace contains it. We sample batches of CUJs app and have the LLM write a synthetic review of each batch, simulating a user that experiences multiple CUJs and writes a review about the features they used.

04

Results

4.1 Activity Coverage

Test generation is bounded by exploration. Static analysis and forced exploration were important components. CodeDroid's ECS also finds more unique screens per token than any LLM baseline, because it plans against a map instead of guessing.

4.2 Feature coverage

05

Closing

The benchmark makes feature coverage a number developers can track, and CodeDroid's ECS moves coverage from 40.3% to 64.1%. Additionally, because CUJ generation compares expected features against the observed UI, the system records what users expect and cannot find. For example, on another testing app, Amazon Shopping, it flagged the missing Prime filter, and a wishlist button that disappeared, which matched real complaints. These are product feedback, generated automatically, before anyone writes the review. This is the direction we are building toward at CodeDroid.