Case Study: Large Screen App Quality

A case study in deterministic agent design, applied to Android adaptive-app testing across a large number of production applications.

The Problem

Android no longer runs on phones alone

Android now runs across a continuous range of screen sizes, from phones to foldables to Chromebooks, and as of Google's May 12, 2026 Googlebook announcement, premium Android laptops. An app no longer has one form factor. It must behave correctly when folded, unfolded, rotated, resized into split-screen, projected onto an external display, and operated by a keyboard, a mouse, or a stylus. Google's Adaptive App Quality Guidelines define what "correct" means across these conditions as a three-tier maturity model:

Each tier is composed of dozens of individually testable checks (e.g. does the app preserve state across a fold). Apps that fail these checks are increasingly demoted in Google Play's tablet and Chromebook surfaces. See our blog post for an in-depth analysis.

The question we wanted to answer: can we evaluate an app's adherence to these guidelines automatically, reliably, and at the scale the ecosystem requires? The short answer is that off-the-shelf LLM agents cannot, even though the test criteria are externally defined by Google and the failure conditions are concrete. The agents have the knowledge. What they lack is execution discipline.

Across our early evaluation runs, the agent's pass/fail judgments agreed with human reviewers approximately 70% of the time on average, with significant per-category variation. At that rate, the report is not useful. A 30% disagreement rate with human reviewers means a developer cannot trust any specific verdict, and a platform team cannot trust aggregate numbers across the ecosystem.

This is the gap we set out to close.

The Bottleneck

The knowledge is there. The execution is not.

A capable LLM agent reads the guideline "the app should preserve state through configuration changes" and broadly understands what that means. It knows what 'state' is. It knows a fold-unfold transition should not erase a text field. What goes wrong is everything between knowing and verifying. When we observed agents running unconstrained, three failure modes recurred:

1. 1The agent picks the wrong surface to test. A guideline like "media playback resumes after configuration change" is meaningless on a screen that has no media. The agent navigates to a plausible-looking page, runs the check, and reports a pass, but the check was not executed on what the guideline cares about.
2. 2The agent tries to complete a journey end-to-end. When an early step fails, instead of stopping and reporting the failure, the agent improvises around it, finishes the journey by a different route, and reports success. The failed step is ignored.
3. 3The agent's per-step judgment drifts. When asked "did this action produce a context menu?", the agent may interpret a tooltip, an autofill suggestion, or no UI change at all as "menu appeared." Without explicit criteria, the model is unreliable.

This is the same pattern we documented in our Android build repair case study. The agent's high-level reasoning is sound, but the translation from reasoning to verifiable execution is lacking.

The Approach

Ground truth from Google. Determinism from us.

An advantage in this domain is that the criteria we are evaluating against are not ours to invent. Google publishes the adaptive-app quality guidelines, defines the three tiers, and enumerates the specific checks within each tier. As summarized by our blog post: Tier 3 (Large Screen Ready) covers reliability under configuration changes, multi-window behavior, basic keyboard and stylus support. Tier 2 (Large Screen Optimized) covers responsive layouts, keyboard shortcuts, hover states, context menus. Tier 1 (Large Screen Differentiated) covers desktop-grade workflows like drag-and-drop, multi-instance, and custom cursors. Each check has a Google-assigned test ID (LS-C1, LS-M1, LS-I1, and so on).

This matters because it gives us something most LLM evaluation work does not have: a stable, externally defined definition of "correct." The model is not asked to decide what good large-screen behavior looks like. It is asked to determine whether a specific criterion is satisfied. The remaining challenge is making that determination reliable.

We restructured the evaluation around the deterministic agent framework: a two-phase loop in which the agent first plans against verifiable expectations, then executes with step-level validation, stopping at an unexplained divergence rather than recovering past it. Determinism enters in two places.

1. CUJ selection (the Explore phase). For each guideline check, we determine ahead of time which Critical User Journey (CUJ) best exercises that check on the app under test. A CUJ in this setting is a structured pair: an exploration trace with an LLM-inferred user intent, anchored to a specific landing page and the specific widget the test will operate on. Rather than letting the agent gravitate toward whatever surface looks plausible, the planner picks one that is known to expose the behavior the guideline asks about (e.g. a media-playback test runs on a page where media is actually playing). This sounds obvious in retrospect but it is also the single largest source of accuracy gain in our methodology, because it removes the most common failure mode: the test that runs cleanly but tests the wrong thing.

2. Step-level verification (the Execute/Validate phase). Once the CUJ is fixed, the agent does not try to "finish the journey." Each step has an explicit assertion attached to it (e.g. "after this rotation, the maximize state should be preserved", "after this right-click, a context menu with labeled items should appear"), and after every action, the assertion is checked against the observed state. When the assertion fails, the agent stops, and the failure point is the output. This is a direct application of our framework's core principle: we are not maximizing completion rate, we are maximizing the informativeness of the result.

The Results

Reliability is the headline

We track two distinct metrics, and keeping them separate is important:

Agreement (model accuracy). How often the evaluator's pass/fail judgment matches a human reviewer. This measures the methodology. It is the metric the deterministic framework is designed to move.

Pass rate. How often the app under test actually satisfies the Google guideline. This measures the app, and is a property of the ecosystem, not of our method.

The headline result is the first metric. It splits into two evaluation tasks:

Before deterministic CUJ selection and step-level verification were in place, the evaluator agreed with human reviewers roughly 70% of the time, with wide per-category variation. With a deterministic layer, LLM-based evaluation is able to achieve 100% accuracy when identifying application issues that do not follow the guideline.

Because agreement is now stable, the pass rate numbers become meaningful to report at scale. The methodology runs across a large number of production Android apps spanning productivity, creative, social, and media categories, on devices ranging from foldables to Chromebook-class large-screen environments. We conducted a large-scale test in May 2026, and the results provide a comprehensive snapshot of the app ecosystem's readiness for adaptive development. A selection of categories across the three tiers:

The Tier-3 results are not uniform. Most Tier-3 categories sit at 90% or above, but the "Ready" tier is supposed to represent a reliability floor, and the floor is not universally met. Multi-Window Resources is failed by all but a handful of apps (12%), and Config Combinations and Camera Preview sit around 70%. The pattern across tiers is what you would expect: basic reliability is mostly, but not always, present; optimized behavior is patchier; and differentiated, desktop-grade behavior (Input Combinations at 56%) is where the ecosystem is weakest.

The value of stopping at the point of divergence is best shown, not described. The report below is representative output for two real checks: a Tier-2 Touch Targets failure and a Tier-3 Configuration Combinations failure on the same app.

Both are developer-grade bug reports. They are ready for developers to follow up on, or for human experts to review.

The Broader Impact

Beyond correctness testing

Traditional automated testing answers one kind of question: does the code execute correctly? With a deterministic layer, LLM-based evaluation does that, and it also does something automated testing never could. It performs subjective review, the kind of judgment that previously required a human expert, with accuracy, consistency, and scale.

Usability studies and compliance reviews have always been bottlenecked on expert human time, which is why large-scale, consistent versions rarely exist. A deterministic agent changes this. It processes the apps, identifies the issues, and makes the initial assessment, while the expert moves from operating the app to reviewing the findings. The same pattern applies anywhere a defined standard meets subjective judgment, from accessibility audits to design-guideline compliance.

This is the direction we are building toward at CodeDroid.