Case Study: Android Build Repair

This paper was first published in collaboration with UC Davis' Computer Science Department in October 2025.

The Problem

Android is invisible to the vibe-coding era

Android accounts for nearly three-quarters of the global mobile OS market.¹ Over 2.2 million apps live on the Google Play Store,² and more than 1.4 million Stack Overflow questions are tagged Android.³ Yet, if you ask a frontier LLM to "build a startup name generator app," every major model, including Google's own, generates a web application. Despite dominating the market, Android has been left behind in AI-assisted development.

A large part of this is a training data problem. Kotlin, the dominant language for Android, represents just 1.45% of open-source code on the web. The consequences are real: a recent study found that only 31.3% of Android apps can be successfully compiled out of the box. Over two-thirds of publicly available Android code simply does not build.

The question we wanted to answer: can LLM agents fix Android build errors reliably? And if not, what is actually holding them back?

The Bottleneck

The knowledge is there. The execution is not.

When you give a capable LLM agent a general-purpose shell and an Android build error, something predictable happens. The agent often correctly diagnoses the root cause. It knows that a cluster of Unresolved reference errors in a single file probably points to a missing dependency. It knows which Gradle commands are relevant. It understands the structure of Android projects.

And then it fails to fix it.

Not because it lacks knowledge, but because translating high-level understanding into a correct, well-sequenced shell interaction is its own skill. Shell usage for Android-specific workflows is likely rare in training data relative to general shell usage, which means the model applies general shell reasoning to a specialized context. The result is what we observed repeatedly in our case studies: agents that correctly identify the problem, then get trapped chasing misleading warnings, sequence commands incorrectly, or, in one striking case, correctly apply a fix and then revert it.

This is the gap we set out to close.

The Approach

GradleFixer: domain-specific tools mapped to the two-phase loop

To validate this approach in a rigorous and reproducible way, we built two things.

AndroidBuildBench is a benchmark of 1,019 real Android build failures, curated from the commit histories of 43 open-source projects. Each failure is paired with a verified fix from a subsequent commit, so every problem is known to be solvable. The benchmark spans human-introduced errors, dependency misconfigurations, and LLM-generated errors, covering the full range of real-world development scenarios.

GradleFixer is our agent, equipped with three domain-specific tools in place of a general-purpose shell. The key design decision was to replace the broad bash interface with purpose-built abstractions that map directly onto the two-phase loop.

These tools are thin wrappers around shell commands. Their raw output is identical to what the shell would produce. What changed is the interface: instead of constructing a multi-step shell interaction, the agent calls run_build(). Execution becomes trivial and reliable, freeing the agent's reasoning capacity for the exploration and validation phases that actually matter.

The Results

Constrained tools, better outcomes

The results were clear. This is exactly the behavior the deterministic agent framework predicts: constrain the action space, reduce execution uncertainty, the agent's existing knowledge can fully express itself.

As we constrained the agent's tools by moving from a general shell, to a Gradle wrapper, to a strict run_build command, performance increased every step of the way, even though the agent had access to strictly fewer raw capabilities. A more constrained action space produced better outcomes.

One further result: GradleFixer running on Gemini 2.5 Flash, a significantly smaller and cheaper model, outperformed the baseline agent running on Gemini 2.5 Pro. The right tools did more for performance than a larger model.

The Broader Impact

Reliability, cost, and the future of intelligent QA

The performance gap has a direct cost implication. Failed attempts consumed nearly four times as many input tokens as successful ones (4.1 million versus 1.1 million) because a failing agent keeps generating new strategies. A more reliable agent is more economical.

But there is a more fundamental point about trust. In a CI/CD pipeline or an enterprise engineering workflow, an agent that fails silently or burns through retries before timing out is expensive and untrustworthy. A deterministic agent that stops at the point of divergence and reports exactly what it expected versus what it observed gives a developer something actionable. "I expected Java 17 to compile this module, but the build failed at dependency resolution" is a useful failure. A loop of hallucinated bash commands does not.

This principle extends beyond build repair to one of the most obvious commercial applications: software testing and QA. Traditional script automation is repeatable but rigid. It breaks the moment the environment changes. Unconstrained AI agents can adapt to interface changes and novel scenarios, but their behavior between runs is unpredictable, which makes their results untrustworthy as a test signal.

Deterministic agents unify both. The agent uses LLM intelligence during the exploration phase to understand the application, navigate workflows, and form expectations. It then executes with step-level validation, confirming that each action produced the expected outcome before proceeding. If the product behavior changes unexpectedly, the agent identifies exactly where. If the agent itself cannot safely proceed, it distinguishes its own limitation from a software defect.

Software testing requires repeatability. A test is only as valuable as its ability to be trusted and explained. Deterministic agents bring AI capability into testing without sacrificing that trust. At CodeDroid, this is the direction we are building toward.