# AndroidBench — Public research protocol

Updated 2026-09-18. This development preview publishes 128 task specifications, four paired pilot cases and three GAgent development regressions. It is not a 128-task evaluation or an overall product ranking.

## Scope

- 64 historical queries are preserved, including their original family/category/level labels.
- 64 new specifications cover eight user domains. They require concrete fixtures, reset procedures and validators before becoming executable instances.
- Four paired cases were observed on the same NX888J Android 16 phone: system settings, travel selection, refund-aware CSV, and buffered calendar scheduling.
- GAgent: official 2.32.20(131), logged in, accessibility enabled. Configured Auto chain: Grok-4.6 through Anthropic protocol with StepFun water18-0910 fallback. Configuration does not prove which provider served every turn.
- Doubao: 15.0.0.1. Native phone mode for settings, work mode for file tasks. Native privileges, conversation histories and observable trace depth differ.

## Four axes

1. User domain: travel, local services, expenses, collaboration, information, media, records, device management.
2. Capabilities: retrieval, comparison, calculation, transformation, organization, configuration, scheduling and drafting.
3. Challenges: conflicts, identity, missing information, time, precision, changes, interruptions and restricted scope.
4. Execution qualification: accounts, permissions, product mode, initial state, materialization/reset and evidence coverage.

## Verification

Settings are checked by independent system reads; unchanged font scale, ring/music volume and do-not-disturb are compared. A timeout setting does not guarantee the physical display switches off after that duration: the device's existing charging keep-awake option was preserved.

Files are retrieved in original bytes and hashed. Both CSVs pass 20 artifact checks. The GAgent ICS passes 10 artifact checks; the Doubao ICS is parseable but has floating start/end times, failing two timezone checks. Its filename also differs from the requested name. Travel Markdown artifacts correctly select A and reject the three alternatives.

No invitation was sent or calendar imported in the tested work cases. Public observations cannot prove all hidden cloud actions absent.

## Timing and limits

- The old observer starts during preparation, not exactly at submission. The final Doubao calendar sample is at submission +293.371 seconds; its state at exactly +300 seconds is unknown.
- The Doubao CSV result was first seen around 227 seconds after the observer began; the viewport did not follow, so this is an observation upper bound, not generation latency.
- The three development regressions took about 148, 138 and 270 observer seconds. They do not establish a stable speed improvement.
- The settings overlay mechanism was reproduced by researcher-controlled same-point taps. It is not an autonomous scored task and does not prove that every historical ineffective gesture had that cause.
- Doubao evidence is public UI only. GAgent stored files sometimes already contain truncated tool inputs and outputs. Missing process data cannot be reconstructed or invented.

## Public evidence

The full R30 internal archive contains 666 sampled screenshots, 22 video segments and 11 selected GAgent stored sessions. The website publishes 17 reviewed, unmodified images, four curated paired traces and six synthetic-task artifacts. Public milestone playback is a reading aid, not a continuous video or synchronized execution replay. The screenshot manifest provides source-relative labels and SHA-256 digests. Credentials, raw sessions, unrelated device logs and holdout tasks are excluded.

## Future paper evaluation

Materialize domain fixtures and reset procedures, obtain human reference executions, use matched repeats and clocks starting at submission, and isolate holdouts by semantic task family. Report domain macro averages and task micro averages separately. The development set currently overrepresents drafts; more isolated, reversible execution and interruption-recovery pairs are required.

Research interface structure reference: https://posttrainbench.com/ . No claim of affiliation, external certification or paper acceptance is made.