Testing

Running

cd android-app
./gradlew testDebugUnitTest          # pure-JVM unit tests — fast, no device
./gradlew connectedDebugAndroidTest  # instrumented (Compose UI) tests — needs device/emulator

For a repeatable phone-sized UI audit, run from the repository root:

git fetch origin main
git rev-parse HEAD origin/main  # compare when auditing latest main
scripts/oneplus12-qa.sh --test --show

Requires Linux with Python 3, setsid (util-linux), the SDK emulator, KVM, the API 36.1 Google Play x86_64 system image and English UI labels. The SDK comes from ANDROID_HOME, then ANDROID_SDK_ROOT, then the Android/Sdk directory in your home folder. The isolated autobudget-oneplus12 AVD uses 1440 × 3168 pixels and 560 logical dpi (about 411 dp wide), with stock Android rather than OxygenOS. --show opens a newly started emulator window; omit it for headless runs. --test clears AutoBudget data on this dedicated emulator, completes onboarding before the instrumented suite, then drives seeded-data navigation, search/filters, transaction CRUD, category creation, review queue, settings, dark mode, 200% text and landscape. The listener grant is automated through ADB after opening system settings. Extra journeys cover duplicate capture, swipe actions, file-picker cancellation, location, currency discard, category exit behavior and the PIN lifecycle. Known-defect observations are recorded for review rather than asserted as desired behavior. The optional rules command records live rule-test output. This harness does not automate backup import or the malformed-file negative probe; use the issue reproduction steps for those cases. Hardware camera, biometrics and real bank notification delivery require separate device checks.

Each invocation gets a separate run directory beneath .cache/oneplus12-qa/: APK provenance/hash, checkout/dirty status, instrumentation output, crash log, action log, and PNG/XML pairs for visual review. The printed run path identifies its artifacts; the AVD lives alongside these run directories. Generated files are ignored by Git. The default builds the current checkout, not the remote branch; --skip-build reuses existing APKs and does not verify source freshness. Fixtures use the emulator's current month; screenshots are review evidence, not byte-stable goldens. Each capture waits for a rendered, settled screen rather than a fixed sleep: Device.wait_ready gates on the app process, a foreground MainActivity, the route anchor (wait_ready(anchor=...) in the launch helpers) and two non-blank frames with a matching sampled-pixel signature, and shot re-checks settle; the wait is bounded and a timeout names whatever never appeared, so a splash/blank frame is never recorded as evidence. A failure still writes a failure PNG/XML that bypasses the gate, so the broken state stays reviewable. QA_DIR and QA_PORT override the output and port (default 5560, an even console port). Both scripts refuse other AVD names; the Python driver also requires a debuggable app. The emulator stays running; stop it with adb -s emulator-5560 emu kill. Individual journey/debug commands are listed by python3 scripts/android_ui_qa.py --help. Run stages in order: onboarding, seed, journey, extras; rerun onboarding/seed before repeating the full journey. Standalone security expects app lock to be off. Restore a clean demo afterwards with the seed command if desired. UI labels are selectors, so update the driver alongside label/navigation changes. A successful script run proves only its asserted flows; manually review the screenshots for visual consistency and accessibility.

Run the driver's host-only device-safety checks without an emulator using python3 -m unittest discover -s scripts -p test_android_ui_qa.py.

Demo data & screenshots

scripts/generate_screenshots.sh seeds a deterministic demo month through the debug-only DEBUG_SEED broadcast (DemoSeeder.kt; see tooling.md). Current-month fixture day-offsets are clamped to the seed day, so a seeded transaction is never dated after the seed clock no matter when the month is seeded — covered by DemoSeederScheduleTest.kt (day 1, mid-month, month end, short month, leap day, year boundary). The review queue is intentional: one explicit Public Bank sample plus one deliberately unparsed Google Wallet notification (two rows), locked in by DemoSeederQueueTest.kt. Regenerate the PNGs locally when a screen changes; the demo dataset drives home, transactions, budgets and reports, while review-queue is fixed at those two rows. The screenshot script pauses briefly after first launch before injecting demo notifications, so startup rule seeding can settle; it is a timing allowance, not a readiness probe.

JVM coverage is measured with JaCoCo (issue #351): ./gradlew :app:jacocoTestDebugUnitTestReport writes the HTML/XML report and :app:jacocoTestDebugUnitTestCoverage enforces a 40 % line-coverage floor over app code (UI/Compose and generated classes excluded; measured 45.8 % on 2026-11-02). CI runs both with test and uploads the jvm-coverage artifact. The floor is a non-regression tripwire, not a target: raise it when the measured baseline improves, never lower it to make a red change pass.

Unit tests are pure JVM (no Robolectric, no Android deps) — that's a deliberate constraint: the parser, money, codecs, and lock logic avoid Android imports so they test in milliseconds. ViewModel aggregation logic is factored into pure top-level functions (aggregateHome, computeReportTotals) so it is unit-testable without a Context.

Recordings

scripts/generate_recordings.sh records the short silent interaction clips embedded in the tour — one per emulator-reproducible flow: onboarding; notification capture + duplicate suppression; Home month navigation + review badge; Activity search/filter + hide/delete; manual add/edit + unsaved guard; itemization/attachment/location; review-queue convert/dismiss; budgets limit/category/swipe; reports month navigation + drill-down; settings theme/currency/export-import; listener revoke/repair; security PIN lock/unlock.

It reuses the screenshot machinery: it builds the debug APK, boots or reuses ONLY the pinned docs-shots AVD (a device reporting any other AVD name is refused), pm clears only the debug package, broadcasts the same DEBUG_SEED demo month, and enters SysUI demo mode. Each flow is driven through scripts/android_ui_qa.py (same rendered-screen readiness gate) while adb shell screenrecord records; the script stops it, pulls the H.264 MP4 and re-encodes it to a silent, faststart clip. A size guard fails the run when a clip exceeds 2 MB, shrinking scale/bit-rate/fps first; clips are 8–20 s.

Regenerate locally (needs adb, ffmpeg and a bootable docs-shots AVD):

bash scripts/generate_recordings.sh          # fresh AVD, build + record all
bash scripts/generate_recordings.sh --list   # clip names
bash scripts/generate_recordings.sh --only home-month

The on-demand docs-screenshots.yml workflow produces both media sets on one emulator and — even though its drift step fails on any PNG change by design — always uploads the screenshots-refreshed and recordings artifacts. Download both and commit them in the UI-change PR (screenshots under docs/assets/screens/, recordings beside them); CI never commits. Two screenshots are intentionally not byte-stable: unsaved-changes and transaction-detail capture a focused text field, so a blinking cursor makes them differ between runs — the screenshot drift guard stays disabled for that reason. The recordings are timing-based, not byte-compared. Hardware-only paths are excluded and documented as out of automated scope: biometric unlock, real bank-notification delivery and camera receipt capture need a physical device.

Test areas

Coverage is organized by area, not by an inventory of individual files — a file list goes stale whenever tests move. On 2026-11-03 the tree declares 1,395 JVM tests (including 13 in app/src/testDebug) and 760 instrumented @Test methods across 120 classes. The bands below are the docs contract: scripts/docs-guard.sh counts the trees and fails when a stated range no longer contains the real number, so adding or deleting a whole band of tests is a deliberate doc update rather than silent drift.

Onboarding and capture tests isolate themselves: each builds its own encrypted in-memory Room database and constructs the ViewModel/repositories directly (never the app's on-device ledger), so a listener grant, tracked package or transaction row never leaks between cases. ManualOnlyJourneyTest covers the permission-denied path end-to-end: manual completion, a first manual row that survives relaunch, then later capture activation that adds rows without resetting the ledger or double-inserting a replayed fixture. It asserts SettingsRepository values rather than touching Android notification-access state, so it needs no grant and cleans up by closing its database.

Golden fixtures

Fixtures live in app/src/test/resources/fixtures/.

One JSON file per package: bigpay, boost, cimb, grab, mae, maybank, misc, pbb, rhb, shopee, tng, wallet, wise. (ParseGoldenTest enumerates the directory, so new files are picked up automatically.)

Shape:

{
  "packageName": "my.com.maybank2u",
  "currency": "MYR",
  "cases": [
    { "title": "…", "body": "RM 25.40 DEBIT from A/C 12345678", "expected": { "amountMillis": 2540, "merchant": null, "direction": "EXPENSE", "status": "COMPLETED", "confidence": "PARTIAL" } },
    { "title": "…", "body": "…unparseable…", "expected": null }
  ]
}

Rules for editing fixtures:

• expected.amountMillis is minor units (25.40 → 2540).
• expected: null means the text must not produce a candidate.
• Keep the ≥90% HIGH-confidence ratio — add PARTIAL cases sparingly.
• misc.json holds cross-package/edge cases.

Writing a new unit test

1. Follow the pure-JVM rule: no android.* imports in the test target.
2. For anything touching settings/security, use an in-memory SettingsStore fake + fake LockClock (pattern in AppLockManagerTest).
3. For classification/override tests, prefer in-memory DAO fakes over Room.
4. Name tests behaviorally (fun masksLongDigitRuns(), not fun testSanitize()).

Design-system verification matrix

The visual contract (ADR-0069) is verified in bounded bands, not on every device. Automated checks run in CI; visual evidence is attached per ticket.

DesignSystemMatrixTest (instrumented) reads the laid-out TextLayoutResult of the Budgets spent amount and the Home hero amount at 320 dp across 100/150/200 % text and both themes through the real AutoBudgetTheme, failing on any ellipsis or lost character. W09RegressionMatrixTest (instrumented) extends that harness across Save, hidden Activity rows, review-queue money, the Reports drill-down lifetime, the month picker, the navigation rail and settings switches at 320/411 dp, short landscape, 100/150/200 % text, light, dark and representative dynamic schemes. It asserts real laid-out text boxes and characters, unclipped bounds, measured 4.5:1/3:1 theme pairs, named 48 dp actions and non-overlap checks for review actions, month cells and rail destinations. Its containment checks read unclipped bounds rather than viewport display, so the 320x640 CI AVD cannot false-fail a correctly laid-out pill that is wholly outside that window while still inside the 1280 dp matrix root; it adds no second fixture system. The matrix is a representative set, not a full width x scale x theme cross-product. Manual evidence — settled captures with fixture metadata (API image, pixel size, density, actual dp window, font scale, locale, theme, navigation mode) — is required for the screens and states the automated bands do not reach; label manual versus automated verification on the ticket.

Constrained viewport tests (CI 320x640)

The instrumented CI job boots AVD test with no hardware profile: the 2026-11-03 run log reports androidboot.qemu.skin=320x640 and Setting display: 0 configuration to: 320x640, dpi: 160x160. A device created the same way locally measures status-bar 24 px + navigation-bar 24 px inside those 640 px.

A wm size 320x640 && wm density 160 override on a Pixel 9 Pro AVD is a stricter stress viewport, not a byte-identical CI clone: that profile keeps its own 52 px status bar and 24 px navigation bar (76 px of vertical system insets instead of 48 px), so its content area is 28 dp shorter. #407 reproduced there — the recovery action started at y 642, its top edge 2 dp below the 640 dp root — while the same original tests pass on the bare 320x640 profile (3/3 locally, and CI's 938/0 artifact). Keep both configurations green; do not claim the override is CI.

Two rules follow from #407:

• Bring the target into view before clicking it. On these screens the filtered-empty recovery surface and the editor body scroll. A performClick() on a node laid out below the fold can silently miss: its clipped bounds are empty, the injected touch lands outside the window and the event is dropped, so only the later state assertion fails. Call performScrollTo(). Where a test journey types into the search field, it also calls Espresso.closeSoftKeyboard() first (dismissing the keyboard is the realistic path), then keeps the behavioural assertion — never a blanket retry or existence-only check.
• Measure, don't assume, whether an off-screen node is a product bug. #407's "Reset filters" measured y 642..682 in a 640 dp root while the production callbacks were correct; a 320x640 seeded-device journey scrolled the recovery viewport and the tap cleared the query and both filter chips. The fix therefore lived in the tests, not in production state or layout.

When a test must simulate a different logical size than the device window (for example the 411x914 Home audit on the 320x640 CI AVD), W12's pattern is the narrow precedent: wrap the simulation in BoxWithConstraints, scale LocalDensity so the requested logical viewport fits the physical px bounds (clamped by a minimum scale), assert the simulated px size is not larger than the physical window and that the root keeps its requested logical size. Document it; do not copy it onto tests that can run at the real viewport, and never use scaling to hide a genuinely clipped control.

CI

PRs run unit tests + lint + ktlint (android-tests.yml), plus an instrumented job (connectedDebugAndroidTest) on a KVM-accelerated emulator — failures block merge. When a merge queue is enabled, its merge-group entries run the same gates; without one, the PR-head run is the accepted evidence for a direct/admin merge (ADR-0081). There is deliberately no duplicate push-to-main rerun. Release artifacts keep their own gates. The JVM step also produces the JaCoCo report and enforces the coverage floor. The job enables KVM first (udev rule per the android-emulator-runner README); without it the emulator falls back to software emulation and never boots within the timeout.

The instrumented job runs the entire app/src/androidTest suite with no include filter, so the manual-only onboarding regression (ManualOnlyJourneyTest) and the cross-screen W09RegressionMatrixTest are reported by test name — no separate emulator job or production permission is needed and no second focused lane duplicates the run. Its JUnit XML and HTML report upload as the instrumented-results artifact on both success and failure (if: always()), so a red PR can be triaged from the retained failure evidence instead of re-running the emulator. Before that upload, scripts/summarize-instrumented-results.sh reads the same XML and writes the totals plus every failing class/method to the job summary, with a re-run hint for one class; it adds no test invocation. The artifact step is continue-on-error: if GitHub's artifact storage quota is exhausted, the lane still reports its test outcome and a warning names the missing artifact.

Discovery is unfiltered and every suite reports under its own name, so a new regression cannot silently escape. The tree currently declares 760 instrumented @Test methods across 120 classes (2026-11-03); the last local full-suite run (2026-11-03, API 36 google_apis, x86_64) reported 938 tests with no skipped entry and no failure (Gradle connectedDebugAndroidTest finished in 26m11s locally). There is no one-for-one class-to-suite mapping to rely on: parameterized classes expand to several cases and one class may hold many @Test methods — always read the JUnit XML. Viewport regression tests that simulate a 411x914 or 320x568 dp shell on a smaller physical CI window scale the simulation density so the logical viewport fits; system-bar insets remain physical pixels, so the simulated shell is marginally more conservative vertically rather than a bit-exact density-1 replica. The persistence and lifecycle suites this release added are: MigrationChainTest, MigrationCoverageTest, Migration8To9Test, CrossSourceMergeInstrumentedTest, CrossSourceInvariantsInstrumentedTest, ImportRollbackInstrumentedTest, CaptureDeliveryInstrumentedTest, CaptureReplayInstrumentedTest, UnmatchedReplayInstrumentedTest, TransactionDeletionUseCaseInstrumentedTest, AppLockGateTest and ScreenshotProtectionActivityTest. The older 2026-10-18 430-test run took 16m04s end to end (the device phase alone 14m48s; excludes AVD boot); the 2026-11-03 938-case local run took 26m11s, leaving only about four minutes under the 30-minute job cap. Shard connectedDebugAndroidTest by package before that cap; never raise the timeout or add retries, which would hide a deterministic failure.

Gate negative control: assert a wrong value in one of those suites — for example make the first assertion in MigrationCoverageTest.everyCommittedSchemaVersionIsCoveredByTheUpgradeChain fail — and run ./gradlew connectedDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.zharif.autobudget.data.db.MigrationCoverageTest. The build fails and androidTest-results/** is still written; revert the probe. A suite missing from the XML, or a non-zero skipped, is a gate bug — not a pass.

Negative control (run on a dedicated emulator, never in CI): break the manual path — for example make OnboardingViewModel.finishManual() require listener access — and confirm ManualOnlyJourneyTest.manualSetupThenCaptureActivationKeepsTheLedger fails; restore the production code afterwards.

Compatibility smoke matrix

The instrumented job above is the full behavior suite on the target platform. Three smaller PR lanes in android-tests.yml cover the gaps it does not — older platform behavior, the newest supported platform and minified release runtime — without multiplying every test across platforms:

Mechanics:

• Each lane pins its system image in the setup-android packages: list, so an unavailable image fails setup with a named error instead of silently skipping the lane. api-level: 26 is the minSdk = 26 floor (targetSdk = 37).
• The release lane builds assembleRelease, then installs the minified APK and exercises it OUT OF PROCESS: launch MainActivity, assert it is the topResumedActivity, assert no FATAL EXCEPTION was logged, and keep a screenshot as evidence. Instrumentation is not possible here — the androidTest source set only builds for the debug variant (AGP exposes no release androidTest task) and a debug-built test APK cannot instrument an R8-minified app (it dies with NoClassDefFoundError: kotlin.jvm.internal.Intrinsics). This workflow passes no RELEASE_* env vars, so the release build signs with the debug key (app/build.gradle.kts); a setup step asserts the APK certificate is CN=Android Debug, so the lane can never quietly require production signing.
• Budgets: Minimum API Smoke 20 min, Target API Smoke 20 min and Release Smoke 50 min job caps. The release lane's 50-minute cap is its job budget and also sets emulator-boot-timeout: 1800 (seconds) so the AVD-boot budget is tracked separately — a 25-minute job cap had cancelled it while it was still polling sys.boot_completed. Both upload evidence with if: always(): the API 26 lane uploads androidTest-results/reports, and the release lane uploads release-smoke.png plus release-crash.log (its smoke runs out of process, so there is no instrumentation transcript).

Reproduce a lane locally on a running emulator. From android-app/:

# API 26: minSdk critical journey (debug variant).
./gradlew connectedDebugAndroidTest \
  -Pandroid.testInstrumentationRunnerArguments.class='com.zharif.autobudget.data.db.SqlcipherEncryptionTest,com.zharif.autobudget.ui.onboarding.ManualOnlyJourneyTest,com.zharif.autobudget.ui.navigation.DeepLinkColdStartTest,com.zharif.autobudget.ui.lock.AppLockGateTest'

# Release: minified APK, debug-key fallback signing. This is the lane's entry
# point: a debug test APK cannot instrument a minified app, so the smoke is out
# of process (install, launch, assert resumed, read the crash log, screenshot).
bash ../scripts/release-smoke.sh

The full API 36 suite stays unfiltered on the Instrumented Testing on Emulator job. scripts/test-ci-smoke-matrix.sh parses the workflow and fails if a lane loses its image pin, selector, timeout or if: always() upload, if a secret creeps in, or if the full suite gains an include filter.

Related

• docs/architecture/tooling.md — build + CI details
• docs/architecture/parser.md — fixture mechanics