Parser Engine

Scope: owns parsing — rules, candidates, amounts, currency detection, dedupe key, golden fixtures.

The parser turns notification text into a TransactionCandidate. It is pure Kotlin — no Android imports — and covered by golden fixtures that enforce ≥90% HIGH-confidence parses.

Model

flowchart LR
    subgraph Input
        TEXT[notification text]
        PKG[source package]
    end

    subgraph ParserEngine
        JSON[EmbeddedJsonParser]
        RULES[per-package ParserRules sorted by priority]
        AP[AmountParser]
        CUR[Currency detection]
        DD[DirectionDetector]
        SD[StatusDetector]
        CUT[cutAtStopWord merchant trimming]
    end

    subgraph Output
        CAND[TransactionCandidate]
        NONE[null → unmatched queue]
    end

    TEXT --> JSON
    JSON -->|EmbeddedTransaction found| AP
    TEXT --> RULES
    RULES -->|first matching rule| AP
    AP --> CAND
    CUR --> AP
    DD --> CAND
    SD --> CAND
    CUT --> CAND
    JSON -->|none| RULES
    RULES -->|no rule matches or no amount| NONE

TransactionCandidate

ParserRule

A rule is one regex per bank alert format:

data class ParserRule(
    val id: String,
    val dbId: Long? = null,   // DB row id for provenance
    val packageName: String,
    val label: String,
    val pattern: Regex,       // Java named groups: amount, merchant, account
    val amountGroup: String = "amount",
    val merchantGroup: String = "merchant",
    val accountGroup: String = "account",
    val directionKind: DirectionKind = AUTO,
    val statusKind: StatusKind = AUTO,
    val priority: Int = 0,
)

• amount group is required; merchant/account are optional.
• DirectionKind: AUTO (detector decides, defaults EXPENSE), DEBIT (→ EXPENSE), CREDIT (→ INCOME).
• StatusKind: AUTO (detector decides, defaults COMPLETED) or an explicit status.
• The static BankRuleSets table seeds the rules table on first launch (62 rules across 12 packages). Runtime matching always reads from the DB so in-app edits take effect immediately (ADR-0004). One exception: debug builds inject 12 static com.android.shell rules straight into the engine map for the fake-notification harness — they are never seeded into or read from the DB. Rules shipped after an install are re-seeded onto existing installs once per seed version, keyed on the stable sourceId (ADR-0019).

Pattern validation (RuleCompiler)

Patterns are validated through one pure-Kotlin seam, parser/RuleCompiler.kt, before they may become a live ParserRule:

sealed interface PatternValidation {
    data class Valid(val regex: Regex) : PatternValidation
    data class Invalid(val error: PatternError) : PatternValidation
}

• PatternError.SYNTAX — the string is not a valid Java regular expression (e.g. an unclosed group).
• PatternError.MISSING_AMOUNT_GROUP — no required (?<amount>…) capture.
• PatternError.CATASTROPHIC_BACKTRACKING — an unbounded quantifier nested inside another (for example (?<amount>(\d+)+X)), which classic ReDoS inputs can turn into seconds of work or a StackOverflowError.

Compilation contains both Exception and Error (the regex engine can raise a recursion Error), so compile always returns a typed result instead of throwing (issue #355).

The rule editor compiles each draft through this seam, so an incomplete or invalid pattern never reaches the stored rules. Capture loading (RuleRepository.enabledForPackage) compiles each enabled stored rule and skips the ones that fail, so a single corrupt row no longer disables a bank's other valid rules (issue #233).

Matching bounds

User-edited rules are still untrusted input to the regex engine. ParserEngine therefore caps the scanned span at MAX_MATCH_CHARS = 2 000 characters and gives the rule loop a best-effort MATCH_BUDGET_NANOS = 100 ms budget (issue #309). StackOverflowError is contained at the rule/compile boundary, and the editor's Test actions run on a background dispatcher, so neither a huge notification nor a pathological pattern stalls capture or ANRs the editor. The stored rawText is the bounded span.

The required amount group is verified structurally, not by searching the pattern text: the compiled platform Pattern is asked whether it declares a group with that name (an optional non-capturing wrapper makes the metadata query matchable without changing group numbering). Escapes, \Q…\E quoted spans, character classes, COMMENTS mode ((?x)) and scoped flags therefore behave exactly as Java does. Compilation never throws for malformed input — it catches PatternSyntaxException and returns a typed result. Resource-exhaustion limits (catastrophic backtracking, oversized patterns) are out of scope: the seam validates shape and never runs a match against notification text.

Match order

1. Embedded JSON — EmbeddedJsonParser lifts merchantName, transactionAmount, transactionCurrency, cardNumber from a balanced {...} object in the text (Maybank/MAE append machine-readable fields). More reliable than prose; wins over regex. Confidence HIGH. Produces parseSource=JSON and preserves the raw {...} span as rawJson.
2. Regex rules — rules for the package sorted by priority; first rule whose pattern matches and whose amount group parses wins. Produces parseSource=REGEX, rawJson=null.

For the JSON path, a currency resolved from the payload is authoritative: a prose RM 45.00 before the JSON span cannot override transactionCurrency: "SGD" (issue #311). A malformed non-primitive field (object/array) makes the payload unusable instead of throwing, so the package's regex rules still run (issue #313). The {...} span scanner tracks JSON strings and backslash escapes, so a brace inside a merchant name does not truncate the payload (issue #355).

Rule amount groups are currency-agnostic: BankRuleSets defines a shared currencyAmount token from Currency.amountAlternation() (every catalog symbol + ISO code), captured inside the amount group, so one rule matches both RM 50.00 and S$ 45.00 and AmountParser resolves the original currency from the symbol (see ADR-0009/money.md). The app-default currency is never assumed for a symbol-carrying amount. The alternation order is load-bearing: RMB precedes RM, and RM carries a trailing non-letter boundary, so 100 RMB cannot be captured as 100 RM (which would store a CNY payment as MYR). Expanding the catalog therefore changes the shipped rule patterns; RuleRepository migrates a prior alternation substring on existing installs once per seed version (ADR-0019 amendment, ADR-0039).

Merchant names are trimmed at trailing qualifiers (cutAtStopWord): stop words like via, through, using, ref, on, for, with, … and trailing declined|failed|pending|completed|….

AmountParser

flowchart LR
    subgraph AmountParser
        PAT[amount token regex: symbol + number or number + symbol]
        TOK[Currency.currencyOfToken on amount span]
        NEAR[Currency.currencyNearestTo full-text fallback]
        SEP[per-value decimal separator detection]
        MIN[minor units via Currency.minorDigits]
    end
    TEXT --> PAT
    PAT -->|currency symbol present| TOK
    PAT -->|bare number| NEAR
    TOK --> SEP
    NEAR -->|default when none| SEP
    SEP --> MIN
    MIN --> RESULT[Result amountMillis, currency]

• Handles ₹1,234.56, 1.234,56 €, $1,234.56, S$ 500.00, Rs 500, INR 500.00, + $12.50, RM 25.40…
• Currency resolution order: symbol in the amount span (currencyOfToken) → symbol nearest to the amount in the full notification body (currencyNearestTo) → app-default currency. The proximity fallback means a foreign-currency charge whose rule captured only the number is still stored in its original currency, and a distant "1 SGD = 3.45 MYR" note cannot mislabel an S$ amount as MYR.
• The decimal separator is detected per value: the last separator is the decimal one when both kinds are present, or when fewer than three digits follow a single separator (500,000 / 1.234 → thousands).
• Currency symbol optional (default currency used), may precede or follow the number. Sign handled (- → negative amount).
• An amount whose digits or minor-unit scaling fall outside Long range is rejected (null) — never coerced to 0 or wrapped to a negative value. Structurally malformed separator runs (0..0, 1,,2, 12.50.60) are rejected too, not stripped into a different number; the structure check is a linear scan, so a very long grouped run is rejected as out-of-range instead of overflowing the regex stack. Long.MIN_VALUE is a valid amount; one unit past either end is not.
• Never uses floating point — everything lands in Long minor units.

Currency (single source of truth)

common/Currency owns three tables consumed by parsing and everywhere else (formatter, settings, export): SPECS (code → minor digits + display symbol), TOKENS (loose symbol regexes matched inside amount spans), and DETECT (letter-boundary detection over the whole notification: RM50 is MYR, while RMB stays CNY). money.md owns the full inventories — consult it before adding a currency; everything else consumes these tables automatically.

Direction & status detectors

• DirectionDetector: scans clauses left to right (split on ., ,, ;, !, ?, |, newline) and the first clause naming a direction decides. Inside a clause the flag that appears first decides, so a mixed alert such as You paid RM50 at Shopee and earned RM2 cashback is EXPENSE while refund received is INCOME. A credit flag directly after non-, no, not, without or for is incidental (non-refundable, for refunds call …) and is ignored, so paid to … non-refundable stays EXPENSE. Only when no clause names a direction does the amount sign decide (-RM → EXPENSE, +RM → INCOME); the sign scan is derived from the shared catalog alternation, so +S$, +A$, +HK$, +CN¥ and every other token participate (issue #355). Returns null → rule directionKind decides.
• StatusDetector: failed/declined/unsuccessful → FAILED, pending/processing/initiated → PENDING, reversed/reversal → REVERSED. Flags match whole words/phrases and are ignored in incidental or negated contexts (a processing fee, processing fees, no pending charges), so a fee line cannot flip an otherwise completed alert (issue #314). Returns null → rule statusKind decides (default COMPLETED).

NotificationKey (dedupe)

sourcePackage + ":" + occurredAtMillis + ":" + sha256(normalize(rawText))

normalize = trim + collapse whitespace. The key is computed per notification in CapturePipeline (with the post time as occurredAtMillis). Duplicate suppression — the session key set and the DB unique index on transactions.notificationKey — is owned by ingestion-pipeline.md § Duplicate suppression.

MerchantNormalizer

Collapses a merchant to a stable form for override matching:

NFD decompose → strip combining marks → lowercase → trim → collapse whitespace → drop chars outside [a-z0-9 .&'-] → trim

NFD decomposition folds accented Latin letters (Żabka → zabka) before the ASCII filter, so merchants from non-English bank alerts still key stably. Stroke letters with no decomposition (e.g. ł) are still dropped by the filter.

Used by KeywordClassifier and TransactionRepository.applyCategoryOverride — normalization happens exactly once, inside the classification module (ADR-0007).

Golden fixtures

app/src/test/resources/fixtures/*.json — one file per package (bigpay, boost, cimb, grab, mae, maybank, misc, pbb, rhb, shopee, tng, wallet, wise).

ParseGoldenTest asserts:

• each fixture parses to exactly the expected amountMillis, currency, merchant, account, direction, status, confidence;
• fixtures marked expected: null produce no candidate;
• each parse's parseSource and (for JSON) the exact rawJson span;
• ≥90% of parseable fixtures must parse with HIGH confidence.

Add new fixtures when adding rules — the ratio gate protects parse quality. The test enumerates the fixtures directory dynamically, so dropping a new file in is enough to include it.

Starter rule sets

Each seeded row carries its stable sourceId (equal to the rule id, e.g. mae_spent_card, pbb_cr); starter-rule re-seeding keys on it (ADR-0019).

Related

• Ingestion pipeline
• Money module
• ADR-0001 occurred-at
• ADR-0002 rule provenance
• ADR-0003 confidence gate
• ADR-0004 per-notification rule loading
• ADR-0019 starter-rule re-seeding
• ADR-0020 cross-source duplicate suppression