The Calendar Gap

Abstract

Aggregate benchmarks report that LLM agents perform worse in Arabic than English; they do not say why. We introduce a mechanism-level diagnostic built on matched task variants: instances byte-identical in tools, semantics and phrasing but for a single toggled property. The pilot isolates Hijri calendar reasoning, entity transliteration and serialization discipline across the language boundary, with an Eastern-numerals control. Five arms are tested: four open-weight (20B to 397B), including a hybrid pair holding identical weights with reasoning toggled, plus one closed lite-tier model. Gregorian dates in Arabic pass at 0.90–1.00 on every arm, while the matched Hijri variants collapse to 0.00–0.20. Across arms, 45 of 50 matched set pairs flip against Hijri and none the other way; the isolated Hijri delta is 0.80–1.00. Two arms survive Holm correction in the pre-specified 25-test primary family (adjusted p = 0.0488 each). A deterministic heuristic finds an attempted conversion in every failure but one, where qwen3.5-397b asks for clarification. Mean absolute error among failed dated records ranges from 145.1 to 20.6 days, never reaching exactness. Transliteration failures show a different, contract-like signature.

Scoring is deterministic, calendar golds derive mechanically from the Umm al-Qura calendar, and hypotheses and thresholds were frozen before any model call. The scorer audit remains below its own 0.95 gate (0.9070), with the author as one annotator. The primary Hijri dates occupy 1448 AH, a near-future window; a separately frozen control repeats the same ten M2 sets in 1440 AH, well before the arms' likely training cutoffs. The matched penalty persists on all five arms at 0.50–1.00. The control protocol did not pre-specify a separate five-test Holm family, so its multiplicity analysis is post-hoc sensitivity only, not confirmatory significance. This weakens a recency-only or cutoff-only account, while heterogeneous movement across arms leaves temporal exposure a possible moderator rather than a settled mechanism.