Immigration Copilot: agents that fill government forms in a real browser

Challenge

Immigration forms are long, stateful and unforgiving: a DS-160 session times out, pages depend on earlier answers, and applicants arrive with a folder of passports, pay slips and old applications in inconsistent formats. Paralegals re-type the same facts dozens of times. The goal was to remove the typing without removing the human.

Approach

Two agents, one harness

1. An extraction agent turns noisy documents into a typed applicant record: schema mapping per document type, and field-level conflict resolution when two documents disagree (passport says one spelling, pay slip another). Conflicts become questions for the user, not silent guesses.
2. A navigation agent drives the real portal through a Playwright CDP harness inside Electron. It reads the DOM, decides the next action, fills fields from the record, and stops at anything that needs a human decision (consent checkboxes, ambiguous questions, payments).

The harness is the product

Most of the engineering is not the prompt; it is the harness around the model. I wrote up the principle in Harness in Agentic AI: mid-session decision handling, form-state recovery after a timeout or crash, deterministic retries, and an audit log of every action. The agent never submits a form; it fills, the human reviews and submits.

Running locally

Electron keeps documents on the applicant's machine. The LLM sees extracted fields, not raw files, and the browser session is the user's own. That decision was as much about trust as about compliance.

Architecture

What shipped

• End-to-end flows for I-485 and DS-160 with checkpoints at every page.
• Document extraction with per-field provenance ("from passport p.1").
• Crash and timeout recovery that resumes from the last confirmed page.

Results

• ~90% less manual entry on test cases compared with a paralegal typing from documents.
• Zero unsupervised submissions by design.

What I'd do differently

• Build the evaluation set of real-world document variants first; extraction quality tracked that set exactly.
• Use structured outputs with strict schemas from the start instead of post-parsing model text; it removed a whole class of bugs when I switched. More on the agent side in AI Agents and RAGs.