Automate data entry from emails, and where it really breaks

Someone in the business spends two hours a day reading emails and typing what is in them into something else. Orders, delivery notes, applications, invoices, enquiry forms that arrive as prose. It is obviously automatable, everyone agrees it should be automated, and a surprising number of attempts quietly stop being used within a few months.

The reason is rarely the parsing. Pulling fields out of a semi-structured email is the part that has improved most in the last few years. What breaks is everything around it: a message that never arrived, a duplicate that arrived twice, and a record that went into the system wrong with nobody looking.

Decide first whether the email is the source of truth

There are two versions of this project and they are not the same size. In the first, the email is a notification of something that also exists elsewhere, and you are saving typing. In the second, the email is the record, and if your system misses one, the business has lost an order.

The second version needs reconciliation, retries, an audit trail and somebody who is accountable for the exceptions queue. The first does not. Teams routinely scope the first and discover they built the second, which is where the budget goes.

What the mail platform actually guarantees

This is the part worth reading the documentation for rather than assuming, because the guarantees are weaker than most designs assume. Google's Gmail API push notifications guide is explicit about its own limits:

• A watch must be renewed. You must call watch at least once every seven days or notifications stop, and Google recommends calling it once per day.
• There is a rate ceiling per mailbox. Each watched user has a maximum notification rate of one event per second, and the service drops anything above that rate.
• Notifications can be lost. The documentation says they may be delayed or dropped in rare cases, and recommends a periodic history.list call as a fallback.
• It is not a direct webhook. Gmail publishes to a Cloud Pub/Sub topic you create, and you must grant publish rights to Google's push service account on that topic.

Read those four together and the architecture writes itself. Push is a latency optimisation, not a delivery guarantee. The thing that makes the system correct is the periodic sweep that reconciles against history, and the renewal job that stops the whole pipeline going silent on day eight. A build that only listens for pushes works perfectly in testing and loses messages in production in a way nobody notices for weeks.

Idempotency, before anything else

Because you now have two paths into the system, push and sweep, every message will sometimes be processed twice. That is not a bug to eliminate, it is a property to design for.

Key every record on something stable from the message itself rather than on your own insert order, and make the write an upsert. Done at the start this is an afternoon. Retrofitted after a month of duplicate orders it is a data cleanup project plus the trust you spent.

Extraction: three tiers, in increasing order of regret

Match the method to how the email is produced, not to what is impressive.

Most projects start at the bottom row because it demos well. Ask first whether the sender can simply send you the data. A surprising share of these pipelines exist because nobody asked a supplier for a CSV.

The confidence threshold and the exceptions queue

Any extraction step should return a value and a confidence, and your system needs a documented rule for what happens below the line. The rule that works is dull: high confidence writes straight through, low confidence goes to a queue a named person clears daily, and nothing is ever silently discarded.

The failure to avoid is a system that is 95 per cent accurate with no way to tell which 95. That is worse than the manual process it replaced, because the errors are now invisible and nobody is reading the source. A person retyping an order at least notices when it makes no sense.

Track one number from day one: what share of messages needed a human. If it does not fall over the first two months, the extraction approach is wrong and more prompt tuning will not fix it.

Where the data lands

Automating entry into a destination that is already disorganised multiplies the mess rather than removing it, and this is the most common reason one of these projects gets switched off. If the target is a CRM, the hygiene rules matter more than the parser does: BDG Labs' piece on CRM hygiene for small sales teams is a good checklist to apply before you point an automated writer at it.

If the destination is a spreadsheet, be honest about whether it should stay one. The volume that justifies automating entry is usually the volume at which a spreadsheet starts failing, and those two thresholds arriving together is not a coincidence. We have written separately on migrating from spreadsheets to a database without losing the history, and doing both at once is often less work than doing them a year apart.

What this should cost, and when not to build it

A single sender, a fixed template, a clean destination and a person who clears exceptions: this is a small build, and it pays back quickly because the saved hours are real and recurring.

Do not build it yet if the volume is under roughly an hour a day, if the formats change every few weeks because the senders are still settling, or if there is no named owner for the exceptions queue. The last one is the real disqualifier. An automation with no owner degrades silently, and silent degradation in a data pipeline is more expensive than the typing was.

The same honest test applies here as to every internal build: it is worth doing when the manual version is both painful and stable. If you are still arguing about what the fields mean, automating the typing just makes the disagreement faster. Our note on internal tools and when to stop using Excel covers that line, and automating invoice generation is the closest neighbour to this problem, with the same three failure points in a different order.

Build the sweep. Make the writes idempotent. Give the exceptions a human. The parser, which is the part that sounded hard, will be the part you think about least.