Here is a promise the app makes that you never think about. If you are invited to something, you find out. Not usually. Not when the network is having a good day. You find out. The unglamorous machinery that makes that promise keepable is a pattern called the transactional outbox, and it is the most boring, most important code we have written.
This is a post about that pattern: the bug it exists to prevent, how it actually works in our backend, and why "boring" is the highest compliment you can pay a notification system.
The bug this pattern exists to prevent
Imagine the naive version. Someone creates an event and invites you. The code does two things: it writes the invite to the database, and it sends you an email. Straightforward. Now watch it break.
You write the invite row. You call the email API. The email API is slow, or down, or it times out. What now? If you send the email before committing the database write and the write then fails, you have emailed someone about an event that does not exist. If you commit the database write first and then the email send fails, you have an invite in the database that nobody was ever told about. There is no ordering of "write to the database" and "call the email service" that is safe, because they are two separate systems and either one can fail while the other succeeds.
This is the dual-write problem. It works perfectly in every test, and then it loses someone's invite in production on a Friday night.
The fix: make the message part of the transaction
The transactional outbox solves it by refusing to talk to the outside world during the request at all. Instead of sending an email, we write a second row to the database that says "an email needs to be sent". That row, which we call an outbox row, is written inside the same database transaction as the invite itself.
That is the whole trick, and it is worth saying slowly. The invite and the intention-to-notify commit together or not at all. If the transaction rolls back, there is no invite and no outbox row, so nobody is told about a thing that did not happen. If the transaction commits, both the invite and the outbox row are durably on disk, so the notification cannot be lost, because it is sitting in the database waiting to be sent. We have converted an unreliable cross-system problem, database plus email API, into a reliable single-system fact: two rows, one transaction.
In our code this lives in a single transactional method. It writes the in-app notification row and then zero or more outbox rows, one for each channel the recipient should get: one for email, one for push. Then the request ends. Not a single network call to an email or push provider has happened yet. The user who triggered it is not left waiting on Resend or on Apple's push servers to answer.
Draining the outbox
Something has to actually send the things. That is a separate background worker, the dispatcher, and its entire job is to drain the outbox table. It wakes up every two seconds and asks a simple question: are there any rows that still need sending?
The way it asks matters, because we run more than one copy of the server and they must never send the same email twice. The query that claims work uses a Postgres feature called FOR UPDATE SKIP LOCKED. In plain terms: "give me up to fifty pending rows, lock them so no one else touches them, and if another worker has already locked a row, do not wait around, just skip it and take the next one". Two workers polling at the same instant get two disjoint piles of work and never collide. No row is sent twice, and neither worker sits blocked behind the other.
Claiming and sending are split into two phases on purpose. The claim is a short database transaction: mark the rows as PROCESSING, commit, release the lock. Only then, with no database transaction open, does the worker make the slow network calls to the email and push providers. This matters more than it looks. Network calls take hundreds of milliseconds and can hang for seconds. If you held a database connection open for that whole time, a provider slowdown would drain your connection pool and take down parts of the app that have nothing to do with notifications. Hold locks for microseconds, do I/O for seconds, and never at the same time.
What happens when sending fails
Providers fail. That is not an edge case, it is a Tuesday. The outbox makes failure boring, because a failed send is just a row that did not reach SENT, and a row that did not reach SENT gets tried again.
Each row gets three attempts, with a deliberately increasing gap between them: five seconds, then thirty seconds, then five minutes. The growing gap is not decoration. If a provider is briefly overloaded, hammering it again immediately makes things worse; backing off gives it room to recover. If all three attempts fail, the row is marked FAILED with the error recorded against it, and it stops retrying. We call that the dead letter state: a row that has earned a human looking at it, rather than an infinite retry loop quietly melting the provider.
There is a second failure the dispatcher guards against: itself. A worker could claim a row, flip it to PROCESSING, and then crash before sending, leaving the row stuck in PROCESSING forever, owned by a worker that no longer exists. So a separate sweep runs every minute, looks for rows that have been PROCESSING for more than five minutes, on the reasonable assumption that no healthy send takes that long, and returns them to PENDING to be claimed again. The system heals its own orphans.
The circuit breaker: knowing when to stop trying
Retrying is the right move for a row that hit a blip. It is exactly the wrong move when the entire email provider is down. Retrying thousands of rows against a dead provider just wastes attempts and burns through each row's three tries against an outage that has nothing to do with that particular email.
So each channel gets a circuit breaker. It watches the recent error rate, and if more than half of recent sends are failing over a short window, it "opens": it stops even attempting that channel for the next thirty seconds, then lets a single probe through to check whether the provider has recovered. While the breaker is open, rows are simply left in PENDING, not consumed, not failed, just politely waiting. When the provider comes back, the breaker closes and the backlog drains. An outage costs you a delay, not a pile of dead-lettered invites.
The paranoid extra: push that falls back to email
For the notifications that genuinely matter, a date getting chosen, an event getting cancelled, an event getting reopened, we go one step further. If the push notification exhausts all three of its attempts and never lands, the dispatcher enqueues an email as a fallback, so the news reaches you down a different pipe. This has one honest cost: if the push later succeeds after the email was already queued, you might get told twice. We decided that being told twice about your event being cancelled is a much better failure than not being told at all.
Why boring is the point
None of this is clever. There is no machine learning, no distributed consensus, no novel algorithm. It is a table, a two-second poll, a lock that skips, three retries, and a breaker that trips. You could have described it to a developer in 2005 and they would have nodded along.
That is exactly why I trust it. The failure modes are all things we have thought about and written down: the provider is slow, the provider is down, a worker crashed mid-send, two workers raced for the same row. Each one has a boring, testable answer. When you tap a date and six friends get told, not one of them thinks about any of this, and that is the whole idea. The best notification system is the one nobody ever has a reason to notice. It just quietly means that when you are invited to something, you find out.