Building a Resilient Offline Layer for Flaky Networks in Vietnam

The Spinner That Never Ended

Our patient app assumed that when you tap a button, the request goes to the server and comes back. Booking an appointment, messaging your care team, uploading a symptom photo: tap, wait, done.

That works in a demo. It does not hold up in Vietnam, where a huge share of our users are on mobile data. Someone books from inside a hospital elevator and the connection dies between floors. Someone at a rural clinic has one bar of 3G that drops every time a truck goes by. Someone in a waiting room is fighting two hundred other phones for the same cell tower. The network isn't off, which would be easier. It's flaky: alive, then dead, then alive, with requests dangling in between.

The symptom that forced our hand was the spinner that never ended. A patient taps "Confirm Booking," the request goes out, the network drops before the response comes back, and the app spins forever. The patient doesn't know if they're booked, so they force quit, reopen, and tap Confirm again. Now maybe they've booked twice, or the first one silently succeeded and the second collides. Either way they call support.

So we stopped treating the network as reliable. That became our offline layer.

What "Offline" Actually Means Here

True airplane mode offline is the easy case: you know you have nothing, you queue everything, you wait. The hard case is the degraded network, connected enough to start a request and not stable enough to finish it. Our layer had to handle the whole range without the patient thinking about which one they were in.

We set a few goals. A request the patient made should never be silently lost, even if they kill the app or the phone dies. A retry should never cause a duplicate booking. The UI should always tell the truth about what's pending, what succeeded, and what failed. And reads, like upcoming appointments and message history, should work with no connection at all, because checking when your appointment is shouldn't need signal.

The Persistent Request Queue

The core is a durable queue of intents. When a patient does something that changes server state (book, cancel, send a message) we don't fire an HTTP call directly. We write an intent into a local persistent store on the device first, then a worker drains that queue.

The queue lives in local storage, not in memory, so it survives the app being backgrounded, killed by the OS, or the battery dying in the middle of a booking. When the app comes back, the worker picks up where it left off. Because the intent outlives the process, we stopped getting "I tapped confirm, then my phone died, and now I have no idea if I'm booked."

Each intent carries what it needs to run on its own: the operation, the payload, a creation timestamp, a retry count, and a status. At tap time the app only records the intent durably and reflects it in the UI; talking to the server happens later, asynchronously.

Idempotency, Or How We Stopped Booking Twice

Retrying a failed request is obvious. The trap is that "failed" is often a lie. The request reaches the server, the server books the appointment, and the response is lost on the way back. From the app's side that's a failure, so it retries, and without protection the same slot gets booked twice.

The fix is idempotency keys. When we create an intent, we generate a unique key once and attach it to every attempt. The server remembers keys it has already processed. If it sees a request with a key it has seen before, it doesn't create a second appointment, it returns the result of the first. A retry after a lost response is then safe: the patient gets one booking, the app gets its confirmation on the second try.

This is the piece I'd build first. Retries without idempotency don't make you resilient, they make you dangerous. The key has to be generated when the intent is created and stay stable across every retry; one that changes per attempt defeats the point.

Retries That Don't Make Things Worse

Queued and idempotent requests are safe to retry, but naive retrying creates a new problem. If ten thousand phones all lose the network during a regional blip and then hammer the server the instant it comes back, at the same cadence, you've caused your own denial of service. That's the thundering herd, and it's real when users share congested infrastructure.

So retries use exponential backoff with jitter. Each failed attempt waits longer than the last, a second, then a few, then longer, up to a cap, so a dead network doesn't get pounded. The randomized jitter means phones that failed at the same moment don't retry at the same moment. Without it, backoff still leaves you with synchronized waves; with it, the load smooths into a trickle.

Syncing Only When There Is a Connection

There's no point sending a request into a network you already know is dead. The queue worker listens to the device's connectivity state, pauses draining when there's no usable connection, and resumes when connectivity returns. The OS saying you have a connection and you actually reaching our servers are different claims, especially on captive portals and on cells that accept a connection and go nowhere. So we treat connectivity as a hint to try, not a guarantee, and still lean on backoff when the OS is optimistic and the network isn't.

The side effect is battery and data: we're not burning the patient's battery or their limited data plan on requests that can't succeed.

Telling the Truth in the UI: Optimistic, But Honest

None of this is visible unless the interface reflects it, so we used optimistic UI with explicit states. The moment a patient taps Confirm, the appointment shows up in their list with a "Pending" indicator. No spinner: they see the thing they wanted, labeled as still syncing.

When the intent succeeds, the pending indicator disappears and it becomes a confirmed appointment. When it genuinely fails, after retries are exhausted or on an error a retry can't fix, it flips to "Failed, tap to retry" instead of vanishing. We never lie about state: showing something as done before it is done is fine, as long as you mark it provisional and say so the moment it goes wrong.

For reads, we keep a local cache, so opening the app with no signal still shows the appointments and messages from your last sync, marked clearly so stale data doesn't look live.

Conflicts on Reconnect

Queuing changes for later means the world can move underneath you. A patient cancels an appointment while offline; meanwhile the clinic reschedules it. When the phone reconnects, the queued cancel meets a slot that has changed. The server is the source of truth, so it rejects an intent whose preconditions no longer hold, and the app surfaces that instead of forcing the stale action through. Conflicts are rare, but with a patient's care involved, rare still has to be handled.

What It Actually Changed

We rolled the offline layer out gradually, behind a flag. Failed booking attempts, where a patient's intent to book never resulted in a booking, dropped by around 70%. Support tickets in the "I booked but it didn't work" and "the app is stuck loading" categories fell by roughly half. Duplicate bookings went to essentially zero.

The deeper win was trust. When patients believe that tapping a button does the thing, even from an elevator, they use the app more and call support less. We didn't make Vietnam's mobile networks better. We stopped pretending they were good.

Key Takeaways

• Design for the degraded network, not the absent one: true offline is easy to detect and handle; the common case is a connection alive enough to start a request and too flaky to finish it.

• Persist intents before you send them: a durable queue on disk that survives app kills and dead batteries turns "I don't know if it worked" into "it will finish when it can."

• Idempotency before retries, always: retrying without idempotency keys doesn't make you resilient, it books the same slot twice. Generate a stable key per intent and let the server dedupe.

• Backoff needs jitter: exponential backoff alone still lets a whole region's phones retry in synchronized waves. Randomized jitter spreads them out so your recovery doesn't become your outage.

• Optimistic UI must stay honest: show the result instantly, label it as pending, and say so the moment it fails. Users forgive "syncing"; they don't forgive an app that silently loses their booking.