A warehouse supervisor clicks Release shipment. The order service commits the release. Before its reply reaches the supervisor, the server loses power. A replacement server opens the order in a second. The supervisor is still signed in. Should the Release button be available?
There are two possible histories behind that same screen. In one, the release committed and the carrier may already have a pickup request. In the other, the release never committed and the pallet is still waiting. Telling the supervisor to try again is a business decision. So is telling her that everything succeeded. This hypothetical shipment illustrates a problem Amazon documents from building distributed services: a caller that loses the response can have less information about the outcome than the service that acted. AWS Builders’ Library: Making retries safe with idempotent APIs
Six things a system can mean by “recovered”
Start with what the screen can establish. Authentication answers who signed in. Session persistence lets another request recover that user’s server-side context and check whether their authority still holds. Browser recovery might bring back the tab and some typed values. Application recovery can reopen a saved draft on a different machine. Database recovery can establish which local transactions committed. End-to-end recovery answers the supervisor’s question: what actually happened to this shipment, at the warehouse and at the carrier, and what action is safe now?
Those distinctions have consequences even on a healthy network. Microsoft documents that ASP.NET Core session data is cache-backed and ephemeral. Its session mechanism is also non-locking: two requests can read a session, make different changes and have the later write overwrite the earlier one. Put that session in a shared cache and a second server can read it. Put an inventory reservation in it and two tabs can still disagree about who owns the last unit. Microsoft: Session and state management in ASP.NET Core
Each storage choice buys something different. Browser storage can make a draft feel instant, but another device cannot ordinarily retrieve it, and the browser cannot confer permission to release stock. Application-server memory is fast and convenient until the process disappears. A distributed cache makes shared reads inexpensive, with a recovery window determined by its persistence and eviction settings. A database-backed draft can carry a revision and ownership record across nodes at the cost of durable writes. None of those choices, by itself, records what the carrier accepted.
A durable workflow engine tackles another part of the problem. Azure Durable Task records orchestration history and can replay it to reconstruct execution after a restart. That introduces a discipline: replay must take the same decisions from the same history, while calls that affect the outside world belong in activities designed for possible repetition. A checkpoint saying “send pickup request” and a receipt saying “carrier accepted pickup 482” are different pieces of evidence. A workflow may need both. Microsoft Azure: Durable orchestrations
The hardest failure happens after the commit
Return to the warehouse. Give the release request an operation ID before its first submission. A deliberate second shipment gets a new ID. A retry of this shipment keeps the first one. Within one database transaction, the order service can check the order revision, reserve stock, record the release and write an instruction to notify the carrier. If any of those local writes fails, the transaction rolls back. If they commit, the replacement node can read a durable fact even though the original browser never received a reply.
The ID alone prevents nothing. The service must enforce uniqueness within the proper tenant and operation scope, bind the ID to the original intent, and record the result atomically with the change. If the retry asks to release twelve cartons under the ID previously used for ten, returning the old success would conceal a material discrepancy. Identical-looking orders can also be legitimate separate purchases. This is why Amazon favors a caller-provided request identifier over guessing intent from matching parameters. AWS: Idempotent API contracts and late requests
Now move the failure one step downstream. The carrier accepts the pickup and the sending worker dies before storing the receipt. The warehouse database has a valid release and a pending outbound instruction. Neither proves that a second pickup is needed. A replacement worker can ask the carrier for the result under the original reference or retry under a deduplication contract the carrier actually supports. If the carrier offers neither, the honest state is confirmation pending. Hold the request for reconciliation. Give the supervisor the release number, the known facts and a person or queue responsible for resolving it.
That produces four distinct decisions. Resume from a recorded completed step. Retry an unconfirmed operation when repeating it is safe. Reject a stale or unauthorized change. Reconcile when another system may already have acted. An interface that puts the same “Try again” button on all four has handed its concurrency policy to the person least able to see the evidence.
Who owns the next write?
Suppose customer service changes the delivery address while the warehouse is recovering. A CRM user is entitled to correct a customer’s address. That does not necessarily entitle the CRM to reroute a shipment already handed to a carrier. Give each business record an owner and define which transitions other applications may request. An address change against order revision 17 should produce an explicit conflict if dispatch has already advanced it to revision 18. Retaining the user’s typed address is helpful. Applying it silently to a dispatched shipment is a different act.
Optimistic concurrency works well when conflicting edits are uncommon: read a version, check it at commit, and show what changed if it no longer matches. Pessimistic concurrency reserves a contested record while a short decision completes. It can simplify the last-unit reservation but introduces waiting and deadlocks. Holding that lock while a person reviews a form or while a carrier answers turns an external delay into a database bottleneck. For invariants spanning several records, stronger isolation can be appropriate. PostgreSQL documents that serializable transactions can fail with a serialization conflict and that the application must retry the whole transaction, including the reads that informed its decision. PostgreSQL: Transaction isolation
Failover adds a competitor that ordinary load tests can miss: the former owner. A disconnected node may still be alive. Kubernetes uses leases for heartbeats and leader election, but the Kubernetes client library explicitly distinguishes leader election from fencing. One way to fence a writer is to give each successive owner a higher generation number and require the storage system to reject writes carrying an older one. The check belongs where the write takes effect. A former leader can believe it is still in charge long after everyone else has moved on. Kubernetes: Leases · Kubernetes client-go: Leader election and fencing limits
Inside the enterprise operational plane at JWilliams & Associates
JWilliams & Associates describes its approach as an enterprise operational plane. In practical terms, the database issues a session identifier, and the server uses it to resolve identity, tenant and current authority. A replacement application node can use the same authority and retrieve recoverable work. The useful property is that neither the right to act nor the evidence of prior work belongs exclusively to the node that first displayed the form.
Authority and progress still have separate lifetimes. A purchase request might outlast its author’s session by a week. A returning employee should be able to find the saved request after signing in again, while an employee whose approval role was revoked should lose the ability to approve it. Across applications, the accounting service needs authority for its own action and a reference to the purchase request. Passing the employee’s browser credential down the entire chain would conflate those responsibilities. These are design principles; the specific enforcement mechanisms used in JWA’s environment were not part of its test report.
Follow the release through three desks. Sales has the customer’s requested delivery address. The warehouse owns the decision to release stock. Accounting needs evidence of what shipped before it invoices. If the warehouse node goes down, each desk can restore its last screen and all three can still tell different stories. The architectural choice JWA describes is to make the work recoverable across those boundaries. The operator’s identity, the saved order revision and the progress of the release must remain intelligible to the next authorized participant, whichever node serves it.
A database-issued session gives the replacement node a place to ask who is acting, for which tenant, and under whose current authority. It cannot tell accounting whether to invoice. That answer belongs to the release itself: which version the operator approved, which changes committed together, and which outside acknowledgments are still missing. Conversely, a complete transaction history cannot authorize its next step. If the operator’s release privilege was withdrawn during the outage, the replacement node should preserve the work and refuse the release. A colleague at another tenant must never acquire the draft merely because their order has the same number.
Coordinating these responsibilities does not require every application to surrender its records to a single workflow engine. It requires agreement on a few consequential facts: who owns each transition, how other services identify the same operation, which outcome counts as final, and where to look when the outcome is uncertain. With those agreements, customer service can resume a draft, a warehouse worker can reconcile a pickup and accounting can wait for dispatch evidence. Each application contributes its own decision without independently reconstructing the entire shipment from a collection of timeouts.
This is the architectural significance of JWA’s approach. Automatic redistribution buys the time and capacity to continue. Durable progress identifies where to continue. Current authorization and concurrency controls determine whether the proposed continuation is legitimate. Synchronization makes a returning node useful again; a deliberate transfer of write authority makes it safe for that node to lead again. The case account describes how the environment behaved. Idempotent requests, version checks, transaction isolation and fencing are established ways to meet parts of that design objective. The account does not specify which of them JWA used.
Two physical interruptions, and the journey back to normal operation
First, the network cable. JWA reports physically disconnecting the active server’s Ethernet connection to test failure detection, workload redistribution and continuity. This is a particularly revealing fault because loss of contact leaves peers uncertain whether the old server has stopped executing. A health check can justify moving traffic. It cannot, on its own, settle the fate of an in-flight business operation.
Then, the power cable. In a separate test JWA reports abruptly disconnecting power from the active server. Restoration raised a second question: could the returning server catch up, rejoin the pool and safely resume its designated master role without an operator reconstructing the environment?
A simulated timeout is valuable because an engineer can place it precisely between a commit and its acknowledgment. Disconnecting a cable asks something broader: will the deployed environment notice a real loss of communication, redirect work, and cope with a server it can no longer observe? Cutting power denies the application the chance to finish a graceful shutdown. Neither test precisely selects which transaction is in flight. Together they exercise parts of recovery that a simulated application error can leave untouched, while leaving room for targeted concurrency tests afterward.
Across the two tests, JWA reports automatic failover, preservation of session authority and recoverable transaction progress, and some workstations needing a browser refresh. Users partway through forms could resume without reentering their information. JWA also reports that the restored server synchronized, rejoined load balancing and automatically resumed its designated master role. The refresh matters: an interruption was visible, yet it did not cost those users their work.
Failover was the move away from the interrupted server. Failback was the return to the intended operating arrangement after restoration. Between those events, the surviving environment could have accepted newer work. A returning server might hold a perfectly coherent, out-of-date view of the order. Rejoining a pool of servers that can answer reads, catching up on intervening writes and regaining authority to make the next write are three distinct milestones. JWA reports automatic restoration through that final milestone. Its account does not establish the order or mechanism of every internal handoff.
Consider the hardest concurrency test to add to those physical exercises. One operator refreshes a form and sees the ten cartons she entered. A second has reduced the order to eight. A replacement worker is checking whether the carrier accepted ten, while the old worker regains connectivity with its original command still in memory. Restoring the first operator’s ten is correct as a draft. Treating ten as committed inventory would overwrite a competing decision. Sending eight as a fresh pickup before resolving the first ten could create two collections. The recovery path needs to identify the current record owner, reject any stale write, preserve both operators’ evidence, and resolve the carrier reference before another release. A valid login answers none of those questions on its own.
These are company-reported observations. The account supplies neither a measured recovery time nor an independent transaction-by-transaction comparison across downstream systems. Restoring a half-completed form is evidence about draft continuity. Proving that two competing releases consumed exactly one unit of inventory would require a different test and different evidence. Likewise, returning a server to service and returning write authority are separate events. Microsoft’s SQL Server availability-group documentation illustrates the distinction: automatic failover depends on configuration, synchronization and quorum. Reachability alone does not qualify a replica to take over. Microsoft: Conditions for SQL Server automatic failover
Different industries have different definitions of “done”
Consider a credit union transfer and a legal filing. In the first, restored balances must agree with the ledger entries that moved the money. In the second, a recovered document and a successful submission are separate milestones; the filing receipt matters. Healthcare operations add a human handoff: a clinician resuming a referral needs to know whether the receiving practice acknowledged it, and whether someone changed the instructions in the meantime. These are illustrative examples, not claims about JWA deployments.
In manufacturing, repeating a materials issue can distort the cost of a production run. In distribution, retrying the same order release can consume the last pallet twice. ERP and supply-chain teams therefore need to trace one movement across reservation, pick, receipt and posting. In telecommunications a provisioning request can succeed at one device and stall at another. In field service and security operations, a technician may finish a job offline while dispatch reassigns it. A timestamp from the returning handset cannot decide which person is authorized to close the incident.
A government benefits office needs to show who authorized a determination and which evidence was current. A multi-tenant SaaS vendor must also prove that a failover lookup cannot retrieve another tenant’s similarly numbered case. In sales intelligence and CRM, merging duplicate contacts is quite different from replaying an email campaign. In media publishing, restoring the article is easy compared with answering whether an external channel accepted the post just before the connection dropped.
Autonomous AI agents sharpen the same question. A recovered agent may reconstruct why it proposed a purchase. Before it calls a purchasing tool again, the platform needs the original approval, the scope of that approval, the operation ID and the tool’s recorded outcome. A persuasive reconstruction of the agent’s reasoning is poor evidence that a supplier never received an order.
What the outage report should tell the executive team
“Service restored” is a timestamp. A credit union still needs a count of transfers whose debit and credit entries agree, and a queue for any instruction awaiting settlement evidence. A law firm facing a filing deadline needs to separate documents recovered from submissions acknowledged by the court. A hospital referral team needs an owner for each referral the receiving practice has yet to acknowledge. In each case, an apparently healthy dashboard can conceal work that someone must resolve before a deadline passes.
The financial exposure can often be measured in familiar operating units. Ask a manufacturer how many units it has promised against stock with an unresolved reservation. Ask a distributor which dispatches have a warehouse release but no carrier receipt. Ask a telecom operator which customers were billed for a service only partly provisioned. In government, an approval that cannot be tied to the responsible official and the evidence they saw can require a case review. In security operations, an unacknowledged reassignment can leave two officers responding to one site while another waits. These are different costs: cash to reconcile, commitments to renegotiate, deadlines to protect, and time during which no one knows who owns the work.
A CIO can ask for four figures after an exercise: how many operations were in flight, how many have a verified final outcome, how many conflicting or duplicate actions were intercepted, and how long the oldest unresolved operation has been waiting for an owner. Break those figures down by tenant and by workflow. In a shared SaaS or AI service, aggregate availability can look excellent while one customer’s orders are stalled or one agent keeps retrying an external action. These are proposed acceptance measures, not measurements from JWA’s tests.
The acceptance test a business owner can run
Bring an operator, an application engineer and the owner of the downstream system to the same test. Interrupt one shipment at three points: just before local commit, just after commit but before the browser gets its answer, and just after the carrier acts but before its receipt is saved. During the interruption, have a second authorized user amend the order. Restore the old node while the replacement is still working. Compare the draft, order revision, inventory movement and carrier receipt. Record what the operator saw and which writes the system rejected. An unresolved carrier outcome is an acceptable test result if the system identifies it and keeps anyone from blindly releasing the shipment again.
The supervisor who clicked Release needs a defensible answer: who can act now, what was saved, what committed, what the carrier knows and what happens next. A resilient enterprise system preserves the authority, context, transaction progress and consistency required to answer those questions after the machines change underneath it.
The goal is business continuity with transactional integrity.
When your infrastructure fails, does your architecture preserve the login—or does it preserve the business operation?