Distributed transactions: two-phase commit and sagas
Why two-phase commit blocks, and sagas of local transactions with compensations, orchestrated or choreographed.
What you will learn
Explain why two-phase commit blocks when its coordinator fails.
Design a saga from compensable steps, a pivot and retryable steps.
Choose between choreography and orchestration, and counter the saga's lack of isolation.
Two-phase commit and why it blocks
Inside one database, a transaction is atomic across tables. With a database per service, one business operation spans several databases.
The classic protocol for a transaction across databases is two-phase commit (2PC):
Prepare: the coordinator asks every participant to promise that it can commit. Each participant must keep holding its locks until it learns the outcome.
Commit: if every participant voted yes, the coordinator tells them all to commit; otherwise, to roll back.
Gray and Lamport state the core problem in one line: "The classic Two-Phase Commit protocol blocks if the coordinator fails." A participant that has voted yes cannot decide on its own, so its locks stay held until the coordinator comes back. Their Paxos Commit protocol removes the block by running consensus on the decision across 2F+1what its symbols mean coordinators, and it makes progress while F+1 of them work. AWS's guidance goes further for microservices: with a database per service, "the two-phase commit is not an option", and it recommends sagas instead.
The saga: a sequence of local transactions
Garcia-Molina and Salem (1987) introduced the saga for long-lived transactions. It is a sequence of transactions T1,T2,…,Tnwhat its symbols mean, each of which commits on its own, with a compensating transactionCiwhat its symbols mean for each Tiwhat its symbols mean. If Tkwhat its symbols mean fails, the steps that already committed are undone in reverse:
T1→⋯→Tk−1→(Tk fails)→Ck−1→⋯→C1what its symbols mean
Richardson defines it for microservices as "a reliably coordinated sequence of local transactions". There is no automatic rollback: "a developer must design compensating transactions that explicitly undo changes made earlier in a saga". A compensation undoes a step's effect for the business. It does not restore the old bytes; a payment, for example, is refunded rather than deleted.
Order the steps: compensable, pivot, retryable
The Azure Architecture Center sorts saga steps into three kinds:
Compensable steps can be undone by another transaction with the opposite effect.
The pivot is "the point of no return". Once it succeeds, the saga must run to completion.
Retryable steps come after the pivot. They are idempotent and retried until they succeed, instead of being undone.
A checkout saga shaped this way:
Step
Kind
Compensation
T1what its symbols mean Create order as PENDING
Compensable
C1what its symbols mean Set status to CANCELED
T2what its symbols mean Reserve stock
Compensable
C2what its symbols mean Release the reservation
T3what its symbols mean Charge the card
Pivot
none needed: if it fails, C2 and C1 run
T4what its symbols mean Create the shipment
Retryable
none: retried until it succeeds
Reserving stock before charging means an out-of-stock failure never needs a refund.
Choreography and orchestration
Choreography: "each local transaction publishes domain events that trigger local transactions in other services" (Richardson). There is no central coordinator.
Orchestration: "an orchestrator (object) tells the participants what local transactions to execute". AWS's guidance shows AWS Step Functions in this role.
Choreography
Orchestration
Suits
Simple workflows with few services
Complex workflows, or ones that gain new steps
Single point of failure
None; responsibility is spread across participants
The orchestrator, unless it is itself highly available
Dependencies
Risk of cyclic dependencies between participants
Avoided, because the orchestrator owns the flow
Understanding the flow
Hard to track which events each participant responds to
Flow is in one place; coordination logic must be written
(Summarized from the Azure Architecture Center and AWS Prescriptive Guidance.)
Either way, each participant must update its database and publish its message atomically (the transactional outbox, chapter 7). It must also be idempotent, because messages can be redelivered.
Lack of isolation
Each step commits at once, so other transactions can see a saga's intermediate states. Azure lists the resulting anomalies as lost updates, dirty reads and fuzzy reads, also called non-repeatable reads. The countermeasures include:
Semantic lock: an application-level flag such as status = 'PENDING' tells other transactions the record is mid-saga. Richardson calls the PENDING state "a semantic lock counter-measure".
Commutative updates: design updates so they give the same result in any order.
Pessimistic view: reorder the saga so updates happen in retryable steps.
Reread values: check the data has not changed before updating it.
Compensations can fail too, which can leave the system inconsistent (Azure), so they need retries and monitoring of their own.
Checkout saga
Local transactions in order; on failure, compensations run in reverse. Steps after the pivot are retried, not undone.
Coordination
T1 · CompensableOrderServiceCreateOrderC1: Set order status to 'CANCELED'Not started
T2 · CompensableInventoryServiceReserveStockC2: Release the reserved unitsNot started
T3 · PivotPaymentServiceChargeCardPoint of no returnNot started
T4 · RetryableShippingServiceCreateShipmentRetried until it succeedsNot started
An orchestrated saga: one orchestrator runs each step and, after a failure, the compensations.
Follow a checkout: the orchestrator runs the saga’s steps in order, each a local transaction in one service: create the order as pending, reserve stock, charge the card, create the shipment. If charging the card fails, it runs the compensations in reverse: release the stock, then cancel the order. Commands and replies travel as messages, and the orchestrator records how far each saga has got, so that it can carry on after a crash.
Shoppers: 1 × Users
Load balancer: 2 × Load balancer
Saga orchestrator: 3 × App server
Order service: 3 × App server
Payment service: 3 × App server
Inventory service: 3 × App server
Message queue: 3 × Queue
Saga state: 2 × Relational database
Shoppers sends requests to Load balancer
Load balancer sends requests to Saga orchestrator
Saga orchestrator sends requests to Order service
Saga orchestrator sends requests to Payment service
Saga orchestrator sends requests to Inventory service
Saga orchestrator sends requests to Message queue
Saga orchestrator sends requests to Saga state
Key takeaways
In two-phase commit a participant that voted yes holds its locks until the coordinator decides, so a failed coordinator blocks it.
A sagacommits each step locally and undoes earlier steps with hand-written compensations; after the pivot, steps are retried instead of undone.
Choreography suits a few services and has no coordinator; orchestration keeps a complex flow in one place.
Others can see a saga's intermediate states, so use semantic locks such as a PENDING status, commutative updates or rereads, and monitor compensations, which can fail too.