Lesson 8.3• 10 min read

Distributed transactions: two-phase commit and sagas

Why two-phase commit blocks, and sagas of local transactions with compensations, orchestrated or choreographed.

What you will learn

  • Explain why two-phase commit blocks when its coordinator fails.
  • Design a saga from compensable steps, a pivot and retryable steps.
  • Choose between choreography and orchestration, and counter the saga's lack of isolation.

Two-phase commit and why it blocks

Inside one database, a transaction is atomic across tables. With a database per service, one business operation spans several databases.

The classic protocol for a transaction across databases is two-phase commit (2PC):

  1. Prepare: the coordinator asks every participant to promise that it can commit. Each participant must keep holding its locks until it learns the outcome.
  2. Commit: if every participant voted yes, the coordinator tells them all to commit; otherwise, to roll back.

Gray and Lamport state the core problem in one line: "The classic Two-Phase Commit protocol blocks if the coordinator fails." A participant that has voted yes cannot decide on its own, so its locks stay held until the coordinator comes back. Their Paxos Commit protocol removes the block by running consensus on the decision across 2F+12F + 1what its symbols mean coordinators, and it makes progress while F+1F + 1 of them work. AWS's guidance goes further for microservices: with a database per service, "the two-phase commit is not an option", and it recommends sagas instead.


The saga: a sequence of local transactions

Garcia-Molina and Salem (1987) introduced the saga for long-lived transactions. It is a sequence of transactions T1,T2,…,TnT_1, T_2, \dots, T_nwhat its symbols mean, each of which commits on its own, with a compensating transaction CiC_iwhat its symbols mean for each TiT_iwhat its symbols mean. If TkT_kwhat its symbols mean fails, the steps that already committed are undone in reverse:

T1→⋯→Tk−1→(Tk fails)→Ck−1→⋯→C1T_1 \rightarrow \dots \rightarrow T_{k-1} \rightarrow \text{(}T_k\text{ fails)} \rightarrow C_{k-1} \rightarrow \dots \rightarrow C_1what its symbols mean

Richardson defines it for microservices as "a reliably coordinated sequence of local transactions". There is no automatic rollback: "a developer must design compensating transactions that explicitly undo changes made earlier in a saga". A compensation undoes a step's effect for the business. It does not restore the old bytes; a payment, for example, is refunded rather than deleted.

Order the steps: compensable, pivot, retryable

The Azure Architecture Center sorts saga steps into three kinds:

  • Compensable steps can be undone by another transaction with the opposite effect.
  • The pivot is "the point of no return". Once it succeeds, the saga must run to completion.
  • Retryable steps come after the pivot. They are idempotent and retried until they succeed, instead of being undone.

A checkout saga shaped this way:

Step Kind Compensation
T1T_1what its symbols mean Create order as PENDING Compensable C1C_1what its symbols mean Set status to CANCELED
T2T_2what its symbols mean Reserve stock Compensable C2C_2what its symbols mean Release the reservation
T3T_3what its symbols mean Charge the card Pivot none needed: if it fails, C2C_2 and C1C_1 run
T4T_4what its symbols mean Create the shipment Retryable none: retried until it succeeds

Reserving stock before charging means an out-of-stock failure never needs a refund.


Choreography and orchestration

  • Choreography: "each local transaction publishes domain events that trigger local transactions in other services" (Richardson). There is no central coordinator.
  • Orchestration: "an orchestrator (object) tells the participants what local transactions to execute". AWS's guidance shows AWS Step Functions in this role.
Choreography Orchestration
Suits Simple workflows with few services Complex workflows, or ones that gain new steps
Single point of failure None; responsibility is spread across participants The orchestrator, unless it is itself highly available
Dependencies Risk of cyclic dependencies between participants Avoided, because the orchestrator owns the flow
Understanding the flow Hard to track which events each participant responds to Flow is in one place; coordination logic must be written

(Summarized from the Azure Architecture Center and AWS Prescriptive Guidance.)

Either way, each participant must update its database and publish its message atomically (the transactional outbox, chapter 7). It must also be idempotent, because messages can be redelivered.


Lack of isolation

Each step commits at once, so other transactions can see a saga's intermediate states. Azure lists the resulting anomalies as lost updates, dirty reads and fuzzy reads, also called non-repeatable reads. The countermeasures include:

  • Semantic lock: an application-level flag such as status = 'PENDING' tells other transactions the record is mid-saga. Richardson calls the PENDING state "a semantic lock counter-measure".
  • Commutative updates: design updates so they give the same result in any order.
  • Pessimistic view: reorder the saga so updates happen in retryable steps.
  • Reread values: check the data has not changed before updating it.

Compensations can fail too, which can leave the system inconsistent (Azure), so they need retries and monitoring of their own.

Checkout saga

Local transactions in order; on failure, compensations run in reverse. Steps after the pivot are retried, not undone.

Coordination
  1. T1 · CompensableOrderServiceCreateOrderC1: Set order status to 'CANCELED'Not started
  2. T2 · CompensableInventoryServiceReserveStockC2: Release the reserved unitsNot started
  3. T3 · PivotPaymentServiceChargeCardPoint of no returnNot started
  4. T4 · RetryableShippingServiceCreateShipmentRetried until it succeedsNot started
Outcome: not run yet
Pick a failure and run the saga.
Shoppers1× instanceLoad balancer2× instancesSaga orchestrator3× instancesOrder service3× instancesPayment service3× instancesInventory service3× instancesMessage queue3× instancesSaga state2× instancesShoppers1× instanceLoad balancer2× instancesSaga orchestrator3× instancesOrder service3× instancesPayment service3× instancesInventory service3× instancesMessage queue3× instancesSaga state2× instances
Figure 3

An orchestrated saga: one orchestrator runs each step and, after a failure, the compensations.

Follow a checkout: the orchestrator runs the saga’s steps in order, each a local transaction in one service: create the order as pending, reserve stock, charge the card, create the shipment. If charging the card fails, it runs the compensations in reverse: release the stock, then cancel the order. Commands and replies travel as messages, and the orchestrator records how far each saga has got, so that it can carry on after a crash.

  • Shoppers: 1 × Users
  • Load balancer: 2 × Load balancer
  • Saga orchestrator: 3 × App server
  • Order service: 3 × App server
  • Payment service: 3 × App server
  • Inventory service: 3 × App server
  • Message queue: 3 × Queue
  • Saga state: 2 × Relational database
  • Shoppers sends requests to Load balancer
  • Load balancer sends requests to Saga orchestrator
  • Saga orchestrator sends requests to Order service
  • Saga orchestrator sends requests to Payment service
  • Saga orchestrator sends requests to Inventory service
  • Saga orchestrator sends requests to Message queue
  • Saga orchestrator sends requests to Saga state

Key takeaways

  • In two-phase commit a participant that voted yes holds its locks until the coordinator decides, so a failed coordinator blocks it.
  • A saga commits each step locally and undoes earlier steps with hand-written compensations; after the pivot, steps are retried instead of undone.
  • Choreography suits a few services and has no coordinator; orchestration keeps a complex flow in one place.
  • Others can see a saga's intermediate states, so use semantic locks such as a PENDING status, commutative updates or rereads, and monitor compensations, which can fail too.
Sources8