Lesson 15.2• 10 min read

Stream processing versus batch recomputation

The primer’s two-step MapReduce, stream windows with watermarks and late data, Lambda versus Kappa, and decaying old sales.

What you will learn

  • Rank products with the primer's hourly two-step MapReduce.
  • Keep running totals with a stream processor, using windows, event time and watermarks.
  • Compare the Lambda and Kappa architectures, and weight recent sales with exponential decay.

The primer's answer: an hourly MapReduce

Results only need to be hourly. So the primer's Sales Rank Service runs a two-step MapReduce over the log files and writes the result to sales_rank:

  1. Sum: the mapper keeps lines from the past week (within_past_week) and emits (category, product_id), quantity. The reducer sums them.
  2. Sort: a second mapper turns each result into (category, total_sold), product_id. MapReduce's shuffle then sorts by that key, giving each category's products in order of sales.
def mapper(self, _, line):
    timestamp, product_id, category_id, quantity, total_price, seller_id, buyer_id = line.split('\t')
    if self.within_past_week(timestamp):
        yield (category_id, product_id), quantity

def reducer(self, key, values):
    yield key, sum(values)

def mapper_sort(self, key, value):
    category_id, product_id = key
    quantity = value
    yield (category_id, quantity), product_id

Recomputing from the raw log has a strong advantage. The log is complete and immutable, so the job counts every transaction in the window, including ones that were delayed in transit, and a buggy run can simply be rerun.


Updating continuously: stream processing

If rankings must change faster than hourly, a stream processor (Flink, Kafka Streams and others) can consume purchase events and keep running totals per window. The window types, as Flink defines them:

  • Tumbling windows "have a fixed size and do not overlap".
  • Sliding windows, which Flink also calls hopping windows, "can be overlapping if the slide is smaller than the window size", so one event is counted in several windows.

Late data. Events arrive out of order. Windows are defined by event time, when the purchase happened. A watermark tells the stream processor how far event time has progressed and triggers the windows that have ended. In Flink, "by default, late elements are dropped when the watermark is past the end of the window". Allowed lateness keeps windows open longer, at the cost of holding more state. The visual shows a late purchase dropped by the stream but counted by the batch recount.


Lambda and Kappa

Running both is the Lambda architecture. Jay Kreps describes it: "an immutable sequence of records is captured and fed into a batch system and a stream processing system in parallel", with results combined at query time. His objection: "maintaining code that needs to produce the same result in two complex distributed systems" is a lasting burden.

His alternative, which he jokingly named the Kappa architecture: keep the log, and to reprocess, "start a second instance of your stream processing job that starts processing from the beginning of the retained data", writing to a new output table.

For this feature, the primer's hourly batch already meets the requirement. A stream helps only if fresher rankings are worth running two systems, or reprocessing from the log.


Weighting recent sales

The primer ranks by total sales over the past week. To favor recent sales smoothly instead, weight each purchase by its age (how long ago it was made) with an exponential decay. With half-life T1/2T_{1/2}what its symbols mean:

w(age)=e−λ⋅age,λ=ln⁡2T1/2w(\text{age}) = e^{-\lambda \cdot \text{age}},\qquad \lambda = \frac{\ln 2}{T_{1/2}}what its symbols mean

With T1/2=24T_{1/2} = 24 hours, a sale from 24 hours ago counts half and one from 48 hours ago a quarter. This is one design option. Amazon says only that recent activity is "weighted more heavily".

Stream windows versus a batch recount

One-hour tumbling windows by event time. The stream's watermark trails the newest event by 5 minutes; events for a window that closed earlier than the allowed lateness are dropped, as in Flink. The batch job recounts every window from the full log.

Allowed lateness:
WindowKettle: stream / batchToaster: stream / batchBlender: stream / batch
00:00–01:003 / 32 / 21 / 1
01:00–02:001 / 12 / 20 / 0
Late events dropped by the stream: 0. Cells where the two disagree are highlighted; the batch recount, or more allowed lateness, fixes them.
Shoppers1× instanceWeb server2× instancesSales API2× instancesObject store (log files)1× instanceMessage queue(purchase events)2× instancesSales Rank Service(hourly)2× instancesStream processor2× instancesSQL database(sales_rank)2× instancesShoppers1× instanceWeb server2× instancesSales API2× instancesObject store (log files)1× instanceMessage queue(purchase events)2× instancesSales Rank Service(hourly)2× instancesStream processor2× instancesSQL database(sales_rank)2× instances
Figure 2

Two ways to keep rankings: an hourly MapReduce recounts the log files, while a stream processor keeps running totals per window.

The Sales API writes each purchase twice: as a line in its log files, kept in an object store, and as an event in a message queue. The batch path is the primer’s: every hour the Sales Rank Service runs a MapReduce over the past week of logs and writes the rankings to the SQL database. The stream path is the option for fresher rankings: a stream processor reads the purchase events, keeps running totals per window by event time, and updates the rankings as it goes. Running both side by side is the Lambda architecture. The Sales API is counted at the lab’s 1,000 requests a second a server for the primer’s 400 purchases a second, plus one spare, and the web server in front of it at 5,000 a second, plus one spare; the primer gives no counts for the rest, so each is drawn with two.

  • Shoppers: 1 × Users
  • Web server: 2 × Load balancer
  • Sales API: 2 × App server
  • Object store (log files): 1 × Object store
  • Message queue (purchase events): 2 × Queue
  • Sales Rank Service (hourly): 2 × App server
  • Stream processor: 2 × App server
  • SQL database (sales_rank): 2 × Relational database
  • Shoppers sends requests to Web server
  • Web server sends requests to Sales API
  • Sales API sends requests to Object store (log files)
  • Sales API sends requests to Message queue (purchase events)
  • Sales Rank Service (hourly) sends requests to Object store (log files)
  • Sales Rank Service (hourly) sends requests to SQL database (sales_rank)
  • Message queue (purchase events) sends requests to Stream processor
  • Stream processor sends requests to SQL database (sales_rank)

Key takeaways

  • Recomputing from the complete, immutable log counts every transaction in the window, late ones included, and a bad run can simply be rerun.
  • A stream processor keeps totals per window by event time; watermarks trigger windows that have ended; later events are dropped unless allowed lateness keeps the window open.
  • Lambda runs batch and stream side by side; Kappa keeps the log and reprocesses by replaying it through a second stream job.
  • The hourly batch already meets the primer's requirement; a stream is worth it only if fresher rankings justify the extra system.
Sources4