Python generators are one-pass: stream data safely

A generator expression produces items on demand. Once a loop finishes consuming it, or sum() has consumed it, another loop over the same generator has nothing left. This matters when a CSV report tries to count rows and total amounts in separate passes.

Watch the second pass disappear

values = (number for number in [2, 3, 5])
print("total", sum(values))
print("remaining", list(values))

Do the work in one pass

The next example reads each CSV row once. It keeps only one count and Decimal total per subject. It does not hold all rows in memory. It still stores a dictionary entry for each distinct subject, so memory depends on the number of subjects.

import csv
import io
from collections import defaultdict
from decimal import Decimal

source = io.StringIO("subject,minutes\nSQL,30\nPython,20\nSQL,15\n", newline="")
reader = csv.DictReader(source)
summary = defaultdict(lambda: {"count": 0, "minutes": Decimal("0")})
for row in reader:
    subject = row["subject"]
    minutes = Decimal(row["minutes"])
    if not minutes.is_finite() or minutes < 0:
        raise ValueError("minutes must be finite and non-negative")
    summary[subject]["count"] += 1
    summary[subject]["minutes"] += minutes
for subject in sorted(summary):
    item = summary[subject]
    print(subject, item["count"], item["minutes"])

A loop consumes one item at a time

The one-pass summary is not merely a shorter way to write two loops. The CSV reader yields each parsed row once, and both aggregates are updated while that row is available. If an exception interrupts the loop, the totals are partial; do not publish them as a complete report. Validate each row first, and only present results after the input has been fully processed.

Where memory still grows

Streaming avoids storing every input row, but the summary dictionary grows with the number of distinct subjects. For a few subjects this is small. If nearly every row has a different key, memory can still grow with input size. A database aggregation, external sort, or disk-backed approach may fit better for very high-cardinality data.

When you really need two passes

For a file on disk, close and reopen it to create a fresh reader, or seek back to the beginning and create a new DictReader if the stream supports seeking. Do not assume a network stream or generator can rewind. If you need random access, materialising rows may be appropriate for a bounded data set.

A generator is a way to control when work happens. It does not by itself make parsing faster, protect against malformed input, or make an unbounded number of groups fit in memory.

If two independent passes are required over a local file, open it again and build a fresh DictReader for the second pass. Reusing the original generator does not rewind it. Keep the file open inside a with block while iterating; returning a lazy reader from a function after its file has closed will fail when callers try to consume it.

Choose the right abstraction

Use a generator when you need sequential processing and can finish the work in one pass. Use a list when the data is small and you genuinely need repeated access, sorting, or indexing. The choice is about the access pattern and memory bound, not a claim that generators always run faster.