Generators and Lazy Iteration: Process Millions of Records

The direct answer

Do not load millions of records into a list. Read, transform, and aggregate records one at a time with a generator, keeping only a count, sum, or bounded buffer. Memory stays roughly flat when each record has a bounded size and you do not retain all the results; appending every output to a list brings the growth back.

The problem, with numbers

range is already lazy: it is a compact sequence object that computes values on demand. It is an iterable, not an iterator, so you can loop over the same range many times. list(range(...)) builds every element up front.

sys.getsizeof measures only the container object itself. For the list, that is mainly its internal array of references to the integer objects. The integers it refers to take additional memory on top, so the real gap is larger than these numbers show. Neither figure is total process memory, and exact sizes depend on the Python build.

Verified in this site's Pyodide 0.27.7 browser worker: the range object is 24 bytes, while the list container is 40,000,028 bytes—about 40 MB before counting the integer objects. This WebAssembly build uses 4-byte references, so the list holds 10,000,000 × 4 bytes of references plus 28 bytes of overhead. Native 64-bit CPython can report different figures.

import sys

print("range object:", sys.getsizeof(range(10_000_000)), "bytes")
print("list object: ", sys.getsizeof(list(range(10_000_000))), "bytes")
print("sum:", sum(range(10_000_000)))

range object: 24 bytes
list object:  40000028 bytes
sum: 49999995000000

Generator expressions vs list comprehensions

The syntax differs only in the brackets. Square brackets build the full list; parentheses create a generator object that yields one value at a time. When you pass a generator expression as the only argument, as in sum(i * i for i in ...), the extra parentheses can be dropped.

import sys

print("generator expression:", sys.getsizeof((i * i for i in range(1_000_000))), "bytes")
print("list comprehension:  ", sys.getsizeof([i * i for i in range(1_000_000)]), "bytes")
print("sum:", sum(i * i for i in range(1_000_000)))

generator expression: 104 bytes
list comprehension:   4224364 bytes
sum: 333332833333500000

Generator functions and yield

A function containing yield returns a generator when called; its body does not run yet. Each next() runs the body until the next yield, hands back that value, and suspends with local variables intact. Here the first next(g) yields 0, the second resumes and yields 1, and the third yields 4. list(squares(5)) consumes a separate, fresh generator.

def squares(n):
    for i in range(n):
        yield i * i

g = squares(5)
print(next(g))
print(next(g))
print(next(g))
print(list(squares(5)))

0
1
4
[0, 1, 4, 9, 16]

Pipelines: chain generators, never materialize

Split processing into small generator stages: read, parse, filter, aggregate. Each stage pulls one item from the previous stage, so only one record is in flight at a time. io.StringIO simulates a file here; a real file object behaves the same way, because iterating a file yields one line at a time.

import io

def read_numbers(lines):
    for line in lines:
        yield int(line.strip())

data = io.StringIO("10\n25\n7\n40\n3\n")
print(sum(n for n in read_numbers(data) if n > 10))

65

itertools for lazy work

itertools.count() is an infinite stream; islice takes the first N items without ever building the rest. chain joins several iterables into one stream without copying them into a new list. dict.items() returns a view of the dictionary rather than a list copy of its pairs.

import itertools

squares = (i * i for i in itertools.count())
print(list(itertools.islice(squares, 5)))

batch_a = range(3)
batch_b = (n * 10 for n in range(3))
print(list(itertools.chain(batch_a, batch_b)))

stock = {"apples": 4, "pears": 9}
print(type(stock.items()).__name__)
for name, qty in stock.items():
    print(name, qty)

[0, 1, 4, 9, 16]
[0, 1, 2, 0, 10, 20]
dict_items
apples 4
pears 9

Chunked file reading

Avoid unbounded read() and readlines() on a large file; both load the whole content. One option is to group lines into batches of a fixed count. Each batch is itself a list, which is fine: memory is bounded by the batch size and line lengths, not the file size. Batches are not a fixed number of bytes, because lines vary in length. The outer list() below is only for displaying this tiny demo; process batches directly in a for loop for a real large file.

For data without useful line breaks, call read(size) in a loop. On a text file, size counts characters; on a file opened in binary mode ('rb'), it counts bytes and returns bytes objects. An empty result means the end of the file. The second example reads fixed-size character chunks without collecting them.

import io

def chunks(lines, size):
    buf = []
    for line in lines:
        buf.append(line.strip())
        if len(buf) == size:
            yield buf
            buf = []
    if buf:
        yield buf

print(list(chunks(io.StringIO("a\nb\nc\nd\ne\n"), 2)))

# Fixed-size character chunks
import io

def read_chunks(f, size):
    while True:
        chunk = f.read(size)
        if not chunk:
            break
        yield chunk

text = io.StringIO("abcdefghij")
for chunk in read_chunks(text, 4):
    print(repr(chunk))

[['a', 'b'], ['c', 'd'], ['e']]
'abcd'
'efgh'
'ij'

When not to use a generator

Use a list when you need random access or len(), when you must iterate more than once, or when the data is small; a list is simpler then and often faster. A generator is single-use: once consumed, it is empty. The example catches the TypeError from len() so the remaining lines still run.

g = (i for i in range(5))
try:
    len(g)
except TypeError as e:
    print("TypeError:", e)
print(sum(g))
print(list(g))

TypeError: object of type 'generator' has no len()
10
[]

FAQ

Is a generator always faster? No. Lazy means less memory, not less time. Per-item work is similar, and for small data a list can be quicker.

Can I get the length of a generator? No. It does not know how many items remain. Count while you consume it, or use a list if you need len().

What is the difference between an iterator and an iterable? An iterable can produce an iterator with iter(); lists, ranges, and dict views are iterables. An iterator returns items with next() and is exhausted when done; generators and file objects are iterators.

Do I need yield, or is range enough? range is already lazy. Use it directly when it fits, and write a generator when each item needs custom logic.

Try it live

Run the range-versus-list comparison in your browser. You can edit the code first; it runs with Pyodide in a Web Worker, and the first run downloads the Python runtime. Sizes may differ from native CPython.