In Post #126, we were introduced to generators as a memory-efficient solution for working with large sequences. We saw how the built-in range() function allows us to loop over millions of numbers without storing them all in memory. But how do we create our own custom generators?
The simplest way is with a generator expression. If you understand list comprehensions from Post #120, you’re already 99% of the way there. In this post, we’ll learn the syntax for generator expressions and see why they are the go-to tool for lazy, memory-efficient iteration.
List Comprehension vs. Generator Expression
The syntax for a generator expression is almost identical to a list comprehension. The only difference is that you use parentheses () instead of square brackets [].
This single character change completely alters the behavior.
List Comprehension (Eager Evaluation):
# The square brackets create a full list in memory immediately.
my_list = [i**2 for i in range(5)]This creates a complete list in memory: [0, 1, 4, 9, 16].
Generator Expression (Lazy Evaluation):
# The parentheses create a generator object. No calculations are done yet.
my_generator = (i**2 for i in range(5))This does not create a list. It creates a special generator object that knows how to produce the values 0, 1, 4, 9, 16 one at a time, but only when asked.
The Immediate Benefit: Memory and Speed
Let’s revisit the memory-hungry example from Post #125 to see the practical difference.
# Eager: This will pause and use a lot of RAM to build the list
list_comp = [i**2 for i in range(10_000_000)]
print("List comprehension is built.")
# Lazy: This is instantaneous and uses almost no RAM
gen_exp = (i**2 for i in range(10_000_000))
print("Generator expression is created.")
print(gen_exp)The output shows the difference:
List comprehension is built.
Generator expression is created.
<generator object <genexpr> at 0x...>The list comprehension forces a delay while it builds the massive list in memory. The generator expression, however, is created instantly because it hasn’t actually done any work yet—it’s just an object waiting for instructions.
Consuming a Generator Expression
A generator object doesn’t do any work until you iterate over it. The most common way to “consume” the values from a generator is with a for loop.
# The generator expression only does work when the for loop asks for a value
squares_generator = (i**2 for i in range(5))
for value in squares_generator:
# Each value is calculated one at a time as the loop proceeds
print(value)Each time the for loop asks for the next item, the generator expression runs just enough to produce that one item and “yields” it to the loop.
An Important Rule: Generators are Single-Use
This is a critical difference from lists. A generator is like a disposable iterator. Once you have looped over it completely, it is exhausted and cannot be used again.
gen_exp = (i for i in range(3)) # Creates a generator for 0, 1, 2
print("First pass:")
for item in gen_exp:
print(item)
print("\nSecond pass:")
for item in gen_exp:
# This loop will not run at all
print(item)
print("Finished second pass.")The output will be:
First pass:
0
1
2
Second pass:
Finished second pass.The second for loop produces no output because the generator has no more values to give. If you need to iterate over the values again, you must create a new generator expression.
What’s Next?
Generator expressions offer the concise power of comprehensions with the memory efficiency of lazy evaluation. By simply changing [] to (), you can switch from eagerly building a list in memory to creating a generator that produces values on demand. This is the Pythonic way to handle large data streams.
Generator expressions are fantastic for creating simple, one-line generators. But what if your generator logic is more complex and requires multiple lines of code? For this, Python provides another way to build generators using a standard function and a special keyword. In Post #128, we will learn how to build our own generator functions using the yield keyword.
Author
Experienced Cloud & DevOps Engineer with hands-on experience in AWS, GCP, Terraform, Ansible, ELK, Docker, Git, GitLab, Python, PowerShell, Shell, and theoretical knowledge on Azure, Kubernetes & Jenkins. In my free time, I write blogs on ckdbtech.com