Skip to main content

Generators in Python: A Complete Guide

Published 3 min readMarkdown

Introduction to Generators

Generators in Python are a special type of iterable that allow lazy evaluation, meaning they generate values one at a time instead of storing them all in memory at once. They are useful for handling large datasets, optimizing memory usage, and improving performance in scenarios where iterating over huge amounts of data is required.

Why Use Generators?

  • Improves Memory Efficiency: Unlike lists, generators do not store all values in memory.

  • Faster Execution: Values are generated as needed, reducing unnecessary processing.

  • Supports Infinite Sequences: You can define generators that run indefinitely.

  • Simplifies Complex Iteration Logic: Avoids using multiple temporary variables.


Creating a Basic Generator

A generator function uses the yield keyword to return values lazily.

Example 1: Simple Generator

def simple_generator():
    yield 1
    yield 2
    yield 3

gen = simple_generator()

print(next(gen))  # Output: 1
print(next(gen))  # Output: 2
print(next(gen))  # Output: 3

How It Works

  • The function simple_generator() does not return all values at once.

  • It pauses execution at each yield and resumes when next() is called.

  • Once all values are yielded, further next() calls raise StopIteration.

Iterating Over a Generator Using for Loop

A generator can be iterated over using a for loop:

def number_generator():
    for i in range(5):
        yield i

for num in number_generator():
    print(num)

Output

0
1
2
3
4

This avoids calling next() manually and ensures graceful handling of StopIteration.

Generator Expressions

Generators can be created using generator expressions, similar to list comprehensions.

gen_exp = (x * x for x in range(5))

print(next(gen_exp))  # Output: 0
print(next(gen_exp))  # Output: 1
print(next(gen_exp))  # Output: 4

This is more memory-efficient than list comprehensions ([x * x for x in range(5)]), which store all values in memory.

Generators vs Lists (Memory Usage)

Consider generating 1 million numbers:

Using Lists (Consumes More Memory)

nums_list = [x * x for x in range(1000000)]

Using Generators (Efficient Memory Usage)

nums_gen = (x * x for x in range(1000000))
  • Lists store all values in memory upfront.

  • Generators compute values only when needed, reducing memory footprint.

Generators for Infinite Sequences

Generators can be used to create infinite sequences:

def infinite_counter():
    num = 0
    while True:
        yield num
        num += 1

counter = infinite_counter()

print(next(counter))  # Output: 0
print(next(counter))  # Output: 1
print(next(counter))  # Output: 2

This is ideal for streaming data, real-time processing, or dynamically generating content.

Using yield with send()

Generators can receive values using the send() method.

def greeter():
    name = yield "Enter your name:"
    yield f"Hello, {name}!"

g = greeter()
print(next(g))       # Output: Enter your name:
print(g.send("Rahul"))  # Output: Hello, Rahul!
  • next() starts the generator.

  • send() sends a value to the yield expression.

Using yield from (Delegating Iteration)

To yield values from another iterable, use yield from:

def sub_generator():
    yield from [10, 20, 30]

for num in sub_generator():
    print(num)

Output

10
20
30

This is cleaner than looping through another iterable manually.

Real-World Applications of Generators

1. Processing Large Files Without Loading into Memory

def read_large_file(file_path):
    with open(file_path, "r") as file:
        for line in file:
            yield line

for line in read_large_file("data.txt"):
    print(line)
  • Processes huge files efficiently without reading the entire file at once.

2. Streaming Live Sensor Data

import random
import time

def sensor_stream():
    while True:
        yield random.uniform(20, 30)  # Simulating temperature sensor
        time.sleep(1)  # Wait for next reading

sensor = sensor_stream()

for _ in range(5):  
    print(next(sensor))
  • Continuously streams live data without storing previous values.

3. Generating Fibonacci Sequence

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib_gen = fibonacci()

for _ in range(10):  
    print(next(fib_gen))
  • Generates Fibonacci numbers without storing previous values.

About the author

  • I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.

No comments yet. Be the first to comment.

More from this blog