Generator Function

A generator function is a specialized Python function that returns a lazy iterator (specifically, a generator object) instead of executing its body immediately and returning a single value. It is distinguished syntactically by the presence of one or more yield expressions. Rather than destroying its local state upon returning data, a generator function suspends its execution, preserves its local variables and instruction pointer, and yields control back to the caller.

def basic_generator():
    # Local state is initialized
    value = "data"
    yield value  # Execution suspends here; value is emitted
    # Execution resumes here on the next iteration

Core Mechanics

1. Instantiation vs. Execution Invoking a generator function does not execute its code block. Instead, it instantiates and returns a generator object, which implements the Python iterator protocol (__iter__() and __next__()).

2. State Suspension (yield) Execution of the function body begins only when next() (or the object’s __next__() method) is called. The interpreter executes the code until it encounters a yield expression. At this point, the function evaluates the expression, returns the result to the caller, and freezes its internal state.

3. Resumption Upon subsequent calls to next(), the generator resumes execution immediately following the last executed yield statement, retaining full access to its previously established local scope and bindings.

4. Termination (StopIteration) When the generator function reaches the end of its code block or encounters a return statement, it automatically raises a StopIteration exception. This signals to the caller (or a consuming for loop) that the iterator is exhausted. If a return statement includes a value (e.g., return "Done"), that value is attached to the StopIteration.value attribute.

def state_machine():
    x = 10
    yield x        # State frozen. Yields 10.
    
    x += 5
    yield x        # State frozen. Yields 15.
    
    return "End"   # Raises StopIteration("End")

gen = state_machine()  # Returns generator object
val1 = next(gen)       # val1 = 10
val2 = next(gen)       # val2 = 15

# next(gen)            # Raises StopIteration

Advanced Generator Interface

Because generator functions maintain state, Python extends their capabilities beyond simple iteration, allowing them to act as lightweight coroutines. The generator object exposes three methods to manipulate its suspended state:

• send(value): Resumes execution and injects value into the generator. The yield expression where the generator is currently paused evaluates to this injected value. (Note: The first call to a generator must be next() or send(None) to advance it to the first yield).
• throw(type, [value, [traceback]]): Resumes execution by raising the specified exception at the exact point of the suspended yield. The generator function can catch this exception using a try/except block.
• close(): Raises a GeneratorExit exception at the suspended yield. If the generator catches this exception, it must either raise GeneratorExit, raise StopIteration, or cleanly exit. A RuntimeError is raised only if the generator attempts to yield another value after catching GeneratorExit. If the generator raises any other exception during closure, that exception is propagated to the caller.

def bidirectional_generator():
    try:
        # Pauses and waits for a value via .send()
        received = yield "Initialized"
        yield f"Processed: {received}"
    except ValueError:
        # Executes if .throw(ValueError) is called at the first yield
        yield "Exception Handled"


# Demonstrating send()
gen1 = bidirectional_generator()
print(next(gen1))              # Outputs: Initialized
print(gen1.send("Data"))       # Outputs: Processed: Data


# Demonstrating throw()
gen2 = bidirectional_generator()
print(next(gen2))              # Outputs: Initialized
print(gen2.throw(ValueError))  # Outputs: Exception Handled

Delegation with yield from

A generator function can delegate part of its operations to another generator (or any iterable) using the yield from syntax. This establishes a transparent, bidirectional communication channel between the caller and the sub-generator, automatically handling next(), send(), throw(), and close() routing, as well as catching the sub-generator’s StopIteration value.

def sub_generator():
    yield 1
    yield 2
    return "Sub Complete"

def delegating_generator():
    yield "Start"
    # Yields all values from sub_generator, captures its return value
    result = yield from sub_generator() 
    yield result

gen = delegating_generator()

# Yields sequence: "Start", 1, 2, "Sub Complete"

Master Python with Deep Grasping Methodology!

Learn More