Objectives

By the end of this chapter, you should be able to:

• Explain what a generator is and how to create one in Python
• Understand what the yield keyword and next function do
• Define and create iterators using the iter function
• Access values and indexes using enumerate

Generators

Generator functions are functions that allow us to return multiple times using the yield keyword. This allows us to generate many values over time from a single function. What makes generators so powerful is that unlike other forms of iteration, the values are not all computed upfront so we can suspend our state using the yield keyword and come back to to the function later to continue on. This makes generators a great choice for things like calculating large data sets.

def gensquares(n):
    for num in range(n):
        yield num**2

for x in gensquares(10):
    print(x)

def fib_with_generator(n):
    a = 1
    b = 1
    for i in range(n):
        yield a
        # a,b = b, a+b -> tuple unpacking instead of the three lines below!
        temp = a
        a = b
        b = temp+b

for num in fib_with_generator(10):
    print(num)

Using the next function

Given a generator, you can obtain the next value by calling a special function called next and passing in the generator. Here's an example:

def use_next():
    for x in range(10):
        yield x

gen = use_next()
print(next(gen)) # 0
print(next(gen)) # 1
print(next(gen)) # 2

In the example above, you can't call next infinitely many times: eventually you'll get a StopIteration error (since x inside of use_next only has finitely many values).

However, if you iterate through a generator using something like a for loop, the loop will catch the error so that it doesn't break your program:

for val in use_next():
    print(val)

# 0
# 1
# 2
# 3
# 4
# 5
# 6
# 7
# 8
# 9

We can also do generator comprehension just like list comprehension by wrapping our comprehension in () to write generator functions with more ease. Here's how use_next would look as a generator comprehension:

def use_next():
    return (x for x in range(10))

You can learn more about generators with these three resources:

http://asmeurer.github.io/python3-presentation/python3-presentation.pdf

http://stackoverflow.com/questions/1756096/understanding-generators-in-python

(http://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do-in-python?rq=1

Iterators

To make something an iterable (i.e. something you can iterate over) we call the iter function on it. For example, if we wanted strings to be iterators, we could do:

str = "hello"
str_iter = iter(str)
next(str_iter) # h
next(str_iter) # e
next(str_iter) # l
next(str_iter) # l
next(str_iter) # o
next(str_iter) # StopIteration Error!

At this point, you may be wondering: what's the difference between an iterator and a generator? An iterator is a more general concept: anything in Python with an __next__ method used to produce some next value is an iterator. Generators are iterators, but not every iterator is a generator.

Enumerate

Sometimes when you iterate through an array, you want access not only to the elements, but also their indices. enumerate exposes both to you. It works by returning a tuple with the (index, value) at each iteration.

list = ["first","second","third"]

# How do we get the indices at each iteration? Enumerate!

for idx, value in enumerate(list):
    print(f"index is {idx} and value is {value}")

# index is 0 and value is first
# index is 1 and value is second
# index is 2 and value is third

all and any

These are both built in functions that all us to check for a boolean matching in an iterable.

all - returns true if all elements are truthy

all([0]) # False
all([0,1]) # False
all([0, "", [1]]) # False
all([1, "a", [1]]) # True

any - returns true if any elements are truthy

any([0]) # False
any([0,1]) # True
any([0, "", [1]]) # True

Further Reading: Itertools

To use some more advanced iterators, you can use the itertools module, which you can learn more about here.