Python – Resolve evaluation time differences in generators

I found myself running into the “assessment time difference” problem from this list today, and I had a hard time solving this problem.

As a short demonstration of my problem, I made infinite generators that skip every n digits, n from [2..5]:

from itertools import count

skip_lists = []
for idx in range(2, 5):
    # skip every 2nd, 3rd, 4th.. number
    skip_lists.append(x for x in count() if (x % idx) != 0)

# print first 10 numbers of every skip_list
for skip_list in skip_lists:
    for _, num in zip(range(10), skip_list):
        print("{}, ".format(num), end="")
    print()

Expected output:

1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 
1, 2, 4, 5, 7, 8, 10, 11, 13, 14, 
1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 

Actual output:

1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 
1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 
1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 

Once I remembered this awesome feature, I tried to “solve” it by binding the if clause variable (bind) to a constant that would become part of the skip_list:

from itertools import count

skip_lists = []
for idx in range(2, 5):
    # bind the skip distance
    skip_lists.append([idx])
    # same as in the first try, but use bound value instead of 'idx' 
    skip_lists[-1].append(x for x in count() if (x % skip_lists[-1][0]) != 0)

# print first 10 numbers of every skip_list
for skip_list in (entry[1] for entry in skip_lists):
    for _, num in zip(range(10), skip_list):
        print("{}, ".format(num), end="")
    print()

But again:

1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 
1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 
1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 

In addition to the actual solution, I’m curious to know why my hack isn’t working.

The value of idx is

never looked up until you start iterating over the generator (the generator is lazily evaluated), at which point idx = 4 is the latest iteration value, which is present in the module scope.

You can make each additional generator stateful in the IDX by passing the IDX to the function and reading the value from the function range at the evaluation time of each generator. This takes advantage of the fact that the iterable source of the generator expression is evaluated at gen. exp’s creation time, so the function is called on each iteration of the loop, and idx is safely stored in the function scope:

from itertools import count

skip_lists = []

def skip_count(skip):
  return (x for x in count() if (x % skip) != 0)

for idx in range(2, 5):
    # skip every 2nd, 3rd, 4th.. number
    skip_lists.append(skip_count(idx))

An iterable source evaluation description of the generator expression. Creation of exp:

>>> (i for i in 5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable

Your case is a bit tricky because the exclusion is actually done in the filter, it is not evaluated at the creation time of the gen exp:

>>> (i for i in range(2) if i in 5)
<generator object <genexpr> at 0x109a0da50>

More reasons why both for loops and filters need to be moved into range to store IDX; More than just filters.

In other words, you can use itertools.islice instead of the inefficient logic of slice that you use to print the builder expression:

from itertools import islice

for skip_list in skip_lists:
    for num in islice(skip_list, 10):
        print("{}, ".format(num), end="")
    print()