可以产生多个连续发电机吗？

Question

以下是将可迭代项目拆分为子列表的两个函数.我相信这种类型的任务是多次编程的.我使用它们来解析由repr('result','case',123,4.56)和('dump',..)等行组成的日志文件.

我想改变这些,以便它们将产生迭代器而不是列表.因为列表可能会变得非常大,但我可以根据前几个项目决定接受或跳过它.此外,如果iter版本可用,我想嵌套它们,但这些列表版本会通过复制部分浪费一些内存.

但是从可迭代源中获取多个生成器对我来说并不容易,所以我请求帮助.如果可能的话,我希望避免引入新课程.

另外,如果您对这个问题有更好的标题,请告诉我.

谢谢!

def cleave_by_mark (stream, key_fn, end_with_mark=False):
    '''[f f t][t][f f] (true) [f f][t][t f f](false)'''
    buf = []
    for item in stream:
        if key_fn(item):
            if end_with_mark: buf.append(item)
            if buf: yield buf
            buf = []
            if end_with_mark: continue
        buf.append(item)
    if buf: yield buf

def cleave_by_change (stream, key_fn):
    '''[1 1 1][2 2][3][2 2 2 2]'''
    prev = None
    buf = []
    for item in stream:
        iden = key_fn(item)
        if prev is None: prev = iden
        if prev != iden:
            yield buf
            buf = []
            prev = iden
        buf.append(item)
    if buf: yield buf

---

编辑:我自己的答案

感谢大家的回答,我可以写出我的要求!当然,至于"cleave_for_change"功能,我也可以使用itertools.groupby.

def cleave_by_mark (stream, key_fn, end_with_mark=False):
    hand = []
    def gen ():
        key = key_fn(hand[0])
        yield hand.pop(0)
        while 1:
            if end_with_mark and key: break
            hand.append(stream.next())
            key = key_fn(hand[0])
            if (not end_with_mark) and key: break
            yield hand.pop(0)
    while 1:
        # allow StopIteration in the main loop
        if not hand: hand.append(stream.next())
        yield gen()

for cl in cleave_by_mark (iter((1,0,0,1,1,0)), lambda x:x):
    print list(cl),  # start with 1
# -> [1, 0, 0] [1] [1, 0]
for cl in cleave_by_mark (iter((0,1,0,0,1,1,0)), lambda x:x):
    print list(cl),
# -> [0] [1, 0, 0] [1] [1, 0]
for cl in cleave_by_mark (iter((1,0,0,1,1,0)), lambda x:x, True):
    print list(cl),  # end with 1
# -> [1] [0, 0, 1] [1] [0]
for cl in cleave_by_mark (iter((0,1,0,0,1,1,0)), lambda x:x, True):
    print list(cl),
# -> [0, 1] [0, 0, 1] [1] [0]

/

def cleave_by_change (stream, key_fn):
    '''[1 1 1][2 2][3][2 2 2 2]'''
    hand = []
    def gen ():
        headkey = key_fn(hand[0])
        yield hand.pop(0)
        while 1:
            hand.append(stream.next())
            key = key_fn(hand[0])
            if key != headkey: break
            yield hand.pop(0)
    while 1:
        # allow StopIteration in the main loop
        if not hand: hand.append(stream.next())
        yield gen()

for cl in cleave_by_change (iter((1,1,1,2,2,2,3,2)), lambda x:x):
    print list(cl),
# -> [1, 1, 1] [2, 2, 2] [3] [2]

小心:如果有人打算使用这些,请务必在每个级别耗尽发电机,正如安德鲁指出的那样.因为否则外部发电机产生回路将在内部发电机离开的地方重新开始,而不是在下一个"块"开始的地方.

stream = itertools.product('abc','1234', 'ABCD')
for a in iters.cleave_by_change(stream, lambda x:x[0]):
    for b in iters.cleave_by_change(a, lambda x:x[1]):
        print b.next()
        for sink in b: pass
    for sink in a: pass

('a', '1', 'A')
('b', '1', 'A')
('c', '1', 'A')

Answer 1

亚当的答案很好.这是为了万一你好奇如何手工完成:

def cleave_by_change(stream):
    def generator():
        head = stream[0]
        while stream and stream[0] == head:
            yield stream.pop(0)
    while stream:
        yield generator()

for g in cleave_by_change([1,1,1,2,2,3,2,2,2,2]):
    print list(g)

这使:

[1, 1, 1]
[2, 2]
[3]
[2, 2, 2, 2]

(以前的版本需要一个hack,或者在python 3中,nonlocal因为我在stream内部分配generator()了(默认情况下也称为第二个变量)stream本地到generator()默认情况下 - 在评论中归功于gnibbler).

请注意,这种方法很危险 - 如果您不"消耗"返回的生成器,那么您将获得越来越多,因为流不会变得更小.