使用Julia生成ngrams

Question

要在Julia中生成单词双字母,我可以简单地压缩原始列表和删除第一个元素的列表,例如:

julia> s = split("the lazy fox jumps over the brown dog")
8-element Array{SubString{String},1}:
 "the"  
 "lazy" 
 "fox"  
 "jumps"
 "over" 
 "the"  
 "brown"
 "dog"  

julia> collect(zip(s, drop(s,1)))
7-element Array{Tuple{SubString{String},SubString{String}},1}:
 ("the","lazy")  
 ("lazy","fox")  
 ("fox","jumps") 
 ("jumps","over")
 ("over","the")  
 ("the","brown") 
 ("brown","dog") 

要生成一个三元组,我可以使用相同的collect(zip(...))习语来获得:

julia> collect(zip(s, drop(s,1), drop(s,2)))
6-element Array{Tuple{SubString{String},SubString{String},SubString{String}},1}:
 ("the","lazy","fox")  
 ("lazy","fox","jumps")
 ("fox","jumps","over")
 ("jumps","over","the")
 ("over","the","brown")
 ("the","brown","dog") 

但是我必须在第3个列表中手动添加以进行压缩,是否有一种惯用的方式使得我可以执行n -gram的任何顺序？

例如,我想避免这样做以提取5克:

julia> collect(zip(s, drop(s,1), drop(s,2), drop(s,3), drop(s,4)))
4-element Array{Tuple{SubString{String},SubString{String},SubString{String},SubString{String},SubString{String}},1}:
 ("the","lazy","fox","jumps","over") 
 ("lazy","fox","jumps","over","the") 
 ("fox","jumps","over","the","brown")
 ("jumps","over","the","brown","dog")

Answer 1

另一种方法是使用Iterators.jl's partition():

ngram(s,n) = collect(partition(s, n, 1))

Answer 2

通过稍微改变输出并使用SubArrays 而不是Tuples，损失很少，但可以避免分配和内存复制。如果底层单词列表是静态的，这可以且更快（在我的基准测试中也是如此）。编码：

ngram(s,n) = [view(s,i:i+n-1) for i=1:length(s)-n+1]

和输出：

julia> ngram(s,5)
 SubString{String}["the","lazy","fox","jumps","over"] 
 SubString{String}["lazy","fox","jumps","over","the"] 
 SubString{String}["fox","jumps","over","the","brown"]
 SubString{String}["jumps","over","the","brown","dog"]

julia> ngram(s,5)[1][3]
"fox"

对于较大的单词列表，内存要求也显着降低。

另请注意，使用生成器可以更快地处理 ngrams 并使用更少的内存，并且可能足以处理所需的处理代码（计算某些内容或传递一些哈希值）。例如，使用@Gnimuc 的解决方案而没有collectie just partition(s, n, 1)。