Langchain：文本分割器行为

Question

我不明白 Langchain 递归文本分割器的以下行为。这是我的代码和输出。

from langchain.text_splitter import RecursiveCharacterTextSplitter
r_splitter = RecursiveCharacterTextSplitter(
    chunk_size=10,
    chunk_overlap=0,
#     separators=["\n"]#, "\n", " ", ""]
)
test = """a\nbcefg\nhij\nk"""
print(len(test))
tmp = r_splitter.split_text(test)
print(tmp)

输出

13
['a\nbcefg', 'hij\nk']

正如您所看到的，它输出大小为 7 和 5 的块，并且仅在新行字符之一上进行分割。我期望输出为 ['a','bcefg','hij','k']

Answer 1

根据RecursiveCharacterTextSplitter中的split_text函数

def split_text(self, text: str) -> List[str]:
    """Split incoming text and return chunks."""
    final_chunks = []
    # Get appropriate separator to use
    separator = self._separators[-1]
    for _s in self._separators:
        if _s == "":
            separator = _s
            break
        if _s in text:
            separator = _s
            break
    # Now that we have the separator, split the text
    if separator:
        splits = text.split(separator)
    else:
        splits = list(text)
    # Now go merging things, recursively splitting longer texts.
    _good_splits = []
    for s in splits:
        if self._length_function(s) < self._chunk_size:
            _good_splits.append(s)
        else:
            if _good_splits:
                merged_text = self._merge_splits(_good_splits, separator)
                final_chunks.extend(merged_text)
                _good_splits = []
            other_info = self.split_text(s)
            final_chunks.extend(other_info)
    if _good_splits:
        merged_text = self._merge_splits(_good_splits, separator)  # Here will merge the items if the cusum is less than chunk size in your example is 10
        final_chunks.extend(merged_text)
    return final_chunks

如果示例中的 cusum 小于块大小 10，这将合并项目