Repository navigation
Conversation
Chinese and Japanese text has no space after a sentence end, so NLTK's Punkt returned the whole text as one sentence, and DocumentSplitter and RecursiveDocumentSplitter could not split it by sentence. Split each sentence after 。, !, ? and 。, except before a closing quote or bracket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: usercjr <92197541+usercjr@users.noreply.github.com>
|
@usercjr is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
| # Full-width sentence terminators of Chinese and Japanese. These scripts put no space between sentences, so NLTK, | ||
| # which looks for whitespace after a sentence ending, never splits there. A terminator that a closing quote or | ||
| # bracket follows ends quoted or bracketed text, not the sentence, so it is not a split point. | ||
| _CJK_SENTENCE_END_RE = re.compile(r"[。!?。]+(?![」』”’)】〕》〉\)\]}\"'])") |
There was a problem hiding this comment.
the backtracking on + lets this split inside quotes when terminators stack, e.g. 「本当?!」と聞きました。 comes out as 「本当? and !」と聞きました。 (same for “好。。”). adding the terminators to the lookahead stops it
| _CJK_SENTENCE_END_RE = re.compile(r"[。!?。]+(?![」』”’)】〕》〉\)\]}\"'])") | |
| _CJK_SENTENCE_END_RE = re.compile(r"[。!?。]+(?![。!?。」』”’)】〕》〉\)\]}\"'])") |
fixed this plus a test in this commit, you can just cherry pick it instead of redoing it: its-amann@008fbb6
There was a problem hiding this comment.
Thanks, confirmed: 「本当?!」と聞きました。 was split into 「本当? / !」と聞きました。, and “好。。”他说。 too. The terminators are now in the lookahead, and the quote test uses ?!」. I made the change in my own commit instead of cherry-picking yours, so that the PR keeps a single CLA signer. (b21146c)
1AbdullahButt
left a comment
There was a problem hiding this comment.
I ran SentenceSplitter(language="en").split_sentences on main and on this branch with more Chinese and Japanese inputs.
| text | main | this PR |
|---|---|---|
我今天去了银行。利息是多少? |
1 sentence | 我今天去了银行。 / 利息是多少? |
「はい。」と彼は言った。次の文。 |
1 | 「はい。」と彼は言った。 / 次の文。 |
等等……真的吗?!是的。 |
1 | 等等……真的吗?! / 是的。 |
价格是3.5元。好的。 |
1 | 价格是3.5元。 / 好的。 |
Hello world. 你好。再见。 |
Hello world. / 你好。再见。 |
Hello world. / 你好。 / 再见。 |
第一句。\n\n第二句。 |
1 | 第一句。 / 第二句。 |
結果は3.5でした.次の文. |
1 | 1 (not split) |
DocumentSplitter(split_by="sentence", split_length=2, split_overlap=1) on 一。二。三。四。 now gives 一。二。 / 二。三。 / 三。四。, and use_split_rules=False also splits. The tests in test_sentence_tokenizer.py, test_document_splitter.py and test_recursive_splitter.py give 177 passed here and 173 on main. The same 38 tests fail on both, because of optional dependencies I did not install (such as tiktoken).
One gap: the full-width full stop . (U+FF0E) is not in _CJK_SENTENCE_END_RE. The half-width 。 (U+FF61) is in it. Japanese technical and academic writing often uses . as the sentence end (often with ,), so text like the last row stays one sentence. If it is added, it needs a guard for full-width numbers such as 3.5, for example a lookbehind or lookahead that skips a . between two digits. Leaving it out on purpose is also fine, but a note in the release note would help.
(I made this review with help from Claude Code. I ran the table above on main and on this branch, and the three preprocessor test files on both. I did not install tiktoken or the other optional dependencies, and I did not test languages other than Chinese and Japanese.)
…ll stop Put the terminators in the lookahead, so that a run like ?!」 is not split in the middle (review by its-amann). Also split at ., except next to a digit, as in 3.5 or 1.はじめに (review by 1AbdullahButt). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: usercjr <92197541+usercjr@users.noreply.github.com>
|
@1AbdullahButt Thanks for the table. I added |
Related Issues
Proposed Changes:
Chinese and Japanese text has no space after
。,!or?, so NLTK's Punkt returns the whole text as one sentence, andDocumentSplitter(split_by="sentence")and the"sentence"separator ofRecursiveDocumentSplittercannot split it. Adding the characters tosent_end_chars(as #13646 did for the danda) is not enough here: Punkt reads a run of Chinese text without spaces as one word.SentenceSplitter.split_sentences()now splits each Punkt sentence after。,!,?(and the half-width。). A terminator followed by a closing quote or bracket (”,」,), ...) is not a split point, so他说:“我走了。”然后离开了。stays one sentence. The whitespace after a terminator goes to the previous sentence whenkeep_white_spaces=True, as for other sentences. These characters are not used as sentence ends in other scripts, so this applies for everylanguage; English output is unchanged.Use case: splitting Chinese PDFs for retrieval. On a Chinese bank terms-of-service page converted with
PDFMinerToDocument,DocumentSplitter(split_by="sentence", split_length=3)gives 1 chunk of 2818 characters onmainand 7 chunks with this change.How did you test it?
test/components/preprocessors/test_sentence_tokenizer.pyfor bothkeep_white_spacesvalues: Chinese and Japanese text with newlines between sentences, and terminators inside quotes. They fail onmainand pass here.hatch run test:unit test/components/preprocessors/: 553 passed.pre-commit runon the changed files: all hooks pass exceptcodespell, which flags the existing Telugu code"te"insentence_tokenizer.py(lines 47 and 73, not touched here).hatch run test:typeson the module: clean.Notes for the reviewer
Because a terminator followed by a closing quote is never a split point,
「行きます。」と言った。is one sentence, as it should be, but so is「行きます。」「はい。」次の文。: a sentence that starts right after such a quote is not split off. I kept the rule simple and conservative (a longer sentence rather than a wrong cut); tell me if you'd prefer a different trade-off.Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.This PR was generated with an AI assistant (Claude Code). I have reviewed the changes and run the tests above.