Skip to content

fix: split Chinese and Japanese sentences at full-width terminators in DocumentSplitter - #13834

Open
usercjr wants to merge 2 commits into
deepset-ai:mainfrom
usercjr:fix/cjk-sentence-split
Open

usercjr wants to merge 2 commits into
deepset-ai:mainfrom
usercjr:fix/cjk-sentence-split

Conversation

@usercjr

@usercjr usercjr commented Nov 20, 2026

Copy link
Copy Markdown

Related Issues

Proposed Changes:

Chinese and Japanese text has no space after 。, ! or ?, so NLTK's Punkt returns the whole text as one sentence, and DocumentSplitter(split_by="sentence") and the "sentence" separator of RecursiveDocumentSplitter cannot split it. Adding the characters to sent_end_chars (as #13646 did for the danda) is not enough here: Punkt reads a run of Chinese text without spaces as one word.

SentenceSplitter.split_sentences() now splits each Punkt sentence after 。, !, ? (and the half-width 。). A terminator followed by a closing quote or bracket (”, 」, ), ...) is not a split point, so 他说:“我走了。”然后离开了。 stays one sentence. The whitespace after a terminator goes to the previous sentence when keep_white_spaces=True, as for other sentences. These characters are not used as sentence ends in other scripts, so this applies for every language; English output is unchanged.

Use case: splitting Chinese PDFs for retrieval. On a Chinese bank terms-of-service page converted with PDFMinerToDocument, DocumentSplitter(split_by="sentence", split_length=3) gives 1 chunk of 2818 characters on main and 7 chunks with this change.

How did you test it?

  • New tests in test/components/preprocessors/test_sentence_tokenizer.py for both keep_white_spaces values: Chinese and Japanese text with newlines between sentences, and terminators inside quotes. They fail on main and pass here.
  • hatch run test:unit test/components/preprocessors/: 553 passed.
  • pre-commit run on the changed files: all hooks pass except codespell, which flags the existing Telugu code "te" in sentence_tokenizer.py (lines 47 and 73, not touched here). hatch run test:types on the module: clean.

Notes for the reviewer

Because a terminator followed by a closing quote is never a split point, 「行きます。」と言った。 is one sentence, as it should be, but so is 「行きます。」「はい。」次の文。: a sentence that starts right after such a quote is not split off. I kept the rule simple and conservative (a longer sentence rather than a wrong cut); tell me if you'd prefer a different trade-off.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

This PR was generated with an AI assistant (Claude Code). I have reviewed the changes and run the tests above.

Chinese and Japanese text has no space after a sentence end, so NLTK's
Punkt returned the whole text as one sentence, and DocumentSplitter and
RecursiveDocumentSplitter could not split it by sentence.

Split each sentence after 。, !, ? and 。, except before a closing
quote or bracket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: usercjr <92197541+usercjr@users.noreply.github.com>
@usercjr
usercjr requested a review from a team as a code owner November 20, 2026 12:21
@usercjr
usercjr requested review from bogdankostic and removed request for a team November 20, 2026 12:21
@vercel

vercel Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

@usercjr is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 20, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

# Full-width sentence terminators of Chinese and Japanese. These scripts put no space between sentences, so NLTK,
# which looks for whitespace after a sentence ending, never splits there. A terminator that a closing quote or
# bracket follows ends quoted or bracketed text, not the sentence, so it is not a split point.
_CJK_SENTENCE_END_RE = re.compile(r"[。!?。]+(?![」』”’)】〕》〉\)\]}\"'])")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the backtracking on + lets this split inside quotes when terminators stack, e.g. 「本当?!」と聞きました。 comes out as 「本当? and !」と聞きました。 (same for “好。。”). adding the terminators to the lookahead stops it

Suggested change
_CJK_SENTENCE_END_RE = re.compile(r"[。!?。]+(?![」』”’)】〕》〉\)\]}\"'])")
_CJK_SENTENCE_END_RE = re.compile(r"[。!?。]+(?![。!?。」』”’)】〕》〉\)\]}\"'])")

fixed this plus a test in this commit, you can just cherry pick it instead of redoing it: its-amann@008fbb6

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed: 「本当?!」と聞きました。 was split into 「本当? / !」と聞きました。, and “好。。”他说。 too. The terminators are now in the lookahead, and the quote test uses ?!」. I made the change in my own commit instead of cherry-picking yours, so that the PR keeps a single CLA signer. (b21146c)

@1AbdullahButt 1AbdullahButt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran SentenceSplitter(language="en").split_sentences on main and on this branch with more Chinese and Japanese inputs.

text main this PR
我今天去了银行。利息是多少? 1 sentence 我今天去了银行。 / 利息是多少?
「はい。」と彼は言った。次の文。 1 「はい。」と彼は言った。 / 次の文。
等等……真的吗?!是的。 1 等等……真的吗?! / 是的。
价格是3.5元。好的。 1 价格是3.5元。 / 好的。
Hello world. 你好。再见。 Hello world. / 你好。再见。 Hello world. / 你好。 / 再见。
第一句。\n\n第二句。 1 第一句。 / 第二句。
結果は3.5でした.次の文. 1 1 (not split)

DocumentSplitter(split_by="sentence", split_length=2, split_overlap=1) on 一。二。三。四。 now gives 一。二。 / 二。三。 / 三。四。, and use_split_rules=False also splits. The tests in test_sentence_tokenizer.py, test_document_splitter.py and test_recursive_splitter.py give 177 passed here and 173 on main. The same 38 tests fail on both, because of optional dependencies I did not install (such as tiktoken).

One gap: the full-width full stop . (U+FF0E) is not in _CJK_SENTENCE_END_RE. The half-width 。 (U+FF61) is in it. Japanese technical and academic writing often uses . as the sentence end (often with ,), so text like the last row stays one sentence. If it is added, it needs a guard for full-width numbers such as 3.5, for example a lookbehind or lookahead that skips a . between two digits. Leaving it out on purpose is also fine, but a note in the release note would help.

(I made this review with help from Claude Code. I ran the table above on main and on this branch, and the three preprocessor test files on both. I did not install tiktoken or the other optional dependencies, and I did not test languages other than Chinese and Japanese.)

…ll stop

Put the terminators in the lookahead, so that a run like ?!」 is not
split in the middle (review by its-amann). Also split at ., except
next to a digit, as in 3.5 or 1.はじめに (review by 1AbdullahButt).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: usercjr <92197541+usercjr@users.noreply.github.com>
@usercjr

usercjr commented Nov 21, 2026

Copy link
Copy Markdown
Author

@1AbdullahButt Thanks for the table. I added . (U+FF0E). A lone . next to a digit is not a split point, so 3.5 and also a list number like 1.はじめに stay intact; 結果は3.5でした.次の文. now gives 結果は3.5でした. / 次の文.. This is a check after the match rather than a lookbehind in the regex: with the alternation in the pattern, the regex alone took 1.96 s instead of 0.45 s on the 50 MB input of test_split_sentences_performance, and that test failed (2.4 s, limit 2 s). The release note mentions . now. (b21146c)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DocumentSplitter: Chinese and Japanese text is not split into sentences, so the whole document becomes one chunk

4 participants