Skip to main content

Function: cjkSentenceTokenizer()

cjkSentenceTokenizer(sentence): string[]

Tokenizes sentences. Suitable for Chinese, Japanese, and Korean. Use instead of defaultSentenceTokenizer.

Parameters

sentence: string

Returns

string[]

string[]

Defined in

packages/llamaindex/src/TextSplitter.ts:41