Class: SentenceSplitter
SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.
One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.
Constructors
new SentenceSplitter()
new SentenceSplitter(
options?):SentenceSplitter
Parameters
• options?
• options.chunkOverlap?: number
• options.chunkSize?: number
• options.chunkingTokenizerFn?
• options.paragraphSeparator?: string
• options.splitLongSentences?: boolean
• options.tokenizer?: Tokenizer
Returns
Defined in
packages/llamaindex/src/TextSplitter.ts:76
Properties
chunkOverlap
chunkOverlap:
number
Defined in
packages/llamaindex/src/TextSplitter.ts:69
chunkSize
chunkSize:
number
Defined in
packages/llamaindex/src/TextSplitter.ts:68
Methods
combineTextSplits()
combineTextSplits(
newSentenceSplits,effectiveChunkSize):TextSplit[]
Parameters
• newSentenceSplits: SplitRep[]
• effectiveChunkSize: number
Returns
TextSplit[]
Defined in
packages/llamaindex/src/TextSplitter.ts:210
getParagraphSplits()
getParagraphSplits(
text,effectiveChunkSize?):string[]
Parameters
• text: string
• effectiveChunkSize?: number
Returns
string[]
Defined in
packages/llamaindex/src/TextSplitter.ts:126
getSentenceSplits()
getSentenceSplits(
text,effectiveChunkSize?):string[]
Parameters
• text: string
• effectiveChunkSize?: number
Returns
string[]
Defined in
packages/llamaindex/src/TextSplitter.ts:152
splitText()
splitText(
text,extraInfoStr?):string[]
Parameters
• text: string
• extraInfoStr?: string
Returns
string[]
Defined in
packages/llamaindex/src/TextSplitter.ts:304
splitTextWithOverlaps()
splitTextWithOverlaps(
text,extraInfoStr?):TextSplit[]
Parameters
• text: string
• extraInfoStr?: string
Returns
TextSplit[]