Class ChunkTransformation<TInput, TOutput>

{{< /rawhtml >}} ```C# public class ChunkTransformation : DataFlowTransformation, IDataFlowTransformation, IDataFlowSource, IDataFlowSource, IDataFlowDestination, IDataFlowDestination, IDataFlowComponent, ILoggableTask ``` {{< rawhtml >}}

Type Parameters

Name	Description
TInput	Type of ingoing data.
TOutput	Type of outgoing data.

Remarks

Exactly one input route is used: either TextSelector (or a single ChunkTextAttribute) for plain text, or DocumentSelector (or a single ChunkDocumentAttribute) for an already structured Microsoft.Extensions.DataIngestion.IngestionDocument. An input type of Microsoft.Extensions.DataIngestion.IngestionDocument is used as the document when no selector is set.

Examples

public class Article {
    public int Id { get; set; }
    public string Text { get; set; }
}
var source = new CsvSource<Article>("articles.csv");
var chunk = new ChunkTransformation<Article> {
    TextSelector = article => article.Text,
    MaxTokensPerChunk = 256,
    OverlapTokens = 32
};
var dest = new MemoryDestination<Chunk<Article>>();
source.LinkTo(chunk).LinkTo(dest);
var fromDocument = new ChunkTransformation<Article> {
    DocumentSelector = article => article.Document,
    Chunker = new SectionChunker(new IngestionChunkerOptions(tokenizer))
};
var custom = new ChunkTransformation<Article, string> {
    TextSelector = article => article.Text,
    CreateOutput = chunk => chunk.Text
};
var dynamic = new ChunkTransformation {
    TextPropertyName = "Text"
};

Constructors

ChunkTransformation()

Declaration

{{< /rawhtml >}} ```C# public ChunkTransformation() ``` {{< rawhtml >}}

Properties

Chunker

The chunker that splits each document. When not set, a Microsoft.Extensions.DataIngestion.Chunkers.DocumentTokenChunker is created from Tokenizer, MaxTokensPerChunk and OverlapTokens.

Declaration

{{< /rawhtml >}} ```C# public IngestionChunker Chunker { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
IngestionChunker<string>

CreateOutput

Creates the outgoing row from the produced Chunk<T>. When not set, the chunk itself is sent to the output.

Declaration

{{< /rawhtml >}} ```C# public Func, TOutput> CreateOutput { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
Func<Chunk<TInput>, TOutput>

DocumentPropertyName

Only for ExpandoObject: property that contains an Microsoft.Extensions.DataIngestion.IngestionDocument. Default is "Document". Used when that property holds a document and neither TextSelector nor DocumentSelector is set. If the property is missing, TextPropertyName is used instead.

Declaration

{{< /rawhtml >}} ```C# public string DocumentPropertyName { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
string

DocumentSelector

Selects an already structured document to chunk. Mutually exclusive with TextSelector. The document is passed to the chunker unchanged.

Declaration

{{< /rawhtml >}} ```C# public Func DocumentSelector { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
Func<TInput, IngestionDocument>

MaxTokensPerChunk

Maximum number of tokens allowed in each chunk. Used by the default chunker. Default is 2000.

Declaration

{{< /rawhtml >}} ```C# public int MaxTokensPerChunk { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
int

OverlapTokens

Number of overlapping tokens between consecutive chunks. Used by the default chunker. Default is 500.

Declaration

{{< /rawhtml >}} ```C# public int OverlapTokens { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
int

SourceBlock

SourceBlock from the underlying TPL.Dataflow which is used as output buffer for the component.

Declaration

{{< /rawhtml >}} ```C# public override ISourceBlock SourceBlock { get; } ``` {{< rawhtml >}}

Property Value

Type	Description
ISourceBlock<TOutput>

Overrides

DataFlowSource<TOutput>.SourceBlock

TargetBlock

TargetBlock from the underlying TPL.Dataflow which is used as input buffer for the component.

Declaration

{{< /rawhtml >}} ```C# public override ITargetBlock TargetBlock { get; } ``` {{< rawhtml >}}

Property Value

Type	Description
ITargetBlock<TInput>

Overrides

DataFlowTransformation<TInput, TOutput>.TargetBlock

TextPropertyName

Only for ExpandoObject: property that contains the text to chunk. Default is "Text". Ignored when TextSelector or DocumentSelector is set.

Declaration

{{< /rawhtml >}} ```C# public string TextPropertyName { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
string

TextSelector

Selects the plain text to chunk. Required unless the input type has exactly one property marked with ChunkTextAttribute, or DocumentSelector is set.

Declaration

{{< /rawhtml >}} ```C# public Func TextSelector { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
Func<TInput, string>

Tokenizer

Tokenizer used to build the default Microsoft.Extensions.DataIngestion.Chunkers.DocumentTokenChunker. When not set, the local gpt-4o tokenizer is used.

Declaration

{{< /rawhtml >}} ```C# public Tokenizer Tokenizer { get; set; } ``` {{< rawhtml >}}

Property Value

Type	Description
Tokenizer

Methods

CheckParameter()

Declaration

{{< /rawhtml >}} ```C# protected override void CheckParameter() ``` {{< rawhtml >}}

Overrides

DataFlowComponent.CheckParameter()

CleanUpOnFaulted(Exception)

Declaration

{{< /rawhtml >}} ```C# protected override void CleanUpOnFaulted(Exception e) ``` {{< rawhtml >}}

Parameters

Type	Name	Description
Exception	e

Overrides

DataFlowComponent.CleanUpOnFaulted(Exception)

CleanUpOnSuccess()

Declaration

{{< /rawhtml >}} ```C# protected override void CleanUpOnSuccess() ``` {{< rawhtml >}}

Overrides

DataFlowComponent.CleanUpOnSuccess()

InitComponent()

Declaration

{{< /rawhtml >}} ```C# protected override void InitComponent() ``` {{< rawhtml >}}

Overrides

DataFlowComponent.InitComponent()

PrepareParameterForCheck()

Declaration

{{< /rawhtml >}} ```C# protected override void PrepareParameterForCheck() ``` {{< rawhtml >}}

Overrides

DataFlowComponent.PrepareParameterForCheck()

Reset()

Declaration

{{< /rawhtml >}} ```C# protected override void Reset() ``` {{< rawhtml >}}

Overrides

DataFlowComponent.Reset()

Implements

IDataFlowTransformation<TInput, TOutput>

IDataFlowSource<TOutput>

IDataFlowSource

IDataFlowDestination<TInput>

IDataFlowDestination

IDataFlowComponent

ILoggableTask