ETLBox ETLBox ETLBox Buy
API
Try Buy

ChunkTransformation<TInput, TOutput>

Class ChunkTransformation<TInput, TOutput>

Splits each incoming row into one or more chunks using a Microsoft.Extensions.DataIngestion.IngestionChunker<T> for string.

Inheritance
object
LoggableTask
DataFlowComponent
DataFlowSource<TOutput>
DataFlowTransformation<TInput, TOutput>
ChunkTransformation<TInput, TOutput>
ChunkTransformation<TInput>
Implements
IDataFlowTransformation<TInput, TOutput>
Inherited Members
Namespace: ETLBox.AI
Assembly: ETLBox.AI.dll
Syntax
    public class ChunkTransformation<TInput, TOutput> : DataFlowTransformation<TInput, TOutput>, IDataFlowTransformation<TInput, TOutput>, IDataFlowSource<TOutput>, IDataFlowSource, IDataFlowDestination<TInput>, IDataFlowDestination, IDataFlowComponent, ILoggableTask
Type Parameters
NameDescription
TInput

Type of ingoing data.

TOutput

Type of outgoing data.

Remarks

Exactly one input route is used: either TextSelector (or a single ChunkTextAttribute) for plain text, or DocumentSelector (or a single ChunkDocumentAttribute) for an already structured Microsoft.Extensions.DataIngestion.IngestionDocument. An input type of Microsoft.Extensions.DataIngestion.IngestionDocument is used as the document when no selector is set.

Examples
public class Article {
    public int Id { get; set; }
    public string Text { get; set; }
}
var source = new CsvSource<Article>("articles.csv");
var chunk = new ChunkTransformation<Article> {
    TextSelector = article => article.Text,
    MaxTokensPerChunk = 256,
    OverlapTokens = 32
};
var dest = new MemoryDestination<Chunk<Article>>();
source.LinkTo(chunk).LinkTo(dest);
var fromDocument = new ChunkTransformation<Article> {
    DocumentSelector = article => article.Document,
    Chunker = new SectionChunker(new IngestionChunkerOptions(tokenizer))
};
var custom = new ChunkTransformation<Article, string> {
    TextSelector = article => article.Text,
    CreateOutput = chunk => chunk.Text
};
var dynamic = new ChunkTransformation {
    TextPropertyName = "Text"
};

Constructors

ChunkTransformation()

Declaration
    public ChunkTransformation()

Properties

Chunker

The chunker that splits each document. When not set, a Microsoft.Extensions.DataIngestion.Chunkers.DocumentTokenChunker is created from Tokenizer, MaxTokensPerChunk and OverlapTokens.

Declaration
    public IngestionChunker<string> Chunker { get; set; }
Property Value
TypeDescription
IngestionChunker<string>

CreateOutput

Creates the outgoing row from the produced Chunk<T>. When not set, the chunk itself is sent to the output.

Declaration
    public Func<Chunk<TInput>, TOutput> CreateOutput { get; set; }
Property Value
TypeDescription
Func<Chunk<TInput>, TOutput>

DocumentPropertyName

Only for ExpandoObject: property that contains an Microsoft.Extensions.DataIngestion.IngestionDocument. Default is "Document". Used when that property holds a document and neither TextSelector nor DocumentSelector is set. If the property is missing, TextPropertyName is used instead.

Declaration
    public string DocumentPropertyName { get; set; }
Property Value
TypeDescription
string

DocumentSelector

Selects an already structured document to chunk. Mutually exclusive with TextSelector. The document is passed to the chunker unchanged.

Declaration
    public Func<TInput, IngestionDocument> DocumentSelector { get; set; }
Property Value
TypeDescription
Func<TInput, IngestionDocument>

MaxTokensPerChunk

Maximum number of tokens allowed in each chunk. Used by the default chunker. Default is 2000.

Declaration
    public int MaxTokensPerChunk { get; set; }
Property Value
TypeDescription
int

OverlapTokens

Number of overlapping tokens between consecutive chunks. Used by the default chunker. Default is 500.

Declaration
    public int OverlapTokens { get; set; }
Property Value
TypeDescription
int

SourceBlock

SourceBlock from the underlying TPL.Dataflow which is used as output buffer for the component.

Declaration
    public override ISourceBlock<TOutput> SourceBlock { get; }
Property Value
TypeDescription
ISourceBlock<TOutput>
Overrides

TargetBlock

TargetBlock from the underlying TPL.Dataflow which is used as input buffer for the component.

Declaration
    public override ITargetBlock<TInput> TargetBlock { get; }
Property Value
TypeDescription
ITargetBlock<TInput>
Overrides

TextPropertyName

Only for ExpandoObject: property that contains the text to chunk. Default is "Text". Ignored when TextSelector or DocumentSelector is set.

Declaration
    public string TextPropertyName { get; set; }
Property Value
TypeDescription
string

TextSelector

Selects the plain text to chunk. Required unless the input type has exactly one property marked with ChunkTextAttribute, or DocumentSelector is set.

Declaration
    public Func<TInput, string> TextSelector { get; set; }
Property Value
TypeDescription
Func<TInput, string>

Tokenizer

Tokenizer used to build the default Microsoft.Extensions.DataIngestion.Chunkers.DocumentTokenChunker. When not set, the local gpt-4o tokenizer is used.

Declaration
    public Tokenizer Tokenizer { get; set; }
Property Value
TypeDescription
Tokenizer

Methods

CheckParameter()

Declaration
    protected override void CheckParameter()
Overrides

CleanUpOnFaulted(Exception)

Declaration
    protected override void CleanUpOnFaulted(Exception e)
Parameters
TypeNameDescription
Exceptione
Overrides

CleanUpOnSuccess()

Declaration
    protected override void CleanUpOnSuccess()
Overrides

InitComponent()

Declaration
    protected override void InitComponent()
Overrides

PrepareParameterForCheck()

Declaration
    protected override void PrepareParameterForCheck()
Overrides

Reset()

Declaration
    protected override void Reset()
Overrides

Implements