ETLBox ETLBox ETLBox Buy
API
Try Buy

MarkdownSource<TOutput>

Class MarkdownSource<TOutput>

Reads one markdown resource as one Microsoft.Extensions.DataIngestion.IngestionDocument. A line-based source emits one row per line; this source emits the parsed document of the current file, HTTP response, blob, or stream.

Inheritance
object
LoggableTask
DataFlowComponent
DataFlowSource<TOutput>
DataFlowExecutableSource<TOutput>
DataFlowStreamSource<TOutput>
MarkdownSource<TOutput>
MarkdownSource
Implements
Inherited Members
Namespace: ETLBox.AI
Assembly: ETLBox.AI.dll
Syntax
    public class MarkdownSource<TOutput> : DataFlowStreamSource<TOutput>, IDataFlowStreamSource<TOutput>, IDataFlowExecutableSource<TOutput>, IDataFlowStreamSource, IDataFlowExecutableSource, IDataFlowSource<TOutput>, IDataFlowSource, IDataFlowComponent, ILoggableTask
Type Parameters
NameDescription
TOutput

Type of outgoing data.

Examples
var source = new MarkdownDocumentSource("docs");
var chunk = new ChunkTransformation {
    DocumentPropertyName = "Document",
    Chunker = new HeaderChunker(new IngestionChunkerOptions(tokenizer) {
        MaxTokensPerChunk = 512,
        OverlapTokens = 50
    })
};
source.LinkTo(chunk);

Constructors

MarkdownSource()

Declaration
    public MarkdownSource()

MarkdownSource(string)

Declaration
    public MarkdownSource(string uri)
Parameters
TypeNameDescription
stringuri

The source of the file. This can be a filename or a web url.Uri

MarkdownSource(string, ResourceType)

Declaration
    public MarkdownSource(string uri, ResourceType resourceType)
Parameters
TypeNameDescription
stringuri

The source of the file. This can be a filename or a web url.Uri

ResourceTyperesourceType

Specifies if data is loaded from a file, a web endpoint or other storage types (e.g. Azure Blob Storage)

Properties

ResultSelector

Creates the outgoing element from the parsed document and the metadata of the current resource. When set, this function is used instead of the ExpandoObject default or the Microsoft.Extensions.DataIngestion.IngestionDocument default.

Declaration
    public Func<IngestionDocument, StreamMetaData, TOutput> ResultSelector { get; set; }
Property Value
TypeDescription
Func<IngestionDocument, StreamMetaData, TOutput>

Methods

CloseReader()

Declaration
    protected override void CloseReader()
Overrides

InitReader()

Declaration
    protected override void InitReader()
Overrides

ReadAllRecords()

Declaration
    protected override void ReadAllRecords()
Overrides

Implements