ETLBox ETLBox ETLBox Buy Try Buy

Text Document Source

This article contains example code that shows the usage of the TextDocumentSource component.

TextDocumentSource reads one text resource as one row. A file, a web response, or a blob is one element. A folder is one element per file. The non-generic source writes Text and StreamMetaData on an ExpandoObject.

Read one file

The whole file is one dynamic row. Text is the text. StreamMetaData carries the path and the resource count.

string sourceFile = "readme.md";
File.WriteAllText(sourceFile, """
    # Warehouse notes

    ETLBox reads this file as one element.
    The second paragraph stays in the same element.
    """);

var source = new TextDocumentSource(sourceFile);
var dest = new MemoryDestination();

source.LinkTo(dest);
Network.Execute(source);

foreach (dynamic row in dest.Data) {
    StreamMetaData metadata = row.StreamMetaData;
    Console.WriteLine($"RequestUri:{metadata.RequestUri}");
    Console.WriteLine($"RequestCount:{metadata.RequestCount}");
    Console.WriteLine(row.Text);
}

//Outputs
//RequestUri:readme.md
//RequestCount:1
//# Warehouse notes
//
//ETLBox reads this file as one element.
//The second paragraph stays in the same element.

Read into your own type

[DocumentContent] receives the text. [DocumentRequestUri] receives the path. Properties without an attribute stay unset.

public class Article
{
    public string Title { get; set; }
    [DocumentContent]
    public string Content { get; set; }
    [DocumentRequestUri]
    public string Source { get; set; }
}

string sourceFile = "catalog.md";
File.WriteAllText(sourceFile, """
    Trail bottle for hiking.
    Fits in a side pocket.
    """);

var source = new TextDocumentSource<Article>(sourceFile);
var dest = new MemoryDestination<Article>();

source.LinkTo(dest);
Network.Execute(source);

foreach (Article row in dest.Data) {
    Console.WriteLine($"Source:{row.Source}");
    Console.WriteLine($"Content:{row.Content}");
}

//Outputs
//Source:catalog.md
//Content:Trail bottle for hiking.
//Fits in a side pocket.

Build the row yourself

ResultSelector gets the text and the StreamMetaData. The attributes are not used, so fields such as Title can be set here.

public class Article
{
    public string Title { get; set; }
    [DocumentContent]
    public string Content { get; set; }
    [DocumentRequestUri]
    public string Source { get; set; }
}

string sourceFile = "catalog.md";
File.WriteAllText(sourceFile, """
    Trail bottle for hiking.
    Fits in a side pocket.
    """);

var source = new TextDocumentSource<Article>(sourceFile) {
    ResultSelector = (text, metadata) => new Article {
        Title = "Catalog",
        Source = metadata.RequestUri,
        Content = text
    }
};
var dest = new MemoryDestination<Article>();

source.LinkTo(dest);
Network.Execute(source);

foreach (Article row in dest.Data) {
    Console.WriteLine($"Title:{row.Title}");
    Console.WriteLine($"Source:{row.Source}");
}

//Outputs
//Title:Catalog
//Source:catalog.md

Read each file in a folder

Folder emits one row per file in that folder. RequestCount numbers the files.

Directory.CreateDirectory("notes");
File.WriteAllText(Path.Combine("notes", "article-a.md"), "Wool socks for winter walks.");
File.WriteAllText(Path.Combine("notes", "article-b.md"), "Coffee grinder for fresh coffee.");

var source = new TextDocumentSource {
    Folder = "notes"
};
var dest = new MemoryDestination();

source.LinkTo(dest);
Network.Execute(source);

foreach (dynamic row in dest.Data) {
    StreamMetaData metadata = row.StreamMetaData;
    Console.WriteLine($"RequestCount:{metadata.RequestCount} File:{Path.GetFileName(metadata.RequestUri)}");
    Console.WriteLine(row.Text);
}

//Outputs
//RequestCount:1 File:article-a.md
//Wool socks for winter walks.
//RequestCount:2 File:article-b.md
//Coffee grinder for fresh coffee.

Split the document

The dynamic text property is Text, which ChunkTransformation reads by default.

string sourceFile = "readme.md";
File.WriteAllText(sourceFile, """
    # Warehouse notes

    ETLBox reads this file as one element.
    The second paragraph stays in the same element.
    """);

var source = new TextDocumentSource(sourceFile);
var chunk = new ChunkTransformation {
    MaxTokensPerChunk = 8,
    OverlapTokens = 0
};
var dest = new MemoryDestination<Chunk<ExpandoObject>>();

source.LinkTo(chunk);
chunk.LinkTo(dest);
Network.Execute(source);

foreach (Chunk<ExpandoObject> row in dest.Data)
    Console.WriteLine($"Index:{row.Index} Text:{row.Text}");

//Outputs
//Index:0 Text:# Warehouse notes
//
//ETLBox reads
//Index:1 Text: this file as one element.
//The second
//Index:2 Text: paragraph stays in the same element.
Edit this page on GitHub
Prev
Pivot Data
Next
Csv to Json