Overview

Supported Formats#

ETLBox offers streaming connectors for the following formats:

• CSV – Structured text with delimiters
• JSON – Flexible structured data
• XML – Hierarchical structured data
• Excel – Spreadsheet-based tabular data
• Text – Unstructured or custom-formatted text, one row per line
• Parquet – Optimized columnar storage

Reading a whole file, web response, or blob as one document — plain text, markdown, HTML, or PDF — is covered in AI.

Each format has a corresponding source (for reading) and destination (for writing), allowing seamless integration into ETL workflows.

Installation#

Each format has a dedicated connector package that must be included alongside the core ETLBox package. For example, to work with JSON, install the following package:

dotnet add package ETLBox.Json

PM> NuGet\Install-Package ETLBox.Json -Version 3.11.1

<PackageReference Include="ETLBox.Json" Version="3.11.1" />

> #r "nuget: ETLBox.Json, 3.11.1"

Connector-Specific Highlights#

Key Features#

Unified Streaming Model#

All connectors share a common architecture, meaning you can use the same API structure to read or write different formats.

For example, switching from a local CSV file to a JSON-based web API requires minimal code changes:

CsvSource source = new CsvSource("data.csv");
source.ResourceType = ResourceType.File;

// Switching to JSON API
JsonSource apiSource = new JsonSource("https://api.example.com/data");
apiSource.ResourceType = ResourceType.Http;

Flexible Resource Types#

Each connector can work with multiple resource types:

• File-based: Local storage or network shares (e.g., "C:/data/file.csv")
• HTTP-based: REST APIs (GET/POST) (e.g., "https://api.example.com/data")
• Azure Blob Storage: Cloud storage integration

Example: Reading from Azure Blob Storage#

CsvSource source = new CsvSource("dataset.csv");
source.ResourceType = ResourceType.AzureBlob;
source.AzureBlobStorage.ConnectionString = "<your_connection_string>";
source.AzureBlobStorage.ContainerName = "data-container";

Streaming & Buffering#

• Data is streamed record-by-record instead of loading the entire dataset into memory.
• Adjustable buffer sizes via MaxBufferSize ensures that the system can buffer processed rows for faster processing speed.

Paging Support for API Requests#

For web-based data sources, ETLBox supports pagination:

JsonSource<MyRow> source = new JsonSource<MyRow>();
int page = 1;
source.GetNextUri = streamMetaData => $"https://api.example.com/data?page={page++}";
source.HasNextUri = streamMetaData => streamMetaData.ProcessedRows > 0;

Schema Flexibility#

ETLBox supports different ways to map data from streaming sources:

POCO Mapping (Plain Old C# Objects)**#

When reading structured data, you can map it directly to a C# class:

public class MyDataRow {
    public int Id { get; set; }
    public string Name { get; set; }
}

CsvSource<MyDataRow> source = new CsvSource<MyDataRow>("data.csv");

Dynamic Objects (ExpandoObject)**#

For unknown schemas, use ExpandoObject to handle flexible structures:

CsvSource<dynamic> source = new CsvSource<dynamic>("data.csv");
source.RowModificationAction = (row, meta) => {
    dynamic r = row;
    Console.WriteLine($"ID: {r.Id}, Name: {r.Name}");
};

Attribute-Based Mapping#

This example demonstrates how CsvHelper’s attribute-based mapping can be used to rename and reorder columns. Since ETLBox uses third-party libraries internally, attribute configurations vary by connector.

public class MyDataRow {
    [CsvHelper.Configuration.Attributes.Name("Identifier")]
    public int Id { get; set; }

    [CsvHelper.Configuration.Attributes.Index(1)]
    public string FullName { get; set; }
}

CsvSource<MyDataRow> source = new CsvSource<MyDataRow>("data.csv");