ETLBox

A lightweight ETL (extract, transform, load) toolbox and data integration library for .NET

Try for free . Get commercial licenses at low rates.

Highly customizable

Tired of using User Interfaces where complex requirements are almost impossible to implement? With ETLBox it's easy to write code that fits your unique needs. Plug in your own logic or modify existing behavior to your specific requirements.

Sophisticated data flows

Read and write data from databases, flat files or web services. Harmonize, filter, aggregate, validate and clean the data in-memory with highly customizable transformations and write them into any destination.

Develop locally

Develop and test your ETL and data integration flows locally with your favorite development and debugging tools. Use the toolset that your team is already familiar with.

Manage your changes

Track your changes with Git (or other source code controls), integrate with your current code review process and connect with your existing CI/CD platform.

Build ETL in .NET

Code your ETL with your favorite .NET language fitting your team’s skills and that is coming with a mature toolset.

Runs everywhere

ETLBox runs on Linux, macOS, and Windows. It is written in the current .NET Standard and successfully tested with the latest versions of .NET Core & .NET.

Fast and efficient

Process your data in-Memory:
Code your ETL with your favorite .NET language fitting your team’s skills and that is coming with a mature toolset.

Made for big data

Process any workload:
ETLBox relies on Microsoft's TPL.Dataflow library and is designed to process efficiently big amounts of any data.

Connect everything

Process data from anywhere:
ETLBox supports a wide range of Sql & NoSql databases, flat files, web services, and more - perfect as a foundation for your tailor made data integregation platform.

ETL made easy

Use a simple syntax to create easy or complex data flows. Works with strongly type and dynamic objects.

string mySqlConnString = 
  @"Server=localhost;Database=ETLBox;Uid=user;Pwd=password;";
string postgresConnString = 
  @"Server=.;Initial Catalog=ETLBox;Trusted_Connection=true;";

var sourceCon = new MySqlConnectionManager(mySqlConnString);
var destCon = new PostgresConnectionManager(postgresConnString);

var source = new DbSource(sourceCon, "SourceTableName");
var dest = new DbDestination(destCon, "DestTableName");                
//Use dest.ColumnMapping to change column name mapping

source.LinkTo(dest); //Create the data flow
Network.Execute(source); //Start the flow synchronously

string connString = 
  @"Source=.;Trusted_Connection=true;Initial Catalog=ETLBox;"
var destCon = new SqlConnectionManager(connString);

var source = new CsvSource(sourceCon, "SourceData.csv");
var dest = new DbDestination(destCon, "DestinationTable");

//Default mapping: Csv header and column name match (case-sensitive)
//Override mapping like this:
dest.ColumnMapping = new[] {    
    new ColumnMap() { 
      DbColumnName = "Id", PropertyName = "OrderNumber"
    }
};

source.LinkTo(dest); //Create the data flow
Network.Execute(source); //Start the flow synchronously

What is ETL

ETL is the foundation of modern Business Intelligence application. It separates the analysis part from everything that comes before it. ETL stands for extract, transform and load, and describes a three-step process:

Extracting data is about acquiring it from a particular source and doing so in a way that all relevant data is collected. For ETL to work well, data needs to come directly from the source and in its rawest form.

Transforming data requires it to be stripped of its various formats, cleaned and harmonized. Put differently, it needs to become easy for the end user to read. Ensuring the highest possible data quality is crucial during this process, including detecting and removing duplicates and sorting the data into relevant, predetermined categories.

In the end, your ETL tool will help harmonize your data and load it into a target location, which essentially is a business intelligence (BI) or database tool of choice. There you can find it ready for your analysis.

ETLBox core concepts

ETLBox is a data processing engine based on .NET Core, giving you the power to create your own ETL processes. This engine runs inside your applications, APIs, and jobs to extract, filter, transform, migrate data on-the-fly. ETLBox speeds up your development by providing an easy-to-use toolbox for integrating your data from different sources within your app and load them into any target you like.

What you can do with ETLBox

Here are a few things you can do with ETLBox:

Convert incoming data to a common format.
Prepare data for analysis and visualization>.
Migrate between databases.
Share data processing logic across web apps, batch jobs, and APIs.
Power your data ingestion and integration tools.
Consume large CSV, Excel or Json files>

Data integration

Data integration solves the problem of moving, transforming, and consolidating information from various sources like databases, applications, files, and web services. Bringing them together enables you to cleanse, standardize, de-duplicate, manipulate, and synchronize them. Data integration can play an important role in any project, and it can be critical to have a rock-solid solution in place that is able to integrate numerous applications, services and databases. ETLBox as an Extract, transform, and load (ETL) tool is a perfect fit when you need to create a data integration solution.

Common data API

ETLBox provides you with a single API for working with data. The API treats all data the same regardless of their source. By developing your applications against a single API, you can use the same components to process data regardless of whether they're coming from a database, Excel file, or 3rd-party API.

Embed inside your App

ETLBox fits well within your applications and services. It is a very lightweight library and has a very small footprint. It's also complication free - requiring no servers, installation, or config files. You just add it as a package reference into your app and start using it.

In-flight Processing

ETLBox can run completely in-memory. In most cases, there's no need to store intermediate results in temporary databases or files on disk. Processing data in-memory, while it moves through the pipeline, can be more than 100 times faster than storing it to disk to query or process later.

Flexible componentens

ETLBox comes with a set of components and tasks that simplify to read data from various sources, transform it on the fly and write it into different destinations. But all components offer you the ability to add your own logic in C# - and you can benefit here from the full power of the .NET framework. Even if your own data source or destination is not supported out of the box, you can easily add it yourself by extending the CustomSource or CustomDestination components.

100% .NET

ETLBox is built on .NET Core and .NET Standard and runs with the latest version of .NET. You write data flows and transformations in C# or any of the other .NET languages you know (e.g. VisualBasic or F#). You can also use your existing tools, IDEs, containers, and libraries. Being built on .NET Core means it can run on all servers, operating systems, and environments.

Try yourself