ETLBox
A lightweight ETL (extract, transform, load) toolbox and data integration library for .NET
Get StartedAsk ChatGPT about ETLBox
Hi ChatGPT, could you give me an introduction into ETLBox?
ETLBoxA lightweight ETL (extract, transform, load) toolbox and data integration library for .NET
Get StartedTired of using User Interfaces where complex requirements are almost impossible to implement? With ETLBox it's easy to write code that fits your unique needs. Plug in your own logic or modify existing behavior to your specific requirements.
Always wanted to create your ETL jobs test driven? Still wondering if your data flow will still work after you latest change? Look no further. ETLBox integrates with all common test framework like xUnit, NUnit or MSTest. Now you can write test for any part or logic of your data pipeline.
Use the toolset that you and your team is already familiar with. Develop your data pipeline locally with your favorite development and debugging tools. No need to learn a new programming language.
Track your changes with Git (or other source code controls), integrate with your current code review process and connect with your existing CI/CD platform.
Code your ETL with your favorite .NET language fitting your team’s skills and that is coming with a mature toolset.
ETLBox runs on Linux, macOS, and Windows. It is written in the current .NET Standard and successfully tested with the latest versions of .NET.
Process your data in-Memory:
Code your ETL with your favorite .NET language fitting your team’s
skills and that is coming with a mature toolset.
Process any workload:
ETLBox relies on
Microsoft's System.Threading.TPL.Dataflow
library and is designed to process efficiently big amounts of any
data.
Process data from anywhere:
ETLBox supports a wide range of Sql & NoSql databases, flat files, web
services, and more - perfect as a foundation for your tailor made data
integregation platform.
Use a simple syntax to create easy or complex data flows. Works with strongly type and dynamic objects.
string mySqlConnString =
@"Server=localhost;Database=ETLBox;Uid=user;Pwd=password;";
string postgresConnString =
@"Server=.;Initial Catalog=ETLBox;Trusted_Connection=true;";
var sourceCon = new MySqlConnectionManager(mySqlConnString);
var destCon = new PostgresConnectionManager(postgresConnString);
var source = new DbSource(sourceCon, "SourceTableName");
var dest = new DbDestination(destCon, "DestTableName");
//Use dest.ColumnMapping to change column name mapping
source.LinkTo(dest); //Create the data flow
Network.Execute(source); //Start the flow synchronously
string connString =
@"Source=.;Trusted_Connection=true;Initial Catalog=ETLBox;"
var destCon = new SqlConnectionManager(connString);
var source = new CsvSource(sourceCon, "SourceData.csv");
var dest = new DbDestination(destCon, "DestinationTable");
//Default mapping: Csv header and column name match (case-sensitive)
//Override mapping like this:
dest.ColumnMapping = new[] {
new ColumnMap() {
DbColumnName = "Id", PropertyName = "OrderNumber"
}
};
source.LinkTo(dest); //Create the data flow
Network.Execute(source); //Start the flow synchronously
This video is also available on youtube, if you want to like or share it.
Yes, of course!
ETLBox is a commercial, .NET-based library for Extract-Transform-Load (ETL) processes. ETL is a critical aspect of data management, and is used to extract data from a variety of sources, clean and transform it into a desired format, and then load it into a target system, such as a data warehouse or a database. ETLBox provides a set of simple, yet powerful components that can be used to automate these tasks.
The key benefits of using ETLBox include:
ETL is the foundation of modern Business Intelligence application. It separates the analysis part from everything that comes before it. ETL stands for extract, transform and load, and describes a three-step process:
Extracting data is about acquiring it from a particular source and doing so in a way that all relevant data is collected. For ETL to work well, data needs to come directly from the source and in its rawest form.
Transforming data requires it to be stripped of its various formats, cleaned and harmonized. Put differently, it needs to become easy for the end user to read. Ensuring the highest possible data quality is crucial during this process, including detecting and removing duplicates and sorting the data into relevant, predetermined categories.
In the end, your ETL tool will help harmonize your data and load it into a target location, which essentially is a business intelligence (BI) or database tool of choice. There you can find it ready for your analysis.
ETLBox is a data processing engine based on .NET, giving you the power to create your own ETL processes. This engine runs inside your applications, APIs, and jobs to extract, filter, transform, migrate data on-the-fly. ETLBox speeds up your development by providing an easy-to-use toolbox for integrating your data from different sources within your app and load them into any target you like.
Here are a few things you can do with ETLBox:
Loved by our customers across the globe
Data integration solves the problem of moving, transforming, and consolidating information from various sources like databases, applications, files, and web services. Bringing them together enables you to cleanse, standardize, de-duplicate, manipulate, and synchronize them. Data integration can play an important role in any project, and it can be critical to have a rock-solid solution in place that is able to integrate numerous applications, services and databases. ETLBox as an Extract, transform, and load (ETL) tool is a perfect fit when you need to create a data integration solution.
ETLBox provides you with a single API for working with data. The API treats all data the same regardless of their source. By developing your applications against a single API, you can use the same components to process data regardless of whether they're coming from a database, Excel file, or 3rd-party API.
ETLBox fits well within your applications and services. It is a very lightweight library and has a very small footprint. It's also complication free - requiring no servers, installation, or config files. You just add it as a package reference into your app and start using it.
ETLBox can run completely in-memory. In most cases, there's no need to store intermediate results in temporary databases or files on disk. Processing data in-memory, while it moves through the pipeline, can be more than 100 times faster than storing it to disk to query or process later.
ETLBox comes with a set of components and tasks that simplify to read data from various sources, transform it on the fly and write it into different destinations. But all components offer you the ability to add your own logic in C# - and you can benefit here from the full power of the .NET framework. Even if your own data source or destination is not supported out of the box, you can easily add it yourself by extending the CustomSource or CustomDestination components.
ETLBox is built on .NET and runs with the latest version of .NET, .NET Core and .NET Framework. You can write data flows and transformations in C# or any of the other .NET languages you know (e.g. VisualBasic or F#). Also use your existing tools, IDEs, containers, and libraries. Being built on .NET means ETLBox can run on all servers, operating systems, and environments.