ETLBox
A lightweight ETL (extract, transform, load) toolbox and data integration library for .NET
Get startedA lightweight ETL (extract, transform, load) toolbox and data integration library for .NET
Get startedTired of using User Interfaces where complex requirements are almost impossible to implement? With ETLBox it's easy to write code that fits your unique needs. Plug in your own logic or modify existing behavior to your specific requirements.
Read and write data from databases, flat files or web services. Harmonize, filter, aggregate, validate and clean the data in-memory with highly customizable transformations and write them into any destination.
Develop and test your ETL and data integration flows locally with your favorite development and debugging tools. Use the toolset that your team is already familiar with.
Track your changes with Git (or other source code controls), integrate with your current code review process and connect with your existing CI/CD platform.
Code your ETL with your favorite .NET language fitting your team’s skills and that is coming with a mature toolset.
ETLBox runs on Linux, macOS, and Windows. It is written in the current .NET Standard and successfully tested with the latest versions of .NET Core & .NET.
Process your data in-Memory:
Code your ETL with your favorite .NET language fitting your team’s skills and that is coming with a mature
toolset.
Process any workload:
ETLBox relies on Microsoft's
TPL.Dataflow library and is designed to process efficiently big amounts of any data.
Process data from anywhere:
ETLBox supports a wide range of Sql & NoSql databases, flat files, web services, and more - perfect as a
foundation for your
tailor made data integregation platform.
Use a simple syntax to create easy or complex data flows. Works with strongly type and dynamic objects.
string mySqlConnString = @"Server=localhost;Database=ETLBox;Uid=user;Pwd=password;"; string postgresConnString = @"Server=.;Initial Catalog=ETLBox;Trusted_Connection=true;"; var sourceCon = new MySqlConnectionManager(mySqlConnString); var destCon = new PostgresConnectionManager(postgresConnString); var source = new DbSource(sourceCon, "SourceTableName"); var dest = new DbDestination(destCon, "DestTableName"); //Use dest.ColumnMapping to change column name mapping source.LinkTo(dest); //Create the data flow Network.Execute(source); //Start the flow synchronously
string connString = @"Source=.;Trusted_Connection=true;Initial Catalog=ETLBox;" var destCon = new SqlConnectionManager(connString); var source = new CsvSource(sourceCon, "SourceData.csv"); var dest = new DbDestination(destCon, "DestinationTable"); //Default mapping: Csv header and column name match (case-sensitive) //Override mapping like this: dest.ColumnMapping = new[] { new ColumnMap() { DbColumnName = "Id", PropertyName = "OrderNumber" } }; source.LinkTo(dest); //Create the data flow Network.Execute(source); //Start the flow synchronously
ETL is the foundation of modern Business Intelligence application. It separates the analysis part from everything that comes before it. ETL stands for extract, transform and load, and describes a three-step process:
Extracting data is about acquiring it from a particular source and doing so in a way that all relevant data is collected. For ETL to work well, data needs to come directly from the source and in its rawest form.
Transforming data requires it to be stripped of its various formats, cleaned and harmonized. Put differently, it needs to become easy for the end user to read. Ensuring the highest possible data quality is crucial during this process, including detecting and removing duplicates and sorting the data into relevant, predetermined categories.
In the end, your ETL tool will help harmonize your data and load it into a target location, which essentially is a business intelligence (BI) or database tool of choice. There you can find it ready for your analysis.
ETLBox is a data processing engine based on .NET Core, giving you the power to create your own ETL processes. This engine runs inside your applications, APIs, and jobs to extract, filter, transform, migrate data on-the-fly. ETLBox speeds up your development by providing an easy-to-use toolbox for integrating your data from different sources within your app and load them into any target you like.
Here are a few things you can do with ETLBox:
Data integration solves the problem of moving, transforming, and consolidating information from various sources like databases, applications, files, and web services. Bringing them together enables you to cleanse, standardize, de-duplicate, manipulate, and synchronize them. Data integration can play an important role in any project, and it can be critical to have a rock-solid solution in place that is able to integrate numerous applications, services and databases. ETLBox as an Extract, transform, and load (ETL) tool is a perfect fit when you need to create a data integration solution.
ETLBox provides you with a single API for working with data. The API treats all data the same regardless of their source. By developing your applications against a single API, you can use the same components to process data regardless of whether they're coming from a database, Excel file, or 3rd-party API.
ETLBox fits well within your applications and services. It is a very lightweight library and has a very small footprint. It's also complication free - requiring no servers, installation, or config files. You just add it as a package reference into your app and start using it.
ETLBox can run completely in-memory. In most cases, there's no need to store intermediate results in temporary databases or files on disk. Processing data in-memory, while it moves through the pipeline, can be more than 100 times faster than storing it to disk to query or process later.
ETLBox comes with a set of components and tasks that simplify to read data from various sources, transform it on the fly and write it into different destinations. But all components offer you the ability to add your own logic in C# - and you can benefit here from the full power of the .NET framework. Even if your own data source or destination is not supported out of the box, you can easily add it yourself by extending the CustomSource or CustomDestination components.
ETLBox is built on .NET Core and .NET Standard and runs with the latest version of .NET. You write data flows and transformations in C# or any of the other .NET languages you know (e.g. VisualBasic or F#). You can also use your existing tools, IDEs, containers, and libraries. Being built on .NET Core means it can run on all servers, operating systems, and environments.