When the file does not fit in RAM

CSV to SQL Server in C#

The file can be larger than memory. The load is still fast. One API for every database.

A partner drops a CSV. It might be bigger than the machine. You still need it in the table tonight — SQL Server, Postgres, MySQL, Oracle, or SQLite.

ETLBox streams the file and bulk-inserts as it reads. Nothing has to fit in memory. The same code hits every supported database.

var conn = new SqlConnectionManager(connString);
var source = new CsvSource("orders.csv");
var dest = new DbDestination(conn, "Orders");

source.LinkTo(dest);
Network.Execute(source);

Swap the connection manager. The source stays.

Stream and bulk-load. Do not load the file first.

A script that reads the whole CSV into a list is simple until the file grows. Then memory is the limit, and row-by-row inserts are slow. ETLBox never holds the file. It bulk-inserts into every database we support — not a one-off for a single engine.

ETLBox

File in, table loaded

Streams — the file does not have to fit in memory
Bulk insert on every supported database
Fast enough for nightly partner files
SQL Server, Postgres, MySQL, Oracle, and the rest
Excel, JSON, XML, Parquet — same pattern

A custom script

Load it, then insert

The whole file sits in RAM
Row-by-row inserts get slow
Each database wants its own bulk path
The next format is another script
Fine for a sample, not for production volume

Every database. The next file, too.

Bulk insert is not a SQL Server trick. Native bulk where the engine has it, ODBC and OLEDB when you need a generic path. When the partner sends Excel or JSON instead, the pattern stays.

Databases

Files and formats

Database connectors · Streaming connectors

Used in production

Commonwealth Bank of Australia, Deloitte, Health Catalyst, SEW Eurodrive, Calpine, FASTEC. Nightly file loads that do not fit in memory are why the library exists.

Try the load