Stream connectors allow to read data that is stored in a common format either from a file or via http.
ETLBox currently supports the following data storage types:
This article covers functionalites that are shared by all of these data types.
There is a connector package for each data type that must be included together with the core package to read the data. E.g. for Json you need to add the ETLBox.Json package.
CustomDestination- these custom connectors allow you to define your own parsing logic.
All these different storage types can be read either from a flat file (e.g. a .csv file on your local machine or a network share) or via a http web service endpoint. So you could read a csv file from a network storage located at
//Volumes/user/data.csv/, but you could also read csv files from a REST endpoint on
Also, all these types (except excel) can also be written into a file or send to a http endpoint. E.g. you can write a text file into you local windows folder at
c:/data/text.csv or send it as a POST into
Once the data is read from one of these types into a data flow, all transformation that ETLBox offers can be used to transform the data. So these connectors make it easy to transfer an xml into a database table or to send an excel as json to a web service.
All data that is read or written with these connectors supports streaming. Streaming for a source means that data will be send into a data flow while it is still read from the source. Imagine you receive a json from a webservice that contains a lot of data - ETLBox will start reading the data from the input stream of the web resonponse. Every element of the data that was already received will be send to the connected components immediately. So while the source server is still transferring data, the ETLBox data will already start processing the data received.
If you source is sending you data faster than your data flow process, each connector will use an internal buffer to store the already received data. The buffer size can be adjusted with the
MaxBufferSize property available on each connector.
Resource Type and Web Requests
All flat files sources and destinations in this article can be set to work either with a file or to use data from a web service. If you want to access a file on your drive or a network share, set the
ResourceType property to
This is default for Csv, Excel and Text, but not for Xml and Json.
The other option is
ResourceType.Http - and allows you to read/write data from a http endpoint.Instead of a filename just provide a url. Furthermore, the components also have
HttpClient and for sources a
HttpRequestMessage property that can be used to configure the http request, e.g. to add authentication, change the request type from GET to POST or to use https instead.
A common scenario is that you need to create multiple calls to a http endpoint in order to receive all data. E.g. your first request to an endpoint will be via a GET request to
http://test.com/data?page=1. To receive all data, you will need to increase the page number and repeat the request until you don’t get any data in the response.
This can be accomplished in ETLBox with using the
JsonSource<Row> source = new JsonSource<Row>(); int page = 1; source.GetNextUri = streamMetaData => $"http://test.com/data?page="+page++; source.HasNextUri = streamMetaData => streamMetaData.ProcessedRows > 0;
Skipping rows at beginning
Each streaming source can skip some rows at the beginning of the stream. These rows will be simple ignored and also not counted as Progress. Set the property
SkipRows to a value greater than 0.
For sources and destinations that deal with text (so every streming source except excel) you can define the used Encoding. The default Encoding is UTF-8 - if you need a different encoding, please overwrite the