Release notes
Version 3.0.0
Features:
- Updating package references to current version, Improving support for .NET 7 and .NET 6
- Using MySqlConnector (instead of MySql.Data connector) when connecting to MariaDb
- Adding MariaDbOdbcConnector which allows Odbc connections to MariaDb
- New connector: Snowflake
- LogProcessTask: Start/End/Abort will modify passed process, and not create new instance
- Adding SnowflakeOdbcConnector which allow Odbc connections to Snowflake
- New connector: Neo4j
- New connector: Azure Cosmos Db
- Renamed package ETLBox.AzureServiceBus to ETLBox.Azure.ServiceBus
- New connector: Azure Tables
- New connector: SAP HANA
- Azure ServiceBusSource now allows to fetch multiple messages with MaxMessageSize
- Adding HanaOdbcConnector which allows Odbc connections to SAP HANA
- Network class now throws exception when trying to run network twice (no network execution/no error message before)
- Improved support for Db2 and different architectures (LUW, zOS, Cloud)
- SqlTask: Removed obsolete static helper methods that contained a “name” properties
- SqlTask: Removed obsolete static helper methods ExecuteReaderSingleLine
- DbDestination / DbMerge: Removing obsolete Wait() method
- [BREAKING] QueryParameter: Renamed DBType -> DbType, DBSize -> DbSize, Type -> SqlType
- [BREAKING] Static class DataFlow renamed to Settings, properties of static class ControlFlow (DefaultDbConnection) moved into Settings, new namespace: ETLBox (was ETLBox.DataFlow before)
- [BREAKING] Prpoerty LogInstance (ETLBox.Logging.Logging) moved into ETLBox.DataFlow.Settings -[BREAKING] New namespace ETLBox, which contains some shared classes that were used in other namespaces before
- [BREAKING] Namespace ETLBox.DataFlow now combines ETLBox.DataFlow, ETLBox.DataFlow.Connectors and ETLBox.DataFlow.Transformations
- [BREAKING] Namespace ETLBox.ControlFlow now combines ETLBox.ControlFlow and ETLBox.ControlFlow.Tasks
- [BREAKING] Moved ObjectNameDescriptor from ETLBox.Helper to ETLBox.ControlFlow
- [BREAKING] Created separate namespace for all classes which are added via separate package (e.g. JsonSource is now in the namespace ETLBox.Json)
- [BREAKING] Renamed Attribute [ColumnMap] to [DbColumnMap]
- [BREAKING] Renamed DbSource. SourceTableDefinition -> TableDefinition and DbDestination.DestinationTableDefinition to TableDefinition
- [BREAKING] DbMerge.MergeProperties: The properties (IdColumns, UpdateColumns, CompareColumns, DeleteColumns) are now direct properties of DbMerge
- [BREAKING] Renamed enum ConnectionManagerType to ConnectionType, Property IConnectionManager.ConnectionManagerType to ConnectionType
Bug
- Fixed issues with spaces in table names for GetTableListTask
- Improving support for dynamic objects in Azure ServiceBusSource
Version 2.7.1
Features:
- Adapter for Azure Service Bus
- Network.Execute now accepts IDataFlowComponent or derived interfaces as arguments
- XmlSource: When using dynamic approach, attribute name prefix can now be configured (default is ‘at_’)
- Fixed issue when using special path expressions with JsonPathConverter
- DbMerge: Overwriting identity/auto increment columns now possible when AllowIdentityInsert is set to true
Version 2.7.0
Features:
- Aggregation: Adding AggregationCondition to only execute predefined aggregation methods when condition is met
- Added async methods for SqlTask (e.g. ExecuteNonQueryAsync), RowCountTask and TruncateTableTask
- DbMerge: ValueGeneratedColumns are now also populated for Exists in DbMerge output
- RowCountTask: Changed return type of Count() and HasRows() to expected return types int and bool
- Improving thrown exceptions- now the original exception is always thrown (was AggregateException before)
- CsvSource: Enabling auto detection of delimiter in default configuration
- Adding RequestCount to StreamMetaData
- Added ValueToSqlConversionFunc for Odbc/OleDb connection managers
- ExcelSource: Exposes property SheetNumber to specify a sheet index instead of a sheet name
- ExcelSource: Sheet name comparison by default is now case-insensitive (can be switched back to case-sensitive)
- ExcelDestination: Default Sheetname set to “Sheet1”
Bug fixes:
- Aggregation: Group columns with the FirstValue aggregation method now properly retrieves the first value for a group
- Fixed slowed ODBC connector execution performance for tables with more than 20 columns
- ExcelSource: Only first sheet will be read from source if no sheet name is specified
- Fixed AccessViolation errors for bulk inserts via Odbc with MySql connector
- DbSource: Fixed issue with no white spaces when parsing column names in sql
Version 2.6.8
Features:
- Network Execute/ExecuteAsync now accepts CancellationToken instead of CancellationTokenSource (BREAKING CHANGE)
Bug fixes:
- LogLevel for messages for each batch of processed rows during processing reduced to Debug (was Information before) - final amount still logged as Information
- DisableLogging now also disables logging for Trace/Fatal message (were always logged before)
Version 2.6.7
Features
- Distinct: now offers LinkDuplicatesTo(..) to forward duplicates into separate output
- DbMerge: Performs implicit type cast when determining equality of CompareColumns
- DbMerge: Records marked as deletion but not present in destination now flagged with ChangeAction NotFound
- DbMerge: Adding ColumnConverters
Bug fixes
- Lookup: When retrieving multiple values from source, unmatched input data was filtered out - this is fixed
Version 2.6.6
Bug fixes
- JsonSource/XmlSource: Fixing issue that limit property is properly observed when collecting unparsed data and using the GetNextUri/HasNextUri functions
Version 2.6.5
Features
- PostgresSql bulk insert now adds flawed data row information to Data dictionary of Exception message
Bug fixes
- DbSource: Fixed issue where sql was provided and properties where mapped to wrong column
- Fixed bug when collecting unparsed data and setting limit on XmlSource or JsonSource
- Fixed bug for GetTableListTask where table names contained an invalid quotation at the beginning
Version 2.6.4
Features
- Network.Execute & ExecuteAsync (and same methods on components) now accept a CancellationTokenSource
Bug fixes
- Fixing issues in XmlSource when collecting unparsed data
- Fixing issues in JsonSource when collecting unparsed data
Version 2.6.3
Features
- Collecting of UnparsedData can now be disabled via the CollectUnparsedData prop
- JsonSource: The json property name that holds the data array can now be set with the PropertyNameOfDataArray prop
- Adding EntityFramework Bulk Operations extensions for MySql
- TableDefinition sql codes: Reduce columns for DbDestination/DbSource meta data retrieval to absolute minimum
Bug fixes
- JsonSource: Fixing bug that UnparsedData is not properly json formatting when using the Limit Property
- Fixed bug where Primary keys were incorrectly dropped/recreated when altering a table with CreateTableTask
- DbDestination/DbSource/DbMerge: Amount of meta data to obtain TableDefinition is reduced
- Adding UnparsedData support for XmlSource (disabled by default)
Version 2.6.2
Features
- Lookups now allow selecting multiple rows with same key from lookup source
- Creating beta version for extending Entity Framework’s DbContext
- Improving support for BulkOperations (preparation for future EntityFramework support)
- ETLBoxNotSupportException replaced with System.NotSupportedException
Bug fixes
- GetDatabaseListTask now properly treats databases with special characters in name
Version 2.6.1
Features
- ETLBox.MySql now references MySql.Data 8.0.29
- XmlSource offers ElementNameRetrievalFunc to change the ElementName while processing Xml data
- DbDestination and DbMerge support instant reading of value generated columns (e.g. computed columns, identites or default values) after inserting/updating data
- Fixing issue where IgnoreColumn set on mutiple props without column name threw an exception
- Meta data retrieval for procedures in SqlServer added: TableDefinition.FromProcedureName(…)
- ETLBoxError: Original exception now retrievable with GetException()
- New connector: (Sybase) SAP ASE
Bug fixes
- Nullable int replaced with int type in ExcelColumn attribute (non basic CLR types lead to issues in attributes)
Version 2.6.0
Features
- New transformation: WaitTransformation waits for other components (or TaskCompletionSources) before any data can pass
- Postgres schema search path logic is now supported (when only a table name is provided, the search path is used to find the first matching schema)
- LookupTransformation: PartialCacheSettings now expose SqlParameter which are applied to the LoadCacheSql
- ETLBoxError extended with additional Information from exception source (e.g. current request URI and progress count)
- Now supporting BulkOperations in DbDestination and DbSource (BulkUpdate, BulkDelete and BulkSelect)
- HttpResponseMessage is now part of StreamMetaData
- All streaming source&destinations now expose same unified constructor (string uri, ResourceType resourceType)
- Breaking Change: SqlTask.ExecuteScalar will now return T as type (was Nullable before)
Bug fixes
- DbMerge: Erroneous rows are now properly forwarded when conversion issue during data lookup occur
- LookupTransformation/Error linking: data conversion exceptions are now properly forwarded with improved error message
- Fixed bug that broke GetTableListTask when table name contained spaces
- HttpClient is not disposed anymore when cleaning up streaming source connectors
- DbMerge: ColumnMap flag IgnoreColumn=true is now properly observed
- DbMerge: identity columns are now ignored by default
- Lookup: ColumnMapping is now properly observed when using DbSource as lookup source in partial cache
- Improved exception message when type conversion of data fails
- QueryParameter extended with additional constructor (name, value) and fixed issue with null values
Version 2.5.2
Features
- Streaming sources enhanced with RowModificationAction that allows to enrich rows with the current stream meta data or to modify rows before they are sent into the output buffer
- LookupTransformation: Property IgnoreDataTypes renamed to UseExactNumericTypes
- DbMerge supports duplicate filtering with the new flag ‘FilterDuplicates’
- TextDestination: Added WriteHeaderFunc which writes a custom header line as first row into output file
- ColumnMap attribute extended with property “IgnoreColumn” which allows to ignore particular columns when reading or writing into a database
Bug fixes
- .NET 3.1 and .NET standard version contains references to 3.1 system packages, .NET 5 only references 5.0 system packages
- DbMerge now ignores flawed rows (redirected via LinkErrorTo) in delta output
- Improved exception handling for SqlTask.ExecuteReader
- SqlServer: fixed bug when reading column definition from tables containing HIERACHYID, GEOGRAPHY or GEOMETRY data type
- CreateTableTask: Fixed bug that prevented keywords to be used as default values
- TableDefinition in SqlServer: Sql specific keywords in default values (e.g. GETDATE) are now correctly read
- DbMerge: will throw exception if a readonly variable is used incorrectly as CompareColumn
- DbSource/DbDestination/DbMerge: Improved exception message if a column mapping with duplicate column names is provided
- Fixed bug when parsing sql statements that contained column names in single quotes
- Using GetNextUri/HasNextUri, error message improved if status code response is not 200
- Fixed in DbMerge: If input row is null, Exception is thrown: “TargetException: Non-static method requires a target”
- Lookup: retrieved column values are now converted automatically into type of destination property
Version 2.5.1
Features
- DbMerge: Added new MergeMode ‘InsertsOnly’
Version 2.5.0
Features
- Removed unnecessary command preparation and removed DoPrepareCommand property in connection managers
- Support that HttpRequestMessage can be changed in StreamMetaData when using GetNextUri/HasNextUri pattern
- Updated third party dependencies to latest version
- Upgraded CsvHelper to latest version (some Configurations are now part of Context object). This change may be a breaking change depending on your usage of CsvHelper specific configuration settings. Please refer to the {< link-ext url=“https://joshclose.github.io/CsvHelper/change-log/" text=“CsvHelper release notes” >} to see what changed here.
- Breaking change: DataFlow.LoggingThresholdRows renamed to LogThreshold
- Breaking change: Combining CreateLogTable and ReadLogTable tasks into LogTask (e.g. use
LogTask.CreateLogTable(..)
instead) - Breaking change: Introducing structured logging - core package now supports general ILogger interface instead NLog. You can now assign your ILogger instance to the static property
Logging.LogInstance
. Now NLog, Serilog and other logger supporting the log abstraction interface will be supported. - LoadProcess: adding SourceId column to ease joins with other tables
- Combining CustomTask/Package/Sequence into LogSection
- Combining Create/Start/Abort/End/Read LoadProcessTasks into one LoadProcessTask (e.g. use
LoadProcessTask.Create()
instead) - Added support for FOREIGN KEY constraints in
CreateTableTask
andTableDefinition
object - Added Parquet source/destination
Bug fixes
- AggregationMethods Min/Max now support DateTime
- LogThreshold set on component now prioritized over global threshold value
Version 2.4.4
Features
- CommandTimeout property limiting timeout for sql commands added to all connection managers
- KeepIdentity renamed to AllowIdentityInsert in DbDestination (Breaking change)
Bug fixes
- Improved support for Db2 - internal sql updated to work on zOS and LUW version
Version 2.4.3
Features
- SqlConnectionManager exposes parameter to use row locks instead table lock for bulk insert operations
- DbSource: Sql property now allows query parameter
Bug fixes
- Improved exception handling for DbMerge: Added check for IdColumn
- XmlSource now correctly reads all tags if xml contains no line breaks
Version 2.4.2
Features
- DataTypeConverter: Added helper method to convert from CLR type to SqlServer specific data type
- ExcelSource exposes ExcelColumns property, allowing to manual set column configuration
- All components now have delegate for progress tracking
- New destination: ExcelDestination
Bug fixes
- ExcelSource: EndColumn value in ExcelRange now starts counting at with 1 instead 0
Version 2.4.1
Features
- Adding GetNextUri/HasNextUri pattern to DataFlowStreamDestinations
- ResourceType.File is now default for all streaming sources and destinations
- Streaming sources: New property CreateStreamReader allows usage of existing streams
- Streaming destinations: New property CreateStreamWriter allows to write into existing streams
- Adding integration to Azure Blob Storage for streaming sources and destinations (ResourceType.AzureBlob)
- ExcelSource&CsvSource: ReadFieldHeaders will read header data based on current configuration (can be called before the actual data flow started)
- DbMerge now exposes ColumnMapping property
- New transformation: FilterTransformation as alternative to linking with predicates
- ColumnRename: RenameFunc let you define a custom function to rename each column/prop separately with a custom function
Bug fixes
- SqlServer: fixing bug for Query parameter types without length (e.g. DECIMAL)
- DbSource: Using a sql query where the column names include white spaces, these white spaces are kept
Version 2.4.0
Features
- TextSource: ParseLineAction changed to ParseLineFunc
- StreamMetaData holds number of processed rows per request
- Added Redis source/destination (package ETLBox.Redis)
- Added Couchbase source/destination (package ETLBox.Couchbase)
- Added MongoDb source/destination (package ETLBox.MongoDb)
- Network class: Network.Cancel(..) allows to cancel a running complete data flow
Bug fixes
- DbMerge: For missing compare columns, the non-Id columns are now properly used detecting exists/updates
- Db2ConnectionManager/UseSqlBulkInsert: Fixed bug when parameter where exceeding Db2 maximum
- Network class: Networks now correctly cancels when exception in error path is thrown
Version 2.3.2
Features
General:
- Auto generated xml documentation added to package - now visible with Intellisense or when browsing package content.
DataFlow:
- ExcelSource exposes parsed header names in FieldHeaders property.
- DbSource has ColumnConverters property
Version 2.3.1
Features
ConnectionManagers:
- ODBC/OleDb now have ConnectionManagerType property settable.
ControlFlow:
- Db2 now has support for schemas
- CreateTableTask: Now offers functionality to alter tables (Alter() / CreateOrAlter())
- CreateSchemaTask supports authorization
- Added GetTableListTask (return all tables in database)
DataFlow:
- UnparsedData property for streaming sources now contains data of skipped rows.
- All executable sources allow to set limit for records to read
- Added property KeepIdentity to DbDestination which allows overwriting of Identity columns
- DataConverters available for DbDestination (allows to add custom column converter)
- (Breaking Change) DbSource: Replace ListColumnNames prop with ICollection ColumnMapping
- (Breaking change) Improved naming in ColumnMap Attribute (NewName = PropertyName, CurrentName = DbColumnName)
- (Breaking change) ColumnRename now uses RenameColumn attribute instead ColumnMapping.
Bug fixes
- Postgres connection manager now has full support for jsonb columns.
- DbSource/Sql property: Whitespace in column aliases are now properly parsed as column names.