Class DbMerge<TInput>
Inserts, updates and (optionally) deletes data in database target.
Before the Merge is executed, all data from the destination is read into memory.
A delta table is generated that stores information if a records was inserted, updated, deleted or hasn't been touched (existed).
Inheritance
object
DbMerge<TInput>
Inherited Members
object.Equals(object)
object.Equals(object, object)
object.GetHashCode()
object.GetType()
object.MemberwiseClone()
object.ReferenceEquals(object, object)
object.ToString()
Namespace: ETLBox.DataFlow
Assembly: ETLBox.dll
Syntax
public class DbMerge<TInput> : DataFlowTransformation<TInput, TInput>, ILoggableTask, IDataFlowLogging, IDataFlowTransformation<TInput, TInput>, IDataFlowSource<TInput>, IDataFlowSource, IDataFlowBatchDestination<TInput>, IDataFlowDestination<TInput>, IDataFlowBatchDestination, IDataFlowDestination, IDataFlowComponent
Type Parameters
Name | Description |
---|
TInput | Type of ingoing data. |
Constructors
DbMerge()
Declaration
DbMerge(IConnectionManager, string, int)
Declaration
public DbMerge(IConnectionManager connectionManager, string tableName, int batchSize)
Parameters
DbMerge(IConnectionManager, string)
Declaration
public DbMerge(IConnectionManager connectionManager, string tableName)
Parameters
DbMerge(string, int)
Declaration
public DbMerge(string tableName, int batchSize)
Parameters
Type | Name | Description |
---|
string | tableName | |
int | batchSize | |
DbMerge(string)
Declaration
public DbMerge(string tableName)
Parameters
Type | Name | Description |
---|
string | tableName | |
Properties
AllowIdentityInsert
By default, identity columns (a.k.a auto increment or serial columns) are ignored
when writing into the destination. If set to true, the DbMerge will try to keep existing identity values if possible.
Declaration
public bool AllowIdentityInsert { get; set; }
Property Value
BatchSize
The batch size used when inserted data into the database table.
Declaration
public int BatchSize { get; set; }
Property Value
CacheMode
The cache mode used for reading data from the destination table.
By default, all data is read into memory before the actual merge is executed (Full mode).
If set to partial, only data is read into memory that is needed for the current merge comparison.
This will keep the memory consumption of the DbMerge much smaller.
Declaration
public CacheMode CacheMode { get; set; }
Property Value
ColumnConverters
Use a column converter to apply a conversion function to each value of a column before writing
into the target
Declaration
public ICollection<ColumnConverter> ColumnConverters { get; set; }
Property Value
ColumnMapping
Column mapping to map property to column names.
E.g. if the value of property "Id" should be written into the database column "Key",
then you can simply add a column mapping: DbColumnName: "Id" -> PropertyName: "Key".
Declaration
public ICollection<DbColumnMap> ColumnMapping { get; set; }
Property Value
Type | Description |
---|
System.Collections.Generic.ICollection<T><DbColumnMap> | |
CompareColumns
Property names that should be use to compare if the values of a column are equal, so that
the DbMerge can decide if the column needs to be updated.
Declaration
public ICollection<CompareColumn> CompareColumns { get; set; }
Property Value
Type | Description |
---|
System.Collections.Generic.ICollection<T><CompareColumn> | |
ConnectionManager
The connection manager used to connect to the database - use the right connection manager for your database type.
Declaration
public virtual IConnectionManager ConnectionManager { get; set; }
Property Value
DeleteColumns
List of property names and a to-be value that tells the DbMerge if this row can be deleted.
Declaration
public ICollection<DeleteColumn> DeleteColumns { get; set; }
Property Value
Type | Description |
---|
System.Collections.Generic.ICollection<T><DeleteColumn> | |
DestinationTableDefinition
The table definition of the destination table. By default, the table definition is read from the database.
Provide a table definition if the definition of the target can't be read automatically or you want the DbMerge
only to use the columns in the provided definition.
Declaration
public TableDefinition DestinationTableDefinition { get; set; }
Property Value
FindDuplicates
By default, the DbMerge expects the IdColumns to be unique.
If another record with the same id values arrives, this record is then identified as a duplicate.
It won't be written into the destination table - the delta output will contain this record, and the
ChangeAction is set to Duplicate
Please note that this may have an impact on the overall memory consumption, as already processed id values need to be stored internally during processing.
Declaration
public bool FindDuplicates { get; set; }
Property Value
IdColumns
Property names that are used to check if the columns match (id values are equal).
Declaration
public ICollection<IdColumn> IdColumns { get; set; }
Property Value
Type | Description |
---|
System.Collections.Generic.ICollection<T><IdColumn> | |
IgnoreDefaultColumnsOnInsert
If set to true, columns that have a default value declartions in the database are
ignored when inserting data
Declaration
public bool IgnoreDefaultColumnsOnInsert { get; set; }
Property Value
MergeMode
Defines the type of target data which affects how deletions or insertions are handled.
Full means that source contains all data, NoDeletions that source contains all data but no deletions are executed,
Delta means that source has only delta information and deletions are deferred from a particular property and
OnlyUpdates means that only updates are applied to the destination.
Declaration
public MergeMode MergeMode { get; set; }
Property Value
SourceBlock
SourceBlock from the underlying TPL.Dataflow which is used as output buffer for the component.
Declaration
public override ISourceBlock<TInput> SourceBlock { get; }
Property Value
Type | Description |
---|
ISourceBlock<><TInput> | |
Overrides
ETLBox.DataFlow.DataFlowSource<TInput>.SourceBlock
TableName
The name of the target database table for the merge.
Declaration
public string TableName { get; set; }
Property Value
TargetBlock
TargetBlock from the underlying TPL.Dataflow which is used as input buffer for the component.
Declaration
public override ITargetBlock<TInput> TargetBlock { get; }
Property Value
Type | Description |
---|
ITargetBlock<><TInput> | |
Overrides
ETLBox.DataFlow.DataFlowTransformation<TInput, TInput>.TargetBlock
TaskName
A name to identify the task or component. Every component or task comes
with a default name that can be overwritten.
Declaration
public override string TaskName { get; set; }
Property Value
Overrides
UpdateColumns
Property names that describe which columns are actually updated (if an update of the row is necessary).
Can be left empty, then all non id columns will be updated.
Declaration
public ICollection<UpdateColumn> UpdateColumns { get; set; }
Property Value
Type | Description |
---|
System.Collections.Generic.ICollection<T><UpdateColumn> | |
UseTruncateMethod
By default, the deletion of records are performed with a bulk delete.
This performs well if the number of records to delete is small compared to the total amount
of records.
If this property is set to true, all records are deleted before the merge operation,
using a TRUNCATE. Subsequently, all records are reinserted again.
In order to make this work, the MergeMode needs to be set to Full
and the CacheMode set to Full as well.
Declaration
public bool UseTruncateMethod { get; set; }
Property Value
ValueGeneratedColumns
Declaration
public ICollection<ValueGenerationColumn> ValueGeneratedColumns { get; set; }
Property Value
Methods
CheckParameter()
Declaration
protected override void CheckParameter()
Overrides
CheckParameterAfterInternalFlowCreation()
Declaration
protected void CheckParameterAfterInternalFlowCreation()
CleanUpOnFaulted(Exception)
Declaration
protected override void CleanUpOnFaulted(Exception e)
Parameters
Type | Name | Description |
---|
System.Exception | e | |
Overrides
CleanUpOnSuccess()
Declaration
protected override void CleanUpOnSuccess()
Overrides
InitComponent()
Declaration
protected override void InitComponent()
Overrides
LinkErrorTo(IDataFlowDestination<ETLBoxError>)
Declaration
public override IDataFlowSource<ETLBoxError> LinkErrorTo(IDataFlowDestination<ETLBoxError> target)
Parameters
Returns
Overrides
ETLBox.DataFlow.DataFlowSource<TInput>.LinkErrorTo(ETLBox.IDataFlowDestination<ETLBox.ETLBoxError>)
PrepareParameter()
Declaration
protected override void PrepareParameter()
Overrides
Implements