Class BatchTransformation<TInput, TOutput>

A batch transformation will transform batches of data. The default batch size are 100000 rows. The batch transformation function allows you to process and modify each batch of data. You can use the BatchSize property to choose a smaller batch size. The batch size must always be smaller than the max buffer size. The default batch size are 1000 rows per batch. The batch transformation is a partial blocking transformation - it will always need at least enough memory to store a whole batch.

Inherited Members
System.Object.Equals(System.Object)
System.Object.Equals(System.Object, System.Object)
System.Object.GetHashCode()
System.Object.GetType()
System.Object.MemberwiseClone()
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.ToString()
Namespace: ETLBox.DataFlow.Transformations
Assembly: ETLBox.dll
Syntax
    public class BatchTransformation<TInput, TOutput> : DataFlowTransformation<TInput, TOutput>, ILoggableTask, IDataFlowLogging, IDataFlowTransformation<TInput, TOutput>, IDataFlowSource<TOutput>, IDataFlowSource, IDataFlowDestination<TInput>, IDataFlowDestination, IDataFlowComponent
Type Parameters
NameDescription
TInput

Type of ingoing data.

TOutput

Type of outgoing data.

Constructors

BatchTransformation()

Declaration
    public BatchTransformation()

BatchTransformation(Int32)

Declaration
    public BatchTransformation(int batchSize)
Parameters
TypeNameDescription
System.Int32batchSize

The size of each batch

BatchTransformation(Int32, Func<TInput[], TOutput[]>)

Declaration
    public BatchTransformation(int batchSize, Func<TInput[], TOutput[]> batchTransformationFunc)
Parameters
TypeNameDescription
System.Int32batchSize

The size of each batch

System.Func<TInput[], TOutput[]>batchTransformationFunc

Sets the BatchTransformationFunc

Fields

WasInitActionInvoked

Declaration
    protected bool WasInitActionInvoked
Field Value
TypeDescription
System.Boolean

Properties

BatchSize

The size of each batch that is passed to the BatchTransformation<TInput, TOutput> Default is 1000.

Declaration
    public virtual int BatchSize { get; set; }
Property Value
TypeDescription
System.Int32

BatchTransformationFunc

The transformation Func that is executed on each array of input data. It returns another array as output data - the output array can have a different length than the input array.

Declaration
    public Func<TInput[], TOutput[]> BatchTransformationFunc { get; set; }
Property Value
TypeDescription
System.Func<TInput[], TOutput[]>

InitAction

The init action is executed shortly before the first data row is processed.

Declaration
    public Action InitAction { get; set; }
Property Value
TypeDescription
System.Action

SourceBlock

SourceBlock from the underlying TPL.Dataflow which is used as output buffer for the component.

Declaration
    public override ISourceBlock<TOutput> SourceBlock { get; }
Property Value
TypeDescription
System.Threading.Tasks.Dataflow.ISourceBlock<TOutput>
Overrides
ETLBox.DataFlow.DataFlowSource<TOutput>.SourceBlock

SuppressNullValueFilter

By default, all null values in the batch returned from the batch transformation func are filtered out. Set this option to true to avoid this behavior.

Declaration
    public bool SuppressNullValueFilter { get; set; }
Property Value
TypeDescription
System.Boolean

TargetBlock

TargetBlock from the underlying TPL.Dataflow which is used as input buffer for the component.

Declaration
    public override ITargetBlock<TInput> TargetBlock { get; }
Property Value
TypeDescription
System.Threading.Tasks.Dataflow.ITargetBlock<TInput>
Overrides
ETLBox.DataFlow.DataFlowTransformation<TInput, TOutput>.TargetBlock

TaskName

A name to identify the task or component. Every component or task comes with a default name that can be overwritten.

Declaration
    public override string TaskName { get; set; }
Property Value
TypeDescription
System.String
Overrides

Methods

CheckAndAdjustBatchSize()

Declaration
    protected void CheckAndAdjustBatchSize()

CheckParameter()

Declaration
    protected override void CheckParameter()
Overrides

CleanUpOnFaulted(Exception)

Declaration
    protected override void CleanUpOnFaulted(Exception e)
Parameters
TypeNameDescription
System.Exceptione
Overrides

CleanUpOnSuccess()

Declaration
    protected override void CleanUpOnSuccess()
Overrides

InitComponent()

Declaration
    protected override void InitComponent()
Overrides

InvokeBatchTransformationFunc(TInput[])

Declaration
    protected virtual TOutput[] InvokeBatchTransformationFunc(TInput[] batch)
Parameters
TypeNameDescription
TInput[]batch
Returns
TypeDescription
TOutput[]

InvokeInitActionOnce()

Declaration
    protected virtual void InvokeInitActionOnce()

Implements