# BatchTransformation {{< rawhtml >}}

Class BatchTransformation<TInput, TOutput>

A batch transformation will transform batches of data. The default batch size are 30000 rows. The batch transformation function allows you to process and modify each batch of data. You can use the BatchSize property to choose a smaller batch size. The batch size must always be smaller than the max buffer size. The default batch size are 1000 rows per batch. The batch transformation is a partial blocking transformation - it will always need at least enough memory to store a whole batch.

Inheritance
object
LoggableTask
DataFlowComponent
DataFlowSource<TOutput>
DataFlowTransformation<TInput, TOutput>
BatchTransformation<TInput, TOutput>
BatchTransformation<TInput>
BatchTransformation<TInput>
BlockTransformation<TInput, TOutput>
BlockTransformation<TInput, TOutput>
CachedBatchTransformation<TInput, TOutput, TCache>
CachedBatchTransformation<TInput, TOutput, TCache>
Implements
IDataFlowTransformation<TInput, TOutput>
IDataFlowSource<TOutput>
IDataFlowSource
IDataFlowDestination<TInput>
IDataFlowDestination
IDataFlowComponent
ILoggableTask
Inherited Members
DataFlowTransformation<TInput, TOutput>.TargetBlock
DataFlowTransformation<TInput, TOutput>.CompleteBuffer()
DataFlowTransformation<TInput, TOutput>.FaultBuffer(Exception)
DataFlowSource<TOutput>.LinkTo(IDataFlowDestination<TOutput>)
DataFlowSource<TOutput>.LinkTo(IDataFlowDestination<TOutput>, Predicate<TOutput>)
DataFlowSource<TOutput>.LinkTo(IDataFlowDestination<TOutput>, Predicate<TOutput>, Predicate<TOutput>)
DataFlowSource<TOutput>.LinkTo<TConvert>(IDataFlowDestination<TOutput>)
DataFlowSource<TOutput>.LinkTo<TConvert>(IDataFlowDestination<TOutput>, Predicate<TOutput>)
DataFlowSource<TOutput>.LinkTo<TConvert>(IDataFlowDestination<TOutput>, Predicate<TOutput>, Predicate<TOutput>)
DataFlowComponent.MaxBufferSize
DataFlowComponent.Predecessors
DataFlowComponent.Successors
DataFlowComponent.Completion
DataFlowComponent.BufferCancellationSource
DataFlowComponent.BufferCancellationToken
DataFlowComponent.WasComponentInitialized
DataFlowComponent.IsReadyForProcessing
DataFlowComponent.SetParentComponent(DataFlowComponent)
DataFlowComponent.LinkErrorTo(IDataFlowDestination<ETLBoxError>)
DataFlowComponent.InternalLinkTo<T>(IDataFlowDestination, object, object)
DataFlowComponent.AddComplementVoidLinkToOtherTargets(Func<Type, IDataFlowDestination>, Delegate)
DataFlowComponent.ResetComponent()
DataFlowComponent.InitBufferObjects(CancellationToken?)
DataFlowComponent.PrepareParameterForCheck()
DataFlowComponent.InitCheckedParameter()
DataFlowComponent.OnInitialization
DataFlowComponent.OnCompletion
DataFlowComponent.OnException
DataFlowComponent.CompleteOrFaultBufferOnPredecessorCompletion(Task)
DataFlowComponent.Exception
DataFlowComponent.ErrorSource
DataFlowComponent.CancelBufferAndThrowOperationCanceledException()
DataFlowComponent.InternalLinkErrorTo(IDataFlowDestination<ETLBoxError>)
DataFlowComponent.ThrowOrRedirectError(Exception, string, string)
DataFlowComponent.ThrowErrorAndFaultNetwork(Exception, string)
DataFlowComponent.CancelComponent()
DataFlowComponent.ErrorCount
DataFlowComponent.ProgressCount
DataFlowComponent.OnProgress
DataFlowComponent.ExecutionStartTime
DataFlowComponent.ExecutionEndTime
DataFlowComponent.Tag
DataFlowComponent.LoggingStartOnce()
DataFlowComponent.LoggingFinishOnce()
DataFlowComponent.LogProgressBatch_AfterThrowOrRedirectError(int)
DataFlowComponent.LogProgress_AfterThrowOrRedirectError()
LoggableTask.TaskType
LoggableTask.TaskName
LoggableTask.DisableLogging
LoggableTask.LogThreshold
LoggableTask.TaskHash
LoggableTask.LogInstance
LoggableTask.ParentTask
LoggableTask.LogTrace(string, params object[])
LoggableTask.LogDebug(string, params object[])
LoggableTask.LogInfo(string, params object[])
LoggableTask.LogInfoOrDebug(string, params object[])
LoggableTask.LogWarn(string, params object[])
LoggableTask.LogError(string, params object[])
LoggableTask.LogFatal(string, params object[])
object.Equals(object)
object.Equals(object, object)
object.GetHashCode()
object.GetType()
object.MemberwiseClone()
object.ReferenceEquals(object, object)
object.ToString()
Namespace: ETLBox.DataFlow
Assembly: ETLBox.dll
Syntax
{{< /rawhtml >}} ```C# public class BatchTransformation : DataFlowTransformation, IDataFlowTransformation, IDataFlowSource, IDataFlowSource, IDataFlowDestination, IDataFlowDestination, IDataFlowComponent, ILoggableTask ``` {{< rawhtml >}}
Type Parameters
Name Description
TInput

Type of ingoing data.
TOutput

Type of outgoing data.

Constructors

BatchTransformation()

Declaration
{{< /rawhtml >}} ```C# public BatchTransformation() ``` {{< rawhtml >}}

BatchTransformation(int)

Declaration
{{< /rawhtml >}} ```C# public BatchTransformation(int batchSize) ``` {{< rawhtml >}}
Parameters
Type Name Description
int batchSize

The size of each batch

BatchTransformation(int, Func<TInput[], TOutput[]>)

Declaration
{{< /rawhtml >}} ```C# public BatchTransformation(int batchSize, Func batchTransformationFunc) ``` {{< rawhtml >}}
Parameters
Type Name Description
int batchSize

The size of each batch
Func<TInput[], TOutput[]> batchTransformationFunc

Sets the BatchTransformationFunc

Fields

WasInitActionInvoked

Declaration
{{< /rawhtml >}} ```C# protected bool WasInitActionInvoked ``` {{< rawhtml >}}
Field Value
Type Description
bool

Properties

BatchSize

The size of each batch that is passed to the BatchTransformation<TInput, TOutput> Default is 1000.

Declaration
{{< /rawhtml >}} ```C# public virtual int BatchSize { get; set; } ``` {{< rawhtml >}}
Property Value
Type Description
int

BatchTransformationFunc

The transformation Func that is executed on each array of input data. It returns another array as output data - the output array can have a different length than the input array.

Declaration
{{< /rawhtml >}} ```C# public Func BatchTransformationFunc { get; set; } ``` {{< rawhtml >}}
Property Value
Type Description
Func<TInput[], TOutput[]>

InitAction

The init action is executed shortly before the first data row is processed.

Declaration
{{< /rawhtml >}} ```C# public Action InitAction { get; set; } ``` {{< rawhtml >}}
Property Value
Type Description
Action

SourceBlock

SourceBlock from the underlying TPL.Dataflow which is used as output buffer for the component.

Declaration
{{< /rawhtml >}} ```C# public override ISourceBlock SourceBlock { get; } ``` {{< rawhtml >}}
Property Value
Type Description
ISourceBlock<TOutput>
Overrides

SuppressNullValueFilter

By default, all null values in the batch returned from the batch transformation func are filtered out. Set this option to true to avoid this behavior.

Declaration
{{< /rawhtml >}} ```C# public bool SuppressNullValueFilter { get; set; } ``` {{< rawhtml >}}
Property Value
Type Description
bool

TargetBlock

TargetBlock from the underlying TPL.Dataflow which is used as input buffer for the component.

Declaration
{{< /rawhtml >}} ```C# public override ITargetBlock TargetBlock { get; } ``` {{< rawhtml >}}
Property Value
Type Description
ITargetBlock<TInput>
Overrides

Methods

CheckAndAdjustBatchSize()

Declaration
{{< /rawhtml >}} ```C# protected void CheckAndAdjustBatchSize() ``` {{< rawhtml >}}

CheckParameter()

Declaration
{{< /rawhtml >}} ```C# protected override void CheckParameter() ``` {{< rawhtml >}}
Overrides

CleanUpOnFaulted(Exception)

Declaration
{{< /rawhtml >}} ```C# protected override void CleanUpOnFaulted(Exception e) ``` {{< rawhtml >}}
Parameters
Type Name Description
Exception e
Overrides

CleanUpOnSuccess()

Declaration
{{< /rawhtml >}} ```C# protected override void CleanUpOnSuccess() ``` {{< rawhtml >}}
Overrides

CompleteBuffer()

Declaration
{{< /rawhtml >}} ```C# protected override void CompleteBuffer() ``` {{< rawhtml >}}
Overrides

FaultBuffer(Exception)

Declaration
{{< /rawhtml >}} ```C# protected override void FaultBuffer(Exception e) ``` {{< rawhtml >}}
Parameters
Type Name Description
Exception e
Overrides

InitComponent()

Declaration
{{< /rawhtml >}} ```C# protected override void InitComponent() ``` {{< rawhtml >}}
Overrides

InvokeBatchTransformationFunc(TInput[])

Declaration
{{< /rawhtml >}} ```C# protected virtual TOutput[] InvokeBatchTransformationFunc(TInput[] batch) ``` {{< rawhtml >}}
Parameters
Type Name Description
TInput[] batch
Returns
Type Description
TOutput[]

InvokeInitActionOnce()

Declaration
{{< /rawhtml >}} ```C# protected virtual void InvokeInitActionOnce() ``` {{< rawhtml >}}

Reset()

Declaration
{{< /rawhtml >}} ```C# protected override void Reset() ``` {{< rawhtml >}}
Overrides

Implements

{{< /rawhtml >}}