ETLBox ETLBox

Class LookupTransformation<TInput, TSource>

The lookup transformation enriches the incoming data with data from the lookup source. Data from the lookup source is read into memory when the first record arrives. For each incoming row, the lookup tries to find a matching record in the loaded source data and uses this record to enrich the ingoing data.

Inheritance
System.Object
LoggableTask
DataFlowComponent
DataFlowSource<TInput>
DataFlowTransformation<TInput, TInput>
LookupTransformation<TInput, TSource>
LookupTransformation
Inherited Members
System.Object.Equals(System.Object)
System.Object.Equals(System.Object, System.Object)
System.Object.GetHashCode()
System.Object.GetType()
System.Object.MemberwiseClone()
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.ToString()
Namespace: ETLBox.DataFlow.Transformations
Assembly: ETLBox.dll
Syntax
    public class LookupTransformation<TInput, TSource> : DataFlowTransformation<TInput, TInput>, ILoggableTask, IDataFlowLogging, IDataFlowTransformation<TInput, TInput>, IDataFlowSource<TInput>, IDataFlowSource, IDataFlowDestination<TInput>, IDataFlowDestination, IDataFlowComponent
Type Parameters
NameDescription
TInput

Type of ingoing and outgoing data.

TSource

Type of data used in the lookup source.

Examples
public class Order
{    
    public int OrderNumber { get; set; }
    public int CustomerId { get; set; }
    public string CustomerName { get; set; }
}
public class Customer
{
    [RetrieveColumn(nameof(Order.CustomerId))]
    public int Id { get; set; }
    [MatchColumn(nameof(Order.CustomerName))]
    public string Name { get; set; }
}
DbSource<Order> orderSource = new DbSource<Order>("OrderData");
CsvSource<Customer> lookupSource = new CsvSource<Customer>("CustomerData.csv");
var lookup = new LookupTransformation<Order, Customer>();
lookup.Source = lookupSource;
DbDestination<Order> dest = new DbDestination<Order>("OrderWithCustomerTable");
source.LinkTo(lookup).LinkTo(dest);

Constructors

LookupTransformation()

Declaration
    public LookupTransformation()

LookupTransformation(IDataFlowExecutableSource<TSource>)

Declaration
    public LookupTransformation(IDataFlowExecutableSource<TSource> source)
Parameters
TypeNameDescription
IDataFlowExecutableSource<TSource>source

Sets the Source of the lookup.

LookupTransformation(IDataFlowExecutableSource<TSource>, Func<TInput, ICollection<TSource>, TInput>)

Declaration
    public LookupTransformation(IDataFlowExecutableSource<TSource> source, Func<TInput, ICollection<TSource>, TInput> retrievalFunc)
Parameters
TypeNameDescription
IDataFlowExecutableSource<TSource>source

Sets the Source of the lookup.

System.Func<TInput, System.Collections.Generic.ICollection<TSource>, TInput>retrievalFunc

Sets the RetrievalFunc

Properties

CacheMode

The cache mode used for loading data into the lookup cache. By default, all data is loaded into memory from the lookup source when the first records arrives at the lookup.

Declaration
    public CacheMode CacheMode { get; set; }
Property Value
TypeDescription
CacheMode

GetInputRecordKeyFunc

This function describe how the key is generated for an ingoing row. If the result of this function match with GetSourceRecordKeyFunc, both records can be identified as a match.

Declaration
    public Func<TInput, object> GetInputRecordKeyFunc { get; set; }
Property Value
TypeDescription
System.Func<TInput, System.Object>

GetSourceRecordKeyFunc

This function describe how the key is generated for a row from the lookup source. This key is used to store the data in the lookup dictionary.

Declaration
    public Func<TSource, object> GetSourceRecordKeyFunc { get; set; }
Property Value
TypeDescription
System.Func<TSource, System.Object>

IgnoreDataTypes

If set to true, data types are ignored when comparing data to identify matching records. E.g. property that stores the integer value 7 would match with a string "7".

Declaration
    public bool IgnoreDataTypes { get; set; }
Property Value
TypeDescription
System.Boolean

MatchColumns

This collection will be used to define the matching columns - will also work with ExpandoObject.

Declaration
    public ICollection<MatchColumn> MatchColumns { get; set; }
Property Value
TypeDescription
System.Collections.Generic.ICollection<MatchColumn>

PartialCacheSettings

Only need if you set the CacheMode to PartialDbCache. This will only work if you use a DbSource as lookup source.

Declaration
    public LookupTransformation<TInput, TSource>.PartialDbCacheSettings PartialCacheSettings { get; set; }
Property Value
TypeDescription
LookupTransformation.PartialDbCacheSettings<>

ProgressCount

Declaration
    public int ProgressCount { get; }
Property Value
TypeDescription
System.Int32

RetrievalByKeyFunc

The retrieval function that describes how the ingoing data can be enriched with the already pre-read data from

  • this one returns not only a list, but a dictionary that provides faster access to the data. The dictionary key is created with the GetSourceKeyFunc. the Source.
Declaration
    public Func<TInput, IDictionary<object, TSource>, TInput> RetrievalByKeyFunc { get; set; }
Property Value
TypeDescription
System.Func<TInput, System.Collections.Generic.IDictionary<System.Object, TSource>, TInput>

RetrievalFunc

The retrieval function that describes how the ingoing data can be enriched with the already pre-read data from the Source.

Declaration
    public Func<TInput, ICollection<TSource>, TInput> RetrievalFunc { get; set; }
Property Value
TypeDescription
System.Func<TInput, System.Collections.Generic.ICollection<TSource>, TInput>

RetrieveColumns

This collection will be used to define the retrieve columns - will also work with ExpandoObject.

Declaration
    public ICollection<RetrieveColumn> RetrieveColumns { get; set; }
Property Value
TypeDescription
System.Collections.Generic.ICollection<RetrieveColumn>

Source

The source component from which the lookup data is retrieved. E.g. a DbSource or a MemorySource.

Declaration
    public IDataFlowExecutableSource<TSource> Source { get; set; }
Property Value
TypeDescription
IDataFlowExecutableSource<TSource>

SourceBlock

SourceBlock from the underlying TPL.Dataflow which is used as output buffer for the component.

Declaration
    public override ISourceBlock<TInput> SourceBlock { get; }
Property Value
TypeDescription
System.Threading.Tasks.Dataflow.ISourceBlock<TInput>
Overrides
ETLBox.DataFlow.DataFlowSource<TInput>.SourceBlock

SourceRecords

Holds the data read from the lookup source. This data is used to find data that is missing in the incoming rows.

Declaration
    public ICollection<TSource> SourceRecords { get; }
Property Value
TypeDescription
System.Collections.Generic.ICollection<TSource>

SourceRecordsByKey

Holds a dictionary containing data from the lookup spurce. The dictionary key is generated using the GetSourceRecordKeyFunc

Declaration
    public IDictionary<object, TSource> SourceRecordsByKey { get; }
Property Value
TypeDescription
System.Collections.Generic.IDictionary<System.Object, TSource>

TargetBlock

TargetBlock from the underlying TPL.Dataflow which is used as input buffer for the component.

Declaration
    public override ITargetBlock<TInput> TargetBlock { get; }
Property Value
TypeDescription
System.Threading.Tasks.Dataflow.ITargetBlock<TInput>
Overrides
ETLBox.DataFlow.DataFlowTransformation<TInput, TInput>.TargetBlock

TaskName

A name to identify the task or component. Every component or task comes with a default name that can be overwritten.

Declaration
    public override string TaskName { get; set; }
Property Value
TypeDescription
System.String
Overrides

Methods

CheckParameter()

Declaration
    protected override void CheckParameter()
Overrides

CleanUpOnFaulted(Exception)

Declaration
    protected override void CleanUpOnFaulted(Exception e)
Parameters
TypeNameDescription
System.Exceptione
Overrides

CleanUpOnSuccess()

Declaration
    protected override void CleanUpOnSuccess()
Overrides

InitComponent()

Declaration
    protected override void InitComponent()
Overrides

LinkErrorTo(IDataFlowDestination<ETLBoxError>)

Declaration
    public override IDataFlowSource<ETLBoxError> LinkErrorTo(IDataFlowDestination<ETLBoxError> target)
Parameters
TypeNameDescription
IDataFlowDestination<ETLBoxError>target
Returns
TypeDescription
IDataFlowSource<ETLBoxError>
Overrides
ETLBox.DataFlow.DataFlowSource<TInput>.LinkErrorTo(ETLBox.DataFlow.IDataFlowDestination<ETLBox.DataFlow.ETLBoxError>)

PrepareParameter()

Declaration
    protected override void PrepareParameter()
Overrides

Implements