Class Distinct<TInput>
The 'Distinct' class in ETLBox is designed to efficiently filter out duplicate records in a data flow. It operates by generating unique hash values for each record based on specified properties, allowing for precise identification and exclusion of duplicates. This class supports various functionalities like setting DistinctColumns for targeted filtering, linking to other transformations or destinations to handle duplicates, and the ability to define custom functions for unique key generation. It is ideal for scenarios where data uniqueness and quality are of paramount importance.
Implements
Inherited Members
Namespace: ETLBox.DataFlow
Assembly: ETLBox.dll
Syntax
{{< /rawhtml >}} ```C# public class DistinctType Parameters
| Name | Description |
|---|---|
| TInput |
Constructors
Distinct()
Declaration
{{< /rawhtml >}} ```C# public Distinct() ``` {{< rawhtml >}}Properties
DistinctColumns
Specifies a collection of property names to identify the uniqueness of an object. This collection dictates which properties are considered for uniqueness evaluation.
Declaration
{{< /rawhtml >}} ```C# public ICollectionProperty Value
| Type | Description |
|---|---|
| ICollection<DistinctColumn> |
DistinctCount
Numbers of rows that were distinct.
Declaration
{{< /rawhtml >}} ```C# public int DistinctCount { get; } ``` {{< rawhtml >}}Property Value
| Type | Description |
|---|---|
| int |
DuplicateCount
Numbers of rows that were detected as duplicates.
Declaration
{{< /rawhtml >}} ```C# public int DuplicateCount { get; } ``` {{< rawhtml >}}Property Value
| Type | Description |
|---|---|
| int |
DuplicatesSourceBlock
Declaration
{{< /rawhtml >}} ```C# public ISourceBlockProperty Value
| Type | Description |
|---|---|
| ISourceBlock<TInput> |
GetUniqueKeyFunc
Allows the specification of a custom function to generate a unique identifier for each row. This function can be used as an alternative to defining distinct properties. Note: When this property is set, it overrides any DistinctColumns settings. The use of this function is optional.
Declaration
{{< /rawhtml >}} ```C# public FuncProperty Value
| Type | Description |
|---|---|
| Func<TInput, object> |
SourceBlock
SourceBlock from the underlying TPL.Dataflow which is used as output buffer for the component.
Declaration
{{< /rawhtml >}} ```C# public override ISourceBlockProperty Value
| Type | Description |
|---|---|
| ISourceBlock<TInput> |
Overrides
TargetBlock
TargetBlock from the underlying TPL.Dataflow which is used as input buffer for the component.
Declaration
{{< /rawhtml >}} ```C# public override ITargetBlockProperty Value
| Type | Description |
|---|---|
| ITargetBlock<TInput> |
Overrides
Methods
AddComplementVoidLinkToOtherTargets(Func<Type, IDataFlowDestination>, Delegate)
Declaration
{{< /rawhtml >}} ```C# protected override void AddComplementVoidLinkToOtherTargets(FuncParameters
| Type | Name | Description |
|---|---|---|
| Func<Type, IDataFlowDestination> | createAutomaticVoidDestination | |
| Delegate | pred |
Overrides
CheckParameter()
Declaration
{{< /rawhtml >}} ```C# protected override void CheckParameter() ``` {{< rawhtml >}}Overrides
CleanUpOnFaulted(Exception)
Declaration
{{< /rawhtml >}} ```C# protected override void CleanUpOnFaulted(Exception e) ``` {{< rawhtml >}}Parameters
| Type | Name | Description |
|---|---|---|
| Exception | e |
Overrides
CleanUpOnSuccess()
Declaration
{{< /rawhtml >}} ```C# protected override void CleanUpOnSuccess() ``` {{< rawhtml >}}Overrides
CompleteBuffer()
Declaration
{{< /rawhtml >}} ```C# protected override void CompleteBuffer() ``` {{< rawhtml >}}Overrides
FaultBuffer(Exception)
Declaration
{{< /rawhtml >}} ```C# protected override void FaultBuffer(Exception e) ``` {{< rawhtml >}}Parameters
| Type | Name | Description |
|---|---|---|
| Exception | e |
Overrides
InitCheckedParameter()
Declaration
{{< /rawhtml >}} ```C# protected override void InitCheckedParameter() ``` {{< rawhtml >}}Overrides
InitComponent()
Declaration
{{< /rawhtml >}} ```C# protected override void InitComponent() ``` {{< rawhtml >}}Overrides
LinkDuplicatesTo(IDataFlowDestination<TInput>)
Establishes a link from the current block to another transformation or destination. This linked component will exclusively receive the duplicates detected by the current block.
Declaration
{{< /rawhtml >}} ```C# public IDataFlowSourceParameters
| Type | Name | Description |
|---|---|---|
| IDataFlowDestination<TInput> | target | The transformation or destination to which this block is linked. |
Returns
| Type | Description |
|---|---|
| IDataFlowSource<TInput> | The component to which the duplicates are linked. |
LinkDuplicatesTo(IDataFlowDestination<TInput>, Predicate<TInput>)
Establishes a link from the current block to another transformation or destination. This linked component will exclusively receive the duplicates detected by the current block.
Declaration
{{< /rawhtml >}} ```C# public virtual IDataFlowSourceParameters
| Type | Name | Description |
|---|---|---|
| IDataFlowDestination<TInput> | target | The transformation or destination to which this block is linked. |
| Predicate<TInput> | rowsToKeep | A predicate to determine which rows to send to the connected target. Rows that satisfy this predicate (evaluate to true) are forwarded. |
Returns
| Type | Description |
|---|---|
| IDataFlowSource<TInput> | The component to which the duplicates are linked. |
LinkDuplicatesTo(IDataFlowDestination<TInput>, Predicate<TInput>, Predicate<TInput>)
Establishes a link from the current block to another transformation or destination. This linked component will exclusively receive the duplicates detected by the current block.
Declaration
{{< /rawhtml >}} ```C# public virtual IDataFlowSourceParameters
| Type | Name | Description |
|---|---|---|
| IDataFlowDestination<TInput> | target | The transformation or destination to which this block is linked. |
| Predicate<TInput> | rowsToKeep | A predicate to determine which rows to send to the connected target. Rows that satisfy this predicate (evaluate to true) are forwarded. |
| Predicate<TInput> | rowsIntoVoid | A predicate to filter out rows. Rows that satisfy this predicate (evaluate to true) are discarded. |
Returns
| Type | Description |
|---|---|
| IDataFlowSource<TInput> | The component to which the duplicates are linked. |