Redis

The ETLBox Redis connector integrates Redis into your ETL pipelines, allowing high-speed access to key-value data for both reading and writing. With support for custom serialization and dynamic objects, it offers a flexible way to work with one of the most popular in-memory databases.

Redis is commonly used for high-performance caching and lightweight storage scenarios. ETLBox supports efficient batch writing and flexible key/value serialization. The connector uses the StackExchange.Redis client library under the hood.

RedisSource#

The RedisSource component scans a Redis database for keys that match a pattern and reads the associated values into your data flow. Values are deserialized from JSON by default, but you can customize the behavior to suit other formats.

Basic Usage#

var source = new RedisSource<MyRow> {
    ConnectionString = "localhost:6379",
    ScanPattern = "user:*",
    ScanCountSize = 1000
};
var dest = new MemoryDestination<MyRow>();
source.LinkTo(dest);
Network.Execute(source);

This example connects to Redis and retrieves all keys that match the user:* pattern.

Deserialization#

By default, values are assumed to be JSON and are deserialized using JsonConvert from the Newtonsoft.Json library.

To customize deserialization, you can either:

• Provide custom JsonSerializerSettings:

  var source = new RedisSource<MyRow>();
  source.DeserializerSettings = new JsonSerializerSettings {
      NullValueHandling = NullValueHandling.Ignore
  };

• Define your own DeserializationFunc:

  var source = new RedisSource<MyRow>();
  source.DeserializationFunc = s => {
      var parts = s.Split("|");
      return new MyRow {
          Col1 = int.Parse(parts[0]),
          Col2 = parts[1]
      };
  };

This allows you to handle plain text or other serialized formats.

Selecting Redis Database#

Redis supports multiple logical databases (typically 0–15). You can set which database to read from:

var source = new RedisSource<MyRow>();
source.DatabaseNumber = 3;

Dynamic Objects#

RedisSource also supports reading into ExpandoObject:

var source = new RedisSource {
    ConnectionString = "localhost:6379"
};
var dest = new MemoryDestination();
source.LinkTo(dest);
Network.Execute(source);

foreach (dynamic row in dest.Data)
    Console.WriteLine(row.SomeProperty);

Connection Options#

Redis connections can be configured in two ways:

Using Connection String#

Provide a Redis connection string (e.g., "localhost:6379"):

source.ConnectionString = "localhost:6379";

ETLBox will automatically create and manage the ConnectionMultiplexer.

Reusing an Existing Connection#

You can reuse an existing ConnectionMultiplexer instance:

var redis = ConnectionMultiplexer.Connect("localhost:6379");
var source = new RedisSource<MyRow> {
    RedisConnection = redis
};

This can improve performance when running multiple operations.

RedisDestination#

The RedisDestination component writes key/value pairs into a Redis database. Each row is serialized and stored using a Redis key derived from a property in the object.

Basic Usage#

var source = new MemorySource<MyRow>();
source.Data = GetRows();

var dest = new RedisDestination<MyRow> {
    ConnectionString = "localhost:6379"
};
source.LinkTo(dest);
Network.Execute(source);

Defining the Key#

Each row must have a key property. You can define this in one of two ways:

KeyColumn Attribute#

public class MyRow {
    [KeyColumn]
    public string Id { get; set; }
    public string Name { get; set; }
}

Manual KeyColumn Setting#

var dest = new RedisDestination<MyRow> ();
dest.KeyColumn = new KeyColumn { KeyPropertyName = "Id" };

This is useful for dynamic objects or when the attribute can’t be used.

Serialization#

By default, each row is serialized into JSON using JsonConvert. You can customize the behavior:

• Use custom settings:

  var dest = new RedisDestination<MyRow> ();
  dest.SerializerSettings = new JsonSerializerSettings {
      Formatting = Formatting.None
  };

• Use a custom serialization function:

  var dest = new RedisDestination<MyRow> ();
  dest.SerializationFunc = row => $"{row.Col1}|{row.Col2}";

This lets you store compact text, binary formats, or custom encoding.

Writing to a Specific Database#

You can write to a different Redis logical database:

var dest = new RedisDestination<MyRow> ();
dest.DatabaseNumber = 1;

Dynamic Objects#

You can also use ExpandoObject for flexible data structures:

var source = new MemorySource();
dynamic row = new ExpandoObject();
row.Id = "user:001";
row.Name = "John Doe";
source.DataAsList.Add(row);

var dest = new RedisDestination {
    ConnectionString = "localhost:6379",
    KeyColumn = new KeyColumn { KeyPropertyName = "Id" }
};
source.LinkTo(dest);
Network.Execute(source);

This setup supports flexible schemas while still enforcing key/value consistency.