ETLBox
Overview
ETLBox.AI adds document reading, text chunking, chat, embeddings, and vector stores to the same data flow as the other connectors. You bring the model client and the vector store. The components only see IChatClient, IEmbeddingGenerator, and VectorStoreCollection.
Chat, embeddings, chunking, and the text and markdown sources are the package ETLBox.AI. HTML reading is ETLBox.AI.Html. PDF reading is ETLBox.AI.Pdf.
| Component | Package | What it does |
|---|---|---|
| Text Document | ETLBox.AI | Reads a whole file, web response, or blob as one row. A folder is one row per file. |
| Markdown | ETLBox.AI | Reads one markdown resource as one parsed document and keeps the headings. |
| HTML | ETLBox.AI.Html | Converts HTML to markdown and emits one parsed document. Script and style elements are left out. |
ETLBox.AI.Pdf | Reads one PDF as one parsed document. Each page stays a section. | |
| Chunk | ETLBox.AI | Splits each row into smaller text pieces and keeps the source row on every piece. |
| Chat | ETLBox.AI | Sends each row to a chat client and writes the returned object downstream. |
| Embedding | ETLBox.AI | Creates one vector per row and stores it on the same object. |
| Chat Batch | ETLBox.AI | Sends a batch of rows to a chat client as one request. |
| Embedding Batch | ETLBox.AI | Creates embeddings for a batch of rows in one generator call. |
| Vector Store | ETLBox.AI | Reads a similarity search or a filtered collection, and upserts records. The destination does not create embeddings. |
Copyable setups for each component are in the recipes, starting with Text Document Source.
Next
Simple Flow