Skip to main content

​
Documents

The document is the unit of ingestion in a Data Source. Documents can be inserted from the Dust interface or by API, and are identified by a unique document_id. Insertion is an upsertion: inserting with an existing document_id replaces the previous version of the document. When a document is inserted, it is automatically pre-processed and prepared for semantic search. See Datasources for the full insertion lifecycle.

​
Chunks

Documents are too large to fit in a language model’s context window as-is, so Dust splits them into chunks:
  • Each document is pre-processed (repeated whitespace removed) and split into chunks of at most max_chunk_size tokens.
  • Each chunk is embedded using the embedding model set on the Data Source (text-embedding-3-large from OpenAI by default; Enterprise plan customers can request a different embedding model).
  • Each embedding vector is indexed in a vector search database along with metadata about the document and the original chunk text.

​
How chunks are retrieved

When a semantic search query runs against a Data Source:
  1. The query is embedded using the same embedding model as the Data Source.
  2. A vector search retrieves the most semantically relevant chunks.
  3. Chunks are aggregated per document: the search returns a list of document objects, each including only the chunks that matched the search.
  4. Documents are sorted by decreasing order of the maximum score of their retrieved chunks.
This structure enables Retrieval-Augmented Generation (RAG): only the most relevant chunks of information are placed into the model’s context to perform a task, instead of entire documents.