osFoundry / Platform
Auto-chunking pipeline
Smart chunking that respects sections, code blocks and lists. Tunable per folder.
What it is
The Auto-chunking pipeline is osFoundry's document splitter for retrieval and embedding. Before text can be searched semantically or fed to a model as context, it has to be broken into pieces. Naive splitters cut on a fixed character count, which routinely severs a function in half, splits a numbered list across two chunks, or breaks a heading away from the paragraph it introduces. osFoundry's pipeline is structure-aware: it reads the document's own boundaries first and cuts along them, so a section stays with its section, a code block stays intact, and a list stays a list.
Inside osFoundry, chunking runs as the step between your knowledge and the embedding/retrieval layer. When you add documents to a folder, the pipeline parses their structure, produces chunks that preserve those natural boundaries, and hands them to the embedding stage. Chunking and retrieval happen local-first, so your source documents can stay on your device rather than being shipped to a third-party indexing service. Embedding is the one platform-paid step (via osFoundry's proxy); the models you bring for chat and agents are your own keys, with no reseller markup. That means the same corpus feeds chat, agents, apps, and search from one place, with no per-seat fee gating who can build against it.
Chunk sizing is not one-size-fits-all, so the pipeline is tunable per folder. A folder of API reference and source code wants different boundaries than a folder of long-form prose or meeting notes. You set the behavior at the folder level, so each body of knowledge is chunked the way that body of knowledge actually reads, and re-indexing a folder re-chunks only what changed. Better chunks mean tighter retrieval: the passages your models see are coherent and self-contained, which reduces the noise that causes weak or hallucinated answers downstream.
Key capabilities
- Splits on real document structure, keeping sections, code blocks and lists intact instead of cutting on a blind character count
- Tunable per folder, so reference docs, source code and long-form prose each get chunk boundaries that suit them
- Feeds osFoundry's embedding and retrieval layer, producing cleaner passages for RAG and semantic search
- Runs local-first, so source documents can stay on your device rather than going to a third-party indexer
- BYOK across chat and agents that consume the retrieved context, with no per-seat fee and no reseller markup
Use cases
- Engineer building an internal RAG assistant: chunks a repo's docs and code without splitting functions mid-body, so answers cite whole examples
- Analyst indexing long reports: keeps each section and its tables together so retrieval returns coherent, self-contained passages
- Team lead managing mixed knowledge: sets prose folders and code folders to chunk differently, tuning retrieval quality per corpus
Frequently asked questions
How is this different from fixed-size chunking?
Fixed-size splitters cut every N characters regardless of content, which breaks code blocks, lists and headings apart. osFoundry reads the document's structure first and cuts on section, code-block and list boundaries, so each chunk stays coherent and self-contained.
Can I control chunk behavior for different kinds of content?
Yes. Chunking is tunable per folder, so a folder of source code or API reference can be chunked differently from a folder of long-form prose or notes. Each body of knowledge gets boundaries that match how it actually reads.
Where does my data go during chunking?
Chunking and retrieval run local-first, so your source documents can stay on your device. Embedding is the one platform-paid step via osFoundry's proxy; the models you use for chat and agents run on your own keys with no reseller markup.