Text chunker → JSONL
Split plain text into fixed-size windows with overlap and download as JSONL for RAG ingestion.
Characters: 262
RAG Chunking Parameters and JSONL Export Guide
RAG systems do not push entire documents into model context. They split content into retrievable chunks and store them in a vector database. Chunks that are too large hurt precision; chunks that are too small lose context.
FormaX RAG Chunker uses fixed character windows with overlap. It is useful for knowledge bases, support docs, product manuals, and Markdown material during prototype and parameter exploration.
Common Use Cases
- Knowledge base building: split product docs and technical manuals into vector-indexable chunks.
- Support chatbots: chunk FAQ and help center articles to improve Q&A retrieval accuracy.
- Code documentation RAG: split API docs and READMEs for developer assistant retrieval and citation.
- Legal/compliance docs: chunk long contracts and policy files by paragraph for clause-level search.
- Multilingual corpora: apply consistent chunking parameters to mixed-language documents and export standard JSONL.
How to Tune Parameters
- Start around 400 characters per chunk with 50 to 80 characters of overlap.
- Increase chunk size for technical specs and API docs; decrease it for chats and FAQ content.
- Keep chunk_index, start, and end after JSONL export so answers can cite the source text.
- Before production, evaluate retrieval using real user questions, not only the number of chunks.
JSONL Output Shape
{"chunk_index":0,"text":"...","metadata":{"start":0,"end":400,"source":"formax-rag-chunker"}}Boundaries
- Character windows are not token windows; re-check lengths against your embedding model.
- Remove headers, footers, repeated copyright text, and navigation noise before chunking.
- More overlap is not always better; it increases vectors, storage cost, and duplicate retrieval.
FAQ
What chunk size should I use?
There is no universal number. Start at 400 characters and adjust after checking whether real queries retrieve complete context.
Why use overlap?
Overlap reduces missed context when sentences are split at boundaries, but too much overlap increases duplication and cost.
What is the difference between characters and tokens?
Chinese is roughly 1-2 tokens per character; English is about 4 characters per token. Re-estimate chunk size in tokens before choosing an embedding model.
What is the exported JSONL format?
One JSON object per line with chunk_index, text, and metadata (including start and end positions), ready for vector database import.
Do Markdown headings affect chunking?
Chunking uses character windows and does not automatically respect Markdown heading boundaries. Pre-split by headings before input for better results.
Can it process PDF or Word documents?
The tool accepts plain text. Extract text from PDF or Word first (using FormaX PDF-to-Word or similar tools), then paste into the chunker.
What overlap setting works best?
Aim for 10%-20% of chunk size. For example, 400-character chunks with 50-80 characters of overlap, then fine-tune based on retrieval quality.
How do I evaluate chunking quality?
Test with 10-20 real user questions and check whether retrieved chunks contain complete answer context, not just the chunk count.