We assume you already separate offline indexing from online query. The failure modes that waste weekends are almost always duplicated: exploding RAM during embedding, path confusion after reboot, and silent regression when you swap chunk sizes without a fixed eval set. Pair this guide with our OpenClaw and automation articles for gateway and pull automation, skim the homepage for platform context, and use the purchase page when you want a dedicated remote Mac instead of your laptop.
Scenarios
Private code + tickets. You need deterministic section boundaries (functions, headings) more than a single token window. Prefer structure-aware splitting with modest overlap so diffs do not shred retrieval.
PDFs and mixed OCR. Fixed character windows win; overlap is your insurance against table rows split across chunks. Expect higher variance in recall—plan an eval harness early.
High-QPS chat over a static corpus. Shrink chunk text, tighten metadata filters, and push batching cost to the indexer so query-time latency stays flat. Apple Silicon benefits from steady, medium batches rather than micro-batches that underfill the accelerator.
Multilingual or legal-heavy corpora. Align chunking with the same tokenizer your embedder uses; mixed CJK and Latin text can shift effective token counts by 30% or more. If retention policies require deletion of a single customer, you need chunk IDs and persistence layout that allow targeted tombstoning without rebuilding the whole HNSW graph.
Data preparation
Normalize whitespace, strip repeated headers/footers, and store source path + byte offset or page in metadata before embedding. For overlap, a practical 2026 starting point on Mac is 10–20% of chunk length for prose, and fixed 64–128 tokens for technical docs when your chunk size sits between 512 and 1,024 tokens. Increase overlap when answers routinely sit on chunk borders; decrease it when redundancy pollutes rerankers.
Choose between sentence-aware windows (better fluency, slower preprocessing) and fixed character windows (predictable throughput). Hybrid pipelines often sentence-chunk Markdown and fixed-window PDFs. Whatever you pick, log the preprocessor version next to the vector snapshot so you can diff regressions without guessing.
Wire those knobs through environment variables so CI and remote jobs stay reproducible:
export RAG_CHUNK_SIZE=768
export RAG_CHUNK_OVERLAP=96
export RAG_EMBED_BATCH=24
export RAG_MAX_CHARS_PER_CHUNK=3200A minimal CLI-style invocation (swap for your stack) keeps audits obvious:
python -m app.rag_index \
--chunk-size "${RAG_CHUNK_SIZE}" \
--overlap "${RAG_CHUNK_OVERLAP}" \
--embed-batch "${RAG_EMBED_BATCH}" \
--persist-dir "${RAG_VECTOR_DIR}"Vector store comparison
The table below is a rules-of-thumb matrix for local-first workflows on a 16–64 GB unified-memory Mac. Treat memory peaks as order-of-magnitude: they scale with batch size, embedding dimensionality, and whether you keep raw documents resident.
| Profile | Chunk overlap | Embed batch | Indexer RAM peak (indicative) | Vector quota / caps | Persistence path |
|---|---|---|---|---|---|
| Embedded SQLite / local file | Low (5–12%) → fewer rows | 8–16 | 2–6 GB + model | Single DB file; cap by max rows / PRAGMA page size | ~/Library/Application Support/yourapp/rag/sqlite_vec.db |
| Chroma persistent client | Medium; dedupe chunk IDs | 16–32 | 4–10 GB + model | Per-collection segment files; watch inode + disk % | ~/Library/Application Support/yourapp/chroma |
| LanceDB (columnar) | Flexible for large corpora | 24–48 | 6–14 GB + model | Shard by table; compaction after bulk upsert | ~/Library/Application Support/yourapp/lancedb |
| Qdrant (localhost) | Tune with payload filters | 32–64 | 8–18 GB incl. daemon | Set max_collection_size / disk watermark in config |
Docker volume or /usr/local/var/qdrant |
Quota discipline: cap collection size with a max document count and max stored bytes; snapshot the persistence directory before destructive reindex. On macOS, keep data under ~/Library/Application Support or a dedicated APFS volume so Time Machine and remote rsync jobs have a single root.
export RAG_VECTOR_DIR="$HOME/Library/Application Support/yourapp/vectors"
export CHROMA_TELEMETRY_DISABLED=1
export OMP_NUM_THREADS=8Evaluation & regression
Freeze a golden question set (30–100 items) with expected source citations. When you change chunking or embeddings, rerun: hit@k, MRR if you have graded relevance, and latency p95 on the same machine class. Log tokenizer hash, model revision, and chunk parameters in the eval artifact.
Remote long-running job acceptance checklist (indexing on a rented Mac or CI agent):
- Pre-flight: disk quota ≥ 2× corpus + vector footprint;
df -handulimit -nrecorded in the job log. - Determinism: pinned dependency lockfile; embedding model checksum verified;
RAG_SEEDfixed if your stack uses sampling. - Resume: idempotent chunk IDs; crash recovery writes a checkpoint file under
RAG_VECTOR_DIR. - Telemetry: wall time, rows inserted/sec, peak RSS from
/usr/bin/time -l(macOS) or sampling profiler. - Exit criteria: eval metrics within agreed delta vs. baseline; spot-check five adversarial queries manually; tarball of
RAG_VECTOR_DIRwith manifest SHA-256.
Automate the last step with the same patterns we outline for dependency sync in the OpenClaw automation topic—your indexer should be as boring and repeatable as a nightly mirror job.
FAQ
Does larger batch always speed up embedding? Not beyond the point where unified memory pressure triggers swap. Sweep batch sizes {8,16,24,32,48} and pick the knee where throughput gains flatten.
One vector DB for everything? Split dev/prod collections at minimum. For multi-tenant SaaS, separate persistence roots per tenant to simplify quota enforcement and backup.
How often to reindex? On embedding model upgrades always; on chunk policy changes only after eval proves net benefit. Partial reindex beats full rebuild when metadata maps chunk IDs to sources cleanly.
Should I force Metal or Core ML for embeddings? If your runtime supports it, yes—after you verify bitwise or near-bitwise parity on a hundred probe vectors. Silent numeric drift between CPU and GPU paths has burned teams that only measured speedups.
What about ANN index parameters? Treat ef_construct, M, or equivalent as part of the same matrix: higher values raise build-time RAM and disk, lower values hurt recall. Snapshot those knobs beside chunk and batch settings in your acceptance manifest.
Summary: treat chunk overlap, embedding batch, memory peaks, and persistence paths as linked controls. Document them in env vars, enforce remote acceptance gates, and regression-test before you promote a new indexer to production traffic.