Skip to content

🤖 Chatbot mode

saltChat is an interactive chat REPL where SALT is the conversation memory: one persistent trie per conversation grows with every exchange (and any attached documents), and each turn it compresses the accumulated history into a query-biased context block under the token budget. The BGE encoder and the chat model stay resident on the GPU for the whole session. The trie lives in process memory and autosaves to disk, so any conversation can be resumed later by its id, recent exchanges included.

Starting a session

Register a model by its HuggingFace name - weights are symlinked from your HF cache into salt/models/, never copied (see salt/models/README.md):

saltChat --add Qwen/Qwen2.5-0.5B-Instruct --alias qwen05

Chat, optionally seeding the trie with a document:

saltChat --model qwen05 --conversation-id demo1 --doc report.txt

The default backend runs the model through HF transformers and works anywhere. --backend vllm serves the same registered weights through an in-process vLLM engine with automatic prefix caching, so the stable prompt head and tail are reused from the GPU KV cache instead of being re-prefilled every turn (install vLLM first, see Installation step 5):

saltChat --model qwen05 --backend vllm --gpu 1

A model too big for one card splits across several with a --gpu list, on either backend, with the encoder riding the last card. Memory shares, window caps and device flags are on the Options page.

Persistent serving

The backends above load the model inside the saltChat process, so the model and its cache disappear on exit. saltServe keeps them alive as a separate long-lived server that chats connect to and resume warm. It has its own page: Serving.

Other models beside this one

A chat talks to one model, and switching with /model unloads the one it has. A roster names other models a session can reach without giving up its own, each one a server of its own that stays loaded. --roster declares them and the /roster and /worker commands below inspect them. It has its own page: Agents.

Conversations without the REPL

These conversations are files, and the prompt is not the only way to reach them. salt-mcp serves the same folder over the Model Context Protocol, so an editor or an agent runtime can add turns to a conversation and read what it remembers, and a session started there can be resumed here. The MCP server page has the setup.

REPL commands

Command Effect
/help list the commands
salt@ list attachable files staged in salt/files/
salt@<file> attach a .pdf/.txt/.md/.rst: whole text, own trie branch
attach@<file> attach in full: uncompressed text rides in every prompt
/model list registered models, /model <name> switches (session kept)
/add <hf_id> [alias] download and register another model
/doc <path> ingest a text or PDF file into the trie
/budget <pct> set the memory budget (0.3 or 30)
/scope which attached files a turn searches: auto lets the rule choose, off searches every file, <file>[,<file>] names them
/summary whether a turn that asks for a summary selects as one: auto reads the wording, off never does, next marks the next turn by hand
/when which time a turn searches: auto reads a day, month, year or span off the question, off searches every row, <time> pins a window until the next /when
/stats session, attachments, compression, and GPU-memory stats
/roster list the models --roster names, /roster probe contacts them, /roster probe --deep NAME reports what one serves
/worker show each worker's connection, calls and mean latency
/worker probe <name> reconnect one worker and report what it serves
/worker start <name> launch a spawn entry's server, start --all does them all
/worker stop <name> stop a server this session started
/offload <task> hand a task to a worker, /offload @NAME <task> picks one
/offload! @NAME put the last delegated task to another worker as well
@NAME <question> let that worker answer this turn instead
/agent <task> answer this turn by planning it out and handing the pieces to workers (see agents)
/new [id], /clear start another conversation, wipe this one
/exit leave (also /quit, /q), the session is saved and resumable by id

TAB completes /commands, @worker names, salt@<file>, and attach@<file> names (see salt/files/README.md).

Letting another model answer

A line that starts with @NAME gives this one turn to a worker from the roster. The prompt is the turn's own, memory and recent messages and question together, so the worker answers exactly what the chat model would have been asked. What comes back is this session's reply: it joins the recent messages, it is remembered, and the next turn goes back to the chat model as though nothing had changed hands. The record of the turn names the model that actually spoke.

you> @qwen05 which of those two options is cheaper to install?
qwen05> The second one, because it reuses the existing inverter ...

That makes it possible to change model in the middle of a conversation without ending it, including with --backend vllm-serve, where the server holds one model and /model cannot switch it.

Scripted turns

Instead of typing, --turns FILE runs a whole conversation from a file, one turn after the next, into the same session so the memory builds across them exactly as it would live. The file is a JSON array or a JSONL file (one JSON value per line). A string item is a user message. An object item takes its message from a common key it finds (question, puzzle, prompt, and a few others), or from --turns-field KEY when you name the key yourself.

saltChat --model QwQ-32B --backend vllm-serve --turns puzzles.json

Every backend works, so the same file can drive a persistent server. Each turn prints its id and the model's reply. Add --turns-out results.jsonl to also append one row per turn, so the run can be reviewed or scored afterward. A row carries the id, the turn number, the question and the answer, the seconds the turn took, the prompt tokens the model saw and, on a served backend, the engine's prefix-cache counts. When a thinking model reasoned first, its reasoning lands under think and the answer holds only what it said, which is also all that memory keeps. An item's other fields, a dataset's category or split for instance, ride along under item.

--turns-mode says how the items relate. The default, conversation, feeds them into one session so the memory builds across them. For a file of unrelated items, such as a set of puzzles, --turns-mode independent runs each item in a fresh session named after the launch id and the item's id, so no item sees another's memory, and each row of --turns-out names the session it ran in. A document item is attached to the session of the item that follows it. Every session stays on disk and can be resumed by its id.

A long run reports itself and survives interruptions. After every turn a line on the error stream gives that turn's time, the elapsed time and an estimate of what is left, so a captured transcript of the run stays clean. --turns-timeout SECONDS gives up on a turn whose server has sent nothing for that long and goes on to the next item, with the row noting the error, which needs --backend vllm-serve since an in-process backend cannot be interrupted mid-generation. --turns-resume keeps an existing --turns-out file and skips every item it already holds an answer for, so a run that was stopped continues where it left off, and an item that failed runs again.

A run scores itself when its items carry a reference answer. An item's gold, answer, solution or expected field is the reference, or name the field with --turns-gold-field KEY. The row then carries gold, a correct verdict, the match rule that decided it and the token f1 against the reference. The rules are conservative and run in order: the whole answer equals the reference, the answer's last line equals it once a label like Answer: is removed, or the reference appears whole inside the answer. The comparison ignores case, punctuation, markup and articles, and a list of references accepts any of them. When the run ends, a line on the error stream gives the count correct overall and per category when the items name one, counting the latest row of every item, so a retried item counts once.

A raw dataset runs without a preprocessing pass. --turns-template composes each message from an item's fields, so --turns-template 'Solve: {puzzle}\nAnswer:' wraps every puzzle the same way, with \n standing for a line break. The fields are the item's keys, a bare string item is used as it is, and an item that lacks a field the template names stops the run before the model loads. The row's question is the message as composed, a field the template used never repeats under item and is never the reference, and the template and --turns-field exclude each other.

An object item can also carry a timestamp, either epoch seconds or an ISO 8601 string like 2026-03-14T09:30. The exchange is then filed in memory at that time instead of the moment of the replay, so a conversation that really happened across days keeps its true gaps even when the file replays in minutes. A timestamp belongs on plain chat turns only.

Launched with --agent, a scripted run plans every plain line out over the roster's helpers, the way it would in the REPL, and each row gains a planned field saying whether that turn went through a round. See agents.

An item can also hand its work to a worker instead of the chat model, which puts a delegation in the middle of a scripted conversation:

[
  "What did we decide about the inverter?",
  {"offload": {"task": "Size the battery bank from that decision",
               "target": "qwen05", "ingest": true}}
]

A line can also attach a document before anything else runs, or hand a turn to the orchestrator to plan out:

[
  {"doc": "salt/files/SALT.pdf"},
  {"agent": "What does that paper claim, and does our sizing agree?"}
]

doc ingests a file exactly as /doc does, and agent runs the turn exactly as /agent does, so a whole session is scriptable rather than only its conversation. Each line does one thing, and a line asking for several is refused when the file loads.

The task goes to that worker with this session's memory as its context, the same way /offload does at the prompt. target picks the worker and can be left out when the roster has only one, and ingest decides whether the answer is remembered, defaulting to whatever the session was launched with. Rows in --turns-out carry kind whenever the line was not an ordinary turn, and a delegated row carries status and worker too, so a run can be told apart from the turns the chat model answered. The Agents page covers what a delegation is and what it leaves behind.

Attachments

A salt@ file becomes its own branch of the session trie, hanging off the conversation's root - so multiple attachments never crowd each other out, and the per-turn budget (default 20%) is taken of the files a question is about plus the conversation (see "Where a turn searches" below), with a floor so a short file is still read. The block is bounded further: --memory-cap auto (the default) sizes it to the space left after the fixed prompt and never lets it pass 4096 tokens, so a huge document cannot fill the window with memory. --memory-cap window keeps only the window fit, and --memory-cap off restores the old unbounded sizing. An attach@ file skips the trie entirely: its full text rides uncompressed in every prompt.

--branch-stats shows how each branch relates to the question. After every query turn, /stats lists the conversation and each attached file of at least three indexed sentences (a shorter one counts with the conversation) with two similarities to the question, the branch as a whole and its best single sentence, plus how many of the question's keywords and names appear in it. Prompts and selection do not change.

Where a turn searches

With several files attached, a question about one of them used to draw its memory block from all of them, and the block grew with every file added. By default (--scope auto) a turn keeps the attached files whose content as a whole is about the question or holds a strong match for it, or that carry the most of the question's names, at least one, plus the conversation, which is always searched. The other files stay out of that turn's selection, and the block is sized from the words inside the scope rather than the whole session, with a floor so a short file is still read. Nothing about the files changes: the next question is routed again from scratch. Two margins set how close to the best file a file must be to stay in, --scope-margin for the whole-file similarity and --scope-peak-margin for the best-sentence one, and /stats prints which files the last turn searched, which it kept out, the words in scope and the budget. It also keeps a census of the session: how many turns were routed, how many had nothing to route, how many files each routed turn kept and how often each file was kept, so a rule that has stopped keeping anything out is visible rather than silent. The files can also be named by hand: /scope notes.pdf,report.pdf searches those two until the next /scope, /scope auto returns the choice to the rule and /scope off searches every file, as --scope off does for the whole session. An unknown name lists what is attached and changes nothing. A session without attachments is not routed and selects exactly as before.

Summary turns

A question that asks for a summary wants breadth. The memory trie's themes are the keywords above a frequency cutoff, and coverage spreads the budget across those themes, so a minor topic that never crossed the cutoff has no branch to be covered by. --summary auto treats a turn that asks for a summary differently: that turn profiles themes at a lower cutoff, set by --summary-themes, and selects with a stronger discount, set by --summary-lam, so more topics get a branch and each branch gives up its sentences sooner. Every other turn selects exactly as before, and the session's own settings never change.

The turn is recognized by its wording, summarize, recap, what did we discuss, catch me up and the like, or marked by hand. /summary next treats the next turn that selects from memory as a summary ask whatever it says, and /summary auto and /summary off set the mode for the session. /stats shows what a summary turn selected under and a census of how many turns were treated that way. The two knobs' defaults are a starting point rather than a measured optimum.

When a turn searches

A question about what was said on a given day is answered from the whole history unless something holds the other days out. Every turn reads the time off the question, a day (on 8 May 2023, May 8, 2023-05-08), a month (in May, May 2023), a year (during 2023) or a span relative to the moment of asking (yesterday, three days ago, last week, the last two weeks, last month, last Monday), and holds the conversation rows filed outside that window out of the turn's selection. A day or a month without a year takes the conversation's own years, latest first. Attached files are never held out, since their rows are filed when they are attached and a date on the line says nothing about them, and a window that holds no conversation rows is not applied. The budget is not resized, so a day that fits inside it is handed over whole. --when off searches every row as before.

/when auto and /when off set the mode for the session, and /when <time> pins a window, /when last week for instance, for every turn until the next /when. /stats shows the window a turn searched, how many rows it held in and out, and a census of how the session's turns came out. The ledger carries the window's label and the rows held out on such turns. A scripted turn's timestamp is the moment of asking for its relative spans.

How PDFs are cleaned

Attached PDFs are read whole and cleaned into proper sentences before they reach the trie: headers, footers and reference lists go, broken lines and hyphenation are repaired, and paragraphs interrupted by a figure or footnote are re-joined. Tables and pseudocode are kept as rows grouped under their captions, and headings, panel labels and equations survive even though they are short. A sentence mentioning a URL keeps its prose with the link stored as <url>.

What conversation text keeps

Messages are stored the way you typed them. Code keeps its generics, tables and pipelines keep their pipes, and short decisive turns like "go with option B" are kept rather than filtered as fragments. Only whitespace is normalized, a link becomes <url>, and lines that are mostly URL or pure junk still drop. Sessions from before this behavior keep their previously stored text as is.

What the model sees

The system prompt carries a reading guide (salt/chat/instructions.md) plus an inventory of attached files, and the compressed memory arrives at the top of the newest user message, grouped by origin: [from attached file 'paper.pdf' - 42 of 358 indexed sentences] versus [from the earlier conversation - turn 12, user, 2h ago].

Conversation excerpts carry their provenance: the turn, the speaker, and how long ago. The model can tell your words from its own, see which of two conflicting statements came later, and answer "what did I decide this morning". --no-turn-labels restores the plain anonymous header.

A worker's answer kept with --offload-ingest is headed [from delegated work - turn 14, qwen05, 2h ago] instead, naming the worker in place of a speaker. The model is told to quote it as that worker's report rather than as something said in this conversation, and the conversation map credits the same turn to worker(qwen05).

/stats prints a conversation map, one line per recent turn with that turn's strongest keywords:

conversation map (all 4 turns):
  t10 user: eviction, indices, mask
  t11 assistant: contract, kvtrace, ledger
  t12 user: budget, decay, retrieval
  t13 assistant: coverage, half-life, themes

--conversation-map puts the map into the prompt as the first section of the memory block, so the model can see a topic came up even on a turn none of its sentences were selected. The map is a signal, never a gate: it changes nothing about which sentences are picked.

The prompt layout

The prompt is KV-cache shaped: everything stable comes first (system prompt, attach@ texts, the verbatim tail, compacting in blocks so it stays byte-identical between compactions), and the only per-turn content, the memory selection and the question, comes last. Attachment order and the tail are saved with the session, so a resumed conversation rebuilds the same prompt bytes and a warm cache still matches. On the vllm backends the engine serves that stable prefix straight from the GPU KV cache, in practice ~95% of prompt tokens on quiet turns, and /stats shows the measured reuse.

Memory knobs

Cross-turn memory has a set of switches. Each is reported in /stats so it can be judged on a real session, and the Options page lists them all with when to reach for each. In concept:

  • Forgetting. A surfaced theme stays discounted for the whole session by default. --coverage-half-life lets that discount fade over turns of silence, so returning topics resurface.
  • Pivots. --shift-damping lifts stale discounts for a detected topic pivot, that turn only, so coming back to an old topic is not fought by its own accumulated suppression.
  • Repetition. --dedup-cos skips near restatements at ingest, so rephrasing does not inflate what counts as a theme. It compares meaning, so a short reversal built from the same words as the answer it corrects ("no, use PostgreSQL" against "yes, use PostgreSQL") can read as a restatement and be skipped. Fused acknowledgements (--short-turns fuse) are exempt, and the gate stays off by default.
  • Recent messages. The last exchanges ride in the prompt verbatim, and selection skips those same sentences while they are visible, so the budget buys older context instead of repeating what is on screen. Their themes start counting as shown once they leave the recent window. --no-tail-exclude restores the old overlapping selection.
  • Key stability. Remembered discounts are keyed to branches of a tree that is rebuilt every turn. --stable-coverage-keys freezes the session's keyword order so those keys keep matching as it grows.
  • Bookkeeping bounds. --coverage-gc collects remembered keys that match nothing anymore, and --coverage-max-keys puts a hard limit on the remembered dictionary.
  • Session bounds. --max-sentences caps how many conversation sentences stay in memory. Past the cap the oldest are masked out of selection rather than deleted, so their text and their numbering survive while a long session stops growing. Attached files are never masked. Reach for one of the bookkeeping bounds above alongside it, which collect the theme keys the masked sentences leave behind.
  • Theme scope. --per-source-themes profiles the conversation and each attached file separately, so one large file cannot crowd the conversation out of memory.
  • Question identifiers. --query-identifiers lets the question's identifier shaped tokens, like dates, versions and numbers, match memory directly. The letters only keyword gate drops them by default, even though memory indexes them, so a question that hinges on a version number can miss the turn that named it.
  • Episodes. --episode-gap 6 splits conversation memory where exchanges sit more than six hours apart and gives each episode its own branch of the memory tree, so selection spreads across time the way it already spreads across attached files. A session that spans days otherwise blurs its epochs into one theme profile. Replayed conversations get real gaps from the timestamp field of a --turns file.
  • Author weight. --assistant-weight 0.5 counts assistant and worker rows as half a row in the theme profile. The model usually writes several times more text than the user, so its restatements can author most of the themes. Down-weighted, the themes track what the user talked about while every sentence stays in memory and stays selectable.
  • Row coverage. --row-coverage remembers what was already shown per sentence instead of per branch of the memory tree. The tree reorganizes as a session grows, and a discount tied to a branch that no longer exists silently disappears, so long sessions repeat material. Tied to the sentence itself, the discount follows it wherever the tree puts it next.
  • Ceiling. --memory-cap auto (the default) fits the memory block to the space the model's window actually has left and stops at 4096 tokens, instead of a percentage that grows without bound.
  • Short turns. Terse decisions like "go with option B" stay in memory by default, and --short-turns fuse stores a bare "yes" together with the question it answers, so the decision is findable by the question's own words.

Background ingestion

Indexing what was said (the keyword and embedding passes) runs on a worker thread: your message is indexed while the model writes its reply, so the next prompt appears immediately however long the paste was. Answers do not change, because the REPL waits for the worker before reading memory. A failed indexing keeps the message text in ingest_failures.jsonl and reports at the next prompt, so nothing is lost silently. --sync-ingest restores the old inline behavior.

Memory bookkeeping is failure safe in the other direction too. The per-turn theme discounts, freshness stamps and topic baseline are applied only once the model has actually answered. A turn that fails partway leaves them exactly as they were, so a retry works from the same memory the failed attempt saw instead of a worse one.

Interrupted saves

A session's memory is saved as separate files, and a crash or kill can land between them. When that happens the session is detected and rolled back to the last complete state the next time it opens, with a one line notice printed and a record appended to load_repairs.jsonl in the session folder. If any sentence text has to be removed because its vector was lost, the text itself is kept in that record, so nothing disappears without a trace.

The kv ledger

Every turn is recorded in a per-conversation KV ledger under salt/chat/sessions/<id>/kvtrace/: an append-only events.jsonl whose usage keys follow the cached-token convention (input = freshly prefilled sentences, input_cached_tokens = context re-selected from the previous turn, output = generated tokens) plus a per-token tokens.npy matrix. /stats shows the running totals. On --backend vllm and vllm-serve every event also records the engine's measured prefix-cache reuse (apc_cached_tokens / apc_prompt_tokens) - the positional ground truth next to the ledger's content-overlap split.

A turn that follows one or more delegations also carries agent_delegations, agent_delegated_tokens and agent_workers. Those sit beside the usage keys and never inside them, so anything already reading the ledger's token accounting sees exactly what it saw before.