🤖 Chatbot mode
saltChat is an interactive chat REPL where SALT is the conversation memory:
one persistent trie per conversation grows with every exchange (and any
attached documents), and each turn it compresses the accumulated history into
a query-biased context block under the token budget. The BGE encoder and the
chat model stay resident on the GPU for the whole session. The trie lives in
process memory and autosaves to disk, so any conversation can be resumed
later by its id, recent exchanges included.
Starting a session
Register a model by its HuggingFace name - weights are symlinked from your HF
cache into salt/models/, never copied (see
salt/models/README.md):
saltChat --add Qwen/Qwen2.5-0.5B-Instruct --alias qwen05
Chat, optionally seeding the trie with a document:
saltChat --model qwen05 --conversation-id demo1 --doc report.txt
The default backend runs the model through HF transformers and works
anywhere. --backend vllm serves the same registered weights through an
in-process vLLM engine with automatic prefix caching, so the stable prompt
head and tail are reused from the GPU KV cache instead of being re-prefilled
every turn (install vLLM first, see
Installation step 5):
saltChat --model qwen05 --backend vllm --gpu 1
A model too big for one card splits across several with a --gpu list,
on either backend, with the encoder riding the last card. Memory
shares, window caps and device flags are on the Options
page.
Persistent serving
The backends above load the model inside the saltChat process, so the
model and its cache disappear on exit. saltServe keeps them alive as
a separate long-lived server that chats connect to and resume warm. It
has its own page: Serving.
Other models beside this one
A chat talks to one model, and switching with /model unloads the one
it has. A roster names other models a session can reach without giving
up its own, each one a server of its own that stays loaded. --roster
declares them and the /roster and /worker commands below inspect
them. It has its own page: Agents.
Conversations without the REPL
These conversations are files, and the prompt is not the only way to
reach them. salt-mcp serves the same folder over the Model Context
Protocol, so an editor or an agent runtime can add turns to a
conversation and read what it remembers, and a session started there
can be resumed here. The MCP server page has the setup.
REPL commands
| Command | Effect |
|---|---|
/help |
list the commands |
salt@ |
list attachable files staged in salt/files/ |
salt@<file> |
attach a .pdf/.txt/.md/.rst: whole text, own trie branch |
attach@<file> |
attach in full: uncompressed text rides in every prompt |
/model |
list registered models, /model <name> switches (session kept) |
/add <hf_id> [alias] |
download and register another model |
/doc <path> |
ingest a text or PDF file into the trie |
/budget <pct> |
set the memory budget (0.3 or 30) |
/scope |
which attached files a turn searches: auto lets the rule choose, off searches every file, <file>[,<file>] names them |
/summary |
whether a turn that asks for a summary selects as one: auto reads the wording, off never does, next marks the next turn by hand |
/when |
which time a turn searches: auto reads a day, month, year or span off the question, off searches every row, <time> pins a window until the next /when |
/stats |
session, attachments, compression, and GPU-memory stats |
/roster |
list the models --roster names, /roster probe contacts them, /roster probe --deep NAME reports what one serves |
/worker |
show each worker's connection, calls and mean latency |
/worker probe <name> |
reconnect one worker and report what it serves |
/worker start <name> |
launch a spawn entry's server, start --all does them all |
/worker stop <name> |
stop a server this session started |
/offload <task> |
hand a task to a worker, /offload @NAME <task> picks one |
/offload! @NAME |
put the last delegated task to another worker as well |
@NAME <question> |
let that worker answer this turn instead |
/agent <task> |
answer this turn by planning it out and handing the pieces to workers (see agents) |
/new [id], /clear |
start another conversation, wipe this one |
/exit |
leave (also /quit, /q), the session is saved and resumable by id |
TAB completes /commands, @worker names, salt@<file>, and
attach@<file> names (see
salt/files/README.md).
Letting another model answer
A line that starts with @NAME gives this one turn to a worker from the
roster. The prompt is the turn's own, memory and recent
messages and question together, so the worker answers exactly what the
chat model would have been asked. What comes back is this session's
reply: it joins the recent messages, it is remembered, and the next turn
goes back to the chat model as though nothing had changed hands. The
record of the turn names the model that actually spoke.
you> @qwen05 which of those two options is cheaper to install?
qwen05> The second one, because it reuses the existing inverter ...
That makes it possible to change model in the middle of a conversation
without ending it, including with --backend vllm-serve, where the
server holds one model and /model cannot switch it.
Scripted turns
Instead of typing, --turns FILE runs a whole conversation from a file,
one turn after the next, into the same session so the memory builds across
them exactly as it would live. The file is a JSON array or a JSONL file
(one JSON value per line). A string item is a user message. An object item
takes its message from a common key it finds (question, puzzle,
prompt, and a few others), or from --turns-field KEY when you name the
key yourself.
saltChat --model QwQ-32B --backend vllm-serve --turns puzzles.json
Every backend works, so the same file can drive a persistent server. Each
turn prints its id and the model's reply. Add --turns-out results.jsonl
to also append one row per turn, so the run can be reviewed or scored
afterward. A row carries the id, the turn number, the question and the
answer, the seconds the turn took, the prompt tokens the model saw and,
on a served backend, the engine's prefix-cache counts. When a thinking
model reasoned first, its reasoning lands under think and the answer
holds only what it said, which is also all that memory keeps. An item's
other fields, a dataset's category or split for instance, ride along
under item.
--turns-mode says how the items relate. The default, conversation,
feeds them into one session so the memory builds across them. For a file
of unrelated items, such as a set of puzzles, --turns-mode independent
runs each item in a fresh session named after the launch id and the item's
id, so no item sees another's memory, and each row of --turns-out names
the session it ran in. A document item is attached to the session of the
item that follows it. Every session stays on disk and can be resumed by
its id.
A long run reports itself and survives interruptions. After every turn a
line on the error stream gives that turn's time, the elapsed time and an
estimate of what is left, so a captured transcript of the run stays
clean. --turns-timeout SECONDS gives up on a turn whose server has sent
nothing for that long and goes on to the next item, with the row noting
the error, which needs --backend vllm-serve since an in-process backend
cannot be interrupted mid-generation. --turns-resume keeps an existing
--turns-out file and skips every item it already holds an answer for, so
a run that was stopped continues where it left off, and an item that
failed runs again.
A run scores itself when its items carry a reference answer. An item's
gold, answer, solution or expected field is the reference, or
name the field with --turns-gold-field KEY. The row then carries
gold, a correct verdict, the match rule that decided it and the
token f1 against the reference. The rules are conservative and run in
order: the whole answer equals the reference, the answer's last line
equals it once a label like Answer: is removed, or the reference
appears whole inside the answer. The comparison ignores case,
punctuation, markup and articles, and a list of references accepts any
of them. When the run ends, a line on the error stream gives the count
correct overall and per category when the items name one, counting
the latest row of every item, so a retried item counts once.
A raw dataset runs without a preprocessing pass. --turns-template
composes each message from an item's fields, so
--turns-template 'Solve: {puzzle}\nAnswer:' wraps every puzzle the same
way, with \n standing for a line break. The fields are the item's keys,
a bare string item is used as it is, and an item that lacks a field the
template names stops the run before the model loads. The row's question
is the message as composed, a field the template used never repeats under
item and is never the reference, and the template and --turns-field
exclude each other.
An object item can also carry a timestamp, either epoch seconds or an
ISO 8601 string like 2026-03-14T09:30. The exchange is then filed in
memory at that time instead of the moment of the replay, so a
conversation that really happened across days keeps its true gaps even
when the file replays in minutes. A timestamp belongs on plain chat
turns only.
Launched with --agent, a scripted run plans every plain line out over
the roster's helpers, the way it would in the REPL, and each row gains a
planned field saying whether that turn went through a round. See
agents.
An item can also hand its work to a worker instead of the chat model, which puts a delegation in the middle of a scripted conversation:
[
"What did we decide about the inverter?",
{"offload": {"task": "Size the battery bank from that decision",
"target": "qwen05", "ingest": true}}
]
A line can also attach a document before anything else runs, or hand a turn to the orchestrator to plan out:
[
{"doc": "salt/files/SALT.pdf"},
{"agent": "What does that paper claim, and does our sizing agree?"}
]
doc ingests a file exactly as /doc does, and agent runs the turn
exactly as /agent does, so a whole session is scriptable rather than
only its conversation. Each line does one thing, and a line asking for
several is refused when the file loads.
The task goes to that worker with this session's memory as its context, the
same way /offload does at the prompt. target picks the worker and can be
left out when the roster has only one, and ingest decides whether the
answer is remembered, defaulting to whatever the session was launched with.
Rows in --turns-out carry kind whenever the line was not an ordinary
turn, and a delegated row carries status and worker too, so a run can
be told apart from the turns the chat model answered. The
Agents page covers what a delegation is and what it leaves
behind.
Attachments
A salt@ file becomes its own branch of the session trie, hanging off the
conversation's root - so multiple attachments never crowd each other out, and
the per-turn budget (default 20%) is taken of the files a question is about
plus the conversation (see "Where a turn searches" below), with a floor so
a short file is still read. The block is bounded further: --memory-cap
auto (the default) sizes it to the space left after the fixed prompt and
never lets it pass 4096 tokens, so a huge document cannot fill the window
with memory.
--memory-cap window keeps only the window fit, and --memory-cap off
restores the old unbounded sizing. An attach@ file skips the trie
entirely: its full text rides uncompressed in every prompt.
--branch-stats shows how each branch relates to the question. After
every query turn, /stats lists the conversation and each attached file
of at least three indexed sentences (a shorter one counts with the
conversation) with two similarities to the question, the branch as a whole and its
best single sentence, plus how many of the question's keywords and names
appear in it. Prompts and selection do not change.
Where a turn searches
With several files attached, a question about one of them used to draw
its memory block from all of them, and the block grew with every file
added. By default (--scope auto) a turn keeps the attached files whose
content as a whole is about the question or holds a strong match for it,
or that carry the most of the question's names, at least one, plus the
conversation, which is always searched. The other files stay out of that turn's
selection, and the block is sized from the words inside the scope rather
than the whole session, with a floor so a short file is still read.
Nothing about the files changes: the next question is routed again from
scratch. Two margins set how close to the best file a file must be to
stay in, --scope-margin for the whole-file similarity and
--scope-peak-margin for the best-sentence one, and /stats prints
which files the last turn searched, which it kept out, the words in
scope and the budget. It also keeps a census of the session: how many
turns were routed, how many had nothing to route, how many files each
routed turn kept and how often each file was kept, so a rule that has
stopped keeping anything out is visible rather than silent. The files
can also be named by hand: /scope notes.pdf,report.pdf searches those
two until the next /scope, /scope auto returns the choice to the
rule and /scope off searches every file, as --scope off does for the
whole session. An unknown name lists what is attached and changes
nothing. A session without attachments is not routed and selects
exactly as before.
Summary turns
A question that asks for a summary wants breadth. The memory trie's
themes are the keywords above a frequency cutoff, and coverage spreads
the budget across those themes, so a minor topic that never crossed the
cutoff has no branch to be covered by. --summary auto treats a turn
that asks for a summary differently: that turn profiles themes at a
lower cutoff, set by --summary-themes, and selects with a stronger
discount, set by --summary-lam, so more topics get a branch and each
branch gives up its sentences sooner. Every other turn selects exactly
as before, and the session's own settings never change.
The turn is recognized by its wording, summarize, recap, what did
we discuss, catch me up and the like, or marked by hand. /summary
next treats the next turn that selects from memory as a summary ask
whatever it says, and /summary auto and /summary off set the mode
for the session. /stats shows what a summary turn selected under and
a census of how many turns were treated that way. The two knobs'
defaults are a starting point rather than a measured optimum.
When a turn searches
A question about what was said on a given day is answered from the
whole history unless something holds the other days out. Every turn
reads the time off the question, a day (on 8 May 2023, May 8,
2023-05-08), a month (in May, May 2023), a year (during 2023)
or a span relative to the moment of asking (yesterday, three days
ago, last week, the last two weeks, last month, last Monday),
and holds the conversation rows filed outside that window out of the
turn's selection. A day or a month without a year takes the
conversation's own years, latest first. Attached files are never held
out, since their rows are filed when they are attached and a date on
the line says nothing about them, and a window that holds no
conversation rows is not applied. The budget is not resized, so a day
that fits inside it is handed over whole. --when off searches every
row as before.
/when auto and /when off set the mode for the session, and /when
<time> pins a window, /when last week for instance, for every turn
until the next /when. /stats shows the window a turn searched, how
many rows it held in and out, and a census of how the session's turns
came out. The ledger carries the window's label and the rows held out
on such turns. A scripted turn's timestamp is the moment of asking
for its relative spans.
How PDFs are cleaned
Attached PDFs are read whole and cleaned into proper sentences before
they reach the trie: headers, footers and reference lists go, broken
lines and hyphenation are repaired, and paragraphs interrupted by a
figure or footnote are re-joined. Tables and pseudocode are kept as
rows grouped under their captions, and headings, panel labels and
equations survive even though they are short. A sentence mentioning a
URL keeps its prose with the link stored as <url>.
What conversation text keeps
Messages are stored the way you typed them. Code keeps its generics,
tables and pipelines keep their pipes, and short decisive turns like
"go with option B" are kept rather than filtered as fragments. Only
whitespace is normalized, a link becomes <url>, and lines that are
mostly URL or pure junk still drop. Sessions from before this behavior
keep their previously stored text as is.
What the model sees
The system prompt carries a reading guide
(salt/chat/instructions.md)
plus an inventory of attached files, and the compressed memory arrives
at the top of the newest user message, grouped by origin: [from
attached file 'paper.pdf' - 42 of 358 indexed sentences] versus
[from the earlier conversation - turn 12, user, 2h ago].
Conversation excerpts carry their provenance: the turn, the
speaker, and how long ago. The model can tell your words from its own,
see which of two conflicting statements came later, and answer "what
did I decide this morning". --no-turn-labels restores the plain
anonymous header.
A worker's answer kept with --offload-ingest is headed [from
delegated work - turn 14, qwen05, 2h ago] instead, naming the worker in
place of a speaker. The model is told to quote it as that worker's
report rather than as something said in this conversation, and the
conversation map credits the same turn to worker(qwen05).
/stats prints a conversation map, one line per recent turn with
that turn's strongest keywords:
conversation map (all 4 turns):
t10 user: eviction, indices, mask
t11 assistant: contract, kvtrace, ledger
t12 user: budget, decay, retrieval
t13 assistant: coverage, half-life, themes
--conversation-map puts the map into the prompt as the first section
of the memory block, so the model can see a topic came up even on a
turn none of its sentences were selected. The map is a signal, never a
gate: it changes nothing about which sentences are picked.
The prompt layout
The prompt is KV-cache shaped: everything stable comes first
(system prompt, attach@ texts, the verbatim tail, compacting in
blocks so it stays byte-identical between compactions), and the only
per-turn content, the memory selection and the question, comes last.
Attachment order and the tail are saved with the session, so a resumed
conversation rebuilds the same prompt bytes and a warm cache still
matches. On the vllm backends the engine serves that stable prefix
straight from the GPU KV cache, in practice ~95% of prompt tokens on
quiet turns, and /stats shows the measured reuse.
Memory knobs
Cross-turn memory has a set of switches. Each is reported in /stats
so it can be judged on a real session, and the Options
page lists them all with when to reach for each. In concept:
- Forgetting. A surfaced theme stays discounted for the whole
session by default.
--coverage-half-lifelets that discount fade over turns of silence, so returning topics resurface. - Pivots.
--shift-dampinglifts stale discounts for a detected topic pivot, that turn only, so coming back to an old topic is not fought by its own accumulated suppression. - Repetition.
--dedup-cosskips near restatements at ingest, so rephrasing does not inflate what counts as a theme. It compares meaning, so a short reversal built from the same words as the answer it corrects ("no, use PostgreSQL" against "yes, use PostgreSQL") can read as a restatement and be skipped. Fused acknowledgements (--short-turns fuse) are exempt, and the gate stays off by default. - Recent messages. The last exchanges ride in the prompt verbatim,
and selection skips those same sentences while they are visible, so
the budget buys older context instead of repeating what is on
screen. Their themes start counting as shown once they leave the
recent window.
--no-tail-excluderestores the old overlapping selection. - Key stability. Remembered discounts are keyed to branches of a
tree that is rebuilt every turn.
--stable-coverage-keysfreezes the session's keyword order so those keys keep matching as it grows. - Bookkeeping bounds.
--coverage-gccollects remembered keys that match nothing anymore, and--coverage-max-keysputs a hard limit on the remembered dictionary. - Session bounds.
--max-sentencescaps how many conversation sentences stay in memory. Past the cap the oldest are masked out of selection rather than deleted, so their text and their numbering survive while a long session stops growing. Attached files are never masked. Reach for one of the bookkeeping bounds above alongside it, which collect the theme keys the masked sentences leave behind. - Theme scope.
--per-source-themesprofiles the conversation and each attached file separately, so one large file cannot crowd the conversation out of memory. - Question identifiers.
--query-identifierslets the question's identifier shaped tokens, like dates, versions and numbers, match memory directly. The letters only keyword gate drops them by default, even though memory indexes them, so a question that hinges on a version number can miss the turn that named it. - Episodes.
--episode-gap 6splits conversation memory where exchanges sit more than six hours apart and gives each episode its own branch of the memory tree, so selection spreads across time the way it already spreads across attached files. A session that spans days otherwise blurs its epochs into one theme profile. Replayed conversations get real gaps from thetimestampfield of a--turnsfile. - Author weight.
--assistant-weight 0.5counts assistant and worker rows as half a row in the theme profile. The model usually writes several times more text than the user, so its restatements can author most of the themes. Down-weighted, the themes track what the user talked about while every sentence stays in memory and stays selectable. - Row coverage.
--row-coverageremembers what was already shown per sentence instead of per branch of the memory tree. The tree reorganizes as a session grows, and a discount tied to a branch that no longer exists silently disappears, so long sessions repeat material. Tied to the sentence itself, the discount follows it wherever the tree puts it next. - Ceiling.
--memory-cap auto(the default) fits the memory block to the space the model's window actually has left and stops at 4096 tokens, instead of a percentage that grows without bound. - Short turns. Terse decisions like "go with option B" stay in
memory by default, and
--short-turns fusestores a bare "yes" together with the question it answers, so the decision is findable by the question's own words.
Background ingestion
Indexing what was said (the keyword and embedding passes) runs on a
worker thread: your message is indexed while the model writes its
reply, so the next prompt appears immediately however long the paste
was. Answers do not change, because the REPL waits for the worker
before reading memory. A failed indexing keeps the message text in
ingest_failures.jsonl and reports at the next prompt, so nothing is
lost silently. --sync-ingest restores the old inline behavior.
Memory bookkeeping is failure safe in the other direction too. The per-turn theme discounts, freshness stamps and topic baseline are applied only once the model has actually answered. A turn that fails partway leaves them exactly as they were, so a retry works from the same memory the failed attempt saw instead of a worse one.
Interrupted saves
A session's memory is saved as separate files, and a crash or kill can
land between them. When that happens the session is detected and rolled
back to the last complete state the next time it opens, with a one line
notice printed and a record appended to load_repairs.jsonl in the
session folder. If any sentence text has to be removed because its
vector was lost, the text itself is kept in that record, so nothing
disappears without a trace.
The kv ledger
Every turn is recorded in a per-conversation KV ledger under
salt/chat/sessions/<id>/kvtrace/: an append-only events.jsonl whose usage
keys follow the cached-token convention (input = freshly prefilled
sentences, input_cached_tokens = context re-selected from the previous
turn, output = generated tokens) plus a per-token tokens.npy matrix.
/stats shows the running totals. On --backend vllm and vllm-serve every event also
records the engine's measured prefix-cache reuse (apc_cached_tokens /
apc_prompt_tokens) - the positional ground truth next to the ledger's
content-overlap split.
A turn that follows one or more delegations also carries
agent_delegations, agent_delegated_tokens and agent_workers. Those
sit beside the usage keys and never inside them, so anything already
reading the ledger's token accounting sees exactly what it saw before.