📦 Installation
Requires Python 3.10 and a CUDA GPU with an NVIDIA driver 570 or newer, since the pinned torch is built for CUDA 12.8 (CPU works for compression, just slower).
1. Clone the repository
git clone https://github.com/oteomamo/SALT.git
cd SALT
2. Create the environment
With conda:
conda env create -f environment.yml
conda activate salt
Or with venv:
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
3. Install SALT in editable mode
pip install -e .
This also installs the four console commands: salt (one-shot compression,
see Usage), saltChat (interactive chat, see
Chatbot mode), saltServe (persistent serving, see
Serving) and salt-mcp (the MCP server, see
MCP server).
4. Authenticate with Hugging Face - the eval model
(meta-llama/Llama-3.1-8B-Instruct) is gated:
hf auth login
Or skip the CLI and export the token directly: export HF_TOKEN=hf_....
5. (Optional) vLLM backend. eval.py defaults to vLLM,
saltChat --backend vllm uses it for prefix caching, and saltServe
launches a persistent model server with it. Install it into the salt
env through the package's extra, which pins vLLM 0.19.1 to the
transformers release it was tested with:
pip install -e ".[vllm]"
The extra also installs a small server package at a version that keeps
the routes healthy next to newer fastapi releases. Skip this and run eval.py --backend hf for a portable run
that needs no vLLM. saltChat already defaults to its HF backend.
saltServe can also run a vLLM installed in a separate environment
through --vllm-bin.
6. (Optional) MCP server. salt-mcp puts SALT behind the Model
Context Protocol, so an editor or an agent runtime can compress text
and keep conversation memory through it. Install the extra:
pip install -e ".[mcp]"
Then point a client at the salt-mcp command, see
MCP server. It needs no GPU.
bash scripts/setup_env.shdoes steps 2–3 in one shot (addWITH_VLLM=1to include vLLM).