Skip to content

📦 Installation

Requires Python 3.10 and a CUDA GPU with an NVIDIA driver 570 or newer, since the pinned torch is built for CUDA 12.8 (CPU works for compression, just slower).

1. Clone the repository

git clone https://github.com/oteomamo/SALT.git
cd SALT

2. Create the environment

With conda:

conda env create -f environment.yml
conda activate salt

Or with venv:

python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

3. Install SALT in editable mode

pip install -e .

This also installs the four console commands: salt (one-shot compression, see Usage), saltChat (interactive chat, see Chatbot mode), saltServe (persistent serving, see Serving) and salt-mcp (the MCP server, see MCP server).

4. Authenticate with Hugging Face - the eval model (meta-llama/Llama-3.1-8B-Instruct) is gated:

hf auth login

Or skip the CLI and export the token directly: export HF_TOKEN=hf_....

5. (Optional) vLLM backend. eval.py defaults to vLLM, saltChat --backend vllm uses it for prefix caching, and saltServe launches a persistent model server with it. Install it into the salt env through the package's extra, which pins vLLM 0.19.1 to the transformers release it was tested with:

pip install -e ".[vllm]"

The extra also installs a small server package at a version that keeps the routes healthy next to newer fastapi releases. Skip this and run eval.py --backend hf for a portable run that needs no vLLM. saltChat already defaults to its HF backend. saltServe can also run a vLLM installed in a separate environment through --vllm-bin.

6. (Optional) MCP server. salt-mcp puts SALT behind the Model Context Protocol, so an editor or an agent runtime can compress text and keep conversation memory through it. Install the extra:

pip install -e ".[mcp]"

Then point a client at the salt-mcp command, see MCP server. It needs no GPU.

bash scripts/setup_env.sh does steps 2–3 in one shot (add WITH_VLLM=1 to include vLLM).