Single-GPU speech and language API

A FastAPI service that runs Whisper transcription, Pyannote speaker diarization, Russian speech emotion recognition, and Qwen chat inference on one NVIDIA GPU. A single FIFO worker serializes every inference request. When the task type changes, the current model is released and the next is loaded; downloaded weights remain in the Hugging Face/container cache and the operating-system page cache (RAM), making later loads faster.

The defaults are chosen for a 16 GB GPU: faster-whisper large-v3-turbo in FP16, Pyannote 3.1, and Qwen2.5-7B-Instruct in 4-bit. Do not start multiple Uvicorn workers because each worker would own a queue and could use the GPU concurrently.

Run

Accept the conditions for pyannote/speaker-diarization-community-1 on Hugging Face and create a read token, then:

cp .env.example .env
# set HF_TOKEN in .env
docker compose up --build

The Docker image uses NVIDIA CUDA 13.3.0 with cuDNN. Override CUDA_IMAGE in .env to select another CUDA 13.3 image variant. Model downloads are stored on the host in /srv/llmserver/models and temporary uploaded data in /srv/llmserver/data; set MODEL_STORAGE_PATH or DATA_STORAGE_PATH in .env to use other locations. These bind mounts keep model weights outside the image and preserve them across container rebuilds.

Set LOG_LEVEL in .env to DEBUG, INFO, WARNING, ERROR, or CRITICAL. The default is INFO; container logs are available through docker compose logs -f api.

Set QWEN_ENABLED=false in the Compose .env file (or MODEL_SERVER_QWEN_ENABLED=false for a host installation) to disable the chat endpoint and skip downloading Qwen during installation.

An API key is optional. Set API_KEY to enable it and send Authorization: Bearer <key>.

API

Requests wait for their queued job by default. Set async_job=true (multipart field for audio, JSON field for chat) to immediately receive a job ID, then poll GET /v1/jobs/{id}.

curl -F file=@speech.wav -F language=en http://localhost:9000/v1/audio/transcriptions

curl -F file=@meeting.wav -F min_speakers=2 http://localhost:9000/v1/audio/diarizations

curl -F file=@speech.wav http://localhost:9000/v1/audio/emotions

curl http://localhost:9000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Hello"}],"max_tokens":128}'

Swagger documentation is at /docs; readiness is at /healthz. Audio uploads default to a 500 MB limit. Configuration uses MODEL_SERVER_-prefixed environment variables; see app/config.py for all settings.

For local development (with a CUDA-enabled PyTorch environment already available):

python -m venv .venv
. .venv/bin/activate
pip install -e '.[test]'
uvicorn app.main:app --host 0.0.0.0 --port 9000 --workers 1
pytest

Production without Docker

On an apt-based Linux host with a working NVIDIA driver, install the service from the repository checkout:

# Pass the token on the first run so the installer can download gated models.
sudo env HF_TOKEN=hf_... ./scripts/install-host.sh
sudoedit /etc/llmserver/llmserver.env
sudo systemctl restart llmserver
curl http://127.0.0.1:9000/healthz

The installer creates a dedicated llmserver account, a virtual environment under /opt/llmserver, and a single-worker systemd service. Configuration is kept in /etc/llmserver/llmserver.env, uploads in /srv/llmserver/data, and the model cache in /srv/llmserver/models. During installation it downloads all configured models, including models referenced by the Pyannote pipeline. Set HF_TOKEN and any model overrides in the configuration, then rerun the installer if a gated model could not be downloaded. It is safe to rerun the installer to update the application; persistent directories and cached files are retained.

Models can also be downloaded manually without starting the API:

sudo -u llmserver /opt/llmserver/venv/bin/llmserver-download-models \
  --env-file /etc/llmserver/llmserver.env \
  --models-dir /srv/llmserver/models

Useful service commands:

systemctl status llmserver
journalctl -u llmserver -f
sudo systemctl restart llmserver

To disable the service while preserving the application, configuration, cached models, and uploaded data:

sudo ./scripts/uninstall-host.sh

The Python environment installs CUDA 12 cuBLAS alongside PyTorch's CUDA runtime. This is intentional: the prebuilt CTranslate2 wheel used by Faster-Whisper currently requires libcublas.so.12. The server preloads that library from the Python package, so no manual LD_LIBRARY_PATH is required.

Notes

• Persist /models so model weights are not downloaded again. The Compose file maps it to ${MODEL_STORAGE_PATH:-/srv/llmserver/models} on the host.
• The queue is in memory. Jobs do not survive process restarts and this service must be deployed as one process/replica.
• Pyannote returns speaker time ranges. Combine its result with Whisper segments in a client if speaker-attributed transcripts are needed.
• Qwen 7B at 4-bit leaves headroom for KV cache on 16 GB. Reduce max_tokens or use a smaller Qwen model for unusually long prompts.