voice-pipeline

voice-pipeline collects and analyzes call recordings, generates local reports, imports transcripts into Bitrix24, and categorizes analyzed calls with a local LLM.

What it does

• collector fetches call metadata and recordings from Bitrix24 / Asterisk sources and writes them into a local spool directory.
• analyzer picks up spooled calls, transcribes them, performs diarization and script compliance checks, and stores artifacts under a dedicated analyzer artifacts root.
• reporter creates HTML reports from analyzed calls, stores them under reports_root, and builds management detail/summary views from analyzer artifacts.
• importer reads analyzer artifacts and writes transcript notes into existing Bitrix24 call activities via crm.timeline.note.save.
• task-import runs daily at 07:00 Asia/Yekaterinburg and creates Bitrix24 follow-up tasks for calls from the previous calendar day. It uses the dashboard filters except employee, dates, and phone; array-valued settings allow multiple choices. The example config keeps it disabled until explicitly enabled.
• voice-category runs every 10 seconds from 22:00 through 08:00, processes one analyzed call, and writes category/YYYY-MM-DD/UNIQUEID/voice_category.json using Ollama.

The processes communicate through the spool directory and the shared meta.json contract stored inside each call folder. After analysis, downstream modules use analyzer artifacts as the primary business contract; DB rows and meta.json remain lifecycle and legacy-fallback storage.

Requirements

• Python 3.11+
• Linux for the systemd deployment
• A configured Bitrix24 webhook
• Optional dependencies depending on your use case:
  • paramiko for Asterisk SFTP collection
  • pyannote.audio for mixed-audio diarization
  • openai, groq, transformers, torch, or faster-whisper for the analyzer, depending on backend choice

Installation

Create a virtual environment and install the project in editable mode:

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Install optional extras when needed:

pip install -e ".[dev,diarize]"   # mixed-audio diarization via pyannote
pip install -e ".[dev,local-asr]" # local Whisper-compatible transcription
pip install -e ".[dev,emotion]"   # local emotion classification

Local faster-whisper model setup

If you configure a local transcription provider with interface = "faster_whisper", the model_path must point to a downloaded local CTranslate2 model directory.

Example:

[[analyzer.providers]]
name = "local-whisper"
provider = "local-whisper"
interface = "faster_whisper"
capabilities = ["transcription"]
model_path = "/opt/models/faster-whisper-large-v3"
device_preference = "auto"

Download the model into that directory:

mkdir -p /opt/models/faster-whisper-large-v3
venv/bin/hf download Systran/faster-whisper-large-v3 \
  --local-dir /opt/models/faster-whisper-large-v3

If hf is not available, use the Python API instead:

venv/bin/python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="Systran/faster-whisper-large-v3",
    local_dir="/opt/models/faster-whisper-large-v3",
    local_dir_use_symlinks=False,
)
PY

Verify that the model loads correctly:

venv/bin/python - <<'PY'
from faster_whisper import WhisperModel
WhisperModel("/opt/models/faster-whisper-large-v3", device="cpu", compute_type="int8")
print("OK")
PY

Configuration

Copy the example config and edit it:

cp config/config.example.toml /etc/voice-pipeline/config.toml

The config path is read from VOICE_PIPELINE_CONFIG, or defaults to:

/etc/voice-pipeline/config.toml

Analyzer provider API keys are loaded from secrets.env next to config.toml.

Important sections:

• [storage] controls spool, analyzer artifacts, reports, runtime state, and database paths.
• [logging] controls the global log level for all services. Supported values: DEBUG, INFO, WARN, ERROR, FATAL.
• [[sources.asterisk]] defines one or more Asterisk SFTP/HTTP sources.
• [sources.bitrix24] defines the Bitrix24 webhook and collector options.
• [sources.megafon_ats] defines read-only Megafon Virtual PBX history access.
• [analyzer] configures provider failover, transcription, diarization, emotions, and script files. Each [[analyzer.providers]] entry separates the provider identity from the API/runtime interface: provider = "openrouter" with interface = "openai" is valid, as are local runtimes like provider = "local-faster-whisper" with interface = "faster_whisper".
• [reporter] controls report size and summary window.
• [importer] controls Bitrix24 transcript-note import behavior.
• [task_import] and [task_import.filters] control the daily Bitrix24 task import.

Run a validation check after editing the config:

voice-pipeline validate-config

Megafon ATS history check

The diagnostic script checks authentication and retrieves the raw call list. Put MEGAFON_ATS_PASSWORD=... in secrets.env next to config.toml, then run:

VOICE_PIPELINE_CONFIG=/etc/voice-pipeline/config.toml \
  venv/bin/python tools/fetch_megafon_calls.py --day 2026-08-17

To keep the raw response for mapping the call and recording fields:

VOICE_PIPELINE_CONFIG=/etc/voice-pipeline/config.toml \
  venv/bin/python tools/fetch_megafon_calls.py --day 2026-08-17 \
  --output /tmp/megafon-calls-2026-08-17.json

Inspect one call and download its recording through the authenticated session:

VOICE_PIPELINE_CONFIG=/etc/voice-pipeline/config.toml \
  venv/bin/python tools/fetch_megafon_calls.py --day 2026-08-17 \
  --uid GC2AS65HTK000046 \
  --download-recording /tmp/GC2AS65HTK000046.mp3

For Asia/Yekaterinburg, that day is sent as 2026-08-16T19:00:00.000Z through 2026-08-17T18:59:59.999Z. Logout is attempted even if processing fails. The client has no transcript-upload method.

The collector processes Bitrix24/Asterisk first and Megafon ATS second. It selects recorded in/out calls, downloads their authenticated recordUrl, and stores them in the source-specific tree:

/var/lib/voice-pipeline/spool/megafon_ats/YYYY-MM-DD/UNIQUEID/
/var/lib/voice-pipeline/analyzer-artifacts/megafon_ats/YYYY-MM-DD/UNIQUEID/
/var/lib/voice-pipeline/category/megafon_ats/YYYY-MM-DD/UNIQUEID/

Existing Bitrix24/Asterisk paths remain unchanged for backward compatibility. Analyzer and category queues prioritize existing sources before megafon_ats; the Bitrix24 transcript importer excludes megafon_ats records entirely.

PostgreSQL

PostgreSQL is the default database backend. Create a database and a dedicated login before starting the services (run these commands as a PostgreSQL administrator):

CREATE ROLE voice_pipeline LOGIN PASSWORD 'replace-with-a-strong-password';
CREATE DATABASE voice_pipeline OWNER voice_pipeline;

Then set the connection URL in /etc/voice-pipeline/config.toml:

[storage]
db_url = "postgresql+psycopg://voice_pipeline:password@localhost/voice_pipeline"

Special characters in the login or password must be percent-encoded. On first startup the application creates its tables; its small additive compatibility migration also works for existing PostgreSQL schemas. To move data from an old SQLite installation, stop all pipeline services and use pgloader, preserving the old file as a backup:

pgloader /var/lib/voice-pipeline/db.sqlite3 \
  postgresql://voice_pipeline:password@localhost/voice_pipeline

Run voice-pipeline validate-config, start one service once to initialize the schema, and verify record counts before enabling all timers and services. During initialization the application also repairs missing PostgreSQL sequences for integer primary keys imported from SQLite.

CLI

The project exposes a single command:

voice-pipeline validate-config
voice-pipeline collector
voice-pipeline analyzer
voice-pipeline reporter
voice-pipeline importer
voice-pipeline task-import --dry-run
voice-pipeline voice-category
voice-pipeline voice-category-backfill
voice-pipeline metrics-exporter
voice-pipeline crm-backfill --dry-run
voice-pipeline crm-backfill
voice-pipeline reload --list --date 2026-06-23
voice-pipeline reload --reanalyze --uniqueid 1782211489.118028
voice-pipeline reload --recollect --status failed
voice-pipeline reload --repair --date 2026-06-23
voice-pipeline reload --repair --from-date 2026-06-23

Prometheus monitoring

The dedicated exporter reads the pipeline database and exposes queue sizes, stage statuses, oldest pending ages, and last-event timestamps without changing call records:

voice-pipeline metrics-exporter
curl http://127.0.0.1:9101/metrics

Its bind address is configured in [metrics_exporter]. For a systemd deployment with an existing Prometheus server, install the service, file-discovery target, and alert rules with:

sudo monitoring/install.sh

The installer prints the two prometheus.yml entries that must be present. It does not overwrite the existing Prometheus configuration. Service and timer alerts also require node_exporter with its systemd collector enabled.

Bitrix24 task import: preview and manual run

The safest production check is a dry run. It applies the configured filters, reads employees through user.get and departments through department.get, and prints the exact prospective task fields, but never calls tasks.task.add. Preview mode is available even when task_import.enabled = false:

sudo -u voice-pipeline \
  env VOICE_PIPELINE_CONFIG=/etc/voice-pipeline/config.toml \
  /opt/voice-pipeline/venv/bin/voice-pipeline task-import --dry-run

By default the previous local calendar day is checked. To inspect a specific day:

sudo -u voice-pipeline \
  env VOICE_PIPELINE_CONFIG=/etc/voice-pipeline/config.toml \
  /opt/voice-pipeline/venv/bin/voice-pipeline task-import \
  --dry-run --date 2026-08-30

Save the same report as UTF-8 JSON for review or comparison:

sudo -u voice-pipeline \
  env VOICE_PIPELINE_CONFIG=/etc/voice-pipeline/config.toml \
  /opt/voice-pipeline/venv/bin/voice-pipeline task-import \
  --dry-run --date 2026-08-30 \
  --output /tmp/task-import-preview-2026-08-30.json

The summary contains selected, would_create, already_created, and failed. Each row has a status such as would_create, already_created, employee_not_found, or department_not_found, together with the employee, department, responsible user, and complete task_fields payload where available.

To create tasks manually for the previous day, first set task_import.enabled = true, review a dry run, and then run:

sudo -u voice-pipeline \
  env VOICE_PIPELINE_CONFIG=/etc/voice-pipeline/config.toml \
  /opt/voice-pipeline/venv/bin/voice-pipeline task-import

A real run for an explicit date is also supported:

sudo -u voice-pipeline \
  env VOICE_PIPELINE_CONFIG=/etc/voice-pipeline/config.toml \
  /opt/voice-pipeline/venv/bin/voice-pipeline task-import --date 2026-08-30

Calls recorded in task_import_records are skipped on repeated preview and real runs. --output is intentionally accepted only together with --dry-run.

The Bitrix24 webhook needs user (or a compatible user-data scope), department, and task permissions. Add crm when CRM bindings in UF_CRM_TASK are used. Tasks are created with the REST method tasks.task.add.

voice-category-backfill creates the current database schema and imports all existing voice_category.json artifacts into voice_categories. It does not call Ollama and does not modify category artifact files. Re-running it safely replaces the database row for each call with the current file contents.

Management dashboard

The read-only dashboard uses categorized rows from PostgreSQL and provides period, manager, category, sentiment, risk, attention, scorecard, and quality score filters. It also exposes a JSON endpoint at /api/statistics.

Configure the local listener in /etc/voice-pipeline/config.toml:

[dashboard]
enabled = true
host = "127.0.0.1"
port = 8080

Initialize/import historical category data before the first start, then run:

voice-pipeline voice-category-backfill
voice-pipeline web-dashboard

The regular dashboard excludes megafon_ats calls. A separate process on port 8081 shows only Megafon ATS calls:

[megafon_dashboard]
enabled = true
host = "127.0.0.1"
port = 8081

voice-pipeline web-dashboard-megafon

For systemd deployments use voice-dashboard.service for existing sources and voice-dashboard-megafon.service for Megafon ATS. Their HTML pages, statistics API, call details, audio access, filter options, and downloadable reports are all isolated by source.

For systemd installations, use voice-dashboard.service. The default listener is intentionally local-only. Put an authenticated HTTPS reverse proxy in front of it before allowing access from other hosts.

Daily DOCX report

At 08:00 local server time, voice-reporter.timer starts the one-shot voice-reporter.service. It builds a report for the previous local calendar day from voice_categories and call_records. The document follows the management structure of samples/report_sample2.docx and is saved by default as:

/var/lib/voice-pipeline/reporter/YYYY-MM-DD/report.docx

Run voice-pipeline reporter to generate the previous day's report manually. The output root can be overridden in /etc/voice-pipeline/config.toml:

[reporter]
output_root = "/var/lib/voice-pipeline/reporter"

reload supports four modes:

• --list shows matching calls without changing anything.
• --reanalyze resets selected calls back to spooled so analyzer can run again.
• --recollect deletes the selected calls from DB and spool so the collector can fetch them again.
• --repair scans for incomplete analyzer outputs and repairs them.
• --repair --from-date YYYY-MM-DD also regenerates daily DOCX reports from the specified date onward.

Collector behavior

The collector currently uses voximplant.statistic.get as the main source of truth.

• Calls are filtered by day and by minimum CALL_DURATION.
• If the Voximplant checkpoint file is missing, the collector falls back to sources.bitrix24.checkpoint_lookback_days days of history.
• The collector enriches each call with CRM entity and Bitrix user data.
• Recording download depends on REST_APP_NAME:
  • "My Calls" calls are downloaded via CALL_RECORD_URL.
  • "My Calls - Android" falls back to crm.activity.get; when CALL_RECORD_URL is empty, the collector reads FILES[].id, resolves each file via disk.file.get, and downloads the first mp3 using NAME/DOWNLOAD_URL.
  • Asterisk calls are resolved in the Asterisk storage by EXTERNAL_CALL_ID.
• Partial Asterisk matches can be spooled when only some of mixed/rx/tx are present.

Each spooled call gets a meta.json file with the structured metadata used by the analyzer and for legacy fallback in downstream modules.

Analyzer behavior

The analyzer is timer-driven and runs in one pass each time the timer fires. It processes the current backlog, writes artifacts, and exits.

• Asterisk split-channel calls use rx.wav and tx.wav.
• Mixed audio calls use audio.mp3 or fallback mixed.wav when available.
• Analyzer artifacts are written under storage.analyzer_artifacts_root/{date}/{uniqueid}.
• conversation.json includes the canonical artifact_context block. Downstream modules rely on it as the primary source for: crm_activity_id, crm_entity_type, crm_entity_type_id, crm_entity_id, user_id, caller_phone, call linkage IDs, and voice_files.
• Cloud gateways that expose OpenAI-compatible APIs can be configured with interface = "openai" plus a custom base_url.
• Groq-compatible gateways can be configured the same way with interface = "groq" plus an optional base_url.
• Local multipart audio APIs can be configured with interface = "local_http". Transcription, diarization, and whole-recording emotion endpoints, language, timeout, retry policy, paths, and multipart file field are configurable:

[analyzer]
mixed_diarization_strategy = "local_http"
emotion_backend = "local_http"
pyannote_enabled = false

[[analyzer.providers]]
name = "local-audio-api"
provider = "local-audio-api"
interface = "local_http"
capabilities = ["transcription", "diarization", "emotion"]
base_url = "http://localhost:9000"
language = "ru"
# Optional endpoint-specific overrides:
# transcription_language = "ru"
# diarization_language = "en"
# emotion_language = "ru"
timeout_s = 300.0
retry_attempts = 3
retry_backoff_s = 2.0

Default paths are /v1/audio/transcriptions, /v1/audio/diarizations, and /v1/audio/emotions; the multipart field defaults to file. The emotion returned for the whole recording is attached to every transcript segment so existing artifacts and reports remain compatible.

Reporter behavior

• reporter uses analyzer artifacts as the primary reporting input.
• It owns reporter_status, reporter_failure_count, and reporter_error and never changes the analyzer-owned main status.
• It indexes transcript, conversation, emotion, and script-analysis artifacts into the analytics store.
• It generates:
  • per-call report_detail.html
  • daily / windowed management summary reports
• reporter does not create Bitrix24 call activities for transcript delivery.

Importer behavior

• importer is a standalone process with its own importer_status and import_error.
• Reporter failures and reporter_status=dead_letter do not block Bitrix24 import once analysis has completed.
• It only processes calls that already have analyzer artifacts and no saved Bitrix24 note id.
• It reads artifact_context and writes a note into the existing Bitrix24 call activity using:
  • ownerTypeId = crm_entity_type_id
  • ownerId = crm_entity_id
  • itemType = 2
  • itemId = crm_activity_id
• The note text format is:

MM:SS user: ...
MM:SS client: ...

• On success it stores importer markers in the DB and writes a local audit artifact bitrix_note.json next to the analyzer artifacts.

Systemd deployment

The repository includes unit files and an installer script:

sudo bash systemd/install.sh

The script creates the voice-pipeline system user, installs the unit files, and prepares the spool and reports directories under /var/lib/voice-pipeline.

Install PostgreSQL, create the application role/database, update storage.db_url, and initialize the schema with the separate idempotent database installer:

sudo bash systemd/install-db.sh

The script prompts for the database password without displaying it. For an unattended installation, pass it through the environment and optionally override the database settings:

sudo DB_PASSWORD='strong-password' DB_NAME=voice_pipeline DB_USER=voice_pipeline \
  bash systemd/install-db.sh

Tests

Run the test suite with:

pytest -q

For targeted runs:

pytest tests/test_collector.py -q
pytest tests/test_integration.py -q

Repository layout

• src/collector/ - collection pipeline, Bitrix24/Asterisk helpers, spooler
• src/analyzer/ - transcription, diarization, emotion scoring, script checks
• src/reporter/ - HTML report generation and analytics snapshots
• src/reporter/api.py - query helpers for call list/detail/snapshots
• src/importer/ - Bitrix24 transcript-note import process
• src/core/ - shared config, database, models, logging, retries
• systemd/ - service units and install script
• config/ - example configuration
• tests/ - unit and integration tests

Notes

• docs/README.md contains additional operator notes for the reload workflow.
• meta.json is written last and acts as the readiness signal for downstream processes.