B24 Call Pipeline

Local pipeline for processing call recordings on Windows and Linux.

What It Does

• Discovers local audio files from a configured input directory.
• Separates calls into incoming, outgoing, or unknown based on configured glob rules.
• Validates files with ffprobe and checksum checks.
• Skips calls shorter than the configured minimum duration and logs them clearly.
• Moves validated files through a spool lifecycle.
• Normalizes audio to WAV PCM 16 kHz mono before transcription.
• Detects silence spans and retries long or gappy transcripts with overlapping chunks.
• Transcribes audio with Faster-Whisper.
• Adds optional offline WhisperX/pyannote diarization and placeholder emotion artifacts.
• Produces rule-based scorecard classification.
• Exports results as JSON, CSV, and Markdown.

Phase 1 Scope

• Local file source only.
• SQLite state tracking.
• Faster-Whisper transcription.
• Optional offline diarization and placeholder emotion.
• Rule-based classification.
• JSON, CSV, and Markdown exports.

Bitrix24 is intentionally not active in Phase 1. The source layer is already shaped so a Bitrix24Source can be added later without changing the rest of the pipeline.

Environment Setup

Use the project virtual environment for development and runtime.

python -m venv .venv
./.venv/bin/python -m pip install -e .

If you are on Windows, use the equivalent py -m venv .venv and .venv\Scripts\python.exe -m pip install -e ..

If you need diarization, install the optional extra as well:

./.venv/bin/python -m pip install -e ".[diarization]"

Required system tools:

• ffmpeg
• ffprobe

If you plan to run the optional heavy Faster-Whisper smoke test, set:

export B24CALLS_RUN_HEAVY_TESTS=1

Configuration

The project expects absolute filesystem paths in the TOML config.

Example paths in configs/example.toml:

• models.directory = "/mnt/c/models" on Linux
• models.directory = "C:\\models" on Windows
• local_source.directory = "/mnt/c/calls/input" on Linux
• local_source.directory = "C:\\calls\\input" on Windows
• local_source.incoming_patterns and local_source.outgoing_patterns define how direction is inferred.
• processing.direction_filter controls whether the pipeline processes incoming, outgoing, or all calls.
• validation.min_duration_seconds sets the minimum audio length accepted for processing.
• audio_preprocess controls normalization, silence detection, chunk size, overlap, and the optional forced-alignment fallback.
• diarization controls optional speaker diarization. Runtime is offline: configure backend = "whisperx" and point model_path at a local pyannote/WhisperX-compatible diarization pipeline directory before setting enabled = true. The cache_dir is used for local model cache and doctor will fail if diarization is enabled but the local pipeline is missing.

The default config lives at configs/example.toml. Keep custom config values absolute as well.

First Run

1. Check the environment.

./.venv/bin/b24calls doctor

2. Discover local audio files.

./.venv/bin/b24calls discover --direction all

3. Validate registered files.

./.venv/bin/b24calls validate --min-duration 5.0

4. Stage validated files into spool.

./.venv/bin/b24calls spool

5. Transcribe staged files.

./.venv/bin/b24calls transcribe

If diarization.enabled = true, this step also writes the diarization object into each transcript artifact. Otherwise the transcript keeps the placeholder diarization payload.

6. Optionally run or re-run diarization for completed transcripts.

./.venv/bin/b24calls diarize --force

7. Export final artifacts.

./.venv/bin/b24calls export --formats json,csv,markdown

For a complete end-to-end local smoke run, execute the commands in this order after placing audio files into the input directory.

CLI Overview

• doctor - validates Python, virtualenv, ffmpeg, ffprobe, directories, and runtime selection.
• discover - registers local audio files in SQLite and applies the direction filter.
• validate - probes audio metadata, applies the minimum duration gate, and records short files.
• spool - stages validated files into spool.
• transcribe - runs Faster-Whisper and writes transcript artifacts.
• transcribe also writes preprocessing metadata and uses chunked retry when the first pass leaves large gaps.
• transcribe also writes offline diarization when the feature is enabled in config.
• diarize - updates only the top-level diarization object inside completed transcript JSON artifacts.
• export - renders JSON, CSV, and Markdown outputs.
• process, run-once - reserved for later orchestration.

Output Artifacts

The pipeline writes artifacts under the configured output directory.

• output/transcripts/<audio_file_id>.json - canonical transcript artifact.
• output/exports/json/<audio_file_id>.json - final JSON export.
• output/exports/csv/<audio_file_id>.csv - flat per-call CSV export.
• output/exports/markdown/<audio_file_id>.md - human-readable report.

SQLite tracks artifact records for these files in the artifacts table.

The transcript and export payloads include the source direction so downstream reporting can distinguish incoming and outgoing calls. When enabled, diarization is stored as a separate top-level diarization object; existing source metadata, preprocessing metadata, and transcription payloads are left unchanged. The diarization object contains backend information, speaker turns, and transcript alignment references.

Developer Workflow

• Use the .venv interpreter for all local commands.
• Run ./.venv/bin/python -m compileall src tests before committing larger changes.
• Run ./.venv/bin/python -m unittest discover -s tests for the full test suite.
• Keep commit steps small and aligned with IMPLEMENTATION_PLAN.md.

Bitrix24 Source

Bitrix24 API ingestion is available through the same discovery command as local files. The source downloads call recordings during discovery, registers the local file in SQLite, and then the existing validate, spool, transcribe, and export commands operate normally.

Required incoming webhook scopes:

• telephony for voximplant.statistic.get.
• disk for recordings that must be resolved through RECORD_FILE_ID.
• user_brief, user_basic, or user only when optional user enrichment is enabled.

See Bitrix24 API user setup for the recommended service user, exact read permissions, webhook scopes, and access checks.

Example discovery for yesterday's calls:

B24CALLS_BITRIX24_BASE_URL="https://portal.example/rest/1/webhook_token/" \
./.venv/bin/b24calls discover --source bitrix24 --direction all

The downloaded recordings are written to bitrix24.recordings_dir. Re-running discovery skips already registered files by source id and checksum. The diagnostic script tmp/get_calls_via_Bitrix24_REST_v1.py remains a prototype for manual API checks; runtime code lives under src/b24calls.

Repeat discovery is effectively idempotent at the application level:

• the Bitrix24 REST call is read-only and can be repeated safely;
• Bitrix24 recordings with already registered source_type + source_id are skipped before download;
• the downloader does not overwrite an existing non-empty local file unless overwrite_existing = true is used in the source implementation;
• SQLite registration skips already known source_type + source_id pairs with the same checksum and size;
• identical content under a different source id is treated as a duplicate and is skipped;
• if a recording was never registered because download failed, the next discovery run tries to download it again.

So the API download itself is not "strictly idempotent" in the HTTP sense, but the pipeline makes repeated discovery runs convergent for the same recordings.

Current Status

The local-file workflow is implemented and covered by unit and smoke tests. Optional offline diarization is implemented with a separate diarize command and transcript-level diarization object. Bitrix24 source ingestion is available as Phase 2 discovery.

Промпт для составления аналитического отчёта с помощью LLM

Подготовить расширенный аналитический отчёт уровня руководителя отдела продаж, включающий:

• сегментацию менеджеров по эффективности;
• анализ соблюдения скрипта (приветствие, выявление потребности, работа с возражениями, завершение);
• рейтинг менеджеров по нескольким KPI;
• анализ причин потери клиентов;
• тепловые карты проблемных зон;
• расчёт интегрального индекса качества коммуникации;
• персональные рекомендации по каждому менеджеру;
• диаграммы и визуализации;
• приложение с примерами лучших и худших переговоров.

Добавить в отчёт:

• Executive Summary по массиву звонков.
• Основные KPI по качеству и длительности переговоров.
• Часто встречающиеся проблемы (missing_next_step, negative_tone_scorecard и др.).
• Рейтинг менеджеров по среднему качеству переговоров.
• Рекомендации по повышению эффективности отдела продаж и сопровождения.
• Оценка потенциальной бизнес-ценности улучшений.