add project README

This commit is contained in:
Ilgiz Kalmetyev 2026-06-27 22:51:51 +05:00
parent 992381f8d9
commit f7a11fa595
2 changed files with 147 additions and 1 deletions

2
.gitignore vendored
View File

@ -28,5 +28,5 @@ dist/
build/
*.egg
*.egg-info
output/

146
README.md Normal file
View File

@ -0,0 +1,146 @@
# voice-pipeline
`voice-pipeline` is a three-process Python service for collecting call recordings, analyzing them, and sending results back to Bitrix24.
## What it does
- `collector` fetches call metadata and recordings from Bitrix24 / Asterisk sources and writes them into a local spool directory.
- `analyzer` picks up spooled calls, transcribes them, performs diarization and script compliance checks, and stores artifacts next to the call.
- `reporter` creates Bitrix24 activities and HTML reports from analyzed calls.
The processes communicate through the spool directory and the shared `meta.json` contract stored inside each call folder.
## Requirements
- Python `3.11+`
- Linux for the systemd deployment
- A configured Bitrix24 webhook
- Optional dependencies depending on your use case:
- `paramiko` for Asterisk SFTP collection
- `pyannote.audio` for mixed-audio diarization
- `openai`, `groq`, `transformers`, `torch`, or `faster-whisper` for the analyzer, depending on backend choice
## Installation
Create a virtual environment and install the project in editable mode:
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```
Install optional extras when needed:
```bash
pip install -e ".[dev,diarize]" # mixed-audio diarization via pyannote
pip install -e ".[dev,local-asr]" # local Whisper-compatible transcription
pip install -e ".[dev,emotion]" # local emotion classification
```
## Configuration
Copy the example config and edit it:
```bash
cp config/config.example.toml /etc/voice-pipeline/config.toml
```
The config path is read from `VOICE_PIPELINE_CONFIG`, or defaults to:
```text
/etc/voice-pipeline/config.toml
```
Important sections:
- `[storage]` controls spool, reports, and database paths.
- `[[sources.asterisk]]` defines one or more Asterisk SFTP/HTTP sources.
- `[sources.bitrix24]` defines the Bitrix24 webhook and collector options.
- `[analyzer]` configures transcription, diarization, emotions, and script files.
- `[reporter]` controls report size and summary window.
Run a validation check after editing the config:
```bash
voice-pipeline validate-config
```
## CLI
The project exposes a single command:
```bash
voice-pipeline validate-config
voice-pipeline collector
voice-pipeline analyzer
voice-pipeline reporter
voice-pipeline reload --list --date 2026-06-23
voice-pipeline reload --reanalyze --uniqueid 1782211489.118028
voice-pipeline reload --recollect --status failed
```
`reload` supports three modes:
- `--list` shows matching calls without changing anything.
- `--reanalyze` resets selected calls back to `spooled` so analyzer can run again.
- `--recollect` deletes the selected calls from DB and spool so the collector can fetch them again.
## Collector behavior
The collector currently uses `voximplant.statistic.get` as the main source of truth.
- Calls are filtered by day and by minimum `CALL_DURATION`.
- The collector enriches each call with CRM entity and Bitrix user data.
- Recording download depends on `REST_APP_NAME`:
- "My Calls" calls are downloaded via `CALL_RECORD_URL`.
- `Asterisk` calls are resolved in the Asterisk storage by `EXTERNAL_CALL_ID`.
- Partial Asterisk matches can be spooled when only some of `mixed/rx/tx` are present.
Each spooled call gets a `meta.json` file with the structured metadata used by the analyzer and reporter.
## Analyzer behavior
- Asterisk split-channel calls use `rx.wav` and `tx.wav`.
- Mixed audio calls use `audio.mp3` or fallback `mixed.wav` when available.
- Transcript and analysis artifacts are written next to the call folder.
## Systemd deployment
The repository includes unit files and an installer script:
```bash
sudo bash systemd/install.sh
```
The script creates the `voice-pipeline` system user, installs the unit files, and prepares the spool and reports directories under `/var/lib/voice-pipeline`.
## Tests
Run the test suite with:
```bash
pytest -q
```
For targeted runs:
```bash
pytest tests/test_collector.py -q
pytest tests/test_integration.py -q
```
## Repository layout
- `src/collector/` - collection pipeline, Bitrix24/Asterisk helpers, spooler
- `src/analyzer/` - transcription, diarization, emotion scoring, script checks
- `src/reporter/` - Bitrix24 push and HTML report generation
- `src/core/` - shared config, database, models, logging, retries
- `systemd/` - service units and install script
- `config/` - example configuration
- `tests/` - unit and integration tests
## Notes
- `docs/README.md` contains additional operator notes for the reload workflow.
- `meta.json` is written last and acts as the readiness signal for downstream processes.