Journal Event Collector

Summary

journal-event-collector is a Linux application that reads journal messages through configured ACL rules, turns matching messages into events, caches them with expiry, and exposes the shared core through a REST API and a CLI.

The project is built around one core layer with thin adapters for the daemon runtime, API, and CLI. Active events live in memory, while persisted snapshots are stored in SQLite so local and remote inspection can read the same state.

Architecture

The application follows a core + adapters layout:

1. ApplicationCore manages lifecycle, cache, collector loop, cleanup, and persistence.
2. JournalctlReader reads JSON journal entries from journalctl and feeds them into the core.
3. EventCache stores active events in memory and removes expired entries.
4. StateStore persists snapshots in SQLite and migrates legacy JSON state to SQLite when needed.
5. REST API and CLI are thin adapters over the same core.

Data flow:

journalctl -> journal reader -> event builder -> in-memory cache -> SQLite snapshot -> REST/CLI

Structure

.
├── docs/
│   ├── deployment-guide.md
│   └── usage-guide.md
├── journal-event-collector/
│   └── example-config.toml
├── README.md
├── src/
│   └── journal_event_collector/
│       ├── api.py
│       ├── cache.py
│       ├── cli.py
│       ├── config.py
│       ├── journal.py
│       ├── models.py
│       ├── runtime.py
│       ├── service.py
│       └── state.py
├── systemd/
│   └── journal-event-collector.service
└── tests/
    └── ...

Components

config.py

• Loads TOML configuration.
• Validates required fields.
• Validates ACL rules, cache timing, API settings, and SQLite state file path.

journal.py

• Launches journalctl.
• Reads JSON journal records.
• Filters entries by configured message ACL rules.
• Builds domain events from journal entries.

models.py

• Defines JournalEvent and ServiceStatus.
• Keeps the shared data shape used by API, CLI, and persistence.

cache.py

• Stores active events in memory.
• Removes expired events by TTL.
• Produces snapshots for publication and persistence.

state.py

• Writes persisted snapshots to SQLite.
• Reads persisted snapshots for CLI and REST.
• Migrates legacy JSON state files to SQLite.

service.py

• Orchestrates collector, cleanup, and persistence.
• Exposes status(), events(), and persisted_snapshot().

runtime.py

• Starts the REST API with uvicorn.
• Binds daemon lifecycle to the core.

api.py

• Exposes /health, /status, /events, and /state.
• Returns live status and persisted snapshot data.

cli.py

• Provides run, status, events, and state.
• Can read local persisted state or query a remote REST API.

Key Decisions

1. One core for all interfaces. This avoids logic drift between daemon, REST, and CLI.
2. In-memory cache for live data. This keeps event handling simple and fast.
3. SQLite for persisted snapshots. This lets API and CLI read the same state without an external service.
4. Legacy JSON migration to SQLite. Old files are converted in place on startup.
5. Explicit virtualenv execution for systemd. The unit points directly to .venv/bin/journal-event-collector.
6. GET /state returns persisted snapshot directly rather than a copy of live cache.

References

• Deployment Guide
• Usage Guide