Observability

How to run Hippo’s optional OpenTelemetry stack, which dashboards to open, and which provisioned alerts fire when capture paths degrade. Companion to otel/README.md ↗ (stack setup and commands) and capture/operator-runbook.md (first-aid when alarms fire).

Telemetry is off by default. Nothing is emitted until you build with OTel support, enable [telemetry] in config, and start the Docker stack.

Quick start

mise run otel:up          # Grafana + Prometheus + collector on localhost
mise run build:otel       # daemon with OTel feature
hippo config edit         # [telemetry] enabled = true
export HIPPO_OTEL_ENABLED=1   # brain + MCP
mise run restart
open http://localhost:3030

Grafana allows anonymous Admin access by default, so startup requires no password export or login. All published ports bind to 127.0.0.1. A new Grafana database uses admin / hippo unless HIPPO_OTEL_GRAFANA_ADMIN_PASSWORD is supplied.

If your Grafana organization was renamed, set HIPPO_OTEL_GRAFANA_ORG_NAME to its name in the ignored otel/.env file. The default is Main Org.. Grafana’s anonymous access configuration ↗ requires the configured name to match the stored organization name.

Optional Grafana authentication

export COMPOSE_FILE=docker-compose.yml:docker-compose.auth.yml
export HIPPO_OTEL_GRAFANA_ADMIN_PASSWORD='<private password>'
mise run otel:up

The override disables anonymous access and requires a nonempty password. Keep both variables available for subsequent OTel tasks, including otel:restart. To restore anonymous access, run unset COMPOSE_FILE and mise run otel:up.

Grafana uses GF_SECURITY_ADMIN_PASSWORD only when the database is first created. For an existing database, change the admin password in Grafana’s user preferences before enabling authentication. Grafana’s configuration reference ↗ documents the first-run behavior.

Architecture

hippo-daemon ──┐
               ├── OTLP ──→ OTel Collector ──→ Tempo (traces)
hippo-brain  ──┤                            ──→ Loki (logs)
hippo-mcp   ──┘                            ──→ Prometheus (metrics)
                                               Grafana (dashboards + alerts)
ServicePortPurpose
Grafana3030Dashboards, Explore, provisioned alert rules
Prometheus9090Metrics storage (30d / 10GB default retention)
OTel Collector4317 (gRPC), 4318 (HTTP)OTLP ingest
Tempo3200Trace storage
Loki3100Log aggregation

Persistent data: ~/.local/share/hippo/otel/. Stack restarts do not wipe state.

Dashboards

All dashboards provision automatically from otel/grafana/dashboards/ into the Hippo folder. No manual import.

DashboardUIDURLWhat it shows
Hippo Overviewhippo-overviewhttp://localhost:3030/d/hippo-overview ↗Health grade, capture lag, probe success/lag, invariant violations, alarm firings, watchdog liveness, daemon drops
Hippo Daemonhippo-daemonhttp://localhost:3030/d/hippo-daemon ↗Event ingest/drop rates, flush latency, redactions, fallback writes, watcher throughput
Hippo Enrichmenthippo-enrichmenthttp://localhost:3030/d/hippo-enrichment ↗Brain queue depth, LLM latency, enrichment throughput, MCP tool metrics, Jev/rules/local decision outcomes, stage latency, Jev tokens, error ratio, classification backlog, and rerank fallback causes, effect, relevance, and deadline overrun (see Jev decisions)
Hippo Processeshippo-processeshttp://localhost:3030/d/hippo-processes ↗process.* CPU/memory for daemon, brain, and MCP, plus recently observed MCP telemetry instances
Hippo — Knowledge Healthhippo-knowledge-healthhttp://localhost:3030/d/hippo-knowledge-health ↗Recall probe (golden-question /ask round-trips), capture alarms/staleness, corpus size and vector coverage, project graveyard and dead-project contamination, identity fragmentation, redaction canary. Fed by the knowledge-health exporter, not OTel.

Metric names in PromQL use Prometheus exporter suffixes (_total, _milliseconds, etc.). The shared OTel name/type contract lives in tests/fixtures/otel-metric-names.json. When adding an instrument, update that contract and exercise its runtime emission in brain/tests/test_otel_dashboards.py (Python) or crates/hippo-daemon/tests/dashboard_metrics.rs (Rust). These tests collect SDK measurements and check names, types, and units; Python validates parsed dashboard and alert references against the contract and collected Python samples. The standalone knowledge-health exporter owns its registry, which the Python tests verify against synthetic-database scrapes. Dashboard selector checks reject the retired service_namespace label, not harmless mentions in descriptions or label values.

Dashboards draw on two metric sources: OTel instruments in the daemon, brain, and MCP (hippo_daemon_*, hippo_brain_*), and the knowledge-health exporter (hippo_kb_*, see below). Both are covered by the same drift tests.

_total is reserved for cumulative counters. A point-in-time reading is a gauge with a bare name — hippo_kb_events, not hippo_kb_events_total — because increase()/rate() over a non-monotonic _total series is silently always zero.

Provisioned alert rules

Alert rules provision from every file in otel/grafana/alerting/ on stack start — hippo-capture-alerts.yml (capture reliability) and hippo-knowledge-alerts.yml (knowledge health). They appear under Alerting → Alert rules in the Hippo folder.

AlertFires whenforSeverity
Daemon events droppedrate(hippo_daemon_events_dropped_total[5m]) > 05mwarning
FS watcher events droppedrate(hippo_watcher_events_dropped_total[5m]) > 05mwarning
Watchdog not runningrate(hippo_watchdog_run_total[5m]) < 0.0015mcritical
Probe failure rate highok=false probe runs > 10% over 15m15mwarning
Capture invariant violationrate(hippo_watchdog_invariant_violation_total[15m]) > 015mcritical

Knowledge health (hippo-knowledge-alerts.yml)

AlertFires whenforSeverity
Recall path downmin_over_time(hippo_kb_recall_up[5m]) < 1 while the exporter scrape target is up5mcritical
Exporter not scrapingup{job="hippo-knowledge-health"} < 1 for 10m10mwarning
Recall path degradedavg_over_time(hippo_kb_recall_latency_milliseconds[15m]) > 15000 (successful probes only)15mwarning
Recall probe failure burstsum(increase(hippo_kb_recall_failures_total[1h])) > 35mwarning
Capture alarm backloghippo_kb_capture_alarms_active above threshold4hwarning
Capture stalemax(hippo_kb_capture_source_last_event_age_milliseconds) above threshold10mwarning
Graveyard contaminationhippo_kb_dead_project_node_ratio above threshold1hwarning
Stranded-hours jumpdelta(hippo_kb_stranded_hours{window="30d"}[7d]) above threshold1hinfo
Secretish env keys presenthippo_kb_env_secretish_keys > 05mwarning
Canary leakmax(hippo_kb_canary_found) > 00mcritical
Collector errorssum(increase(hippo_kb_collector_errors_total[15m])) > 05mwarning

Two further rules (hippo_kb_epitaph_unconfirmed, hippo_kb_push_useful_floor) ship isPaused: true — they target snowball metrics whose backing tables do not exist yet. Unpause them when the feature lands.

All rules use noDataState: OK. Dead-exporter detection moved out of hippo_kb_recall_down (which previously used noDataState: Alerting for that) into the dedicated hippo_kb_exporter_down rule on up{job="hippo-knowledge-health"} — that keeps the OTel-stack-without-exporter deployment ([telemetry] enabled = false) from paging critical, while still surfacing the dead-exporter case as a warning.

All capture rules use noDataState: OK so a stack with telemetry disabled does not page. When OTel is enabled but a rule has no series, treat that as “instrument not emitting” rather than healthy silence.

Notification routing: this repo provisions rules only. Wire contact points and notification policies in Grafana UI (or add provisioning YAML) when you want Slack/PagerDuty delivery.

Knowledge-health exporter

scripts/hippo-metrics-exporter.py is a stdlib-only Python bridge that reads hippo.db read-only (mode=ro + PRAGMA query_only=ON; it can never write or lock capture) and serves Prometheus text on 127.0.0.1:9835. It covers what the OTel capture instruments cannot: what is in the knowledge base and whether it can still be recalled.

Vector coverage is the share of live knowledge nodes with vector rows, not a measure of answer accuracy. The enrichment dashboard also uses this exporter’s stored ready-classification signals: accepted-topic coverage, accepted-label count, and mean accepted Noul score. These do not establish classification correctness or calibrated confidence. The recall probe exercises brain /ask, not MCP credentials.

Endpoints: /metrics (Prometheus), /metrics.json (same samples as JSON), /healthz.

Enabling telemetry

Daemon (Rust)

# ~/.config/hippo/config.toml
[telemetry]
enabled = true
endpoint = "http://localhost:4317"

Build: mise run build:otel or cargo build --features otel.

Brain / MCP (Python)

export HIPPO_OTEL_ENABLED=1
# optional: export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

New hippo-mcp processes also honor [telemetry] enabled = true in ~/.config/hippo/config.toml when HIPPO_OTEL_ENABLED is unset. An explicit HIPPO_OTEL_ENABLED=0 disables MCP telemetry. Restart existing MCP processes after changing settings. Python telemetry assigns a unique service.instance.id on initialization so concurrent clients export distinct series.

The enrichment dashboard aggregates MCP calls, latency, errors, lookup outcomes, and returned result counts across recently observed processes. Lookup outcomes distinguish hit, empty, and degraded responses; a hit means a nonempty result or source list, not a correct answer. These cumulative MCP panels reset with process restarts, rather than showing a fixed-window rate. MCP request logs omit query text and filters, and lookup spans record query length or repository presence instead of their contents.

Commands

mise run otel:up       # start stack
mise run otel:down     # stop stack
mise run otel:status   # container health
mise run otel:logs     # tail compose logs
mise run otel:backup   # snapshot persisted data

See otel/README.md ↗ for retention overrides, reset workflow, and process-metric details.

When alerts fire

  1. hippo doctor --explain — isolated checks with CAUSE/FIX per failure
  2. hippo alarms list — unacknowledged capture alarms from SQLite ground truth
  3. capture/operator-runbook.md — version mismatch, enrichment wedge, probe failures

SQLite source_health remains the correctness source during OTel outages; dashboards and alerts are time-series views on the same invariants.