Redaction Reference

How hippo’s secret-redaction layer works, what it catches by default, and what it explicitly doesn’t. Companion to the README’s Privacy and Security section.

For the bigger threat model (data flow, encryption, MCP trust boundary), read the README section first. This doc is the regex-pattern deep-dive.

What redaction is

A regex-based filter that runs over event text before storage. Implemented in crates/hippo-core/src/redaction.rs::RedactionEngine. Applied to:

Redaction is best-effort. It catches known secret formats; it cannot catch secrets in arbitrary positions. Treat it as a noise filter, not a security guarantee. If you need stronger guarantees, don’t paste secrets into your terminal in front of a tool that captures stdout.

What redaction isn’t

Derived index exclusion

Capture redaction and derived index exclusion have different contracts. Raw source events, session fields and memory files keep their existing capture behavior. Full PATs, actual JWTs and recognized credential assignments still redact at capture or existing Python credential boundaries.

brain/src/hippo_brain/redaction.py::index_text replaces preserved 64-hex digests, incomplete GitHub/AWS prefixes, JWT lookalikes and standalone base64 blobs with type placeholders such as <hash> and <github-token>. UUIDs and git SHAs are plain identifiers, not credentials, and stay in derived text so commits and sessions remain searchable. The same policy covers enrichment inputs, returned node fields (including FTS content JSON), and both knowledge and command embedding inputs. Re-embedding and the orphan reaper use the shared embedding boundary; successful re-derivation also cleans the node text used by FTS. The excluded shapes can represent harmless values; search by an excluded value can lose recall, so use raw event display for it.

The standalone base64 heuristic matches slash-free blobs only, so absolute and relative paths survive. An encoded blob that contains / is treated as a path and is not replaced. Other credential patterns and structured Authorization protections still apply to path text.

This source change does not rebuild an installed corpus. Existing derived rows need separately authorized re-derivation. mise run re-embed stops the brain before running scripts/hippo-re-embed.py; it requires operational authorization. The script replaces both vector columns, cleans derived node text, tags and FTS, and rebuilds affected entity links. It removes unreferenced credential-shaped entities but preserves entities referenced by raw events or relationships and classification-owned entities. It publishes model metadata even for an empty corpus. Publication uses one transaction; failure rolls back the whole replacement. The command stages both validated vector arms in an anonymous temporary file before acquiring the SQLite write lock. It retains one vector pair at a time, then rechecks corpus inputs and model metadata before atomic publication. The input snapshot still scales with corpus size; vector payloads do not accumulate in Python memory. The temporary file closes on success or failure. Publication retains one transaction across all nodes and rolls back if any stored pair or cleanup fails. Regression coverage is in brain/tests/test_index_exclusion.py ↗. Re-derivation does not remove raw sources, logs or backups, and it does not prove resistance to vector inversion. The synthetic regression checks both embedding inputs and a controlled nearest-neighbor probe with an unfiltered positive control. It does not use a live inference model.

Pattern format

~/.config/hippo/redact.toml is a TOML file with an array of pattern tables:

[[patterns]]
name = "aws_access_key"
regex = 'AKIA[0-9A-Z]{16}'
replacement = "[REDACTED]"

[[patterns]]
name = "github_pat"
regex = 'ghp_[a-zA-Z0-9]{36}|github_pat_[a-zA-Z0-9_]{82}'
replacement = "[REDACTED]"
FieldRequiredBehavior
nameyesRule identifier. Surfaces in metrics (hippo.daemon.redactions{rule=<name>}) and in hippo redact test output. Must be unique.
regexyesRust regex crate ↗ syntax. No PCRE backreferences or lookaround. Compiled once at daemon startup (and RegexSet-bundled for fast match dispatch). Use (?i) for case-insensitivity.
replacementnoThe replacement string for matched substrings. Defaults to "[REDACTED]" when omitted (see RedactConfig in crates/hippo-core/src/config.rs); can include capture-group references ($1, $2) per the regex crate’s Replacer.

The whole redact.toml is the rule set; patterns are loaded in file order. See crates/hippo-core/src/config.rs::RedactConfig for the deserialization shape.

Evaluation model

Default patterns

The authoritative patterns and evaluation order are in config/redact.default.toml ↗, mirrored by RedactConfig::builtin() for capture paths using built-in rules. Do not copy regex definitions into documentation. Existing user redact.toml files are not automatically upgraded; compare them with the template to adopt the PEM-body and quoted-assignment rules. Regression coverage lives in crates/hippo-core/src/redaction.rs and brain/tests/test_redaction.py.

Known false-negatives

The default rules do not catch:

If your workflow involves any of the above, write custom patterns. (See Writing custom patterns.)

Writing custom patterns

The hippo redact test "<input>" CLI compiles your live redact.toml and reports which rules fire on the given input. Use it iteratively while writing a new pattern.

Example session

You want to catch your team’s internal API tokens, which look like xtok_<32-hex>:

# Step 1: confirm the default rules don't catch it
hippo redact test "deploy --token xtok_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6"
# No patterns matched.
# Redacted (0 replacements):
#   deploy --token xtok_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

# Step 2: add the pattern
$EDITOR ~/.config/hippo/redact.toml
[[patterns]]
name = "internal_xtok"
regex = 'xtok_[a-f0-9]{32}'
replacement = "[REDACTED]"
# Step 3: verify
hippo redact test "deploy --token xtok_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6"
# Matched patterns: internal_xtok
# Redacted (1 replacements):
#   deploy --token [REDACTED]

# Step 4: confirm it doesn't false-positive on benign text
hippo redact test "git checkout xtok-feature-branch"
# No patterns matched.
# Redacted (0 replacements):
#   git checkout xtok-feature-branch

More example sessions

# Stripe live keys
hippo redact test "STRIPE_KEY=sk_live_AbCdEfGhIjKlMnOp"
# No patterns matched.
# Redacted (0 replacements): ... (STRIPE_KEY isn't on the keyword list)

# After adding a pattern named `stripe_secret_key` with regex `sk_live_[a-zA-Z0-9]{24,}`:
hippo redact test "STRIPE_KEY=sk_live_AbCdEfGhIjKlMnOp"
# Matched patterns: stripe_secret_key
# Redacted (1 replacements):
#   STRIPE_KEY=[REDACTED]
# Slack incoming webhooks (placeholder shown — replace with your team's URL when testing)
hippo redact test "curl -X POST https://hooks.slack.com/services/<TEAM>/<CHANNEL>/<TOKEN>"
# No patterns matched.
# Redacted (0 replacements): ...

# After adding a pattern with regex = 'https://hooks\.slack\.com/services/[A-Z0-9/]+'
# the same call would report:
# Matched patterns: slack_webhook_url
# Redacted (1 replacements): curl -X POST [REDACTED]
# Database connection strings
hippo redact test "DATABASE_URL=postgres://admin:hunter2@db.example.com:5432/prod"
# No patterns matched.
# Redacted (0 replacements): ... (DATABASE_URL isn't on the keyword list)

# Add: regex = '(?i)(database_url|postgres|mysql)://[^@\s]+:[^@\s]+@'
hippo redact test "DATABASE_URL=postgres://admin:hunter2@db.example.com:5432/prod"
# Matched patterns: db_connection_string
# Redacted (1 replacements): DATABASE_URL=[REDACTED]db.example.com:5432/prod

After adding any new pattern, restart the daemon: mise run restart. The engine compiles patterns at startup and doesn’t reload on redact.toml change.

Browser URL redaction

Separate from redact.toml. Configured in [browser.url_redaction] in ~/.config/hippo/config.toml:

[browser.url_redaction]
strip_params = ["session_id", "auth_token", "access_token", "api_key", "token"]

Implemented in crates/hippo-daemon/src/native_messaging.rs::strip_sensitive_params. For each URL passed by the Firefox extension, query parameters whose names match strip_params are removed before storage. Path components are preserved.

What it catches:

What it doesn’t catch:

Threat model

Hippo defends againstHippo does NOT defend against
Accidental capture of common token formats (AWS, GitHub PAT, JWT, PEM private keys) into the LLM contextSecrets in positional arguments, non-standard env-var names, or arbitrary file content the user pastes
Common URL-borne tokens in browser visit URLsTokens in URL path segments or page content
Secrets in Authorization: Bearer … HTTP headers logged to stdoutSecrets in custom auth schemes
Storing recognized PEM private-key blocks, including their bodiesUnmarked key material without a recognized credential assignment
Replay of [REDACTED] strings across the LLM/MCP path (since the secret has been replaced before storage)Secrets that were already stored before a pattern was added

For threats outside this list, the answer is “don’t paste secrets into your terminal.” Hippo’s job is to catch the common case.

See also