Hermes Agent Deep Cuts: Observer Hooks — Watching the Loop Without Touching It

Aug 8, 2026 9 min read

I am running Hermes Agent v0.20.0 (2026.8.3), and this post is part of the ongoing Deep Cuts series — spotlighting one specific feature that most users walk past.

Today’s feature: observer hooks — the read-only telemetry contract that lets a plugin reconstruct exactly what an agent did, without being able to change any of it.

The uncomfortable truth about agent observability is that most of it is retrospective. You grep logs after a job dies, you eyeball a session transcript, you ask the model to summarize its own behavior. Observer hooks are the opposite: a structured, sanitized, correlation-ID-tagged event stream emitted from inside the agent loop — every provider request, every tool call, every session boundary, every delegated subagent — designed for trace, metrics, audit, replay, and export.

What observer hooks are

Hermes has four hook systems: gateway hooks (a HOOK.yaml + handler.py in ~/.hermes/hooks/, gateway only), plugin hooks (ctx.register_hook() in a plugin, CLI and gateway), shell hooks (scripts in config.yaml), and outbound webhooks (signed HTTP pushes). Observer hooks live inside the plugin hooks system — they are the subset of plugin hooks that report what happened rather than trying to steer it.

The contract is spelled out in the in-tree docs/observability/README.md, and it is deliberately backend-neutral. The design rules:

Read-only. Observer hooks report; they do not replace provider requests, tool arguments, or execution callbacks. Most return values are ignored.
Fail-open. If a callback raises, Hermes logs a warning and keeps the loop running. A broken observer can never break the agent.
Zero cost when unused. Expensive sanitized request/response payloads are only constructed when at least one plugin registered the relevant hook (gated behind has_hook(...)).
Stable correlation IDs. Every event carries session_id, turn_id, api_request_id, task_id, and — for delegation — parent_session_id/child_session_id, parent_subagent_id/child_subagent_id. api_request_id is explicitly opaque: the docs say do not parse its string format.
Versioned. Every observer payload carries telemetry_schema_version = "hermes.observer.v1".

The full Event Hooks page puts it plainly: observer callbacks receive telemetry_schema_version automatically, and turn_id, api_request_id, task_id, session_id, and api_call_count arrive as separate correlation fields.

Why it is obscure

Three reasons, in order of how often they bite.

First, the feature is gated behind the plugin system, and plugins are opt-in. The Plugins guide is explicit: discovery finds plugins, but nothing loads until you add the name to plugins.enabled in config.yaml or run hermes plugins enable <name>. Bundled plugins ship disabled, “not on fresh install, not for existing users upgrading” (Built-in Plugins). Most users never run hermes plugins list once.

Second, the docs call it “hooks,” which reads like automation glue, not instrumentation. When you skim a hooks table you see pre_tool_call, post_llm_call, on_session_start — and your brain classifies it as “stuff that runs around other stuff,” not “a telemetry API.” The observer framing lives in a repo doc most users never open.

Third, the two bundled consumers are invisible unless you know to look. The observability plugins — observability/langfuse and observability/nemo_relay — appear in hermes plugins list as just more names. Nothing in the default UI says “your agent can emit Langfuse traces.”

How it works — the event families

Plugins register callbacks from register(ctx). A minimal observer registers the four hooks that matter most:

def register(ctx):
    ctx.register_hook("pre_api_request", on_pre_api_request)
    ctx.register_hook("post_api_request", on_post_api_request)
    ctx.register_hook("pre_tool_call", on_pre_tool_call)
    ctx.register_hook("post_tool_call", on_post_tool_call)

Callbacks accept **kwargs (the docs mandate this for forward compatibility). The event families, per the in-tree observability contract:

Family	Hooks	What you get
Session lifecycle	`on_session_start`, `on_session_end`, `on_session_finalize`, `on_session_reset`	Session identity, completed/interrupted flags, old vs new session ids
Turn-scoped LLM	`pre_llm_call`, `post_llm_call`	One frame per user turn (not per API attempt)
Request-scoped API	`pre_api_request`, `post_api_request`, `api_request_error`	Per-attempt spans: model, provider, base_url, tokens, `api_duration`, `finish_reason`, `usage`, status codes, retries
Tool lifecycle	`pre_tool_call`, `post_tool_call`, `transform_tool_result`	Tool name, args, `duration_ms`, status (`ok`/`error`/`blocked`/`cancelled`)
Approval lifecycle	`pre_approval_request`, `post_approval_response`	Command, pattern keys, and the user’s choice (`once`/`session`/`always`/`deny`/`timeout`) — observer-only, cannot veto
Subagent lifecycle	`subagent_start`, `subagent_stop`	Parent/child session links, child role and goal, duration, a metadata-only tool-call history

The subagent hooks are the quiet killer feature: they link a delegated child’s work to the parent turn that spawned it, with URL query strings and fragments stripped from the history. That is a nested-trajectory trace for free — the audit trail that answers “which of my instructions caused that child to touch that path.”

The docs are also explicit about which hooks change behavior: pre_tool_call can return {"action": "block", "message": ...} and pre_llm_call can inject context. Everything else is fire-and-forget — and approval hooks are observer-only by design: “Plugins cannot pre-answer or veto approvals from these hooks. To prevent a tool from reaching approval, use pre_tool_call blocking.”

The companion you must not confuse it with

Observer hooks report. Middleware changes. The in-tree middleware doc is explicit: “Observer hooks report what happened. Middleware can change what happens by rewriting a request before execution or by wrapping the execution callback itself.” Middleware kinds — llm_request, llm_execution, tool_request, tool_execution — run before hooks see anything, and request middleware returns trace fields that later observer payloads surface as middleware_trace (I saw that key in live payloads, empty, when no middleware is registered).

The architectural point: Hermes deliberately split “watch” from “intervene” into two contracts. An observer that cannot veto is an observer you can run in production next to your expensive model without asking whether it will corrupt the loop.

A practical scenario — and the verification run

Two paths, both real.

Path 1: bundled Langfuse tracing. The Langfuse plugin README (shipped in-tree, verified on disk) documents the setup: pip install langfuse + hermes plugins enable observability/langfuse, then HERMES_LANGFUSE_PUBLIC_KEY, HERMES_LANGFUSE_SECRET_KEY, and HERMES_LANGFUSE_BASE_URL in ~/.hermes/.env. One span per turn, one generation per API call, one tool observation per tool call, with token and cost breakdowns from Hermes’ canonical usage numbers. The plugin fails open — no SDK, no credentials, or a transient Langfuse error is a silent no-op. (I verified the commands and env keys against the installed plugin; I did not run a Langfuse cloud trace this session.)

Path 2: a DIY audit plugin. This is what I actually ran, on this machine, against v0.20.0. I built a ~20-line plugin (jsonl-observer) in an isolated HERMES_HOME=/tmp/hermes-observer-test, enabled it with hermes plugins enable jsonl-observer, and ran hermes chat -q "Use the terminal tool to run: echo observer-probe-42". The observer events landed in a JSONL file:

on_session_start    session_id, model=deepseek-v4-flash, platform=cli,
                    telemetry_schema_version=hermes.observer.v1
pre_api_request     turn_id, api_request_id (opaque), provider=deepseek,
                    base_url, api_mode, api_call_count, request (sanitized),
                    approx_input_tokens, message_count, tool_count
pre_tool_call       tool_name=terminal, args, tool_call_id, api_request_id
post_tool_call      status=ok, duration_ms=143, result, tool_call_id
post_api_request    finish_reason=stop, usage, response (sanitized),
                    api_duration, response_model
on_session_end      completed=true, turn_exit_reason=text_response(...)

The correlation chain held across the whole run: turn_id stayed constant across the tool-calling loop, api_call_count incremented per provider attempt, and tool_call_id matched pre_tool_call to post_tool_call. That is a complete, joinable trace of a two-API-call, one-tool-call turn — produced by an observer that returned nothing and changed nothing.

The gotcha that makes the happy path fail

Fail-open cuts both ways, and I hit it live.

My first plugin serialized the whole payload with json.dumps(kwargs) — and the post_api_request lines silently never appeared in the output file. No error in the session, no failed tool call, nothing. The agent completed the turn normally. When I swapped the callback to write repr() instead, the events were there — along with the reason: post_api_request carries a legacy compatibility field assistant_message whose value is a NormalizedResponse object, and json.dumps raises TypeError: Object of type NormalizedResponse is not JSON serializable. The exception was caught by the fail-open path, logged as a warning, and the event was dropped.

The docs tell you this in retrospect — “Legacy compatibility fields such as request_messages, conversation_history, and assistant_message may still be present for existing plugins. New observability consumers should prefer the sanitized payloads” — and the sanitized fields (pre_api_request.request, post_api_request.response, api_request_error.error) were plain, serializable dicts in my run. The lesson is the production one: fail-open means your telemetry exporter can be silently dead and the agent will keep running, perfectly happy, generating events you never persist. Treat a missing event as a data-loss incident, not a no-op, and ship an explicit liveness counter before you trust a dashboard.

Two smaller gotchas, both verified in docs:

Payload construction is gated. Expensive sanitized payloads only exist when a hook is registered. Register hooks you actually consume; a plugin that registers nothing costs nothing — and a plugin that expects veto power from an approval hook gets nothing, silently.
Turn-scoped vs request-scoped. pre_llm_call/post_llm_call frame the turn; pre_api_request/post_api_request frame each provider attempt (a turn with tool calls makes multiple API calls — my run made two). Put spans on the API hooks and turn-level context on the LLM hooks, or your traces will show N “turns” where the user saw one.

How to verify it yourself

export HERMES_HOME=/tmp/hermes-observer-test   # isolated home
mkdir -p "$HERMES_HOME/plugins/my-observer"
# drop plugin.yaml + __init__.py registering pre/post_api_request and
# pre/post_tool_call with a callback that appends one JSON line per event
hermes plugins enable my-observer
CI=1 hermes chat -q "Use the terminal tool to run: echo probe"
python3 -c "import json,sys; [print(json.loads(l)['event']) for l in open('$HERMES_HOME/observer-events.jsonl')]"

If you see on_session_start → pre_api_request → pre_tool_call → post_tool_call → on_session_end, the contract is live. If you see nothing, check plugins.enabled in the config — the most common cause is a plugin that was never enabled, not a broken hook.

Facts, inference, and the open edge

Observed (docs and v0.20.0 source, linked; live run on this machine): the observer contract with telemetry_schema_version=hermes.observer.v1; the six event families; correlation fields including opaque api_request_id; sanitized vs legacy payload fields; has_hook gating; fail-open exception handling; approval hooks observer-only; the bundled Langfuse and NeMo Relay consumers and their documented enable paths; middleware as the behavior-changing companion; my JSONL run producing the full event sequence with matching turn_id/tool_call_id/api_call_count; the NormalizedResponse serialization failure in assistant_message dropping events silently.

Inference: the read-only/fail-open design is a deliberate safety boundary — an observer is the only piece of agent code you can run in production with zero blast radius, and the split from middleware means “watch” can never secretly become “intervene.” The cost is that observability failures are invisible by construction, which is why the liveness check matters more than the exporter.

Open questions: whether the legacy assistant_message compatibility field will be dropped in a future version (the docs already steer consumers away); whether the public docs will ever surface the full observer contract beyond the repo README; and whether Hermes will ship a first-party JSONL/OTLP exporter so users get the audit trail without writing a plugin — the contract is the hard part, and it is already done.

An agent you cannot replay is an agent you cannot debug, bill, or defend. Hermes solved that problem with a contract most users never hear about — and the reason it works is precisely what makes it hard to discover: an observer that cannot touch the loop is the only instrumentation you will ever trust to watch it.