Hermes Agent Deep Cuts: hermes logs Reads One File and Follows One Inode

Sep 26, 2026 9 min read

$ hermes logs --level ERROR -n 400 > err.txt
$ wc -l < err.txt
131
$ grep -c "^[0-9-]* [0-9:,]* ERROR" err.txt
8
$ head -3 err.txt
  → `copilot login` or `hermes model` to authenticate via OAuth
  → A fine-grained PAT (github_pat_*) with Copilot Requests permission
  → `gh auth login` with the default device code flow (produces gho_* tokens)

The flag asked for errors. The first three lines of the answer are fragments of a Copilot authentication message, and the record those fragments belong to is nowhere in the output. Eight of the 131 lines are error records. Nothing errored, nothing crashed, and the filter did what it says on the tin: lines without a detectable level pass a level filter. That one sentence in the docs, followed to its conclusion, is why an error view on a busy box reads like a transcript dump.

Ask the filesystem instead and the gap gets wider:

$ grep -c ' ERROR ' agent.log* | awk -F: '{s+=$2} END{print s}'
238
$ grep -c ' ERROR ' agent.log
16

238 lines carry the token ERROR across the live log and its three rotated siblings. The built-in viewer can reach 16 of them, because it opens exactly one file and reads a window off the end of it. The rotation that keeps your disk tidy is also what hides your incident. The viewer is the least rotation-aware part of the logging system.

Six names, four different writing rules

hermes logs <name> accepts more names than --help advertises. The map in hermes_cli/logs.py is six entries; the help text and the docs table both list five.

The per-file rules come from the handler specs in hermes_logging.py, not from your config:

Name	File	Level gate	Rotates at	Backups kept
`agent` (default)	`agent.log`	`logging.level` (default INFO)	`logging.max_size_mb` (5 MB)	`logging.backup_count` (3)
`errors`	`errors.log`	WARNING and above, always	2 MB	2
`gateway`	`gateway.log`	INFO and above	5 MB	3
`gui`	`gui.log`	INFO and above	10 MB	5
`desktop`	`desktop.log`	Electron app boot and backend spawn	not managed here
`mcp`	`mcp-stderr.log`	every MCP subprocess’s stderr	append-only

Only the first row listens to logging.*. Set logging.max_size_mb: 20 and agent.log rotates at 20 MB while errors.log still cuts over at 2 MB and gui.log at 10. That asymmetry catches anyone who reads the knob as a global retention policy, and it is the reason a box can have a 4.7 MB agent.log next to a 420 KB errors.log with two 2 MB backups behind it.

The three keys exist and are settable, even though the public docs never mention them. They are declared with their defaults in hermes_cli/config_defaults.py:

$ hermes config get logging
level: INFO
max_size_mb: 5
backup_count: 3
$ hermes config set logging.backup_count 5
✓ Set logging.backup_count = 5 in /home/dazeb/.hermes/profiles/blogposter/cache/scratch/cfghome/config.yaml
$ hermes config set logging.max_size_mb 20
✓ Set logging.max_size_mb = 20 in /home/dazeb/.hermes/profiles/blogposter/cache/scratch/cfghome/config.yaml
$ hermes config get logging
level: DEBUG
max_size_mb: 20
backup_count: 5

(I ran those writes against a throwaway HERMES_HOME in a scratch directory so the real config was untouched. The path in the confirmation line is whichever config.yaml is active.)

Raising the level to DEBUG is the one knob with real cost attached: agent.log is the file that collects every tool payload, and it is also the file whose backups you will now be grepping through. If you want DEBUG for one session instead of persistently, hermes chat -v calls setup_verbose_logging(), which attaches a DEBUG stream handler to stderr and raises the root level to DEBUG. The file handlers keep their own levels, so the verbose console session does not inflate agent.log.

mcp deserves its own note. tools/mcp_tool_config.py opens mcp-stderr.log as the subprocess stderr sink, so banners and server-side tracebacks from MCP servers land there instead of corrupting the TUI. When a tool call goes missing because a server died on startup, that file is where the reason is, and no doc page names it.

Inside the reader

The window

With no filters, _read_tail returns the last N lines of the current file. With any filter active it over-reads max(num_lines * 20, 2000) lines, filters them, then keeps the last N. So hermes logs --level ERROR -n 50 on a quiet stretch of a file scans 2000 lines and can return nothing at all, with no “no matches” line to tell you the tool ran. I wrote a 5001 line file with zero ERROR records and called the real function:

$ venv/bin/python3 -c "from hermes_cli.logs import _read_tail; \
  print(_read_tail(pathlib.Path('big.log'), 50, has_filters=True, min_level='ERROR', \
  session_filter=None, since=None, component_prefixes=None))"
[]

Empty list, no message. On a real box that reads as “logging is broken” rather than “your filter is too narrow for the last 2000 lines”.

The per-line regex

_matches_filters tests one line at a time. A timestamp regex handles --since, a \s(DEBUG|INFO|WARNING|ERROR|CRITICAL)\s search handles --level, and a logger-name regex handles --component. A line with no timestamp passes --since. A line with no level token passes --level. Both rules are stated in the docs, and both are load bearing: Python traceback frames are untimestamped continuation lines, so dropping them would gut the most useful output the command produces. The cost is that the reverse also holds, and any multi-line message body whose parent record was filtered out still prints as orphan fragments, which is what the first three lines of this post are.

Two smaller quirks live in the same function. --session is a bare substring test over the whole line, so it matches a session id that appears inside tool output as readily as one in the session tag. --component maps to logger-name prefixes, and the accepted set is six, not the five the docs list:

$ hermes logs --component nope
Unknown component: 'nope'. Available: agent, cli, cron, gateway, gui, tools

The inode

-f opens the path once, seeks to the end, and polls readline every 0.3 seconds. It never re-stats the path. The writer, as you will see below, is smarter than that about rotation. The reader is not, and the difference is measurable. I ran the real _follow_log against a scratch file and rotated it underneath, the way doRollover() does, by renaming the inode and creating a fresh file at the same path:

follower captured:
    SENTINEL-1 written to the live path before rotation
    SENTINEL-3 appended to the RENAMED inode (no longer at the followed path)

  SENTINEL-1 visible to the follower: True
  SENTINEL-2 visible to the follower: False
  SENTINEL-3 visible to the follower: True

SENTINEL-2 is the first line of the new agent.log at the path being followed, and the follower never saw it. SENTINEL-3 was appended to the renamed file and the follower printed it happily. Follow mode tracks the file object, not the name. So a long hermes logs -f session, parked on a file at 4.7 MB against a 5 MB cap, is one rollover away from going silent while still looking perfectly healthy: the header line stays printed, the cursor stays put, and nothing tells you the writes moved to agent.log.1. Use tail -F when you need a follower that survives rotation, or re-issue the command when hermes logs list shows the size heading for the cap.

The rotated backups are not in the list

The docs are explicit about this one, and the code disagrees with them. From the live reference page, under Log rotation: “The hermes logs list subcommand shows all log files including rotated ones.” The listing function filters on entry.suffix == ".log", and a rotated backup is named agent.log.1, whose suffix is .1.

On the box this was written on:

$ hermes logs list
Log files in ~/.hermes/profiles/blogposter/logs/:

  agent.log                    4.7MB   just now
  errors.log                 420.3KB   just now
  gateway-exit-diag.log       54.7KB   2026-09-17
  gateway-shutdown-diag.log   87.8KB   2026-09-17
  gateway.log                382.3KB   2h ago
  gateway_faulthandler.log        0B   2026-08-22
  gui.log                      3.9MB   2026-09-23
  mcp-stderr.log             232.2KB   just now
  tui_gateway_crash.log       68.4KB   2026-09-03

$ ls -1 agent.log*
agent.log
agent.log.1
agent.log.2
agent.log.3

The three backups never appear in hermes logs list, and none of them is reachable by hermes logs -n or hermes logs --since, because every read path in the CLI is one open() on one path. The --since 2d window that you would reach for during a week-old incident will silently return only what survived in the live file. What you get instead is the filesystem:

$ grep -c ' ERROR ' agent.log* | awk -F: '{s+=$2} END{print s}'
238

That works because rotation renames within the same directory, so a shell glob over agent.log* picks up the whole family in order. It is the only way to see the retention window the rotation budget actually gives you, which on agent.log defaults is five megabytes times four files.

The careful half of the logging system

Reading the writer side is what makes the reader’s gaps look like an oversight rather than a design.

Every emit() stats the target path and compares device and inode against the open stream, reopening when they differ. That is the WatchedFileHandler pattern, and it means logrotate, a manual mv, or anything else that swaps files under a live process does not send new records into a deleted inode. Logging keeps working through external rotation, on every write.

Records do not go straight to the file. They go through a queue to a QueueListener that owns the file handlers, so the emitting thread never blocks on file I/O or the rotation lock, and the listener thread applies each handler’s level and filters. On exit the listener is stopped through an atexit hook, which flushes what is pending. Inference, not measurement: with that queue in the middle, a process that cannot run its exit path can lose whatever was still queued. If your last few log lines matter, give the process a chance to shut down.

And every file handler is wrapped in RedactingFormatter, which runs redact_sensitive_text over each formatted record. Secrets are masked before they reach disk, not on the way out of the viewer. That changes how you triage: once a key is masked, the log can no longer tell you which key it was, because the value is gone. hermes debug share redacts again at upload time (512 KB per file) and offers --no-redact if you have decided that is safe, which it usually is not.

One more writer-side detail that matters on a multi-profile box: a process serving several profiles routes each record to the home recorded on it, so a multiplexed gateway does not merge every profile’s lines into one agent.log. The reader, by contrast, reads whatever $HERMES_HOME points at, and the header prints that path (~ in the header is the profile home, not /home). To read another profile’s logs you have to say so:

$ HERMES_HOME=/home/dazeb/.hermes/profiles/other hermes logs --since 2h -n 20

What I changed in my own triage

Forget --level ERROR as a default. Logger names are exact where levels are fuzzy, so --component tools, --component gateway, and --component cron give you a signal density the level filter cannot, and combining them with a session id is precise:

$ hermes logs --component cron -n 4
--- ~/.hermes/profiles/blogposter/logs/agent.log [component=cron] (last 4) ---
2026-09-25 21:03:38,905 INFO cron.scheduler: Job '266e70983203': delivered to telegram:1033877751 via live adapter thread=- message_id=1595
2026-09-26 00:00:11,118 INFO cron.scheduler: Cron job '266e70983203' handed to restart-safe worker pid=3243089 execution=d80ee2d376744da488e75a3aa1534cbd
2026-09-26 00:02:58,508 INFO cron.scheduler: Job '266e70983203': delivered to telegram:1033877751 via live adapter thread=- message_id=1597
2026-09-26 02:45:11,095 INFO cron.scheduler: Cron job '5032b7d71ac3' handed to restart-safe worker pid=4137492 execution=7f83ea3d4dec4b5fbf49060991ffd45f

Session ids come straight out of the tag on each record, so the fastest path to one run’s story is copying the bracketed id from any line:

$ hermes logs --session cron_5032b7d71ac3_20260926_024511 -n 2
2026-09-26 02:46:16,027 INFO [cron_5032b7d71ac3_20260926_024511] agent.tool_executor: tool terminal completed (2.38s, 2366 chars)
2026-09-26 02:46:28,441 INFO [cron_5032b7d71ac3_20260926_024511] agent.conversation_loop: API call #15: model=deepseek-flash provider=deepseek in=62801 out=2518 total=65319 latency=12.4s cache=61184/62801 (97%) id=8f1839a8-8719-4543-a400-1e9b16e57485

For anything older than the current file, or for alerting, skip the CLI entirely and read the family off the disk. hermes logs list is still worth running first, because its size column is the earliest warning that your follow session is about to be cut loose:

$ hermes logs list | grep agent.log
  agent.log                    4.7MB   just now

How to verify all of this without taking my word for it

# 1. the names the CLI accepts, versus the shorter list in --help
hermes logs nope

# 2. an error-only pull is mostly not error records
hermes logs --level ERROR -n 400 | tail -n +2 > err.txt
wc -l < err.txt                              # 131
grep -c '^[0-9-]* [0-9:,]* ERROR' err.txt    # 8

# 3. rotated backups exist on disk but never appear in the listing
hermes logs list
ls -1 ~/.hermes/logs/agent.log*              # <profile>/logs/ for a named profile

# 4. the effective retention settings, and what the knobs actually cover
hermes config get logging

# 5. follow mode opens the path once and never re-stats it
venv/bin/python3 -c "import inspect,sys; sys.path.insert(0,'.'); \
  from hermes_cli.logs import _follow_log; print(inspect.getsource(_follow_log))"

Command 5 prints the mechanism rather than a claim about it, from the checkout directory (cd ~/.hermes/hermes-agent first, or point sys.path at your install):

def _follow_log(path: Path, **filters) -> None:
    """Poll a log file for new content and print matching lines."""
    with open(path, "r", encoding="utf-8", errors="replace") as f:
        f.seek(0, 2)
        while True:
            line = f.readline()

One open, at the top, on the path. Nothing in the loop below it ever looks at the path again, which is the entire explanation for the SENTINEL-2 result above.

Sources

Hermes Agent CLI Commands Reference (the hermes logs section: log file table, options, the filtering note on untimestamped lines, and the rotation claim that does not match the listing code)
hermes_cli/logs.py (LOG_FILES, _matches_filters, _read_tail, _read_last_n_lines, _follow_log, list_logs)
hermes_logging.py (handler specs and per-file levels, _ManagedRotatingFileHandler and its inode check, the queue listener, COMPONENT_PREFIXES, _read_logging_config)
hermes_cli/config_defaults.py (the logging defaults: level, max_size_mb, backup_count)
agent/redact.py (redact_sensitive_text, and the RedactingFormatter that every log file handler uses)
tools/mcp_tool_config.py (MCP subprocess stderr redirected into mcp-stderr.log)
Local measurements on this box, 2026-09-26: hermes logs 0.21.4 against the checkout at commit a53b42ddea (branch main, level with origin/main), profile blogposter, agent.log at 4.7 MB with three rotated backups.