Hermes Agent Deep Cuts: The terminal Tool Is a Shell Session, Not a Command Runner

Aug 26, 2026 9 min read

export PERSIST_TEST=snapshot-works in one tool call, then echo $PERSIST_TEST in the next. The variable is there. cd /tmp, then run any later command: it starts in /tmp. Two separate shell processes, because every single terminal call spawns a brand-new bash, and the state carries anyway.

The trick is not a persistent shell. The trick is a file. Look at /tmp after any session has run a few commands:

-rw------- 1 dazeb dazeb 11309 Aug 26 00:09 /tmp/hermes-snap-06121ed33651.sh
-rw------- 1 dazeb dazeb  9213 Aug 26 00:09 /tmp/hermes-snap-589250d16db2.sh

Those are session snapshots, mode 0600, one per environment. Every command is wrapped in a script that sources the snapshot, cds to the recorded directory, runs your command, then dumps the entire environment back into that same file under umask 077 before exiting. Spawn-per-call execution with file-based state: that is the whole mechanism, and it explains the weird behavior you have seen from the terminal tool.

The mechanism: spawn-per-call with a snapshot file

LocalEnvironment is a subclass of BaseEnvironment in tools/environments/local.py, and its class docstring says it plainly: “spawn-per-call: every execute() spawns a fresh bash process. Session snapshot preserves env vars across calls. CWD persists via file-based read after each command.”

The lifecycle, traced in the installed v0.20.5 source:

Environment creation. On the first terminal call, terminal_tool() lazily builds one LocalEnvironment for the task id and caches it in _active_environments. The constructor calls init_session(), which runs a login bash (bash -l) that captures the full interactive environment into /tmp/hermes-snap-<12-hex-id>.sh: exported vars, declare -F function definitions (private _-prefixed functions filtered by name, not by line), alias -p, shopt -s expand_aliases, set +e, set +u. The file is assembled in a mktemp-allocated temp and published with mv -f, so concurrent writers can never source a half-written snapshot (#38249).
Every command is wrapped. _wrap_command() builds a script that sources the snapshot (source /tmp/hermes-snap-...sh >/dev/null 2>&1 || true, the redirect is there because macOS bash 3.2 leaks declare -x lines to stdout), exports the cross-agent AI_AGENT=hermes-agent and HERMES_AGENT=true markers, builtin cd -- <quoted cwd> || exit 126, runs your command with eval, captures __hermes_ec=$?, re-dumps the env vars back to the snapshot file atomically, and prints pwd -P inside a session-specific marker line so the parent can parse where the command ended up.
The cwd is a record, not a process. After a command that actually reported a cwd marker (cwd_observed), the result is written to an in-memory session-cwd record keyed by session. _resolve_command_cwd() is explicit: an explicit workdir= overrides everything; otherwise the session’s own cwd record wins; a session with no record yet runs in the config/TERMINAL_CWD default. That record is why cd in one call changes later calls, and why a workdir= override is transient by contract: recording it would hijack the session’s durable cwd for every later command.
The environment is shared across the process. One env per task id, cached for the lifetime of the process. On container backends the default task ids of top-level agent and delegate_task children collapse to "default" so subagents share one sandbox; under per-session isolation a register_container_alias() chain maps child task ids back to the parent’s container.

The local backend also sanitizes what the child sees. _build_provider_env_blocklist() strips every provider credential (OPENAI_API_KEY, ANTHROPIC_API_KEY, DEEPSEEK_API_KEY, XAI_API_KEY, FIRECRAWL_API_KEY, …), tool/messaging tokens, and gateway relay secrets from subprocess env; VIRTUAL_ENV, CONDA_PREFIX, and PYTHONHOME are stripped so a project’s uv/poetry never rebuilds into the Hermes venv; AUXILIARY_*_API_KEY/AUXILIARY_*_BASE_URL and GATEWAY_RELAY_*_SECRET dynamic names are removed unconditionally. General AWS credentials are deliberately left inheritable: SECURITY.md §3.2 treats the local terminal as the user’s trusted operator shell.

Advanced usage: what the flags actually gate

The model-facing surface is seven parameters in TERMINAL_SCHEMA (command, background, timeout, workdir, pty, notify_on_complete, watch_patterns), but three of them carry real machinery.

timeout is capped for foreground. FOREGROUND_MAX_TIMEOUT defaults to 600 seconds (env override: TERMINAL_MAX_FOREGROUND_TIMEOUT). A foreground call requesting more is rejected before execution with “Use background=true with notify_on_complete=true”. timeout=0 or negative is rejected outright: zero would silently become the default through timeout or default, and a negative value would sail into deadline = now + timeout and fire an immediate nonsense timeout.

background=true does not just fork a process. It routes to process_registry.spawn_local(), which wraps the command in _rewrite_compound_background(). That rewriter exists because bash parses A && B & as a subshell (A && B) &, and when B is long-running, the subshell holds the stdout pipe open forever (#68915). The rewriter turns it into A && { B & } so no subshell fork happens. The spawned worker is a login shell bash -lic "set +m; ..." with start_new_session=True; inside the supervised systemd gateway it also gets its own transient systemd scope, so an OOM in the worker kills the worker cgroup, not the gateway control plane (#70716).

Every tracked background session is checkpointed atomically to ~/.hermes/processes.json. On gateway startup, recover_from_checkpoint() re-validates each PID against its recorded kernel start time (so a recycled PID is never adopted and later tree-killed as a stranger) and re-attaches the survivors as detached sessions.

pty=true changes the exec shape. It spawns via ptyprocess with bash -lic "set +m; ...", giving interactive CLIs (Codex, Claude Code, REPLs) a real TTY. Two carve-outs: the schema says local and SSH backends only, and _command_requires_pipe_stdin() silently forces pty=false for gh auth login --with-token, because under a PTY process(submit) only sends a newline and the command waits forever for the EOF that piped stdin would have delivered.

The exit-code meaning table

The model-facing result of a foreground call is JSON: output, exit_code, plus optional annotations. Non-zero exit codes that are not errors get a human-readable exit_code_meaning:

$ grep nosuchstring /dev/null      # rc=1
exit_code_meaning: No matches found (not an error)

The table in _interpret_exit_code() covers grep/rg/ag/ack/find/diff/test/curl/git (1 = no matches or files differ, curl 6/7/22/28), and signal deaths map through _SIGNAL_EXIT_NOTES: negative codes (subprocess semantics, stated definitively) and 128+signum shell encodings (hedged with “usually”, since an application can legitimately exit 139). The note exists so the model does not burn turns investigating an expected grep 1, and 137 = OOM-kill is the one that matters operationally.

The gotcha: masked success

Watch what rc comes back as for a failed command piped through a truncator:

$ python3 -c "import definitely_not_a_real_module_xyz" 2>&1 | tail -3
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'definitely_not_a_real_module_xyz'
rc=0
hint: exit_code 0 here is the status of the last pipeline command
(tail/head/cat/...), NOT of the command before the pipe — and the output
contains failure indicators. Treat this run as FAILED until proven otherwise:
re-run the command WITHOUT the pipe ...

bash without pipefail reports the last pipeline command’s status, so cargo build 2>&1 | tail -20 returns tail’s 0 even when the build failed. Hermes does not fix the exit code: annotate_masked_success() in tools/terminal_hints.py attaches an advisory hint when BOTH the command shape can mask a failure (... | tail/head/cat/tee/less/more/wc/sort/uniq as the last segment, or || echo/|| true) AND the output carries a strong tool-specific failure shape (rustc, cargo, python tracebacks, pytest, gcc, npm ERR!, make). grep ... | head is excluded because search pipelines legitimately contain error text. The hint matters because the model treats exit_code 0 as a success signal, and the “full output is saved to a file” claim in the hint is real: _BoundedOutputCollector tees the full stream to a spill file, redacted, and the result carries full_output_path so you can page the middle instead of re-running.

The same terminal that annotated my probe commands has other guards you will hit as errors, not hints. A foreground command that looks long-lived is refused outright: my python3 -m http.server 8899 came back with status: error and “This foreground command appears to start a long-lived server/watch process. Run it with background=true…” (the detector is a curated regex list: npm run dev, docker compose up, uvicorn, python3 -m http.server, …). Shell-level background wrappers get the same treatment: nohup/disown/setsid and trailing & are rejected with instructions to use background=true instead, because the process registry is what makes the process killable, pollable, and checkpointed. And in a gateway, systemctl/hermes gateway restart-style lifecycle commands are hard-blocked inside the gateway process itself, because the SIGTERM would kill the very subprocess executing the command before it could complete.

The snapshot mechanism has its own failure shape. If the configured cwd was deleted by an earlier command, _resolve_safe_cwd() walks up to the nearest existing accessible ancestor rather than letting subprocess.Popen(cwd=...) throw before bash starts, which would wedge every later call until restart (#17558). And init_session() failure does not disable the tool: it falls back to bash -l per command, or to non-login bash -c when login bash itself is broken.

How to verify it is actually doing what you think

You do not need to trust the wrapper script. Verify the mechanism from the outside:

# 1. Environment persistence is file-based, not process-based
$ export PERSIST_TEST=snapshot-works    # first call
$ ls -la /tmp/hermes-snap-*.sh          # a new/updated 0600 file appears
-rw------- 1 dazeb dazeb 11309 Aug 26 00:09 /tmp/hermes-snap-06121ed33651.sh
$ echo $PERSIST_TEST                    # next call: snapshot-works

# 2. The cwd record follows cd, and only cd
$ cd /tmp && pwd                        # /tmp
$ pwd                                   # next call: /tmp
# a workdir= override must NOT change the session record

# 3. Exit-code meaning annotations
$ grep nosuchstring /dev/null           # rc=1, exit_code_meaning: No matches found

# 4. Masked-success detection
$ python3 -c "import nonexistent_mod" 2>&1 | tail -3
# rc=0 + hint telling you the pipeline masked the failure

# 5. Foreground guard
$ python3 -m http.server 8899
# status: error, "appears to start a long-lived server/watch process"

# 6. Background processes survive restarts via the checkpoint
$ ls -la ~/.hermes/processes.json       # atomic checkpoint, redacted commands

The one thing to remember when reading output: the wrapper sources the snapshot at the start of every call, so your exports persist, but a backgrounded process spawned with background=true does not run through _wrap_command(). It gets the registry’s sanitized env instead, with PYTHONUNBUFFERED=1 so its progress actually shows up in process(action="log"). Two different state models, one mental model of “the terminal remembers things.” That is the feature: a fresh shell per call plus a 0600 file that pretends to be a session, so you never notice the difference until the file goes away.

Facts, inference, and open questions

Observed (installed v0.20.5 source plus live runs in this session on 2026-08-26): LocalEnvironment is documented as spawn-per-call; init_session() captures env, functions, and aliases into /tmp/hermes-snap-<id>.sh via login bash; _wrap_command() sources the snapshot, cds, runs the command, re-dumps env under umask 077 with atomic mv, and emits a pwd -P marker; the session cwd is a per-session record updated only when cwd_observed is set and no workdir override is present; FOREGROUND_MAX_TIMEOUT is 600s (env-overridable); background spawns go through _rewrite_compound_background() (A && B & → A && { B & }), a login-shell bash -lic "set +m; ..." with start_new_session=True, and an atomic checkpoint at ~/.hermes/processes.json with PID start-time re-validation on recovery; _interpret_exit_code() annotates expected non-zero exits (grep 1, diff 1, curl 6/7/22/28, git 1) and signal deaths via _SIGNAL_EXIT_NOTES; annotate_masked_success() fires only when a masking command shape AND a tool-specific failure shape co-occur, and never modifies the exit code; the long-lived foreground guard refuses python3 -m http.server and friends with an error result; _sanitize_subprocess_env() strips provider keys and venv markers while preserving general AWS credentials by design (SECURITY.md §3.2).

Inference: the snapshot design is a portability trade: one state model that works identically across local, SSH, Docker, Modal, Daytona, Singularity, and Vercel backends, at the cost of exporting/re-importing the whole environment on every call. It is also why the docs’ shell-startup guidance (the non-interactive case $- in *i*) guard in .bashrc) exists: init_session() sources login rc files once, but every wrapped command sources the snapshot, and slow or TTY-expecting init code makes every single tool call slow or hung. The workdir-transient-by-contract behavior is a deliberate isolation choice, not an omission.

Open questions: I verified the 0600 snapshot, the cwd record, exit-code meaning, masked-success hints, and the foreground guard live in this session; I did not exercise pty=true, the systemd-scope isolation, or the checkpoint recovery path end to end (this box’s gateway is supervised, but no background process was killed mid-flight to prove re-attachment). env_passthrough and _HERMES_FORCE_ overrides exist as an escape hatch for blocklisted variables but were not tested here.