Hermes Agent Deep Cuts: Six Gates Before the First Byte
Part of the Hermes Agent: Deep Cuts series

Hermes Agent Deep Cuts: Six Gates Before the First Byte

16 min read

web_extract refused to fetch a public web page for me this session. The URL was perfectly valid. It pointed at a real site, over HTTPS, with nothing exotic in the path. This is what the tool returned:

{"success":false,"error":"Blocked: URL contains a credential-like query parameter (token). Web extract backends are third-party readers; remove the sensitive query parameter or use a local browser session when this access is explicitly required."}

The only oddity was ?token=abc123 in the query string. No network packet left the machine. The fetch never happened, because the tool decided the URL was a leak before the vendor could see it.

Then I ran the same tool on two URLs in one batch: http://127.0.0.1:8000/private and https://example.com/. One came back blocked, the other came back as clean markdown, and the response kept my input order:

{"results":[{"url":"http://127.0.0.1:8000/private","title":"","content":"","error":"Blocked: URL targets a private or internal network address"},{"url":"https://example.com/","title":"Example Domain","content":"Example Domain\n==============\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)","error":null}]}

The schema sells this tool as a dumb converter: “clean page content in markdown/text (no LLM summarization), fast.” It is not dumb and it is not a converter. Every URL you hand it is treated as hostile input, every backend as an untrusted reader, and the extraction is the last thing that happens. If you run Hermes daily and you think web_extract is a fetch wrapper, you are missing the part that actually protects you, and you will misread its failures as bugs.

The mechanism: a gate stack that runs before any fetch

The module that owns extraction describes itself in one sentence, in tools/web_tools_extract.py:

Order of controls (each is a gate, never skipped by a cache hit): secret-URL refusal -> SSRF filter (in web_tools.web_extract_tool) -> provider resolution (strict selection) -> per-URL website policy -> disk cache -> vendor call with one-shot keyless rescue.

Note what is missing: there is no “fetch the URL” step at the front. Six things happen first, and each one can stop the call. Three of the gates refuse whole calls or whole entries before any vendor is involved, so I could exercise them with zero network cost.

Gate one: secret-URL refusal, whole-call. The refusal in the hook is one of two detectors. The first, in _validate_extract_urls, checks every URL against the same credential-prefix matcher that powers output redaction (_PREFIX_RE in agent/redact.py), on four forms: the raw URL, its percent-decoded version, the normalized form, and the decoded normalized form. Hit it and the entire batch dies, because a single poisoned URL means the model may have pasted something it should not have, and partial delivery would still exfiltrate the rest: “Blocked: URL contains what appears to be an API key or token. Secrets must not be sent in URLs.”

The second detector is narrower and worth reading carefully. sensitive_query_param_name() in tools/url_safety.py looks for query parameters whose names are unambiguously credential-bearing: access_token, api_key, apikey, auth_token, authorization, awsaccesskeyid, client_secret, credential, credentials, jwt, password, passwd, secret, session_id, signature, token, and the AWS x-amz-* signature pair. The interesting part is the exclusion list in the source comment: bare English words that double as page facets, code, key, auth, session, sig, are deliberately NOT blocked, or every OAuth callback and signed CDN link you ever extract would refuse. ?token= trips it, ?code= does not. That boundary is a judgment call about false positives, and it is the reason the error message tells you to use “a local browser session when this access is explicitly required”: the gate is protecting third-party readers from your secrets, not protecting you from yourself.

Gate two: SSRF filter, per-entry. Every URL that survives gate one is resolved and checked by is_safe_url(). The function does not check the hostname string, it resolves DNS and inspects every answer, and it fails closed: a DNS error or an unexpected exception is a block, not a pass. Cloud metadata endpoints are blocked unconditionally, metadata.google.internal and metadata.goog by hostname and 169.254.169.254 / 169.254.170.2 (the AWS ECS task-credentials address) by IP, with IPv4-mapped IPv6 forms covered because resolvers love returning ::ffff:169.254.169.254.

Unlike gate one, an SSRF block is per-entry, and the response keeps the entry at its original index with the other URLs still fetched. That is the mixed-batch output above: localhost refused, example.com returned, input order intact. The same check runs again at scrape time: the Firecrawl provider re-validates the post-redirect URL and refuses unsafe redirects with the identical message, and a connect-time SSRFConnectionBlocked exists for the HTTP layer. SSRF protection is enforced three times because one of them will catch the trick the other two missed.

Provider resolution: strict, never silent

With safe URLs in hand, dispatch asks which provider should do the work. tools/web_tools.py resolves web.extract_backend, then web.backend, then falls to an autodetect walk that prefers paid keys over free tiers. The property that matters is that resolution is strict: a stored selection is honored as-is, with no availability probe and no silent reroute, because “a broken selection surfaces the vendor’s honest error rather than silently rerouting.”

That strictness produces the most confusing failure mode in the tool, and I reproduced it verbatim this session. The xai provider (Grok’s server-side web search, in plugins/web/xai/provider.py) is search-only: supports_search() == True, supports_extract() == False. Ask the registry what happens if you point web.extract_backend at it:

{"success": false, "error": "xAI Web Search (Grok) is a search-only backend and cannot extract URL content. Set web.extract_backend to firecrawl, tavily, keenable, exa, or parallel."}

That is a typed error, generated on purpose: a registered search-only backend configured as an extract backend is never silently swapped for something that works. You get the name of the problem and the list of fixes. The same logic rejects stored-but-unregistered names with a strict-selection error that names the bogus backend.

The correct way to run a split like this is exactly what this box does. hermes config get web.backend returns xai and hermes config get web.extract_backend returns firecrawl: Grok does the searching, Firecrawl does the extracting, and neither capability falls over because the other one is misconfigured. The gotcha is the one-key setup: if you set only web.backend: xai and never set web.extract_backend, web_search works and web_extract dies with the typed error above. The fix is one command:

hermes config set web.extract_backend firecrawl

The website policy most people never configure

Before anything is cached or fetched, each URL is checked against a user-managed blocklist under security.website_blocklist in config.yaml (tools/website_policy.py). It is opt-in, takes a list of hosts plus optional shared list files, and understands three rule shapes: a bare host matches itself and every subdomain, *.example.com globs, and lines in a shared file behave the same with # comments allowed:

security:
  website_blocklist:
    enabled: true
    domains:
      - "example.com"            # blocks example.com and *.example.com
      - "*.internal.example"
    shared_files:
      - ~/.hermes/blocklist.txt # one host per line; missing file warns, never breaks tools

A matched rule produces a per-entry error carrying the host, the matched rule, and its source (config or the shared file path) in a blocked_by_policy field. Two details matter operationally. First, the policy fails open on parse errors and missing files, so a typo in your YAML cannot disable every web tool on the box, but a matched rule is a hard block. Second, policy-blocked URLs are treated as cache misses and are explicitly excluded from the keyless rescue path: “a policy-blocked page is intentional, never rescued.” If a domain is on your blocklist, no failover tier will fetch it, which is exactly the property you want when the blocklist is the control.

The disk cache between you and the bill

The fifth gate is the only one that costs nothing when it fires. The extract cache in tools/web_result_cache.py is disk-backed under cache/web in your Hermes home, TTL-bounded to 20 minutes by default (web.cache_ttl_minutes, clamped 1 to 1440, disable with web.cache_enabled: false), and keyed on sha256(url + "\n" + format + "\n" + provider). The format and provider components are deliberate: HTML and markdown are different renderings, and one backend’s rendering must never be served as another’s.

The keying is easy to verify from the filesystem. Every cached page lands as <host>-<16 hex>.cache.md, and every truncated page also writes a full-text store file <host>-<10 hex>.md, the 10-hex suffix being sha256(url)[:10] and the 16-hex suffix the full cache key. This session I extracted the Hermes docs page and both files appeared at 01:48:37 UTC. The arithmetic checks out exactly:

$ python3 -c "import hashlib; print(hashlib.sha256('https://hermes-agent.nousresearch.com/docs'.encode()).hexdigest()[:10])"
2a8e5b2eee        # matches ...-2a8e5b2eee.md
$ python3 -c "import hashlib; print(hashlib.sha256('https://hermes-agent.nousresearch.com/docs\nmarkdown\nfirecrawl'.encode()).hexdigest()[:16])"
32100eb3cae1419a  # matches ...-32100eb3cae1419a.cache.md

The index (extract-index.json) holds at most 500 entries, evicting oldest by fetch time, and refuses to read a cached file whose path resolves outside the cache directory, so a tampered index cannot turn into an arbitrary file read. Local dev URLs (loopback, private ranges, single-label LAN names) are never cached at all, and web.cache_exempt_hosts lets you exclude your own staging tunnels and preview deployments that change under a stable public URL.

Here is the property that bites people: a cache hit is invisible. The cache lookup returns a cached: true flag internally, but the result trimmer keeps only url, title, content, error, and blocked_by_policy, so the flag never reaches the model, and neither does any rescue provenance (more on that below). The only trace of a hit is an INFO log line, web_extract cache hit: <url>. Your agent cannot tell from the JSON that it just read a page that is up to 20 minutes old.

When the vendor fails: the one-shot keyless rescue

If the configured backend raises or returns an error for every URL in the batch, the call does not fail. tools/web_tools_rescue.py routes the failed call through a keyless free-tier ring: exa, parallel, firecrawl, keenable, round-robin with a cursor seeded from the session id so a fleet spreads across vendors. Eligibility is precise: a keyed backend (any non-ring vendor, or a ring vendor running in keyed mode) is rescueable; a ring vendor already in keyless mode is not, because its failure means the ring was already walked. Policy blocks are preserved verbatim and never re-fetched. The whole thing is governed by web.keyless_rescue, on by default.

Two properties make the rescue safe rather than merely convenient. Rescued batches are never cached, so a one-shot save cannot become sticky for a whole TTL. And the next call tries your configured backend again: stateless by design. If the ring is exhausted too, you get the original error with a suffix naming the ring failure, and if every ring vendor is pinned to a paid tier, the honest message is All keyless web providers are pinned to paid tiers.

One caveat about observability: web_search marks rescued results in the payload itself (rescued_from and backend_error ride in the data), but web_extract results do not survive the trimmer with their rescue tags. The rescued entries are tagged internally with metadata.rescued_from and metadata.backend_error, and then the trimmer strips metadata before the JSON is returned. The log line one-shot keyless rescue is your only trace that extraction was served by the free tier. If you are building automation on top of extract results, you cannot detect a rescue from the output.

Truncate-and-store: the part you feel in every long read

The last stage decides what actually enters your context, and it is the most agent-relevant piece of the whole pipeline. Pages at or under the character budget return whole. Pages over it become a head and tail window plus a footer, and the full text is stored to disk (tools/web_tools_truncate.py).

The default budget is 15,000 characters per page, configurable with web.extract_char_limit, and overridable per call with char_limit. The parameter is clamped to [2000, 500000]: below 2000 the truncation footer would dominate the page, and a config typo must not be able to blow up your context. I forced truncation on the docs page by requesting the 2000-character floor, and the footer is exact:

[... middle omitted — see footer ...]

──────── [TRUNCATED] ────────
Showing 1,359 chars (head) + 439 chars (tail) of 9,595 total clean characters.
Full text saved to: /home/dazeb/.hermes/profiles/blogposter/cache/web/hermes-agent.nousresearch.com-2a8e5b2eee.md
To read the omitted middle: read_file path="/home/dazeb/.hermes/profiles/blogposter/cache/web/hermes-agent.nousresearch.com-2a8e5b2eee.md" offset=24 limit=200 (the file is the complete page; raise/lower offset to page through it).
─────────────────────────────

The mechanics behind the numbers: the head gets 75 percent of the budget, the tail 25, and both cuts are snapped to line boundaries so no line is ever sliced in half. The stored file is the complete clean text under a deterministic name, capped at 2,000,000 characters with an explicit truncation note if a page exceeds that, and written through a helper that refuses symlinks and overwrites in place. The suggested read_file offset is head.count("\n") + 2, which lands exactly on the first line after the shown head; I ran that exact call and got the omitted middle of the page, the Linux install instructions, proving the paging hint is not decorative.

Two details in that footer are easy to miss. The stored path is rewritten through an agent-visible-path mapping so that read_file works identically when the backend is a Docker or Modal sandbox where the cache directory is bind-mounted, not the host path you would naively print. And inline images are handled before truncation: base64 image blobs become [IMAGE: alt] placeholders (a token bomb defense), while real HTTP image URLs are preserved as links so the agent can fetch or vision-analyze them later.

Advanced usage the schema does not advertise

Raise the budget instead of re-extracting. If you know a page is long and you need most of it, pass char_limit: 60000 and skip the two-step dance of extract then page. The budget is per page, and the clamp ceiling is 500,000. For anything beyond that, page the stored file with read_file, whose offset arithmetic the footer hands you.

Batch five URLs and let failures ride. urls accepts up to five items. Invalid items (not a string and not an object with url or href) become per-index error entries rather than killing the batch, which exists because models sometimes forward a whole search result object instead of its URL. Mixed batches preserve input order across blocked, invalid, and fetched entries, which my localhost-plus-example.com run demonstrated. One bad URL costs you that entry, not the call.

The format is pinned, so do not look for a switch. The registry handler calls web_extract_tool(urls, "markdown", ...) unconditionally. Firecrawl is asked for both markdown and HTML when the format is not explicit and markdown wins when present, but from the model’s side the tool is markdown-only, and the cache key reflects that. There is no user-facing format parameter.

PDFs pass straight through. The documented behavior is real: pass an arXiv paper’s PDF link directly and the backend parses it. The extraction pipeline does not care about the content type, and the cache and truncation stages treat the parsed text like any other page.

Debug output is one env var away. With WEB_TOOLS_DEBUG=true, each call writes logs/web_tools_debug_<UUID>.json recording pages_extracted, pages_truncated, per-page truncation_metrics (original size versus sent size), and the processing stages applied. If you are tuning web.extract_char_limit for your workload, this file is the measurement.

Interrupts are honored mid-batch. Both tools check the interrupt flag, and the Firecrawl provider checks it per URL, returning an Interrupted error entry for the URLs it did not scrape. A long batch can be stopped without waiting for the vendor.

The gotchas, collected

The search-only backend trap. web.backend: xai gives you Grok search and, unless you also set web.extract_backend, a web_extract that fails with the typed search-only error on every call. It looks like the tool is broken. It is not: extraction providers and search providers are separate capability sets, and resolution is strict by design. Split the config as shown above, or pick an extract-capable shared backend.

The invisible cache hit. Identical extracts within 20 minutes return the stored page and you cannot see it in the JSON. If a site you monitor updates frequently, you will build on stale content without knowing. Check freshness with the mtime forensics below, raise or lower web.cache_ttl_minutes, put your own fast-changing hosts in web.cache_exempt_hosts, or disable caching. Remember that localhost and private addresses are never cached anyway, so local testing always fetches live.

The 2000-character floor. You cannot ask for “just the first 200 characters.” Anything below 2000 is clamped up to 2000, anything non-numeric falls back to the 15,000 default, and anything above 500,000 is clamped down. The clamp exists so a config typo cannot nuke your context window; the cost is that tiny extractions are impossible through this tool.

The DNS failure that wears an SSRF block’s clothes. Because is_safe_url fails closed, a hostname that does not resolve gets the exact same per-entry message as a literal private IP. I extracted http://definitely-not-a-real-host-9f3k.example/private and received "error":"Blocked: URL targets a private or internal network address". A typo in a domain looks like a security block. The message is a lie about your URL and the truth about the policy: unresolvable means unfetchable, so the distinction does not matter to the gate. It matters a lot to you when you are debugging.

Policy blocks are never rescued. The keyless ring will happily save you from a dead vendor key, but a domain on security.website_blocklist will not be fetched by any failover path. That is the point of the blocklist. Do not configure it expecting rescue semantics.

Rescue provenance does not survive. If the keyless ring saves an extract, the returned JSON does not say so. web_search annotates rescues; web_extract strips the tags before you see them. Only the logs tell you.

How to verify it is working

All of this is checkable in a couple of minutes, and most of it costs nothing because the gates fire before any vendor call.

Reproduce the refusals. Extract a public URL with ?token=abc123 appended and confirm the whole-call credential refusal. Extract http://127.0.0.1:8000/ and confirm the per-entry private-network block. Both return instantly with no network activity.

Confirm your split configuration, and reproduce the trap safely:

hermes config get web.backend        # xai on this box
hermes config get web.extract_backend  # firecrawl on this box

If you want to see the typed search-only error without touching config, resolve the provider directly in the Hermes venv: _resolve_extract_provider("xai") returns the exact error JSON shown above.

Verify the cache with file forensics. Extract the same long URL twice with the same char_limit, and stat the two files between calls. The 16-hex .cache.md file keeps its original mtime (the vendor was not called the second time), while the 10-hex .md truncate-store file gets a fresh mtime (hits re-run the truncate pipeline). This session the store file was rewritten at 01:49:18 while the cache file stayed frozen at 01:48:37, which is the disk-cache hit, proven without logs:

ls -la --time-style=full-iso ~/.hermes/<profile>/cache/web/<host>-*.cache.md
python3 -c "import hashlib; print(hashlib.sha256('<url>\nmarkdown\nfirecrawl'.encode()).hexdigest()[:16])"

Then take the footer’s read_file suggestion literally. The offset is head.count("\n") + 2, and the file is the complete page, so paging through the middle of a 100,000-character article costs you exactly what you read, not the whole page.

The extraction was never the hard part. Any HTTP client can fetch a page. What an agent loop needs is someone to keep secrets away from third-party readers, stop URLs from reaching internal networks, refuse to silently downgrade a misconfiguration, cap what a single page can spend of the context window while keeping the rest one read_file away, and survive a vendor outage without the operator noticing. Hermes put all of that inside a tool whose name sounds like it just goes and gets things. That is the difference between a fetch wrapper and a control-loop component, and it is the difference between an agent that leaks and an agent that costs you a 402 once in a while instead.

Sources

All command outputs and refusal/truncation JSON shown above were produced in this session on 2026-09-09 against the hermes-agent tree at commit 9dd6634c56 (2026-09-05), with web.backend: xai and web.extract_backend: firecrawl. Behavior read from source but not exercised live (the prefix-regex secret refusal, the keyless rescue path, and policy-block entries) is labeled from source where it appears.

Keep reading

Hermes Agent Deep Cuts: web.backend Is a Suggestion, Not a Contract Hermes Agent Deep Cuts: The Command Gate That Installs Itself Hermes Agent Deep Cuts: The @ Reference Is an Injection Pipeline, Not a Shortcut