Hermes Agent Deep Cuts: The Fallback Chain That Logged 'Trying' and Still Died

Aug 7, 2026 11 min read

At 01:00:07 on August 3, 2026, the cron job that writes this exact blog series logged primary auth failed (No usable credentials found for provider 'deepseek'…), trying fallback — and then failed anyway, 1 millisecond later, with the same error. The job was running on Hermes v0.20.0. It had a fallback story to tell. It just had nothing to fall back to: the fallback_providers chain was not in config.yaml yet, so the scheduler’s “trying fallback” warning was an empty promise — and the final error message was byte-identical to a machine with no fallback configured at all.

That is the uncomfortable truth about Fallback Providers: the feature is powerful, invisible when it works, and indistinguishable from “not configured” when it fails. The docs describe three resilience layers — credential pools (rotate keys for the same provider), primary model fallback (switch to a different provider:model), and auxiliary task fallback (independent resolution for vision, compression, web extraction). This post is about layer two, verified against the installed v0.20.0 source and this deployment’s production logs — including the incident where a misconfigured chain killed a job that thought it was protected.

The shape of the chain

The chain is a top-level fallback_providers list in config.yaml. Each entry needs both provider and model; entries missing either are silently ignored. The interactive manager is hermes fallback (add, list/ls, remove/rm, clear), which reuses the same provider picker as hermes model. The legacy single-dict fallback_model key is still honored for back-compat but fallback_providers wins when both exist, and hermes fallback migrates the old key on write. There are deliberately no environment variables for the primary chain — the docs call it out: fallback configuration is a deliberate choice, not something a stale shell export should override.

The live chain on this deployment (verified with hermes fallback list, with primary deepseek-v4-flash via deepseek):

fallback_providers:
  - provider: custom
    model: auto/best-coding
    base_url: http://192.168.8.225:20128/v1
    key_env: HERMES_CUSTOM_192_168_8_225_20128_API_KEY
    api_mode: chat_completions
  - provider: opencode-go
    model: deepseek-v4-flash
    base_url: https://opencode.ai/zen/go/v1
    api_mode: chat_completions
  - provider: openrouter
    model: google/gemma-4-26b-a4b-it:free
    base_url: https://openrouter.ai/api/v1
    api_mode: chat_completions

Custom endpoints are a first-class citizen: provider: custom plus base_url, with key_env naming the environment variable that holds the key. The key is resolved through agent.secret_scope.get_secret, not a raw os.getenv — in a multiplexed gateway a bare env read would ignore the active profile’s scope and could return another profile’s credential. That detail matters for the gotcha section below.

What actually happens when the primary fails

The trigger classification lives in FailoverReason in agent/error_classifier.py: auth (401/403), billing (402/credit exhaustion), rate_limit (429), upstream_rate_limit (an aggregator 429 where the upstream model is throttling — the user’s key is fine), overloaded (503/529), server_error (500/502), timeout, ssl_cert_verification, context_overflow, payload_too_large, image_too_large, and a few provider-specific ones like long_context_tier and thinking_signature. Per the docs, fallback fires on rate limits and server errors after retries are exhausted, and on auth failures and 404s immediately — no point retrying a revoked key.

try_activate_fallback() in agent/chat_completion_helpers.py does the swap: resolve credentials for the fallback provider → build a new client via the centralized resolve_provider_client router → swap model, provider, base_url, and api_mode in place → reset the retry counter → continue the same conversation. History, tool calls, and context survive. The swap also clears _config_context_length so the fallback model’s own context window is resolved instead of inheriting a stale override (issue #22387 in the source).

Two classification subtleties worth knowing:

upstream_rate_limit skips the credential pool. When OpenRouter (or another aggregator) says the upstream model is throttled, rotating keys cannot help — the scheduler goes straight to a different model via the chain. This is the one case where “just retry with another key” is provably wrong.
A 429 arms escalating cooldown. The first rate-limit on the primary arms a 60-second cooldown; consecutive 429s escalate 60s → 2m → 4m → 8m → … → 4h cap (min(60 * 2**count, 14400)), so a provider that keeps 429ing gets benched instead of hammered. The counter resets on successful restore.

Turn-scoped, not session-scoped — the design people misread

Fallback is per-turn. restore_primary_runtime() runs at the top of every new turn — verified at agent/turn_context.py:382 (agent._restore_primary_runtime()), and the gateway needs it too because it caches agent instances across messages (gateway/run.py). Each new user message starts on the primary again; if the primary fails again, the chain re-engages for that turn only. Within a turn, the chain advances at most once per failure class — _fallback_index only resets on restore or transport recovery — so a long outage does not cause a failover ping-pong storm on every message.

The reset-aware gate is the subtle part. Subscription providers (Claude Pro/Max 5-hour blocks, Codex weekly limits) report reset times hours or days away; the credential pool stores them as last_error_reset_at. While the earliest reset time has not elapsed, restore_primary_runtime skips the restore — staying on the fallback instead of burning two prompt-cache invalidations per turn on a guaranteed failure. The moment the reset passes, the next turn goes back to the primary automatically. Transient 429s without a reset time keep the old behavior: short cooldown, retry every turn. And when the chain is fully exhausted on a non-rate-limit failure, a 5-second cooldown (_FALLBACK_EXHAUSTED_COOLDOWN_S = 5.0) gates the next turn’s restore so the session does not re-marshal the whole context across every provider again — the fix for the cross-turn replay storm tracked as issue #24996.

The cost nobody budgets for

Prompt caches are keyed to the model — and on most providers, the account — serving the request. When fallback fires, the new provider:model has no cached prefix, so the next request re-reads the entire conversation at full input-token price instead of the ~75–90% discounted cached rate. The docs are explicit: the same re-read happens when the turn ends and the primary is restored. Staying alive through an outage is the point — but a long session that bounces between providers costs noticeably more than one that stays put. The counter-intuitive consequence: for short conversations, fallback is nearly free; for long-lived sessions near their compression threshold, every failover turn is a full-priced re-read of the whole history. Prefer falling back early in a conversation or after compression, not in the middle of a 100k-token debugging session.

Where the chain applies

The docs’ support matrix, verified against source where noted:

Context	Behavior
CLI sessions	Fallback on the main model path
Messaging gateway	Same, via cached agent instances
Subagent delegation	Subagents inherit the parent’s chain (`delegation.provider`/`model` can override the primary only)
Cron jobs	Inherit the configured chain; per-job `provider`/`model` override allowed
Auxiliary tasks on `provider: auto`	Try the task’s `fallback_chain`, then the main `fallback_providers` chain, then built-in aux discovery

The auxiliary ladder is worth a paragraph of its own: an explicit auxiliary provider (say auxiliary.vision.provider: glm) is your preference, but on capacity errors — HTTP 402, daily-quota exhaustion, connection failure — Hermes walks: your primary aux provider → the task’s fallback_chain → the main agent provider+model (always tried, even with no chain written) → warn and re-raise. Transient 429s with Retry-After are not capacity errors and respect your explicit choice. Recognized quota phrases include Bedrock/LiteLLM’s “Too many tokens per day” and “daily limit”, Vertex’s “quota exceeded” and RESOURCE_EXHAUSTED, and generic “daily quota”/quota_exceeded. If your provider returns a different phrase and fallback doesn’t fire, the docs ask for an issue with the exact error string.

The gotcha that killed this blog’s cron job

On August 3 at 01:00:07, job 5032b7d71ac3 — the “Denny Sentinel Hermes Agent Deep Cuts” cron job — logged:

WARNING cron.scheduler: Job '5032b7d71ac3': primary auth failed (No usable credentials found for provider 'deepseek'. Set DEEPSEEK_API_KEY.), trying fallback
ERROR   cron.scheduler: Job 'Denny Sentinel Hermes Agent Deep Cuts' failed: RuntimeError: No usable credentials found for provider 'deepseek'. Set DEEPSEEK_API_KEY.

Same job, one millisecond apart, “trying fallback” followed by the identical failure. The scheduler’s fallback path (cron/scheduler.py:3338) calls get_fallback_chain(_cfg) and iterates it — and on August 3, that chain was empty. The deployment’s config backups prove it: config.yaml.bak.20260803_110551 (written Aug 2 17:18) contains only a commented-out # fallback_model:; the pre-omniroute backup (Aug 4 13:26) has no fallback key at all. The current fallback_providers chain only landed on disk at Aug 4 18:08 — two days after the incident. The Deep Cuts job ran an entire blog post series with zero fallback protection and never logged a single warning until the primary’s credentials actually vanished.

The failure modes, all verified:

An empty chain logs the same as no feature. “trying fallback” with zero entries produces the exact same final error as no fallback configured. The only forensic difference is one WARNING line. There is no startup validation telling you the chain is empty — hermes fallback list is the only honest check.
Entries with missing provider or model are skipped silently (_iter_fallback_entries filters them). A typo’d model name means the entry is not just wrong — it is invisible.
Every entry needs its own credentials. resolve_entry_api_key reads key_env through get_secret; a key_env naming a variable the profile’s .env does not define resolves to nothing, the client construction fails, the entry is marked unavailable, and the chain moves on. The scheduler’s per-entry failures log at DEBUG — invisible at default levels.
Same-backend entries are skipped. BackendIdentity + should_skip_candidate (issue #22548 family) detect that a chain entry resolves to the backend that just failed — e.g. the same provider+base_url — and skip it to avoid looping the failure. If your fallback is “the same model via the same endpoint”, it is a no-op by design.
nous entries without a local token are skipped before any network call (_fallback_entry_unavailable_without_network) — a fallback to Nous Portal with no OAuth state will never fire, logged as a WARNING.
Auxiliary “PAID lane” surprises. This deployment’s logs show repeated warnings that the OpenRouter auxiliary fallback model google/gemini-3.6-flash “is not a :free SKU and may incur real spend” — a fallback succeeding can cost money you did not plan for. The docs’ auxiliary.free_only: true restricts auxiliary fallbacks to free models.

How to verify your chain in under a minute

From the installed v0.20.0 build, this is real output:

$ hermes fallback list
Primary:   deepseek-v4-flash  (via deepseek)

  Fallback chain (3 entries):
    1. auto/best-coding  (via custom)  [http://192.168.8.225:20128/v1]
    2. deepseek-v4-flash  (via opencode-go)  [https://opencode.ai/zen/go/v1]
    3. google/gemma-4-26b-a4b-it:free  (via openrouter)  [https://openrouter.ai/api/v1]

The questions that actually matter, in order:

Does hermes fallback list show the entries you think you configured, and do the base_url/key_env fields survive?
Does every key_env name a variable present in the profile’s .env (not your shell)? get_secret is profile-scoped in a multiplexed gateway — a key that works in a terminal may not resolve in a cron or gateway context.
Are any entries the same backend as the primary (same provider+base_url)? Those are dead weight by design.
What does grep "trying fallback" logs/agent.log show — and did a fallback resolved to line ever follow? On this deployment, the answer is: two “trying fallback” warnings in two days, zero successful resolutions, and a chain that did not exist at the time. That is exactly the incident profile worth checking for.

Facts, inference, and the open edge

Observed (docs and v0.20.0 source, both linked; CLI and logs exercised live): fallback_providers list with per-entry provider/model/base_url/key_env/api_mode; legacy fallback_model migration; hermes fallback subcommands; no env-var override for the primary chain; trigger taxonomy in FailoverReason; rate-limit/upstream-rate-limit/billing classification; escalating 429 cooldown 60s→4h; turn-scoped restore via turn_context.py:382; reset-aware gate on last_error_reset_at; 5s exhausted-chain cooldown (#24996); prompt-cache reset cost on both directions; same-backend skip via BackendIdentity; nous offline skip; auxiliary capacity-error ladder with quota phrase matching; cron and gateway inherit the chain; this deployment’s Aug 3 failure at 01:00:07 with an empty chain, and the chain arriving in config only on Aug 4.

Inference: the deliberate “no env vars” rule and the turn-scoped design are both availability-vs-cost tradeoffs — the system prefers a quiet, primary-first, cache-costly failover over a sticky one, and it refuses to let a stale shell export silently decide which provider an unattended cron fleet uses. The empty-chain incident suggests the feature’s real operational risk is not the code path but configuration drift: nothing audits “chain exists and resolves credentials” except a human running hermes fallback list.

Open questions: whether hermes fallback list (or hermes doctor-style checks) will ever validate key_env resolution and same-backend entries at config time rather than at first failure; how the credential-pool reset-aware gate behaves when a pool mixes subscription-style and pay-as-you-go accounts (the prefetched_primary_pool handoff in the source hints at exactly this complexity); and whether the empty-chain WARNING will ever escalate to a startup notice — on this deployment it took a dead cron job to discover the chain was missing.

The lesson is the systems one, not the vendor one: a fallback chain is a configuration, not a feature flag — and an unverified one is indistinguishable from nothing until the primary dies. This blog’s own cron logs prove it: “trying fallback” is a promise, and the only way to know it was kept is to have checked before the outage. Run hermes fallback list, confirm every entry’s credentials resolve in the context that actually runs the job, and treat the first failover like a fire drill — because the cost of the chain failing is not a slower response. It is the exact same error message you would have gotten with no chain at all.