OctaFuse 2.3.0: keep upstream selection stable across requests

OctaFuse Gateway 2.3.0 is now available.

After failover finds a working upstream, sending the next request to the same Provider can materially improve Prompt Cache continuity and reduce unnecessary switching.

Version 2.3.0 adds Provider Sticky to Route Pools. The gateway can remember the last successful Target for an idle TTL, while Admin exposes binding distribution, per-user unbind, and Pool-wide invalidation.

In one sentence:

The four strategies decide how to route without a binding; Provider Sticky remembers the successful result.

Provider Sticky

Sticky is optional and off by default. The key point: sticky binding is evaluated before the four routing strategies. Once Sticky is enabled, a valid binding keeps the user’s requests on the Provider that was last allocated and succeeded—whether the Pool or priority tier uses hash_affinity, weighted_random, weight_priority, or weighted_round_robin. Tier strategies only run when there is no binding yet, or after a binding is cleared and routing must choose again. That is how a Route Pool can reliably get high Prompt Cache hit rates.

How it works:

1. Without a valid binding, routing still follows priority and the effective tier strategy.
2. A successful request records the Target for the user, model, route group, and protocol.
3. While valid, that Target is tried before normal priority tiers and in-tier strategies.
4. Each success renews the idle TTL; the default is 3600 seconds.
5. A 429, 401 / 403, 5xx, 524, or network failure clears the binding and continues normal failover.

Enable Provider sticky on a Route Pool and set its idle TTL. The dialog explains that the last successful Provider is tried first and other Providers are used after it fails.

Bindings live in D1, Postgres, or MySQL, so they are shared across Cloudflare Worker isolates and multiple Node instances. Storage failures fail open to the regular route plan instead of blocking inference.

Sticky versus hash_affinity

hash_affinity is the default stateless tier strategy. It gives the same user, model, route group, and protocol a stable preferred Provider.

Provider Sticky records the Target that actually succeeded:

• It can prefer that Target across priority tiers.
• It expires after an idle TTL.
• Provider-attributable failures clear it.
• A sticky_epoch bump can invalidate the entire Pool.

The two features work together. When no binding is valid, routing returns to hash_affinity or the strategy configured for that tier.

Topology and binding operations

Summary works well for a quick scan. When one model has many Providers split across multiple priority tiers, switch to Topology to see request surfaces, route groups, priority tiers, and failover order more clearly.

Topology keeps complex Route Pools readable when they contain many Providers and priority tiers, with strategy and sticky status visible in the same view.

Open the Provider sticky dialog from a Route Group or Route Pool node in Topology to:

• Enable or disable Sticky and set its idle TTL.
• Compare active binding share with each Target’s route weight.
• Look up a binding by user email or user ID.
• Unbind one user so their next request selects normally.
• Invalidate the entire Pool with an epoch bump.

After enabling Sticky, operators can compare binding share with route weight, look up individual user bindings, and invalidate all Pool bindings when traffic must be reassigned.

Topology also shows effective policies and priority tiers so operators can compare configuration with the actual failover order.

route_trace.sticky

Request logs can now include:

{
  "sticky": {
    "lookup": "hit",
    "attempted_target": "route-target-id",
    "result": "kept"
  }
}

• lookup records outcomes such as hit, miss, expired, invalid_epoch, or invalid_circuit.
• attempted_target is the bound Target tried before normal tiers.
• result shows whether the binding was kept, cleared, created, or rebound, and can report storage errors.

Compare these fields with cache-read tokens and upstream failover counts to verify that Sticky is improving cache continuity.

Strategy IDs now match Admin labels

Version 2.3.0 renames two strategy IDs:

2.2.0 ID	2.3.0 ID
cache_affinity	hash_affinity
fixed_order	weight_priority

weighted_random and weighted_round_robin are unchanged.

There are no aliases for the old IDs. Automation, ROUTE_STRATEGY, model route_policy, Route Pool defaults, and priority-tier overrides must use the new names. Writing an old ID returns 400.

Upgrade to 2.3.0

This release includes two database migrations:

• 0020 adds Route Pool Sticky columns and route_pool_sticky_bindings.
• 0021 rewrites persisted strategy IDs to hash_affinity / weight_priority.

When upgrading from 2.2.0:

1. Back up the database and pause Proxy traffic plus Admin configuration writes.
2. Apply 0020 → 0021 in order.
3. Immediately deploy Proxy, Admin, and migrate artifacts from the same release.
4. Verify that no old strategy IDs remain, then restore traffic.
5. Sticky stays off by default; enable it Pool by Pool and observe request logs.

Do not mix old and new Proxy / Admin versions during the maintenance window because migration 0021 has no old-ID compatibility aliases.

Get 2.3.0

• GitHub Release v2.3.0
• Full changelog
• Migration 0020: Provider Sticky
• Migration 0021: strategy IDs
• Routing guide

Learn more about OctaFuse or explore the source:

• OctaFuse website
• GitHub repository

If OctaFuse is useful to your project, consider giving the repository a Star. Your interest and feedback help us keep improving routing, failover, and operations.