Spaces:
Running
Running
Commit History
-
NIM speed optimization — adaptive rate limiting and increased throughput
- Add AdaptiveRateLimiter with auto-backoff on 429s (starts at 100 req/min, backs off to 10, recovers gradually) - Increase NIM rate limit to 100 req/min with 40 max concurrency (configurable via NIM_RATE_LIMIT, NIM_MAX_CONCURRENCY) - Tighten NIM timeouts: connect=8s, first_chunk=20s, fallback_first_chunk=12s - Lock-free fast path in StrictSlidingWindowLimiter (reduces contention under load) - Faster retry delays: base=0.3s, max=20s, jitter=0.1s - Max out HTTP connection pool: 100 keepalive, 500 connections, 5s expiry - Aggressive HTTP pool warmup on startup (forces TCP+TLS connection establishment) - Record 429s to ModelHealthTracker for adaptive recovery - Update CLAUDE.md with per-provider health params and session cleanup notes Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>aa9c0b0
-
Fix stale sessions in admin dashboard and improve auto-routing health
- Add session cleanup background task (runs every 60s) to remove inactive sessions - Wire cleanup loop into AppRuntime startup/shutdown lifecycle - Per-provider health parameters: NIM gets tighter TTL (15s/2 failures), Zen looser (60s/5) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>188ffa9
-
Performance optimizations for faster proxy routing
- Remove cleanup from hot path in session tracking - Skip _cleanup_old_sessions on every track_request call - Add track_request_async for contexts needing async guarantees - Add provider warmup on startup to eliminate cold-start penalty - Pre-establish HTTP connections before first request Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>49813da -
Deploy claude-code-nvidia proxy to Hugging Face Spaces
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>0157ac7