Concepts
How chomp works, and why it’s designed this way.
What are free models?
Free models come from multiple sources across chomp’s router network. OpenRouter offers 20–30 free models at any time (identified by a :free suffix). Groq provides a generous free tier with blazing-fast inference. OpenCode Zen offers free access to premium models. Cerebras, SambaNova, Together, and Fireworks each have their own free tiers or promotional models.
The roster rotates — a model that’s free this week might not be next week, and new ones appear regularly. Each router has different strengths: Groq is the fastest, Zen gives access to premium models at no cost, OpenRouter has the widest variety, and providers like Cerebras and SambaNova offer specialized hardware-accelerated inference.
These aren’t toy models. Free models across these routers include DeepSeek R1 (a reasoning model comparable to o1), Llama 3 70B on Groq (with sub-second latency), Qwen3 Coder 480B, and various 70B–120B parameter models.
How model auto-selection works
When you dispatch a prompt with router: "auto" or omit the router field, chomp walks a router registry in priority order: Zen → Groq → Cerebras → SambaNova → Together → Fireworks → OpenRouter. It picks the first router that has a configured API key and dispatches using that router’s default model.
This means the best available router wins automatically. If you have a Groq key configured, you get Groq’s speed. If you only have an OpenRouter key, you still get a working free model. Each router defines its own default model — typically the strongest free option available on that platform.
You can also specify a router explicitly (e.g. router: "groq") to bypass auto-selection and target a specific provider. For full control, set both the router and the model.
Why async dispatch?
Chomp’s dispatch endpoint returns immediately with a job ID, rather than waiting for the model to respond. This is a deliberate design choice for two reasons:
Parallelism. A calling agent can dispatch five tasks simultaneously and poll for all of them. This is much faster than making five sequential synchronous calls.
Timeout resilience. Free models can be slow. DeepSeek R1 regularly takes 30–60 seconds. Some providers have cold starts. An async design means the caller never hits HTTP timeouts, and can implement its own backoff strategy.
Rate limits and fair use
Each router has its own rate limits set by the underlying provider. These are typically per-minute (requests and tokens) and per-day limits. When you hit them, the provider returns HTTP 429.
Chomp does not currently handle rate limit retries automatically — the error propagates to the job result. The calling agent is responsible for retry logic. A future version may add automatic fallback to the next router in the registry on 429.
Some providers also enforce data policies. For OpenRouter specifically, if you see a 404 about “no endpoints matching your data policy”, visit OpenRouter privacy settings and allow free model training data usage.
The context window problem
Chomp exists because AI agents have finite context windows. A coding agent like Shelley (Claude) has ~200k tokens of context. Every file it reads, every tool output, every conversation turn eats into that budget.
The pattern chomp enables: an orchestrating agent breaks a large task into bounded subtasks, dispatches each to a free model, and collects only the summarized results. The orchestrator’s context stays lean while the grunt work happens elsewhere at zero cost.
This is the “staff of agents” model. The orchestrator is the senior engineer who delegates; the free models are junior engineers who do bounded, well-scoped tasks.
BYO key model
Chomp never stores or manages AI provider subscriptions. You bring your own API keys for whichever routers you want to use. Each router has its own environment variable — GROQ_API_KEY, OPENROUTER_API_KEY, ZEN_API_KEY, and so on. When you register, chomp stores your keys in Cloudflare KV and gives you a chomp token. Every dispatch call uses your key for the selected router. Your keys, your rate limits, your usage.
You don’t need keys for every router — just the ones you want. If you only configure Groq, auto-selection uses Groq. If you configure all seven, you get maximum flexibility. Most routers offer free API keys.
This means chomp is free to operate for the platform owner and free to use for users (since the models themselves are free). There’s no billing layer, no margin, no middleman markup.
The entire codebase is open source. If you don’t trust the hosted version, fork and deploy your own — it’s one Astro project on Cloudflare Workers with a single KV namespace.
The router registry
Chomp treats every router as an OpenAI-compatible chat completions API. Under the hood, a router is just a base URL and an API key. This makes adding new routers trivial — if a provider exposes an OpenAI-compatible endpoint, it can be a chomp router in a few lines of config.
Currently there are 7 routers, each with different strengths:
- OpenCode Zen — Free access to premium models, first in priority order
- Groq — Ultra-fast inference on custom LPU hardware, ideal for latency-sensitive tasks
- Cerebras — Wafer-scale hardware-accelerated inference
- SambaNova — High-throughput inference on custom silicon
- Together — Wide selection of open-source models with competitive pricing
- Fireworks — Optimized serving for open-source models
- OpenRouter — Aggregator with the widest model variety, last in priority order
The registry is ordered by priority. Auto-selection walks this list top to bottom and picks the first router with a configured key. You can override this by specifying a router explicitly in your dispatch call.
Jobs and persistence
Jobs are stored in Cloudflare KV with a 24-hour TTL. After 24 hours, job results are garbage collected. If you need to retain results, save them on your end when you poll.
The job index (the list of recent job IDs) holds the last 100 entries. Older jobs may still be accessible by ID if they haven’t expired, but won’t appear in /api/jobs.