# ForgeModel: frontier accuracy, a fraction of the cost

Source: https://forgemodel.org/ (single page). Contact: abhishekg@forgemodel.org. Last updated 2026-09-05.

We forge frontier-model capability into small private models: open weights fine-tuned on your highest-volume task, proven head-to-head against the API you pay for today, served on infrastructure you control.

A fine-tuned 4B model beat GPT-4o-mini on 5,000 held-out examples at 14.3x lower cost. Every knob was tuned on a separate dev split.

## Proof before promises

Frontier-level is a claim we prove, not assert. Every engagement begins with a benchmark you can interrogate. This is ours: a Qwen3-4B fine-tuned on Amazon's ESCI relevance task, against GPT-4o-mini, identical prompts, one held-out test set. Measured August 2026.

| Setup | micro-F1 | Cost per 1M calls |
| --- | --- | --- |
| GPT-4o-mini only | 0.6116 | $28.26 |
| ForgeModel only (fine-tuned Qwen3-4B) | 0.6424 | $1.98 |
| ForgeModel with confidence router | 0.6536 | varies by threshold |

- Higher accuracy: a 4-billion-parameter model, fine-tuned on the task, outscoring the frontier API on the held-out test set.
- Self-hosted cost: one rented RTX 4090, model resident in memory, continuous batching. Real GPU rate, measured throughput.
- 14.3x lower cost at higher accuracy. The model answers with a calibrated confidence score. The unsure slice escalates to your frontier API automatically, so the system as a whole scores higher than either model alone. In production the router is one `if` statement on first-token logprobs.

How we measure:

- Head-to-head, on your data. Your fine-tuned model against the API you use today, on a test set your team defines.
- Honest methodology. Tuning decisions are made on a separate development set, never on the data we report. Baselines run the way a skeptic would run them.
- The report is yours either way. If your current setup wins, the benchmark says so and you keep the full analysis.

## Built for repetitive work. Honest about the rest.

Our specialty is high-volume, structured work where you already know the output shape: one fixed prompt, short outputs, millions of calls.

What we take on:

- Document and product classification
- Entity and attribute extraction
- Metadata tagging and moderation
- Request routing and dedupe

What stays with frontier models: open-ended conversation, coding, research, agentic reasoning. Anything where every request is different. They win there, and we will tell you to keep them. Use frontier models for exploration. Use ForgeModel for production execution.

What happens to the unusual cases: every answer comes with a confidence score. Below your threshold, the request escalates to your frontier API automatically, so weird inputs never get a confident wrong answer from a small model.

Example requests from the ESCI benchmark: query "wireless earbuds noise cancelling" against "Sony WF-1000XM5 earbuds" is labeled Exact match at confidence 0.94 and answered locally. Query "gift for a ten year old" against "LEGO Technic crane, 1,200 pieces" is labeled Substitute at confidence 0.41 and escalated to the API.

## Run your numbers (cost calculator)

The site has an interactive calculator. Pick the API model you use today; its published token prices, a matched open-weights replacement, and that model's GPU are prefilled. Every field stays editable and the math updates live.

Inputs: current API model, requests per day, input tokens per request, output tokens per request, API $ per 1M input tokens, API $ per 1M output tokens, GPU $ per hour, throughput (requests per second per GPU), target utilization.

Math: monthly API cost = requests per month x (input tokens x input price + output tokens x output price) / 1,000,000. GPUs needed = ceil(average requests per second / (throughput x target utilization)), minimum 1. Monthly self-hosted cost = GPUs x GPU $ per hour x 730. Savings = API cost minus self-hosted cost. At low volume the API is cheaper and the calculator says so.

Default example (gpt-4o-mini, 1,000,000 requests per day, 185 input tokens, 1 output token, $0.15 / $0.60 per 1M tokens, one RTX 4090 at $0.74 per hour, 60 requests per second, 70% target utilization): current API cost $863 per month, ForgeModel cost $540 per month, estimated savings $323 per month, $3,873 per year, 1 GPU at 19.3% actual utilization.

Prefilled replacements by API model (open-weights model, GPU, GPU $ per hour, estimated throughput in requests per second):

- gpt-4o-mini, gpt-5-nano, Gemini 2.5 Flash-Lite: Qwen3-8B (AWQ) on 1x RTX 4090 24GB, $0.74/h, 60 rps
- gpt-4.1-mini, gpt-5-mini, Claude Haiku 4.5, Gemini 2.5 Flash, DeepSeek V4 Flash, Mistral Small 4: Qwen3-32B (AWQ) on 1x L40S 48GB, $0.99/h, 20 rps
- gpt-4o, gpt-5, Claude Sonnet 5, Gemini 3.7 Flash, Grok 4.3, DeepSeek V4 Pro, Mistral Medium 3.5: Llama-3.3-70B (AWQ) on 1x H100 80GB, $2.89/h, 15 rps
- gpt-5.4, Claude Opus 5, Gemini 3.1 Pro, Grok 4.6: Qwen3-235B-A22B (AWQ) on 1x H200 141GB, $4.59/h, 10 rps

Directional, not a quote. The token prices and GPU rates are real and current; the throughput and utilization figures are honest estimates. Every workload is different, which is exactly what the fixed-price pilot is for: it replaces every estimate with numbers measured on your data, before you commit to anything. Excludes one-time fine-tuning (a few GPU-hours) and your time.

## Forge, prove, ship

A disciplined path to production, in the order it actually happens.

1. Fine-tune. We tune a small open model on your task using labeled examples you choose to share: your historical labels, or answers generated by the API you already pay for. Hours of GPU time, not weeks.
2. Prove. Head-to-head against your current model on a held-out test set you define, tuning knobs only on a separate dev split. Per-class results and confusion matrices before anything ships.
3. Route and serve. The model answers with a calibrated confidence score; the unsure slice escalates to your frontier API automatically. Served behind an OpenAI-compatible endpoint, so switching is a one-line change.

## Start your pilot

Tell us about one high-volume pipeline and we will scope a fixed-price pilot that proves the savings on your actual data. The report is yours either way.

Request a pilot at https://forgemodel.org/#contact (name, work email, company, highest-volume task) or email abhishekg@forgemodel.org directly.

## About

Private, fine-tuned models built on open source, with frontier-level results for repetitive AI tasks. Open weights, open method. Calculator prefills use providers' published token prices and market GPU rates; all figures are directional until measured on your workload.