Open-weights ≠ open-source for agent stacks

Closed APIs win on convenience and frontier performance. Self-hosted weights win when data must stay put, you need real fine-tuning, or token volume is high enough that GPUs beat per-token fees. “Which is better?” is the wrong question. Ask which layer of the stack needs which trade-off. Most shipped agents already mix both.

Definitions

Closed / API-first model - a proprietary model you call over a vendor API. Weights, training data, and serving stay with the vendor; you send prompts and get completions.

Open-weights - trained parameters you can download, run, and fine-tune yourself. Training code and data are often still closed.

Open-source (strict) - source code, and ideally training data, released under an open license in addition to weights. Rare for frontier-scale LLMs; do not equate marketing “open” with this bar.

TCO (total cost of ownership) - all-in cost: per-token API fees (operating expense, OPEX) versus GPU purchase or rental (capital and operating expense, CAPEX/OPEX), serving software, and the people who keep it up.

Inference engine - the runtime that loads weights and produces tokens (examples: Ollama on a laptop; vLLM or TensorRT-LLM on a server).

The stack is four layers, not one switch

An agent is not “on Claude” or “on Llama.” IBM Technology frames the AI stack as model → data → orchestration → application. Each layer can be open or closed independently.

Sensitive retrieval can sit on open infra while a closed frontier model handles hard reasoning and an open framework owns the agent loop. That mix is normal. It is not a half-measure.

Comparison at a glance

Do the TCO math before the philosophy

“Open-weights win when token volume is high enough” is the standard claim and it is useless without a number. The number is computable, and it is usually larger than people expect.

The formula is one line:

break-even tokens/month ≈ GPU monthly cost ÷ blended API price per token

Blended price means weighting input and output by your actual ratio - agent workloads skew heavily toward input, which makes the API side cheaper than an output-only comparison suggests.

Worked example. A single high-end GPU VPS runs on the order of $1,000/month. Against frontier API pricing, that pays for itself somewhere around 160–256M tokens/month (roughly 5–8M/day) at 60–70% utilization (Cloudzy’s 2026 breakdown runs the same arithmetic across tiers). Below that, the API is cheaper and someone else carries the ops.

Two multipliers decide whether that estimate survives contact with production:

A rented GPU bills identically whether it is saturated or idle, so utilization is the variable that actually decides the outcome - not the model, not the engine. Bursty traffic against owned compute is the worst case in this table: you pay peak capacity to serve average load.

The second multiplier is the one that kills most business cases: against a budget open-weight API rather than a frontier one, self-hosting almost never wins on cost. When a hosted DeepSeek-class model bills $0.14–$0.50 per million tokens, break-even moves to roughly 2.5B–7B+ tokens/month. For throughput scale, vLLM serving Llama-70B across 4×H100 at 256 concurrent requests lands near 3,400 output tokens/second aggregate - saturating that for a month is a serious traffic commitment.

The conclusion is sharper than “it depends,” and it is the opposite of the usual framing:

If a cheap hosted open-weight API can serve your workload, self-hosting the same weights is a data-path decision, not a cost decision. Make that argument on residency and control, and stop pretending the spreadsheet supports it.

That is the same conclusion DeepSeek V4 Flash reaches from the hardware side: local throughput lands well below the hosted API, so the reason to run it yourself is a perimeter that data never leaves, not speed and not price.

How you call each side

The request code is nearly identical on both sides - most serving stacks expose an OpenAI-compatible endpoint, so the client differs by a base_url. That symmetry is misleading, and it is why “just swap the endpoint” undersells the decision.

What actually differs is everything upstream of the request. Closed path: there is no upstream. You send a request.

Open-weights path: you own the serving configuration, and the flags below are the ones the TCO math above is sensitive to.

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \        # shard across 4 GPUs - must match the box you rented
  --max-num-seqs 256 \              # concurrency ceiling; this is your throughput knob
  --quantization fp8 \              # halves memory, costs some quality - measure it
  --max-model-len 32768 \           # longer context = more KV cache = fewer concurrent slots
  --gpu-memory-utilization 0.92     # too high and you OOM under burst, too low and you waste the rental

Every one of those flags moves cost per token. --max-num-seqs and --gpu-memory-utilization decide how close you get to the saturation the break-even assumed; --quantization and --max-model-len trade quality and context against how many requests fit on the card. None of them exist on the closed path, and none of them tune themselves.

That is the real asymmetry: the model file is an artifact on your disk, and so is the responsibility for batching, quantization, KV-cache sizing, auth, and multi-tenancy. The API price you were comparing against already had all of that priced in.

When closed APIs win

Choose closed when:

1. Frontier quality now - closed models still debut ahead on many hard tasks; downloadable weights typically trail by months.
2. Bursty traffic - serverless APIs absorb spikes; a fixed GPU fleet bills you while idle.
3. No ML ops - nobody to run vLLM, watch GPU memory (VRAM), rotate secrets, or patch the host.
4. Demo first - ship the product loop; revisit openness when volume or compliance forces the issue.

Caching, batching, and tier routing inside a closed stack are a separate problem from open vs closed - see Stop paying full price for repeat LLM calls and Why most agents default to the wrong Claude tier.

When open-weights win

Self-host when:

1. Data must not leave the perimeter - regulated or proprietary corpora that cannot ride a third-party API without a hard enterprise contract.
2. Real fine-tuning - domain jargon, style, or knowledge must live in the weights, not only in prompts or a retrieval index.
3. GPU math wins - steady high token volume where per-token OPEX beats owned or rented compute.
4. Model inspection - inspect behavior, audit weights, or debug with white-box access no API will give you.
5. Vendor lock-in is a product risk - a pricing or policy change would break a core feature.

You own security patches, capacity planning, and serving quality. Risk moves from vendor diligence onto your ops team.

Decision checklist

Score each row for the use case. Majority “open” → self-host that slice; majority “closed” → API that slice; split → hybrid by layer.

Hybrid pattern for agents

A split that shows up in production:

• Closed frontier model as orchestrator, or for the single hardest reasoning step.
• Self-hosted workers for high-volume classify / extract / draft work where “good enough” quality is fine and tokens dominate cost.
• Open orchestration so tool loops and retries stay under your control.
• Data layer colocated with whatever residency rule you already have.

Same idea as worker-tier routing inside one vendor (Claude model tiers), except the axis is openness and hosting - not just price and capability.

Next move

Take one real agent path (ingest → plan → tools → answer) and map it onto the four layers. Mark each layer open or closed with the checklist. If everything stays closed for convenience, fine - then cut the bill with caching and batching. If one layer fails the data or TCO test, swap only that layer. Do not rewrite the whole stack to prove a point.