# How a Demo Outage Made Us Rebuild Our AI Inference Stack

A demo outage forced us to rebuild our entire AI inference architecture. We went from Mistral to Nebius Token Factory and saved 75% on our reasoning costs.


I was mid-demo. A prospect was watching me walk through Orbiq's trust center live. Then the answers stopped coming. Mistral was down. Not degraded — gone.

For a compliance platform where buyers evaluate your security posture in real time, "the AI is down" is not recoverable. We lost the demo slot. I opened my notebook and wrote one line: _we are never in this position again._

## The problem was Mistral

After that call, I pulled up Mistral's status page: service unavailable. That's not a great look. Claude went down recently too. The problem was a single point of failure in the part of our stack that customers, and their prospects, touch directly sales conversations and won deals.

We fixed two things at once: added a new primary provider, and right-sized every model in the process.

## Why Nebius Token Factory

We landed on **Nebius Token Factory** as our new primary for three reasons that matter specifically to us:

**EU sovereignty.** Nebius runs on EU infrastructure under EU jurisdiction. When we're selling to FinTech and HealthTech companies navigating GDPR and NIS2, "where does your AI inference run?" is a real question in security reviews. We now have a cleaner answer.

**Zero data retention.** Prompts and completions are not logged or retained for any purpose. Non-negotiable when security questionnaires and NDA-gated compliance materials flow through the system.

**Open models at commodity prices.** No licensing premium. We ended up running _larger, better_ models at _lower_ cost than before, which forced the model audit below.

## The model audit

When you switch providers, you're forced to ask a question you should have asked earlier: is each AI call using the right model for the job?

Orbiq runs seven distinct AI tasks... for now. Here's what changed and why.

**Language detection** (Mistral Small 3.1 24B → Qwen3-14B): Three-way classification — English, German, or French. We were using a 24B multimodal model for it. Qwen3-14B handles this at lower latency and cost. One honest note: Mistral Small 3.1 actually beats Qwen3-14B on AIME-25 math reasoning (85.0 vs 73.7). For a language classifier, that doesn't matter.

**Question parsing and evaluation** (Mistral Medium 3 → Qwen3-30B-A3B-Instruct-2507): A MoE model with 30B total and 3B active parameters per pass. Faster than a dense equivalent, better structured output in our tests, roughly the same price.

**Answer generation** (Mistral Large 3 → Qwen3-235B-A22B-Instruct-2507): This one deserves the honest comparison. Mistral Large 3 is a genuinely strong model: 85.5% on MMLU, Chatbot Arena Elo ~1418, second among open-weight non-reasoning models. For general knowledge tasks, it's excellent. But on GPQA Diamond (graduate-level scientific reasoning), it scores 43.9%. Qwen3-235B-Instruct scores 95.6 on ArenaHard. For generating precise answers grounded in compliance documentation, the harder reasoning benchmark matters more. Output cost: $0.60/M vs $1.50/M.

**Reasoning tasks** (Magistral Medium → Qwen3-235B-A22B-Thinking-2507): Deep search and questionnaire evaluation — tasks that require actual chain-of-thought. The benchmarks are unambiguous:

| Benchmark        | Magistral Medium | Qwen3-235B-Thinking  |
| ---------------- | ---------------- | -------------------- |
| AIME-24 (pass@1) | 73.6%            | 85.7%                |
| GPQA Diamond     | 70.83%           | significantly higher |
| LiveCodeBench v5 | 59.36%           | 70.7%                |

Magistral Medium is a capable reasoning model. Qwen3-235B-Thinking is in a different tier. Output cost: $0.80/M vs $5.00/M.

## The fallback architecture

Every AI call goes through a provider chain. Before each call, the system checks the provider's status page; if it detects a major or critical outage, it skips immediately and caches the decision for 5 minutes. If the primary call fails at runtime, exponential backoff kicks in, then falls through to the next provider. Mistral stays in the chain as our fallback.

Adding a provider or swapping a model is a config change, not a code change.

## What to take from this

**Right-size your models.** A 24B multimodal model for language classification is waste. Audit every AI call and ask what the minimum viable model actually is: the latency improvement alone is worth it for synchronous user-facing features.

**MoE models are the production default now.** Qwen3-235B-A22B activates 22B parameters per forward pass. You get a 235B model's knowledge at a fraction of the inference cost.

**Never benchmark on the wrong dimension.** Mistral Large 3 wins on MMLU. Qwen3-235B wins on GPQA and ArenaHard. Your task profile determines which number matters.

**Build the fallback in from day one.** Outages are likely to occur more frequently as geopolitical volatility and compute demand increase.

**Make data residency a first-class property.** If you sell to regulated EU companies, "EU infrastructure, zero data retention, sovereignty-aligned provider" belongs in your security questionnaire responses. It's a product decision as much as an infrastructure one.

_Orbiq is a trust center platform that helps EU companies in regulated industries close enterprise deals faster by centralizing security documentation, automating questionnaire responses, and giving buyers self-service access to compliance posture. [orbiqhq.com](https://app.orbiqhq.com/signup)_