Skip to main content
Platform & Infrastructure

Rebuilding the cost structure of long-context inference

From GPU clusters and disk-level KV cache to a multi-model gateway — we operate a full self-hosted inference stack and supply tokens at scale.

Read the Docs →See benchmark data ↓
Inference Stack

Three pieces: cluster, cache, gateway

Capacity comes from the cluster, cost advantage comes from the cache, deliverability comes from the gateway.

🖥️

GPU Cluster

  • A100-class clusters running in production, with tensor-parallel (TP8) multi-GPU inference
  • Dual vLLM / SGLang stacks with multi-model serving
  • Long-context requests served continuously (400K-class server-side context configuration; ≤100K per request recommended in production)
💾

Cascade Disk-Level KV Cache

  • Lifts KV cache from GPU memory to NVMe and cluster storage
  • Cache match skips repeated prefill — saving compute and cutting TTFT
  • Dual storage backends: POSIX and GDS/NvFile, with automatic fallback
🔀

Inference Gateway

  • Multi-model routing with load-aware scheduling (cluster side)
  • Platform gateway: quota billing, key and user management
  • OpenAI-compatible APIs; existing apps can plug in directly
Core Technology

Cascade: KV cache that outgrows GPU memory

The bottleneck of long context is not compute — it is GPU memory. Cascade moves cold and reusable data down to disk and cluster storage.

📉

Why disk

GPU memory is ~80 GB per card and extremely expensive; a single node of NVMe holds terabytes at an order of magnitude lower cost. Trade a little storage latency for a big relief in memory pressure.

🎯

A hit saves compute

Each request starts with prefix matching: matching KV loads directly from storage, skipping repeated prefill and slashing TTFT; cluster KV bandwidth reaches 20 GB/s (depending on network).

🔗

Reuse across nodes

A shared cache pool (MVP) lets different nodes and requests reuse the same KV — one context is computed once.

Measured Data

Public Benchmark Snapshot

Numbers come from Cascade’s public benchmark docs and can be independently verified.

ScenarioMetricResult
Repeated-prefix reuse (POSIX backend)TTFT (cold → cache hit)8.105s → 0.390s (20.8×)
Repeated-prefix reuse (POSIX backend)Cache hit rate96%
A100 validation hostRepeated-request TTFT1.704s → 0.199s (8.6×)
Storage clusterAggregate throughputTB-class
KV load / saveBandwidth20 GB/s (depends on network)

* KV bandwidth depends on network equipment and deployment; storage throughput is cluster-aggregate. TTFT and hit rate are measured on public benchmarks — refer to the Cascade repository docs for exact conditions.

Want to talk deployment or partnership?

Token services, private model deployment, or product adoption — get in touch.

Read the Docs →Try Web App