Overview

Tandemn is an open-source, algorithmic planner for self-hosted AI inference that unlocks more usable capacity from your existing GPU fleet. It places and configures workloads across heterogeneous hardware, clouds, and regions so you can run more jobs and serve more demand without expanding your infrastructure.

It brings fragmented GPU capacity across clusters, cloud providers, and serving regions into one cohesive pool for planning. Different GPU types and hardware configurations retain their individual capabilities, while Tandemn matches each workload to suitable capacity wherever it is available across the fleet.

The algorithm models inference end to end, connecting model configurations and hardware characteristics to runtime behavior and serving performance. It uses this understanding to navigate a space of billions of possible deployment configurations and continuously refines its planning from observed results. As workloads and demand change, it jointly selects configurations and placements that make better use of available resources while meeting each job’s serving requirements. The result is higher goodput and greater effective fleet capacity: more useful inference work from the GPUs you already have.

For an enterprise, usable capacity is the work your fleet can deliver within its performance commitments. Customer-facing services need responsive first tokens and consistent generation speed; internal evaluations and processing campaigns need to finish by their deadlines. These requirements determine how much demand you can support and which new workloads you can bring into production.

Tandemn finds headroom by examining what limits each deployment. Input-heavy requests can be constrained by compute, while long contexts and extended generation put pressure on memory and the KV cache. Splitting a model across more GPUs can relieve memory pressure but introduce communication overhead or leave pipeline stages waiting. The algorithm evaluates GPU selection, parallelism, and replica counts against these behaviors to identify configurations that deliver the required performance with a more efficient resource allocation.

An improvement to one deployment can create room for another. A service that meets its targets with fewer replicas can release GPUs for a waiting batch job. A workload that runs effectively on an alternative accelerator can leave scarce hardware available for a more demanding model. Tandemn evaluates these options together, accounting for job priorities, shared resource limits, and the effect each change has on the rest of the fleet.

The plan also preserves the headroom needed to handle demand and accounts for the cost and risk of changing a running deployment. As traffic, request lengths, and workload mixes evolve, observed serving performance informs the next allocation. This lets the enterprise recover avoidable overprovisioning while protecting the latency targets and completion commitments that make the compute useful.

The business impact is room to support more customer demand, bring additional services online, and complete more internal work before procuring more hardware. More output from the same infrastructure spend improves cost per useful token, while better allocation reduces the need to reserve dedicated capacity for every workload. Your fleet becomes a resource that can support more of the enterprise’s priorities as they change.

Tandemn connects two open-source components: a planning algorithm that determines how to use your fleet, and an execution engine that puts those decisions into operation. Together, they connect a global view of capacity to the deployments running in your environment.

The planning algorithm evaluates workloads together, using each job’s requirements, available hardware, and evidence about inference performance to select configurations and placements. It produces a cluster-wide deployment plan that specifies where models should run, which resources they should use, and how they should be configured.

The execution engine is built on Kubernetes and translates that plan into workload launches, scaling, and configuration updates. It can connect to any lower-level serving framework through that framework’s deployment interfaces, allowing your existing systems to continue serving models. Observed performance feeds back into the planning algorithm, so the next plan reflects how deployments actually behave as workloads and conditions change.

Both components are open source and run in your environment. Your team can inspect the planning logic, adapt execution integrations, and use the complete system with the infrastructure you already operate.