How Retailers Run Batch Inference on Product Catalogs

The Business Problem

Modern retailers rely on clean, structured product data to power search, filtering, recommendations, merchandising, and conversion. The problem is that most catalogs are not clean.

Product data arrives from dozens of sources: suppliers, brands, distributors, internal merchandising teams, and third-party marketplace sellers. It arrives incomplete, inconsistent, and poorly formatted. A product may have a clean title but missing attributes. Another may have useful information buried in a long description or only visible in the product images.

A product might arrive in the catalog as nothing more than:

Nike Air Zoom Pegasus 41 Men's Road Running Shoes White/Volt

But the retailer still needs to know the category and subcategory, gender, sport and surface type, material and closure type, cushioning category, and primary and secondary colors. Without those fields, the product is effectively invisible to large segments of shoppers.

Why This Matters

Missing or low-quality attributes degrade performance across the entire stack.

Search
Reduced relevance and missing keyword coverage

Faceted Navigation
Products don't appear in filtered views

Recommendations
Poor signal quality and bad affinity matches

Merchandising
Manual tagging burden and inconsistent categorization

SEO
Missing long-tail attribute coverage

Conversion
Shoppers can't find what they're looking for

If a product is missing attributes like waterproof, organic cotton, trail running, or wide fit, it simply won't appear for the shoppers who need it. This is why major retailers run large-scale catalog enrichment pipelines. The goal is not just better metadata, but rather better product discoverability and better business performance.

---

The Inference Problem

The retailer wants to run an AI analysis over every product in the catalog and infer structured metadata that can be written back into the system.

This case study focuses on one specific task: structured product attribute extraction. The model reads available product information and predicts the structured fields that are missing or inconsistent.

Text signals include:

• Title
• Description and bullet points
• Specifications
• Vendor metadata and taxonomy hints

Image signals include:

• Hero product images
• Alternate angle shots
• Packaging imagery
• On-model photography

Common enrichment tasks beyond attribute extraction include taxonomy classification, style and material inference, title normalization, quality scoring, and product embedding generation. Attribute extraction remains the most common and operationally expensive batch inference workload in retail.

---

Input and Output Structure

A real retailer's product records are semi-structured, noisy, and incomplete. A typical input record may look like:

{
  "product_id": "SKU_48291",
  "seller_id": "SELLER_1042",
  "brand": "Nike",
  "title": "Nike Air Zoom Pegasus 41 Men's Road Running Shoes",
  "description": "Responsive daily trainer",
  "bullet_points": [
    "Engineered mesh upper",
    "ReactX foam cushioning",
    "Road running",
    "Men's sizing"
  ],
  "category_hint": "Footwear",
  "price": 139.99,
  "images": [
    "gs://catalog-images/sku_48291_1.jpg",
    "gs://catalog-images/sku_48291_2.jpg"
  ],
  "vendor_metadata": {
    "department": "Men",
    "sport": "Running",
    "color_raw": "White/Volt/Black"
  },
  "existing_attributes": {
    "color": null,
    "material": null,
    "surface": null,
    "closure": null
  }
}

The output needs to be structured, machine-readable, and trustworthy enough to write directly back into the catalog. Here is the desired product output structure:

{
  "product_id": "SKU_48291",
  "enriched_attributes": {
    "category": "Shoes",
    "subcategory": "Running Shoes",
    "gender": "Men's",
    "sport": "Running",
    "surface": "Road",
    "closure": "Lace-Up",
    "material": "Mesh",
    "cushioning": "Neutral",
    "primary_color": "White",
    "secondary_color": "Volt",
    "use_case": "Daily Trainer"
  },
  "confidence_scores": {
    "category": 0.99,
    "subcategory": 0.98,
    "surface": 0.94,
    "material": 0.91,
    "use_case": 0.87
  },
  "model_metadata": {
    "model_name": "RetailAttrExtractor-v1",
    "model_version": "2026-04-01",
    "inference_timestamp": "2026-04-06T03:02:17Z"
  }
}

This structured output feeds into search index pipelines, recommendation engines, merchandising systems, analytics pipelines, PDP rendering logic, and internal catalog QA workflows.

---

How the Batch Job Is Packaged

For large-scale offline inference, retailers typically export products to JSONL and process them asynchronously. Each line is one product record, and may look like:

{"product_id":"SKU_48291","brand":"Nike","title":"Nike Air ...}

{"product_id":"SKU_77420","brand":"Levi's","title":"Levi's 501 Origi...}

{"product_id":"SKU_99213","brand":"KitchenAid","title":"KitchenAid Art...}

In practice, a real retailer might run hundreds of thousands, millions, or tens of millions of records in a single nightly or weekly job.

The critical point: not every row in that file is equally difficult to process. Some products are simple. Others are long, ambiguous, image-dependent, or poorly structured. That variability is the core issue that most infrastructure completely ignores.

---

How Self-Hosted Inference is Run Today

Most retailers hosting their own models build a system whose focus is solely on getting the job done. They do not build a highly optimized inference scheduler.

An example inference pipeline may look like:

1. Data prep. A nightly or weekly job exports products into JSONL, Parquet, or object storage.
2. Batch orchestration. A scheduler kicks off the enrichment run.
3. Inference cluster. The model runs on self-managed GPU infrastructure using vLLM, TGI, Ray, or an internal serving layer the company has configured.
4. Worker fleet. Workers shard the input file, dispatch requests to the model endpoint, handle retries, and collect results.
5. Post-processing and writeback. Outputs are validated, normalized, and written back into the catalog and downstream systems.

This setup is widely deployed. It is also where most of the operational inefficiency begins.

---

Operational Constraints

This workload is not real-time, but that does not mean it is unconstrained. Retail batch inference is deadline-driven. The question is not whether the system can return a response in 200ms. The question is whether the system can finish the entire catalog before tomorrow morning.

Example Batch Objective

For a job like this, the team cares about total completion time, sustained throughput, retry behavior and failure recovery, cost per completed job, GPU availability, schema correctness at writeback, and deadline confidence.

The goal is not low latency. It is finishing on time, reliably, and without overspending.

---

How Teams Size the Job

Step 1: Define the Service Level Objective (SLO)

Required products per hour:
12,000,000 / 8 hours = 1,500,000 products/hr

Required products per second:
1,500,000 / 3,600 = ~417 products/sec

The system must sustain roughly 417 products per second for the entire 8-hour window.

Step 2: Estimate workload shape

Average input tokens per product:   180
Average output tokens per product:   60
Total tokens per product:           240

Total token volume:
12,000,000 x 240 = 2.88 billion tokens

Step 3: Backsolve infrastructure requirements

If one replica processes approximately 15 products per second:

Required replicas:
417 / 15 = ~28 replicas

In practice, teams round up replicas to build in a safety margin for traffic variability, retries, token skew, model instability, and infrastructure failures. In other words, they overprovision for the necessary resources. This is common practice, and is exactly where costs start to climb.

---

The Current Approach Is Inefficient

Most retailers can get this workload running with relatively little engineering effort. The problem is that they run it in a way that is expensive, rigid, and operationally fragile. The inefficiency is structural: the workload is treated like a generic batch job instead of a workload-aware inference problem.

Issue 1: The workload is not uniform

Not every product record costs the same to process.

Easy records have a clean title with strong metadata, obvious taxonomy classification, and short descriptions with a single image.

Hard records have an ambiguous product type, poor or missing vendor metadata, long descriptions with image-dependent classification, and multiple plausible categories with inconsistent formatting.

Most teams process all of these rows identically. The result is paying the cost of the hardest cases across the entire workload.

Issue 2: One hardware class for everything

A common decision process: "We know this model runs safely on A100 80GB, so let's just use that."

That works, but it is far from optimal. Many records could be processed on cheaper hardware, but most teams do not dynamically separate those execution paths. It is like using a drill when a screwdriver would have been perfectly fine. You end up paying for unnecessary resources.

Issue 3: Deadline-driven jobs get overprovisioned

Retail teams do not want to miss the overnight window. So they plan around the worst case. They size for worst-case records, retries, throughput drift, cluster instability, and failure recovery. This solves the operational anxiety, but is not an efficient way to run the workload.

Issue 4: Static deployments cannot adapt

Most jobs are configured with a fixed replica count, a fixed hardware class, and a fixed batch strategy. Once launched, they run without any dynamic adjustment.

As you can imagine, workloads are not static. For example, easier rows often finish first, harder rows cluster later in the queue, retries create throughput spikes, instance availability changes, spot interruptions occur, etc.

A static deployment has no ability to respond.

Issue 5: Cheap capacity goes unused

Batch inference is one of the best candidates for lower-cost infrastructure: spot instances, heterogeneous GPU pools, non-premium accelerators, and elastic worker fleets. Most teams avoid these because using them reliably requires complex orchestration. Without that orchestration layer, the default is a more expensive but more stable route.

---

What Workload-Aware Orchestration Does Differently

The insight at the center of this case study is that retail catalog enrichment is not simply a data problem. It is a deadline-constrained resource optimization problem.

Instead of "run the entire batch on a fixed cluster until it finishes," a workload-aware system inspects the workload shape before launch, understands that records have different processing costs, estimates throughput against the target deadline, selects infrastructure based on cost and completion requirements, and dynamically reallocates capacity based on live progress. It optimizes for deadline completion, not just deployment success.

This shifts the core question from "where can I host this model?" to "what is the cheapest and most reliable way to finish this workload before the deadline?" That is the real problem underneath retail batch inference.

---

The Shared Cluster Problem

In practice, the catalog enrichment job is rarely the only workload running on the retailer's infrastructure.

The same GPU cluster may also be serving:

• Live online inference for shopper-facing experiences
• Other batch enrichment jobs for search, recommendations, or embeddings
• Internal analytics and experimentation workloads
• Fine-tuning, evaluation, or QA runs from adjacent teams

This creates a second layer of complexity that simple batch sizing math does not capture: workloads are competing for the same finite pool of accelerators.

Why this creates operational tension

If the retailer reserves too much capacity for the batch job, online systems can suffer. If the retailer protects online systems too aggressively, the batch job slips and misses its deadline. The problem is not just how large the enrichment job is in isolation. The problem is how that job behaves inside a shared cluster with other priorities.

What competition looks like in practice

• A nightly catalog job launches at the same time a recommendation backfill starts
• A high-priority online service needs burst capacity because live traffic spikes
• A batch workload fills the fastest GPUs, forcing latency-sensitive services onto worse hardware
• One long-running enrichment job fragments the cluster and makes placement harder for every workload behind it
• Teams create manual reservation rules to protect online inference, but those reservations leave batch capacity stranded when live demand is low

This is where utilization becomes deceptive. A company may think it is "using the cluster efficiently" because GPUs are busy, while in reality the wrong workloads are occupying the wrong hardware at the wrong time.

Why static scheduling breaks down

Most internal schedulers treat these jobs as separate queues or fixed reservations. That is manageable when utilization is low, but it breaks down once multiple jobs and service classes share the same infrastructure.

The scheduler now has to answer harder questions:

• Which workloads can tolerate interruption?
• Which jobs need the highest-throughput GPUs versus acceptable lower-cost hardware?
• How much capacity must remain reserved for online services?
• When should a batch job slow down so a real-time service can stay within latency targets?
• When spare online capacity appears, which batch workload should consume it first?

Without workload-aware orchestration, the usual outcome is a mix of over-reservation, manual intervention, and underutilized hardware. Teams either isolate workloads too aggressively and lose efficiency, or let them compete too freely and create deadline risk and service instability.

For retailers operating a shared inference cluster, the real challenge is not only finishing the catalog job cheaply. It is finishing the catalog job cheaply without disrupting every other model workload that depends on the same pool of compute.

---

How Tandemn Approaches This Workload

Tandemn treats catalog enrichment as a deadline-constrained optimization problem. Rather than deploying a static task and hoping it finishes, Tandemn continuously reasons about the workload as it runs and adjusts resource allocation to protect the SLO while minimizing cost.

Initial execution planning

When a batch job is launched, Tandemn analyzes the workload and the target SLO together before provisioning anything. It profiles the input file to characterize the distribution of record complexity, estimates total token volume, and selects an initial resource configuration grounded in those actual workload characteristics.

Continuous SLO monitoring

Tandemn does not treat the deployment as fixed. As the job runs, the intelligence layer continuously measures actual throughput against the projected completion trajectory. It tracks whether the workload is on pace to finish on time, and uses that live signal to decide how aggressively it can optimize for cost without putting the deadline at risk.

If job is ahead of schedule  ->  Explore cheaper configurations
If job is drifting behind    ->  Add or rebalance resources

This happens in a continuous loop throughout the job, not just at launch.

The core intelligence loop

1. Profile workload and parse the SLO target
   Tandemn starts by understanding the shape of the job and the deadline it has to hit.

2. Select the initial execution plan by workload tier
   It chooses an initial mix of hardware, batching strategy, and placement based on expected record complexity.

3. Launch the batch across available cluster resources
   Work is distributed across the infrastructure that best fits the initial plan.

4. Measure live throughput against completion trajectory
   As the job runs, Tandemn continuously checks whether the workload is ahead of schedule or at risk of missing the SLO.

5. Adapt the plan in real time
   If the job is ahead, Tandemn explores cheaper configurations. If it is behind, Tandemn adds or rebalances resources to recover schedule confidence.

6. Update the performance database
   The observed behavior of the model on the actual infrastructure is recorded for future jobs.

7. Improve future launch decisions
   Each completed run makes the next planning cycle more accurate, cheaper, and more reliable.

Performance database

Underneath the control loop, Tandemn maintains a performance database that captures how specific models behave on the customer's GPU cluster over time. This is profiling data specific to the models you run on the infrastructure you run them on.

Rather than starting from scratch on every run, Tandemn builds an increasingly accurate map of which workloads perform best on which resources, under which serving configurations, and at what cost. Each completed job feeds back into this database and makes the next launch decision more accurate.

Over time, the system develops a detailed picture of throughput curves, cost per token by hardware tier, retry patterns, and optimal batch sizes for different record types. This compounds. The longer you run Tandemn, the better its launch decisions become.

Heterogeneous capacity management

Tandemn is built to run across mixed GPU pools. Rather than requiring a homogeneous cluster, it can coordinate work across A100, L4, A10G, and other accelerator types simultaneously, routing records to the appropriate hardware based on their complexity tier and the current cost profile of each instance type.

It also manages spot and preemptible capacity without requiring the team to build retry and rebalancing logic from scratch. When a spot instance is reclaimed, Tandemn detects the throughput drop, reassigns the affected work, and rebalances across remaining capacity. This is what makes it practical to use cheaper instance types at scale rather than just in theory.

---

The Cost Difference

The cost difference between running this workload naively and running it intelligently is not marginal.

For a retailer running nightly enrichment across a 50 million SKU catalog, that cost difference compounds to roughly $700,000 per year on a single classification pipeline.

Most self-hosting teams can get this workload running. Very few run it intelligently.

---

Final Takeaway

Retail catalog enrichment is one of the clearest examples of how AI inference becomes an infrastructure problem at scale.

On the surface, the job is simple: read product records, run the model to predict structured attributes, write outputs back into storage. Underneath, the team is solving for deadline completion, sustained throughput, hardware planning, failure recovery, cost efficiency, and operational reliability.

The naive approach treats the batch as a monolithic job on fixed infrastructure. It works, but it overspends, underutilizes, and cannot adapt.

The intelligent approach treats the batch as what it actually is: a heterogeneous workload with a deadline, a cost constraint, and a live signal that can be used to continuously improve resource allocation while the job is running.