Published

May 5, 2026

12 min read

AWS Fast Networking Is Deep Tribal Knowledge

A practical walk-through of the hidden AWS networking stack behind fast multinode inference on A100 and L40S clusters.

Quan Hao Ng

Founding Engineer

The backstory

We were trying to profile some vLLM inference workloads on Nvidia L40S and A100 GPUs in AWS, testing different tensor-parallel (TP) and pipeline-parallel (PP) configurations. That meant a lot of inter-GPU and inter-instance communication. For example, PP=3 means three nodes, and TP=8 means eight GPUs in one node.

I was given the very vague task of making sure that fast networking was working. I knew this would not be a trivial ping cia.gov kind of task, but I definitely did not expect the networking monster that was waiting for me.

AWS does a very good job of telling you how nice and easy it is to set everything up optimally. The truth is that there are many hidden steps to configure, plus random technologies you need a basic understanding of just to squeeze the best networking out of the instance you are already paying for. In no particular order, I had to wrestle with AWS's custom network fabric (EFA), Nvidia's GPU configurations, AWS's network card configurations, libfabric, and more. In this post, we will walk through the happy and fun things that ensure you get the fastest networking out of your instances.

Fast networking in AWS

If you look through the EC2 specifications for GPU-accelerated instances, such as g6e or p4d, you might see terms like EFA, GPUDirect RDMA, NVLink, and NVSwitch.

Here is an example of the card for p4d instances:

Figure 1. AWS EC2 accelerated computing instance specifications, including network bandwidth, GPUDirect RDMA support, and GPU peer-to-peer bandwidth.

We can break these technologies down into two families:

Within-instance networking: NVLink, NVSwitch
Between-instance networking: EFA, GPUDirect RDMA

Let's talk about each component and why it matters for inference workloads. We will zoom into an individual GPU and work our way outwards into a full multinode inference setup.

NVLink and NVSwitch

Other than being good at making GPUs, Nvidia is also good at coming up with proprietary names. NVLink and NVSwitch are a good example. AWS's specification for p4d instances mentions that the GPU peer-to-peer bandwidth is 600 GB/s NVSwitch. But what does that really mean?

In brief, NVLink and NVSwitch are Nvidia's technologies for enabling GPU-to-GPU networking within a machine or instance.

To really understand this, it helps to take a quick look at how Nvidia GPUs are actually configured by datacenters. Growing up, GPUs were always the PCIe cards I plugged into my PC to improve gaming. Unfortunately, datacenter-grade GPUs do not always use PCIe. They do not even have display ports.

Take the A100. It comes in two form factors: PCIe and SXM. SXM is a proprietary Nvidia socket, and you will more commonly encounter it as eight A100-SXM cards bundled into an HGX A100 baseboard.

Here is an image of a HGX A100 8-GPU system:

Nvidia HGX A100 8-GPU system Figure 2. HGX A100 baseboard with eight SXM GPUs and the NVSwitch fabric.

The eight big blocks are the A100 SXM GPUs themselves. What is more interesting are the six little cubes laid out in a row on one side. These are the NVSwitches. The idea is that each GPU uses NVLink ports to connect to each of the six NVSwitches, and collectively the eight GPUs form a big network domain that allows any A100 GPU to talk to any other A100 GPU at 600 GB/s. This is where AWS gets its "GPU Peer to Peer - 600 GB/s NVSwitch."

For inference workloads, when I use TP=8 tensor parallelism across all GPUs in the machine, collective communication primitives like all-reduce and all-scatter run much faster because each GPU effectively has a direct link to any of the other GPUs. Compare this against a hypothetical setup where I have eight A100 GPUs installed as PCIe cards in the same machine. Inter-GPU communications would have much lower bandwidth and would depend on the PCIe topology, like the PCIe switch and root complexes. An A100 on PCIe Gen4 is 64 GB/s, and that is only between the card and the PCIe fabric.

At this point, you might be thinking: eight A100 SXM GPUs are faster for inference, but AWS only provides that anyway, so why should I care? I just have to spin up the instance and I will magically get the 600 GB/s between GPUs. Right? Wrong. Thankfully, that part is an easy fix: you need Nvidia Fabric Manager, and then you really get the unified memory fabric that you are paying $22/hr for.

The point is simple: all A100s are equal, but some A100s (SXM) are more equal than others (PCIe). AWS has the optimal configuration for A100s. GCP does too. Azure does not, because they also provide A100 PCIe instances. Making inference efficient depends not only on the GPU type, but also on the GPU configuration and the cloud.

EFA

It is well known that TCP/IP was designed for the common, unreliable internet, and is not always the best fit for reliable datacenter networking. That is why technologies like InfiniBand exist.

AWS took it a step further and decided that existing network fabric technologies like RoCE and InfiniBand still were not good enough. Instead, AWS created its own network transport protocol: Scalable Reliable Datagram (SRD). It comes with multipath load balancing, reduced jitter, and faster response to network congestion fluctuations. It is implemented in custom AWS Nitro network cards and exposed to the EC2 host as an EFA PCIe network device.

Why do we care? Accelerated EC2 instances usually support two kinds of network interface: ENA and EFA. Think of ENA as the classic TCP/IP path, giving you common facilities like IP addressing, while EFA is the fast network fabric. For multi-node inference configurations, we want our tensors to move from one instance to the next using the fastest possible network path so that token latency is reduced and throughput is increased.

AWS EFA software stack comparison Figure 3. Traditional EC2 networking stack compared with the EFA-based HPC and ML stack.

Beyond the network fabric, EFA also has optimizations in the OS stack. EFA is available as a provider in libfabric. libfabric abstracts away the diverse network fabric backends, such as EFA, Cray Slingshot, and InfiniBand, and is used by many HPC and ML applications like MPI and NCCL. Using the EFA kernel driver as a coordinator, userspace applications like NCCL (vllm uses this) can write data directly to the EFA device, bypassing the OS and its traditional TCP/IP stack. Again, this means faster inter-instance networking, and therefore faster inference.

All in all, the stack looks like this for vLLM:

PyTorch/vLLM -> NCCL -> aws-ofi-nccl -> libfabric -> EFA provider (fi_efa) -> EFA device/NIC -> AWS EFA/SRD fabric

Of course, as you know by now, things do not just work. To really connect this pipeline, we need to tell NCCL to use the aws-ofi-nccl plugin, and tell libfabric to use the fi_efa provider. To do so, set the following environment variables:

# Ubuntu: taken from "Get started with EFA and NCCL for ML workloads on Amazon EC2"
export LD_LIBRARY_PATH=/opt/amazon/ofi-nccl/lib:/opt/amazon/efa/lib:$LD_LIBRARY_PATH

# Good for forcing the OFI plugin on newer EFA releases
export NCCL_NET_PLUGIN=ofi

# Debug / validation
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=NET

# Relevant on p4d/p5 in AWS examples
export FI_EFA_USE_DEVICE_RDMA=1

# Optional but often useful in AWS launch recipes
export FI_PROVIDER=efa

With everything set up, AWS claims an instance network bandwidth of 400 Gbps over EFA.

GPUDirect RDMA

Among the jargon we saw in the beginning of the post, GPUDirect RDMA is probably the easiest to explain, especially for people who already know what DMA is.

Nvidia's documentation says that it "enables a direct path for data exchange between the GPU and a third-party peer device... Examples of third-party devices are network interfaces, video acquisition devices, and storage adapters." Essentially, it allows the network card to directly read the GPU memory and send it over the network, without going through host memory. This is almost the same idea as old-school DMA.

Here is the path without GPUDirect RDMA:

GPU VRAM -> Host memory (RAM) -> NIC -> Network fabric

And the path with GPUDirect RDMA:

GPU VRAM -> NIC -> Network fabric

There is no additional setup step here. The software pipeline for EFA takes care of this.

Are we done?

Fortunately, many of the software dependencies can be handled by using AWS's Deep Learning AMI (DLAMI) VM image. The base image contains the essential software, such as Nvidia drivers, Nvidia Fabric Manager, aws-ofi-nccl, libfabric, fi_efa, and the EFA kernel driver. I just had to do the right plumbing to get the components connected.

At this point, I thought I was done. I understood how the GPU was configured and set it up: 600 GB/s between each GPU. I knew how EFA worked and set that up too: 400 Gbps between machines. I grabbed a beer and prepared to retire from a distinguished career of setting environment variables.

That happiness lasted until I decided to look at the instance specification one last time:

AWS p4d instance networking specifications Figure 4. p4d instance networking table showing four network cards and 4 x 100 Gigabit bandwidth.

It was not 400 Gbps, like the first picture at the top of the post implied. It was 4 x 100 Gbps, and then I noticed "Network cards: 4". A quick check confirmed my suspicions: each network card only supported 100 Gbps. The marketing of "400 Gbps EFA and ENA" was really more like "get an aggregate bandwidth of 400 Gbps IF you set up all these things right AND activate four networking cards,"... which, of course, I had not done.

AWS p4d deep dive networking diagram Figure 5. AWS p4d deep dive diagram showing the relationship between GPU groups, PCIe switches, and the NVSwitch fabric.

In the diagram, you can see that every two GPUs is connected to one PCIe switch that has a Nitro 100g network card attached to it. The fix was straightforward: when launching EC2 p4d A100 GPU instances, attach multiple EFA network interfaces.

The same applies to other instance types. For example, the g6e family of instances with Nvidia L40S does not have GPUDirect RDMA or NVLink/NVSwitch, but it does support EFA and multiple network cards for some instances, up to 100 Gbps per network interface.

Inference on the cloud

The truth is that inference already worked on the A100 instances even without fast networking. We spun up individual p4d instances, put vLLM on them, and let it go. But we knew we could not stop there, not with so much more performance we could tap into to improve throughput on the instances we were already paying for.

All of this work was AWS-specific, but the real challenge comes in comparing the performance across multiple clouds, and using that knowledge to intelligently choose how to run inference jobs.

At Tandemn, our orchestration software knows that A100s on AWS and GCP are A100 SXM while A100s on Azure are A100 PCIe, and that the inter-GPU bandwidth is a 10x difference. Tandemn's orchestration software knows that AWS A100s have inter-instance bandwidth of 400 Gbps, GCP A100s have 100 Gbps at the highest tier, and Azure has dedicated 200 GB/s InfiniBand connections for each A100.

Tandemn's fully open source software looks at your cluster and helps you choose the right set of GPUs for the right set of jobs. It is software running on your cluster, deploying your inference jobs and saving your money. It is deep domain knowledge expertise coupled with smart orchestration.

In a future post, we will share metrics, benchmarks, and side-by-side comparisons of performance before enabling these networking technologies and after. For now, our inference profiling jobs are happily running on the GPUs.

References

Copy status