PayPal Tech Blog

Let's assume a cloud datacenter context, using GCP as an example for real, verifiable numbers.

An L4 GPU node (with 4 CPU cores) is approximately the cost of a 16-core VM. So for the price of 64 cores, you could have ~4 L4 nodes (or perhaps one VM with 4 L4s). Each GPU has 300 GB/s of memory bandwidth, which is approximately 60 cores worth (at ~5GB/s per core). In fact, the 64 "cores" are typically 32 physical cores only, so you would expect lower than 300 GB/s for a 64-core VM.

Let's say network I/O is between 10 Gbits/s and 800 Gbits/s. PCIe 4.0x16 is 64 GB/s, i.e. 512 Gbits/s. PCIe 5.0x16 is double that, which would easily saturate the network link. Also, L4 GPUs have RDMA enabled, bypassing the CPU when ingesting data. I also highly doubt that a 64-(v)core VM can access ">100 PCIe lanes" unless you have some kind of sole tenancy setup.

Then, what if: instead of 4 L4 GPUs (which would have a similar cost to the 64-core node), you used 2 L4 GPUs and found that setup to process just as fast as your 64-core node? You'd have just halved your cost.

Of course, this depends on whether or not the GPU nodes actually process your data with the same SLA, but generally GPU network ingest > CPU network ingest (per $ of processing).

Here's some benchmark results from Voltron Data (I'm not affiliated with them): https://voltrondata.com/benchmarks/theseus

Here's more data from PayPal:…