S1E17. Serving It: Dynamo, Dynamo-Triton, and NIM

A queue of people waiting, and the chip sitting idle at the same moment. That's what Ana's dashboard showed when every warehouse opened and her team's new assistant crawled. Ana and Kestrel exist only because Shelley and Rob made them up, but the mistake is common: a chip built to do an enormous number of small things at once was handed one request at a time. Rob explains what a serving layer is and isn't, why a compiler like TensorRT comes first, and how dynamic batching works like a lift holding its doors. Learn how Dynamo Triton differs from Dynamo, which splits reading a question from writing the answer, and what goes inside an NVIDIA NIM package, with a sober look at NVIDIA's own comparison of about twice as much per second. Next, a vendor pitching Ana a model of her own, trained on years of shelf photos.