Skip to content
Artwork for NVIDIA Generative AI
NVIDIA Generative AI · August 25 · 8 min

S1E10. More Than Text: Modalities, CLIP, and One Shared Space

Oh, brilliant, said a warehouse manager on a support call, in the flattest voice you've ever heard, after a camera marked his whole aisle empty again. The summary said the customer was very pleased. He'd rung Kestrel, the shelf-camera company Ana runs, and Shelley and Rob made up both of them. Rob explains that the tone died when the call became a transcript, and what multimodal really means: one model holding the voice and the words together, not two models passing a note. Then CLIP, trained on four hundred million picture-and-caption pairs from the web, which lets Ana write a sentence like a photo of a crushed pallet instead of retraining her cameras. You'll also learn what CLIP can't do, from counting boxes to word order, and why its original release isn't for shipping. Coming up, a camera that reads photos the way chatbots read sentences.

0:00-8:54

transcript

No transcript — this publisher did not publish one.

show notes

Oh, brilliant, said a warehouse manager on a support call, in the flattest voice you've ever heard, after a camera marked his whole aisle empty again. The summary said the customer was very pleased. He'd rung Kestrel, the shelf-camera company Ana runs, and Shelley and Rob made up both of them. Rob explains that the tone died when the call became a transcript, and what multimodal really means: one model holding the voice and the words together, not two models passing a note. Then CLIP, trained on four hundred million picture-and-caption pairs from the web, which lets Ana write a sentence like a photo of a crushed pallet instead of retraining her cameras. You'll also learn what CLIP can't do, from counting boxes to word order, and why its original release isn't for shipping. Coming up, a camera that reads photos the way chatbots read sentences.