
S1E10. More Than Text: Modalities, CLIP, and One Shared Space
transcript
show notes
Oh, brilliant, said a warehouse manager on a support call, in the flattest voice you've ever heard, after a camera marked his whole aisle empty again. The summary said the customer was very pleased. He'd rung Kestrel, the shelf-camera company Ana runs, and Shelley and Rob made up both of them. Rob explains that the tone died when the call became a transcript, and what multimodal really means: one model holding the voice and the words together, not two models passing a note. Then CLIP, trained on four hundred million picture-and-caption pairs from the web, which lets Ana write a sentence like a photo of a crushed pallet instead of retraining her cameras. You'll also learn what CLIP can't do, from counting boxes to word order, and why its original release isn't for shipping. Coming up, a camera that reads photos the way chatbots read sentences.