Have a question about something you can see but can’t describe? Just ask Google.
Google has unveiled what seems like a really useful and almost scarily advanced new way to search with images.
Google Lens already lets you search based on an image. For example, if you take a picture of an elephant, you’ll probably get Google Lens search results back for “elephant.” But now, you can tap a picture you’ve taken, or one that you’ve saved in your library, and ask a question about it. Take the elephant: Just tap the photo for the ability to “Add Questions,” and a text box will pop up where you can plumb Google for more information about that specific image, like “What kind of elephant is this?,” or “How many of these elephants are left in the world?”
That involves so many layers of AI processing it’s actually hard to comprehend. It understands what’s in the picture, it understands your question, and it understands how your question relates to the picture. And, of course, it (ostensibly) gives you the answers you’re looking for. What?!?
Making it all possible is a new, more advanced AI system called Multitask Unified Model (or MUM), announced in May, that is beginning to power Search. Google has been slowly rolling out applications for the new tech that’s capable of processing queries in more complex ways, and delivering results that Google believes will be more relevant or instructive than before.
The change to Lens is one of the most eye-popping yet, and the examples for queries Google provides show off just how smart MUM is. You can ask Google Lens about a pattern on a shirt, and whether that same pattern comes available in socks: Et voila, you get the exact product listing.
Or here’s an example of someone snapping a picture of a broken bicycle component. Showing Google Lens the picture of the broken part, and asking “how to fix,” delivers both what the exact broken part is, AND how to fix it.
Both of these sorts of questions would be difficult to answer without the visual component. For the pattern, you’d likely get non-specific results for a verbal search for, say, socks in a floral pattern. And for the bike example, you’d probably have to figure out that the specific bike part that’s broken is called a “derailleur” (Who knew?) before you could even start worrying about how to fix it.
The machines are getting smarter, which is a possibly scary prospect for the world, but great news for finding answers to tricky questions.
