Until Google formally announced the Googlebook at the Android Show, we had little idea how Gemini would be a part of the experience. But now not only do we know how Magic Pointer, its signature feature, will work, but we can try it right now. And while there’s a lot wrong with it, I can’t help but be a little excited.
Google’s Magic Pointer turns your cursor into an AI assistant, and you can test it now
Or wait until the Googlebook’s arrival this fall
Magic Pointer is Gemini on speed dial
No need to use your keyboard
The concept of Magic Pointer is simple. Point your cursor at anything, give it a wiggle, and Gemini will prime itself for input.
These inputs can be as simple as “Where is this?” or involve more complex actions like combining different images to show how different outfits might look on the same person.
Magic Pointer is meant to understand whatever you’re pointing at and react accordingly. For example, it could recognize the separate text and images on a concert poster.
But Google’s goals with this tool are much more ambitious than simple recognition. Instead, Magic Pointer (theoretically) can understand why something matters to you and how to act on it. So, to use the concert poster example above, selecting it with Magic Pointer could create a calendar event with all the information filled out.
The best way to think about Magic Pointer’s intent is to imagine someone else is in control of your mouse and you’re giving them verbal instructions. You would point at different areas of the screen, instructing them on what to do with each element you point at.
Fortunately, Google isn’t leaving it all to our imaginations and edited promotional material. It has two experiments live now where you can try it in action. I gave them a go, and while my experiences were mixed, I’m more intrigued than ever.
Magic Pointer has a learning curve, but it does work
Like any AI tool, you won’t get it right the first time
The first Magic Pointer experiment from Google involves planning out a trip using a variety of reference images of London landmarks and Google Maps. I pointed to an image of the London Eye, asked Gemini where it was, then followed up with a request to find restaurants nearby.
Within seconds, Gemini had understood my request, and it was all the more impressive that the only tools I used were my voice and my cursor. But when I tried to complete the third task (asking Gemini to get directions), things fell apart.
What should have happened is that after asking Gemini, “How do I go from here to there?” while moving my cursor between two images, it would create directions in Maps. Instead, I got a complete meltdown.
Not only did the app completely fail to understand that I was pointing between two locations and repeatedly gave me directions from Hyde Park to Hyde Park, but it also thought I was trying to interact with the other experiment open in a different tab, despite my cursor being nowhere near the window.
If Magic Pointer is meant to interact only with the objects under my cursor, why was Gemini triggering image edits in the other tab? I’m worried this will cause mass confusion if I have multiple windows open on a Googlebook.
Eventually, I managed to complete the app’s final task by waiting five seconds after saying “here” before finishing my instruction. It’s another example of an AI tool that’s advertised to work for you but ends up acting like a dog that needs instructions spoken in the most basic of terms.
Nevertheless, after playing with Magic Pointer for an hour, I was able to complete the tasks in both experiments repeatedly with no hiccups. Usually, I’m much more critical of AI tools failing, but then I think back to coworkers asking “Sorry, what were you pointing to? Can you repeat that?” and realize that perhaps it’s not so dumb.
Magic Pointer might be more fun than useful
On the surface, Magic Pointer looks to be a more precise version of Circle to Search, but it’s much more than that. I haven’t adopted Circle to Search as a habit due to a series of unsuccessful experiments, and it’s too slow to be helpful.
But if Magic Pointer on Googlebooks can be as responsive as a person, then I can see it saving hours.
But what intrigues me most about Magic Pointer is far more childish.
Since ChatGPT launched, my initial enthusiasm about this exciting technology was destroyed by an unstoppable stream of broken AI tools that completely failed to live up to my sci-fi expectations.
But lounging in my chair, waving my hand like a dissatisfied Roman emperor? Perhaps Magic Pointer will finally fulfill the AI fantasies I thought long dead.
