Reposting from my comment https://lemmy.world/post/37758804/20109240 which I recommend to check, as someone did a test with Dark Souls 1 and IMHO was unsurprisingly disappointing, namely it does recognize the game (honestly, not bad) and get the right boss (which name is literally on screen) and make kind of sometimes useful suggestions. But like... what's the point? Who would play a game and... NOT know its name? Or not be able to search based on a boss name or a weapon name with existing dedicated good online guides?

Anyway... if you still want to try yourself WITHOUT relying on Microsoft consider :

"If someone somehow wants to test this locally I suggest

• install locally a vision model, e.g. Moondream (which Ollama supports but alternatives too), then
• take a screenshot of your game,
• write a prompt like "How can I play this game better"
• query the vision model with the image and your prompt

marvel at how pointless and costly the whole setup is and how a basic query on e.g. DuckDuckGo with "game name" + prompt would yield way WAY better results from actual human, uninstall the whole, keep on playing with your actual brain.

At least now you can say you tried before you complain, rightfully, that it sucks.

For more check https://fabien.benetou.fr/Content/SelfHostingArtificialIntelligence

PS: I didn't actually try this, I'm too lazy for that right how but feel free to report back if you do!

Edit : 2 potential optimization (despite not being sure it ever makes sense in the first place!)

• do so automatically, e.g. ~/gaming_screenshots directory (via e.g. Spectacle shortcut) monitored via inotify then notify-send the suggestion, thus stay in game during the whole process
• fine tune on specific visual datasets, e.g. rely on fextra as mentioned in https://lemmy.world/post/37758804/20113877

" and again feel free to share back results.