When we say "Sidekick AI watches your screen," most people imagine something like OCR reading text off a HUD. The reality is more interesting. The AI looks at your game the way you do: as a complete visual scene with spatial relationships, movement, color, and context. Here's what that actually means.
What the AI sees (and doesn't see)
Sidekick AI uses a vision language model (VLM) to analyze your screen. Unlike traditional computer vision that looks for specific patterns (health bar pixels, minimap icons), a VLM understands the scene holistically. It processes the entire frame as an image and generates a text description of what it observes.
When you're fighting Malenia in Elden Ring, the VLM doesn't just see "boss health bar at 40%." It sees: a humanoid boss with a prosthetic arm in a large arena, the player character at medium range with a greatsword, the boss mid-animation in what looks like the wind-up for an aerial attack, health bars for both characters, and the general visual language of a Soulslike boss fight.
What it does NOT see: game memory, internal state variables, exact damage numbers, or frame data. It works purely from the visual output on your screen, the same pixels you see.
The pipeline: screen to voice in under 3 seconds
Here's what happens every time Sidekick AI gives you a tip:
- Screen capture. A frame is grabbed from your display. This happens outside the game process, no injection, no hooks, no anti-cheat risk. Think of it like a screenshot tool running in the background.
- Vision analysis.The frame goes to the VLM, which outputs a structured understanding: what game is this, what's happening right now, what phase is the boss in, what is the player doing, and is there something the player should know.
- Context integration.The current frame analysis is combined with context from recent frames. The AI remembers what just happened (you dodged, you healed, the boss transitioned phases) and uses that history to give advice that's relevant to the trajectory of the fight, not just a single snapshot.
- Tip generation. Based on the analysis and context, the AI decides whether to say something and what to say. Not every frame triggers a tip. The AI is trained to speak up when something actionable is happening: a dangerous attack wind-up, a heal window, a phase transition, a puzzle element you might have missed.
- Voice synthesis. The tip text is converted to natural speech and played through your headset. Low-latency TTS means the voice arrives while the information is still relevant, not 10 seconds after the boss already hit you.
Total time from screen capture to voice in your ear: roughly 1-3 seconds. Fast enough for boss fight coaching. Not fast enough for frame-perfect reaction calls, but that's not the goal. The goal is strategic guidance: "she's about to do the dive attack, get ready to dodge sideways" not "press B now."
Why vision AI, not game integration?
The obvious question: why not just read the game's memory directly? If you hooked into Elden Ring's process, you could read exact boss HP, attack IDs, and frame data with perfect accuracy. Three reasons we don't do this:
- Anti-cheat. Reading game memory triggers anti-cheat in most games. Players would get banned. Vision AI works from outside the game process, like a screen recorder. No game files are touched, no memory is read, no code is injected.
- Universal compatibility. Game memory integration requires reverse-engineering each game individually. Vision AI works with any game that runs on your screen. One system, every game. No per-game plugins, no updates when games patch.
- It's closer to how humans help each other. When a friend watches you play and says "dodge now," they're reading the same screen you are. Vision AI does the same thing. This means the advice is grounded in what you can actually see and react to, not hidden game state you'd never know about.
What makes gaming hard for vision AI
Gaming is one of the harder applications for vision AI. Unlike analyzing a photo or reading a document, game screens are:
- Constantly moving. Every frame is different. The AI needs to track state changes across time, not just analyze a single image.
- Visually dense. A boss fight has particle effects, multiple characters, UI overlays, environmental detail, and motion blur. Separating signal from noise is hard.
- Context-dependent.The same visual can mean different things in different games. A red flash might mean "parry window" in Sekiro and "incoming damage" in Hollow Knight. The AI needs game-level context, not just pixel-level recognition.
- Time-sensitive. A tip that arrives 5 seconds late is worse than no tip at all. The latency budget is tight.
These challenges are exactly why this wasn't possible until recently. Vision language models capable of real-time game scene understanding at consumer-grade latency are a 2025-2026 development. The technology that makes Sidekick AI work literally did not exist two years ago.
When vision AI gets it wrong
Honesty time: the AI makes mistakes. Common failure modes include:
- Phase misidentification.The AI might think a boss is in phase 2 when it's still in phase 1, leading to premature strategy advice.
- Visual occlusion. If particle effects cover the boss, the AI might miss an attack wind-up it would normally catch.
- Unfamiliar games. The AI is better with popular, well-known games where the VLM has seen similar visuals in training. Obscure indie games with unique visual styles get less accurate analysis.
- Speed vs. accuracy tradeoff.To stay under 3 seconds, the AI sometimes sacrifices analysis depth. It might give a correct but generic tip ("heal now") instead of a specific one ("heal now because she's in the recovery animation after the third combo hit").
The system is designed so that wrong tips are unhelpful, never harmful. The AI only speaks suggestions through voice. It never takes actions in your game. A bad tip wastes a few seconds of attention. A good tip saves you from a death you didn't see coming.
What's next for vision AI in gaming
The current system is v1. What gets better over time:
- Faster models. As VLMs get more efficient, the latency drops. Sub-1-second response times would enable reaction-level coaching, not just strategic advice.
- Game-specific fine-tuning. The current system uses general-purpose vision. Fine-tuning on specific games (Elden Ring boss animations, BG3 puzzle layouts) would dramatically improve accuracy for those titles.
- Multi-frame reasoning. Better temporal understanding means the AI can track combos across multiple frames, predict attack sequences, and give earlier warnings.
- Player modeling. Learning your patterns across sessions. If you always get hit by the same attack, the AI prioritizes coaching on that specific mechanic.
The gap between "AI that can understand a game screen" and "AI that coaches like a skilled friend" is closing fast. We're building Sidekick AI at the intersection of that gap.