AI & ML
StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval
arXiv cs.IR0 views
arXiv:2611.18516v1 Announce Type: cross Abstract: Screen history can serve as searchable visual memory for people and agents, but unlike an offline corpus, it must be encoded and indexed as each screenshot arrives. OCR-based indexing is fast but loses visual information; visual encoding takes over two seconds. Yet screens change incrementally: most pixels repeat within and across screenshots. This raises a question: do visual encoders need to recompute their representations from scratch for every new screenshot? We present StreamEmbed, a system for incremental visual embedding that exploits this redundancy. Guided solely by pixel differences, it combines spatial token pruning, which skips blocks that repeat a neighbor, with temporal KV reuse, which reuses cached keys and values for blocks unchanged across screenshots. A small learned adapter compensates for pruning and reuse to preserve fidelity to full-screenshot embeddings. On ScreenMemEval, our benchmark of 10,426 questions over 184 hours of screen histories from two humans and a computer-use agent, StreamEmbed sets a new accuracy-latency Pareto frontier on an M2 Pro GPU: it encodes up to 3.7x faster than full-screenshot encoding without sacrificing accuracy and improves accuracy over OCR-based indexing by up to 6.3 Recall@10 points at matched latency. A memory-efficient vector index stores a simulated year of screenshots in 0.13 GB of RAM. Together, these results make real-time indexing of screen history in its native visual form practical, laying a foundation for fully local visual assistants.
Source:https://arxiv.org/abs/2611.18516PreviousRecMem: Recurrent Memory Compression for Long-Sequence Generative RecommendationNextScrapeGraphAI-100k: Dataset for Schema-Constrained LLM Generation
Related posts

Add ChatGPT-style AI chat to your Expo app in 5 minutes
0dev.to OpenSource
sheet_route: نوافذ bottom sheet في Flutter كـ routes حقيقية مع snap points و go_router
0dev.to OpenSource
Every family WhatsApp group in India has the same problem
0dev.to OpenSource

How to Use Git Stash Step by Step
0dev.to Git
GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers
0arXiv cs.DB
Larch: Learned Query Optimization for Semantic Predicates
0arXiv cs.DB