AI & ML
Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests
arXiv cs.HC0 views
arXiv:2611.25200v1 Announce Type: new Abstract: Experimentation teams must decide which proposed website changes to test with live traffic. We evaluated whether forecasts from simulated shoppers could inform that choice. We screened 412 historical website A/B tests and evaluated four systems on the 318 that ran longer than 21 days. Each system predicted an original win, a variant win, or an inconclusive result. Page roles were masked and recorded outcomes were withheld during inference. Squoosh UX-1, a model fine-tuned on human browsing data, simulated 1,000 shoppers per arm and correctly classified 267 of 318 outcomes (84.0%, 95% Wilson interval 79.5-87.6%, treating experiments as independent). Qwen3 Next 80B, the strongest general-purpose configuration on outcome accuracy, correctly classified 196 of 318 (61.6%). UX-1 favored the variant in 126 experiments. In the live tests, 118 of those variants won (93.7%), four lost, and four had inconclusive results. The remaining 192 experiments included 19 additional variant wins.
Source:https://arxiv.org/abs/2611.25200PreviousOpen Response Explorer: a Structured Collaborative Writing System for Peer-Informed Open-Response AssignmentsNextTowards Restorative Urban Spaces: An Exploratory Field Study of Nature Videos on Public Multi-Display Signage
Related posts
Trying to get our Arms around AI efficiency
0Stack Overflow Blog
Why an Embedded Game Fails on Some Phones, and How to Debug It
0dev.to
Microsoft 365 vs Office 2024: The Break-Even Math
0dev.to
Master AI Evals: Build Reliable, Scalable Model Evaluation Pipelines
0dev.to
What to give someone who misses a place
0dev.to
Google is pushing OEMs against GrapheneOS and bootloader unlocking
0Hacker News