| I built a small library called whitetree for exact Mahalanobis nearest-neighbour search on low-dimensional sensor data that keeps arriving. The idea is old. Whiten with the Cholesky factor of the covariance so Mahalanobis becomes Euclidean, then keep several scipy cKDTrees instead of one so inserts and deletes never force a full rebuild. Three measurements came out of it that I haven't seen stated plainly anywhere, so I'm posting those rather than a pitch. The short version first. On the static side it is 40 to 300x faster than sklearn's BallTree(mahalanobis) and 7 to 60x faster than FAISS Flat at 500k points, and on the interleaved side it is the only exact option I found that keeps up with one insert and one delete per query. It's numpy and scipy only, one writer thread with any number of readers, and results match a static cKDTree exactly (distance error 0.0) after any mix of inserts and deletes.
Setup, briefly. Covariance in float64 with a scale-relative ridge and Ledoit-Wolf shrinkage only when n < 5d. Trees kept largest-first, each at least 32x the next, merged and rebuilt when a new one breaks that. The largest tree's k-th distance bounds the rest. Deletes are tombstones. Benchmarks follow the ann-benchmarks and big-ann-benchmarks streaming protocols, recall against float64 brute force. Code, tests, benchmark scripts, and a design note with the numbers behind each decision are at https://github.com/whitetree-dev/whitetree A question for people who run exact low-dimensional kNN on streams. Is there a dynamic exact index I should've benchmarked against and missed? I compared FAISS IndexFlatL2 with IDMap2, scipy cKDTree and sklearn BallTree rebuilt per query, and numpy brute force. If something beats ~1,100 insert/delete/query steps per second at 200k points on one core, I'd like to know. [link] [留言] |
Cloud
Read original: https://www.reddit.com/r/MachineLearning/comments/1wfg8e3/got_scipys_kdtree_to_handle_inserts_and_deletes/
Got scipy's KD-tree to handle inserts and deletes without rebuilding. Three things I learned [P]
/u/monononon34 Reddit r/MachineLearning
3 views
← Previous
Apple is reportedly working on iPhone game controllers
Next →
Show Reversibility Proof Before Pasting Agent Undo Copy
Related
Why I'm Migrating My Blog to Astro
Cloud
0
Dev.to (EN Zone)
Se acabó el limite de los 15 minutos: un análisis a fondo del tiempo de espera de 90 minutos de AWS Lambda
Cloud
3
Dev.to (EN Zone)
Design YouTube — Video Upload, Transcoding & CDN Delivery at Scale (with Production .NET Code)
Cloud
4
Dev.to (EN Zone)
Send Transactional Email from a Verified Domain with an API or SMTP
Cloud
5
Dev.to (EN Zone)
Comments0
No comments yet — be the first