AI & ML
Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering
arXiv cs.IR0 views
arXiv:2605.24297v3 Announce Type: replace Abstract: Two questions regarding practitioners' use of patent embeddings arise: (i) Does one fine-tuning recipe suffice for all downstream applications? (ii) Is fine-tuning on one patent landscape sufficient for downstream application on other landscapes? By evaluating 22 pre-trained embedding models (ranging from 22M to 12B parameters) on three tasks -- information retrieval, classification, and clustering -- on 113,148 WIPO patents for assistive technology (46,069 citation queries) and on an external DAPFAM dataset, we find that two results cast doubt on the prevailing wisdom. (i) The optimal fine-tuning recipe depends on the downstream task: cross-sectional alignment (recipe R3) provides the largest improvements to retrieval performance (+7.1% nDCG@10), whereas a combined signal recipe (recipe R4) is better suited to classification (+7.1 F1) and clustering (+10.9 V-measure); a matched data control confirms that differences in training dataset size are not a contributing factor. (ii) Single-landscape fine-tuning hampers cross-landscape information retrieval: fine-tuning on one landscape significantly degrades cross-domain retrieval for 5 of 8 model-recipe combinations on the DAPFAM corpus, with the stronger zero-shot models suffering most. While within-family scaling is consistent (Qwen3 0.6B->4B->8B; Llama-Nemotron 1B->8B), cross-family scaling is erratic; the 12B KaLM-Gemma3 is ranked 8th on TAC retrieval performance, following prefix modification. Title+Abstract+Claims is the ubiquitous best text view, and all models suffer from a 55-65% gap between IN and OUT-of-domain performance which cannot be mitigated by hybrid BM25-dense fusion. Code and evaluation framework are publicly available.
Source:https://arxiv.org/abs/2605.24297PreviousMCGI: Manifold-Consistent Graph Indexing for Billion-Scale Disk-Resident Vector SearchNextPULSAR: Pooled Unified Late-Interaction Search and Retrieval for Enterprise Visual Document RAG
Related posts
Why can't my AI agent use mobile apps?
0dev.to AI
24 KI-Agenten für einen Film – und was das über das Sehenlernen verrät
0dev.to AI
How AI Agent Memory Architectures Split Working Memory from Episodic Storage
0dev.to AI
Your Page Listened for Messages From the Checkout Frame and Took Them From Any Window
0dev.to Security
Field report: 20359 real pull requests scanned, 6.5% flagged, 46 of our own mistakes fixed in public
0dev.to DevOps
Evaluating API Portals for Autonomous Agents: Schemas, Perks, and Registry Data
0dev.to DevOps