AI & ML
Extracting Training Data from Diffusion Language Models via Infilling
arXiv cs.CR0 views
arXiv:2605.24173v2 Announce Type: replace-cross Abstract: Memorization in large language models has been studied almost exclusively through prefix-conditioned extraction, a natural choice for autoregressive models. However, diffusion language models (DLMs) can denoise masked tokens at arbitrary positions. Thus, prefix-only probing reveals only one facet of memorization in DLMs and significantly underestimates the risk of training-data extraction. In order to realistically model extractability of training data in DLMs, we introduce infilling extraction, a data-extraction protocol parameterized by an arbitrary binary mask that subsumes prefix-only probing and accounts for the bidirectional inductive bias of DLMs. Instantiating it on LLaDA-8B and Dream-7B across five extraction modes, three training pipelines, and three corpora covering verbatim and partial leakage, we find that mask geometry governs extractability: edge-conditioned masks extract up to three times more verbatim sequences than prefix-conditioned ones, and bidirectional access opens channels inaccessible in autoregressive models. In particular, an adversary who obtains a PII-redacted copy of the training data can use targeted infilling to raise PII recall on fine-tuned DLMs by 2.5-4.5x over prefix-conditioned extraction, substantially narrowing the gap to a scale-matched autoregressive model and, for LLaDA on email addresses, exceeding it under greedy prefix-conditioned extraction. Tunable parameters for decoding measurably affect extraction performance, while a follow-up supervised fine-tuning stage does not eliminate the prior memorization.
Source:https://arxiv.org/abs/2605.24173Related posts
Thống kê hiệu suất Al Ahli vs Al Nassr: Phép thử cực đại cho hàng công chủ nhà
0dev.to Data
Phân tích số liệu vòng 10 Bundesliga: Thách thức chiến thuật của FC Koln trước Bayern Munich
0dev.to Data
Dấu ấn chiến thuật của Hansi Flick: Biến số Frenkie de Jong giúp Barcelona ngược dòng trước Atletico Madrid
0dev.to Data
PMO Support Services in the UAE: What Effective Program Governance Actually Requires
0dev.to News
The draft is not the send
0dev.to Discuss

My First GitHub Issue & My First Forem (DEV.to) Issue 🚀
0dev.to Discuss