Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspaper

https://aclanthology.org/2024.inlg-main.14/

Transcript

1. Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspaper Shotaro Ishihara (Nikkei Inc.) and Hiromu Takahashi Research Question: 1) Do Japanese PLMs memorize the training data as well as the English PLMs? 2) Is the memorized training data detectable as well as the English PLMs? Approach: 1. Pre-training GPT-2 models using Japanese newspaper articles. 2. Quantifying memorization using the generated candidate and reference. 3. Membership inference attacks using the generated candidate. Findings: 1. Japanese PLMs sometimes “copy and paste” on a large scale (max 48 chars). 2. We replicated the English empirical finding that memorization is related to duplication, model size, and prompt length. 3. Experiments demonstrated that the training data was detected from PLMs even in Japanese (AUC 0.60). The more duplicates and the longer the prompt, the easier the detection was. The more epochs (more duplication), the larger the model size, the longer the prompt, the more memorization. The more duplicates and the longer the prompt, the easier the detection was.