NeurIPS 2020

De-Anonymizing Text by Fingerprinting Language Generation


Meta Review

This paper generated a significant amount of discussion. SCIENTIFIC: Regarding the purely scientific aspects, the reviewers discussed about the novelty of the contribution. On the one hand, if one takes the point of view of the security community, the proposed attack and defense are known and the vulnerability is not surprising since any data-dependent accesses is prone to side-channel attacks. On the other hand, from the point of view of the machine learning community where these concerns are currently not well known, the paper presents very clearly a reasonable approach to start thinking about security of machine learning and NLP code using actual algorithms that text generation researchers and practitioners use. The paper can thus serve a useful cross-discipline discussion. In the end, there was a consensus to say that the latter aspect outweighs the former. ETHICS: The paper was also flagged by one reviewer as raising potential ethical concerns due to the use of data scrapped from the infamous Silk Road forum and (to a lesser extent) the lack of clear policy related to responsible vulnerabilities disclosure communication towards Hugging Face. The paper was therefore sent to three ethics experts for review, which then generated further discussion among reviewers, myself and the senior area chair. Many perspectives were considered regarding the issues such as (quoting the ethics reviews): - data provenance: "The URL for the Silk Road data does not clarify the provenance much". - offensiveness of the content: "the archive dataset used for this case study contains inappropriate and offensive text content, detailing now known illegal activity and explicit racial/misogynistic slurs. But it was also recognized that many/all forum-like datasets such as comments in Fox News articles, Washington Post articles, and on Amazon would have on the order of .01-10% content one can find offensive. - setting a potentially dangerous precedent: "If we ban this dataset because it has some offensive content, will we ban all datasets that have any offensive content?" - appropriateness of the match between the choice of dataset and the goals of the research: the dataset was seen as a use-case where "individuals are truly speaking their mind and wish for anonymity", which is important for the purpose of the research. It was pointed out that other sources such as text from online health forums (e.g., https://zenodo.org/record/1479354) could also be appropriate although they may be more formal and not as individualized communication. DECISION AND ACTIONS FROM THE AUTHORS: In the end, although there were scientific/ethical pros and cons, the final decision is to accept this paper conditionally on the following actions being taken by the authors: 1/ In the final version of the paper, include an explicit note warning people that the dataset has offensive material, and that their use of it is because it is representative of anonymous informal communications, and that the data should not be construed to represent the opinions of the authors. 2/ Notify Hugging Face of the vulnerability before publication. Following the suggestions of R1, we also recommend the authors to add a further case study to assess the influence of the sequence length. ******************************* Note from Program Chairs: The camera-ready version of this paper has been reviewed with regard to the conditions listed above, and this paper is now fully accepted for publication.