Nicholas Carlini
Nicholas Carlini | |
|---|---|
| Alma mater | University of California, Berkeley (PhD) |
| Scientific career | |
| Fields | Computer Security |
| Institutions | Google DeepMind |
| Thesis | Evaluation and Design of Robust Neural Network Defenses (2018) |
| Doctoral advisor | David Wagner |
| Website | nicholas.carlini.com |
Nicholas Carlini is a researcher affiliated with Google DeepMind who has published research in the field of computer security and machine learning. He is known for his work on adversarial machine learning.
Education
[edit]Nicholas Carlini obtained his Bachelor of Arts in Computer Science and Mathematics from the University of California, Berkeley in 2013.[1] He then continued his studies at the same university, where he pursued a PhD under the supervision of David Wagner, completing it in 2018.[1][2][3]
Research
[edit]Nicholas Carlini is known for his work on adversarial machine learning. In 2016, he working alongside David Wagner to develop the Carlini & Wagner attack, a method of generating adversarial examples against machine learning models. The attack was proved to be useful against defensive distillation, a popular mechanism where a student model is trained based on the features of a parent model to increase the robustness and generalizability of student models. The attack gained popularity when it was shown that the methodology was also effective against most other defences, rendering them ineffective.[4][5] In 2018, Carlini demonstrated a attack against Mozilla Foundation's DeepSpeech model where he showed that by hiding malicious commands inside normal speech input the speech model would respond to the hidden commands even when the commands were not discernible by humans.[6][7] In the same year, Carlini and his team at UC Berkeley showed that out of the 11 papers presenting defenses to adversarial attacks accepted in that year's ICLR conference, seven of the defenses could be broken.[8] More recently, he and his team have worked on large-language model creating a questionnaire where humans typically scored 35% whereas AI models scored in the 40 percents, with GPT-3 getting 38% which could be improved to 40% through few shot prompting. The best performer in the test was UnifiedQA a model developed by Google specifically for answer question and answer sets.[9] Recently, Carlini has also developed methods to cause large language models like ChatGPT to answer harmful questions like how to construct bombs.[10][11]
He is also well known for his work studying the privacy of machine learning models. In 2020 he showed for the first time that large language models would memorize some text data that they were trained on. For example, he found that GPT-2 could output personally identifiable information.[12] He then lead an analysis of larger models and studied how memorization increased with model size. Then, in 2022 he showed the same vulnerability in generative image models, and specifically diffusion models, by showing that Stable Diffusion could output images of people's faces that it was trained on.[13] Following on this, Carlini then showed that ChatGPT would also sometimes output exact copies of webpages it was trained on, including personally identifiable information.[14] Several of these studies have since been referenced by the courts in debating the copyright status of AI models.[15]
Awards
[edit]- Best Student Paper Award, IEEE S&P 2017 ("Towards Evaluating the Robustness of Neural Networks")
- Best Paper Award, ICML 2018 ("Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples")
- Distinguished Paper Award, USENIX 2021 ("Poisoning the Unlabeled Dataset of Semi-Supervised Learning")
- Distinguished Paper Award, USENIX 2023 ("Tight Auditing of Differentially Private Machine Learning")
- Best Paper Award, ICML 2024 ("Stealing Part of a Production Language Model")
- Best Paper Award, ICML 2024 ("Considerations for Differentially Private Learning with Large-Scale Public Pretraining")
Other Work
[edit]Carlini received the Best of Show award at the 2020 IOCCC for implementing a tic tac toe game entirely with calls to printf, expanding on work from a research paper of his from 2015. The judges commented on his submission "This year’s Best of Show (carlini) is such a novel way of obfuscation that it would be worth of a special mention in the (future) Best of IOCCC list!"[16]
References
[edit]- ^ a b "Nicholas Carlini". nicholas.carlini.com. Retrieved 2024-06-04.
- ^ "Nicholas Carlini". AI for Good. Retrieved 2024-06-04.
- ^ "Graduates". people.eecs.berkeley.edu. Retrieved 2024-06-04.
- ^ Pujari, Medha; Cherukuri, Bhanu Prakash; Javaid, Ahmad Y; Sun, Weiqing (2022-07-27). "An Approach to Improve the Robustness of Machine Learning based Intrusion Detection System Models Against the Carlini-Wagner Attack". 2022 IEEE International Conference on Cyber Security and Resilience (CSR). IEEE. pp. 62–67. doi:10.1109/CSR54599.2022.9850306. ISBN 978-1-6654-9952-1.
- ^ Schwab, Katharine (12 December 2017). "How To Fool A Neural Network". Retrieved 4 June 2023.
- ^ Smith, Craig S. (2018-05-10). "Alexa and Siri Can Hear This Hidden Command. You Can't". The New York Times. ISSN 0362-4331. Retrieved 2024-06-04.
- ^ "As voice assistants go mainstream, researchers warn of vulnerabilities". CNET. Retrieved 2024-06-04.
- ^ Simonite, Tom. "AI Has a Hallucination Problem That's Proving Tough to Fix". Wired. ISSN 1059-1028. Retrieved 2024-06-04.
- ^ Hutson, Matthew (2021-03-03). "Robo-writers: the rise and risks of language-generating AI". Nature. 591 (7848): 22–25. Bibcode:2021Natur.591...22H. doi:10.1038/d41586-021-00530-0. PMID 33658699.
- ^ "AI chatbots can be tricked into misbehaving. Can scientists stop it?". 2024-02-01. Retrieved 2024-07-26.
- ^ Metz, Cade (2023-07-27). "Researchers Poke Holes in Safety Controls of ChatGPT and Other Chatbots". The New York Times. ISSN 0362-4331. Retrieved 2024-07-26.
- ^ "What does GPT-3 "know" about me?". MIT Technology Review. Retrieved 2024-07-26.
- ^ Edwards, Benj (2023-02-01). "Paper: Stable Diffusion "memorizes" some images, sparking privacy concerns". Ars Technica. Retrieved 2024-07-26.
- ^ Newman, Lily Hay. "ChatGPT Spit Out Sensitive Data When Told to Repeat 'Poem' Forever". Wired. ISSN 1059-1028. Retrieved 2024-07-26.
- ^ J. DOE 1 (United states district court northern district of California), Text.
- ^ "The 27th IOCCC". www.ioccc.org. Retrieved 2024-07-26.