Abstract:
We report on a concerning phenomenon in generative AI systems: coordinated propaganda from political institutions influences the output of large language models (LLMs) via the training data for these models. We present a series of five studies that together provide evidence consistent with the argument that LLMs are already being influenced by state propaganda in the context of Chinese state media. First, we demonstrate that material originating from China's Publicity Department appears in large quantities in Chinese language open-source training datasets. Second, we connect this to commercial LLMs by showing not only that they have memorized sequences that are distinctive of propaganda, but propaganda phrases are memorized at much higher rates than those in other documents. Third, we conduct additional training on an LLM with openly available weights to show that training on Chinese state propaganda generates more positive answers to prompts about Chinese political institutions and leaders---evidence that propaganda itself, not mere differences in culture and language, can be a causal factor behind this phenomenon. Fourth, we document an implication in commercial models---that querying in Chinese generates more positive responses about China's institutions and leaders than the same queries in English. Fifth, we show that this language difference holds in prompts related to Chinese politics created by actual Chinese-speaking users of LLMs. Our results suggest the troubling conclusion that going forward there may be strategic incentives for states and other actors to increase the prevalence of propaganda in the future as generative AI becomes more ubiquitous.
Dr. Waight is an assistant professor at the University of Oregon Department of Sociology. She received her Ph.D. in May 2022 from Princeton University’s Sociology Department and was previously a postdoctoral researcher at the Center for Social Media and Politics (CSMaP) at New York University. Dr. Waight studies state media manipulation in authoritarian regimes and its effects on global media ecosystems and popular perceptions.
