Correcting Inferences in AI-Assisted Content Analysis

Content analysis methods measures social categories and constructs in unstructured data. In social science, human annotators interpret and categorize communication data such as behaviors, messages, or images to build or test theories. Machine learning classifiers, increasingly in the form of language models, can help automate this process. The trouble is that machines and humans may interpret the data in different ways.  Divergence between machine and human classifiers creates problems for inference, but methods that use human and machine classifications together can reduce bias and variance. Practical trade-offs between methods will be discussed with an emphasis on suitability for causal inference estimators.