Description:Description: Causal inference methods allow us to learn about the effects of interventions, which is crucial for evidence informed decision making. However, interpretational errors may threaten the successful translation from scientific findings to actionable implications. A common such error, termed identity slippage was described and formalized by Sarvet et al. (2023). The idea behind identity slippage is relatively straightforward: Researchers are interested in one causal estimand, but target a different estimand, likely because they believe that the assumptions required for the identification of the original estimand do not hold in their setting. Then, they interpret results in a way that suggests the original estimand was targeted, and not the one that was actually estimated. Sarvet et al. (2023) argue that each causal estimate corresponds to a set of valid statements, which they call the interpretive map. Empirical measurement of identity slippage requires assessing whether c laims fall within the interpretive map defined by a study’s targeted estimand and its estimation results. In previous studies where this phenomenon was studied (Sarvet et al. (2023) and Locher, Stensrud and Sarvet (unpublished manuscript)), authors have relied on human classification of relevant variables. The goal of this research project is to assess the potential of large language models (LLMs) to automate this task. This may facilitate the measurement of interpretational errors in future studies, and if deployed in the publication process, prevent such errors altogether.

Contact + direct supervisor: miquel.serraburriel@uzh.ch
Internal supervisor: torsten.hothorn@uzh.ch

References:
Aaron L. Sarvet, Mats J. Stensrud, and Lan Wen. Interpretational errors in statistical causal inference, December 2023. URL http://arxiv.org/abs/2312.07610. arXiv:2312.07610 [stat].

Description:The pursuit of causal knowledge lies at the heart of many scientific domains, yet establishing causal relationships with observational data remains a challenging task that requires assumptions on the structure of the data generating process. Unwilling to make such assumptions, researchers frequently abandon their original causal objectives and instead restrict their analyses to associational parameters, a practice supported, or even enforced, by editorial policies of many journals (Hernan, 2018). While this shift may spare researchers from making assumptions whose truth is attached to a considerable degree of uncertainty, it carries the risk of interpretational errors. Findings based on associations may be presented, or understood, as if they reflect causal effects, thereby creating the impression that the original causal objective was achieved. Such errors represent instances of what Sarvet et al. (2023) termed identity slippage. Research is consumed by diverse stakeholders throug h multiple outlets, including scientific journals or traditional news media, and identity slippage may occur on different steps on this pathway. In this research project, we focus specifically on claims made in news reporting. We plan to study whether news coverage of studies that refrain from causal language and limit themselves to associational parameters conveys results in a way that clarifies the limited scope of the study in terms of causality, or whether reports lead readers to believe that scientists have produced causal knowledge.

Contact + direct supervisor: miquel.serraburriel@uzh.ch
Internal supervisor: torsten.hothorn@uzh.ch

References:
Miguel A. Hernan. The C-Word: Scientific Euphemisms Do Not Improve Causal Inference From Observational Data. American Journal of Public Health, 108(5):616–619, May 2018. ISSN 0090-0036. doi: 10.2105/AJPH.2018.304337. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5888052/
Aaron L. Sarvet, Mats J. Stensrud, and Lan Wen. Interpretational errors in statistical causal inference, December 2023. URL http://arxiv.org/abs/2312.07610. arXiv:2312.07610 [stat].

Description: Peer review is the cornerstone of scientific publishing, yet its tangible contributions to research quality remain underexplored. This study aims to quantify the added value of peer review by analyzing matched preprint–publication pairs across disciplines. The project will compare preprints to their final published versions using natural language processing to detect textual and structural changes and relate these changes to time under review. Data will be collected via public APIs (e.g., bioRxiv, arXiv, CrossRef) and processed using semantic similarity metrics and change classification models. The study will provide empirical evidence on how, when, and to what extent peer review alters scientific manuscripts, offering insights for improving transparency and efficiency in the publication process.

Contact + direct supervisor: miquel.serraburriel@uzh.ch
Internal supervisor: torsten.hothorn@uzh.ch