Examining the generalizability of research findings from archival data

Author(s)

, Andrew Delios, Elena Giulia Clemente, Tao Wu, Hongbin Tan, Yong Wang, Michael Gordon, Domenico Viganola, Zhaowei Chen, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Eric Luis Uhlmann, Ahmad M.Abd Al-Aziz, Ajay T. Abraham, Jais Trojan, Matus Adamkovic, Elena Agadullina, Jungsoo Ahn, Cinla Akinci, Handan Akkas, David Albrecht, Shilaan Alzahawi, Marcio Amaral-Baptista, Rahul Anand, Kevin Francis U. Ang, Frederik Anseel, John Jamir Benzon R. Aruta, Mujeeba Ashraf, Bradley J. Baker, Xueqi Bao, Ernest Baskin, Hanoku Bathula, Christopher W. Bauman, Jozef Bavolar, Secil Bayraktar, Stephanie E. Beckman, Aaron S. Benjamin, Stephanie E.V. Brown, Jeffrey Buckley, Ricardo E. Buitrago, Jefferson L. Bution, Nick Byrd, Clara Carrera, Eugene M. Caruso, Minxia Chen, Lin Chen, Eyyub Ensari Cicerali, Alexander Mohr, Leopold H.O. Roth, Hao Sun

Abstract

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Organisation(s)

Quantum Optics, Quantum Nanophysics and Quantum Information, Department of Accounting, Innovation and Strategy, Department of Occupational, Economic and Social Psychology

External organisation(s)

National University of Singapore (NUS), Stockholm School of Economics, Chinese University of HongKong, Tongji University, Xi'an Jiaotong University (XJTU), Massey University, The World Bank, Leopold-Franzens-Universität Innsbruck, INSEAD Asia Campus, Swedish House of Finance, British University in Egypt, Seattle University, Keele University, University of Prešov, National Research University Higher School of Economics, University of Western Ontario, University of St. Andrews, Ankara Science University, Maastricht University Medical Center, Stanford University, Iscte-University Institute of Lisbon , Aarhus University, Value Care Health Systems, University of New South Wales, De La Salle University, University of the Punjab, Temple University, Philadelphia, INSEAD, Europe, Saint Joseph's University, University of Auckland, University of California, Irvine, Pavol Jozef Šafárik University in Košice, Toulouse Business School, Madison College, University of Illinois at Urbana-Champaign, Texas A&M University, KTH - Royal Institute of Technology, Del Rosario University, Instituto Oceanográfico, Stevens Institute of Technology, UCLA Anderson School of Management, Nisantasi University, Wirtschaftsuniversität Wien (WU), Xiamen University

Journal

Proceedings of the National Academy of Sciences of the United States of America

Volume

119

ISSN

0027-8424

DOI

https://doi.org/10.1073/pnas.2120377119

Publication date

07-2022

Peer reviewed

Yes

Austrian Fields of Science 2012

501021 Social psychology

Keywords

ASJC Scopus subject areas

General

Portal url

https://ucrisportal.univie.ac.at/en/publications/88893a44-8203-4bc1-b265-6750007bb1c4