Bernard J. Koch and David Peterson, researchers at the intersection of artificial intelligence and computational social science, have published a provocative study that traces the evolution of AI research from its foundational era to the present day. Their work, titled “From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolution,” offers a critical examination of how the field of AI has been shaped by shifting priorities, funding mechanisms, and evaluation systems.
Koch and Peterson begin by outlining the early years of AI research, from the 1950s to the late 1980s, when the field was approached as a “basic science” driven by autonomous exploration and organic assessments of progress. During this period, researchers relied on peer review and theoretical consensus to gauge advancements, but the lack of clear, measurable outcomes led to a significant funding retrenchment in the 1980s—a period now known as the “AI Winter.”
The turning point came with an intervention by the U.S. government, which reoriented AI research toward measurable progress on tasks of military and commercial interest. This shift introduced a new evaluation system called “benchmarking,” which focused on increasing predictive accuracy on example datasets. Benchmarking provided a clear, objective way to quantify progress, clarifying the roles of scientists, accelerating talent integration, and offering clear signals of significance and advancement.
However, the researchers argue that this streamlined approach to science has come at a cost. The consolidation around external interests and the inherent conservatism of benchmarking have disincentivized exploration beyond scaling monoculture. This has led to an “epistemic monoculture” in AI research, where the field has become heavily focused on building ever-larger deep learning models, often at the expense of addressing long-standing limitations in explainability, ethical harms, and environmental efficiency.
Koch and Peterson’s study draws on qualitative interviews and computational analyses to trace the creation of this monoculture back to the late 1980s. They highlight how benchmarking, while effective in driving rapid progress, has also stifled innovation by prioritizing measurable outcomes over exploratory research. The result is a field that has achieved remarkable feats in predictive accuracy but struggles to address broader challenges such as bias, transparency, and sustainability.
The implications of this monoculture extend beyond AI, raising questions about the future of scientific progress in an era where generative AI is becoming increasingly prevalent. Koch and Peterson suggest that the AI monoculture challenges the belief that basic, exploration-driven research is essential for scientific advancement. Instead, they argue that the field must strike a balance between measurable progress and the pursuit of more fundamental, exploratory research to ensure long-term innovation and ethical responsibility.
As AI continues to evolve, the lessons from this study could inform how other scientific disciplines approach research priorities, funding mechanisms, and evaluation systems. The defence and security sector, in particular, stands to benefit from a more nuanced understanding of how AI’s historical trajectory has shaped its current capabilities and limitations. By re-evaluating the trade-offs between scalability and exploration, the field can better address the complex challenges of modern warfare and national security. Read the original research paper here.