Artificial intelligence can generate many suggestions – but it does not understand which ones truly matter. A new study published in Decision Analysis, one of the world's leading journals in decision research, published by INFORMS, provides clear evidence. The findings are illuminating and relevant to anyone seeking to make better decisions with the help of AI.
Johannes Ulrich Siebert, Professor at the MCI, and Jay Simon, Professor at American University in Washington D.C., jointly investigated how well generative AI tools can develop objective systems for complex decision-making situations – and where their limitations lie. The transatlantic research collaboration underscores MCI's growing international reach and research strength in the fields of decision analysis and artificial intelligence.
To investigate this, Simon and Siebert compared AI-generated objective systems with those developed by professional decision analysts. Several leading AI models were tasked with developing objective systems for six real-world decision situations: ranging from building evacuations and cybersecurity to investment decisions in pharmaceutical research. The results were evaluated against nine criteria of a structured decision-making methodology and compared with published objective systems developed by experienced experts.
The findings are nuanced: AI frequently produces plausible and well-formulated individual suggestions. On criteria such as comprehensibility, measurability, and operational feasibility, the AI tools performed solidly. Yet as soon as the objective system is considered as a whole, clear weaknesses emerge. The systems are often incomplete, contain redundancies, and conflate fundamental objectives with so-called means objectives – aspects that are only indirectly relevant. Ralph Keeney, a pioneer in decision research, put it plainly after reviewing the AI-generated lists: "Both lists are better than what most individuals could create. However, they should not be used for a high-quality decision analysis."
Significantly better results only emerge when AI is guided through targeted prompting strategies and its suggestions are critically scrutinized by human expertise. Particularly effective was the combination of a step-by-step guidance procedure that leads the AI toward structured thinking, and a feedback mechanism in which experts critically assess the AI output and request revisions – findings with direct relevance for organizations, public authorities, and policymakers who must make important decisions under uncertainty.
From these insights, Simon and Siebert derive a practical four-step model: First, decision-makers independently develop an initial set of objectives, after which the AI generates its own proposals. Both lists are then merged and consolidated. In the third step, the AI improves the quality of individual objectives while experts ensure the coherence of the overall system. Finally, a holistic quality review takes place – ideally together with all relevant decision-makers – ensuring that in the end, it is not AI but human judgment that has the final word.
"This study exemplifies what MCI stands for: research at an international level that simultaneously delivers practical answers to the pressing questions of our time," said Head of Research & Development at MCI Martin Pillei.
"Generative AI should complement human expertise, not replace it," emphasizes MCI Professor Siebert. "When humans and AI work together, each contributing their respective strengths, significantly better decision-making foundations emerge – for organizations as well as for policymakers." Simon adds: "AI can list what might be relevant – but deciding what truly matters remains a human task." All findings and the complete four-step model are available at: https://doi.org/10.1287/deca.2025.0387
MCI study shows how AI and human expertise work together to create a better basis for decision-making ©MCI/Christian Kasper
Discover the program that suits you.