EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants

Franck Cappello, Sandeep Madireddy, Robert Underwood, Neil Getty, Nicholas Lee-Ping Chia, Nesar Ramachandra, Josh Nguyen, Murat Keçeli, Tanwi Mallick, Zilinghan Li, Marieme Ngom, Chenhui Zhang, Angel Yanguas-Gil, Evan Antoniuk, Bhavya Kailkhura, Minyang Tian, Yufeng Du, Yuan-Sen Ting, Azton Wells, Bogdan Nicolae, Avinash Maurya, M. Mustafa Rafique, Eliu Huerta, Bo Li, Ian Foster, Rick Stevens

preprint (2025)

preprint PDF

Summary

EAIRA is an Argonne methodology for evaluating LLMs as scientific research assistants through four evaluation classes—multiple-choice recall, open-response reasoning, lab-style experiments, and field-style interactive studies.

Keywords

AI evaluation large language models scientific research assistants benchmarks methodology AI for science

Cite

@article{2025_cappello_eaira,
  title = {EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants},
  author = {Franck Cappello and Sandeep Madireddy and Robert Underwood and Neil Getty and Nicholas Lee-Ping Chia and Nesar Ramachandra and Josh Nguyen and Murat Keçeli and Tanwi Mallick and Zilinghan Li and Marieme Ngom and Chenhui Zhang and Angel Yanguas-Gil and Evan Antoniuk and Bhavya Kailkhura and Minyang Tian and Yufeng Du and Yuan-Sen Ting and Azton Wells and Bogdan Nicolae and Avinash Maurya and M. Mustafa Rafique and Eliu Huerta and Bo Li and Ian Foster and Rick Stevens},
  journal = {preprint (2025)},
  year = {2025}
}