Rutgers graduate students have presented research on the use of artificial intelligence (AI) in evaluating United Nations Development Programme (UNDP) project reports. The findings, titled “Effectiveness of AI-assisted report assessments: A case study of the United Nations Development Program,” were shared at the American Association for the Advancement of Science (AAAS) Annual Meeting’s poster competition in Phoenix, Arizona.
The team includes Raquel Padilla (PhD Planning and Public Policy), Sandra Jae-Ah Chung (MPP/MCRP candidate), Hemali Angne (PhD Computer Science), and Professor Hal Salzman from the Edward J. Bloustein School of Planning and Public Policy at Rutgers University. All are members of the SOCRATES research group, which focuses on socially aware robotics. The project is a collaboration with UNDP’s Independent Evaluation Office (IEO).
The study examined how an AI tool based on a large language model performs when assessing the quality of UN project evaluation reports. According to their initial results, “the language used in AI-generated assessments is more constrained than that of human reviewers.” The researchers also found that “AI outputs lack nuance and context-awareness.”
These issues limit how useful AI-generated quality assessments can be for improving project reports or giving feedback to development projects. The team notes this raises questions about whether using AI will actually lower operating costs or simply introduce new technology expenses while still requiring human oversight. They suggest these observations could apply broadly to other knowledge work where AI tools are being considered.
UNDP oversees a variety of international projects, such as those focused on biodiversity conservation and local governance participation. Country offices prepare evaluation reports for these initiatives, which are then reviewed for quality—a process currently handled by external reviewers at an annual cost near $180,000. To cut costs, IEO has explored replacing human reviewers with an AI tool using general-purpose large language models.
The Rutgers team conducted both qualitative and linguistic analyses on output from the AI system and found notable shortcomings. Specifically, they reported that “the AI struggled with implicit and culturally-specific content, which is essential for an international organization such as the UNDP.” In addition, “AI-generated comments were monotonous and repetitive, not reflecting the diversity of the reports.” This was further confirmed through linguistic mapping analysis: “the pool of AI comments was much closer together than the human pool.”
The Edward J. Bloustein School of Planning and Public Policy at Rutgers University is recognized for its research centers advancing social impact across fields like community development, transportation, health, workforce development, and energy policy (source). It operates as part of Rutgers University—the State University of New Jersey—and aims to foster just and sustainable communities globally (source). The school has earned national rankings for its programs in urban planning and public health (source). Stuart Shapiro became dean in 2023 (source).