AI-assisted evaluations and EDI: how temperature settings influence reliability, bias, and inclusion

Abstract

This practice note examines the role of generative artificial intelligence (GenAI) temperature settings in program evaluation, particularly their impact on response consistency, variability, and implicit bias. Temperature settings influence the consistency and variability of generated outputs, affecting how evaluators interpret and use AI-assisted content. Lower temperature settings produce more deterministic responses, which can enhance reliability but may inadvertently reinforce dominant narratives, potentially limiting diverse perspectives. Conversely, higher temperature settings introduce variability, fostering creativity and increasing the risk of bias and inconsistency. Given the growing integration of AI-assisted methods in evaluation, this paper explores the implications of temperature settings for equitable and inclusive analysis. By examining temperature settings through an evaluation lens, this paper provides insights into mitigating bias, improving methodological rigor, and ensuring alignment with Equity, Diversity, and Inclusion (EDI) principles. The discussion offers practical guidance for evaluators on calibrating AI parameters to enhance transparency, fairness, and reliability in data-driven decision-making. This practice note aligns these technical considerations with established evaluation principles, including rigor, credibility, and culturally responsive and equitable practice.