In today's digital age, large language models such as ChatGPT are increasingly becoming vital sources of healthcare information. This is particularly significant for individuals who encounter barriers like insurance issues or language constraints. As part of the IMPACT-NJ project, efforts are underway to ensure that chatbot technologies remain accessible to everyone in New Jersey and beyond. A recent paper titled “Ensuring Accuracy and Equity in Vaccination Information From ChatGPT and CDC: Mixed-Methods Cross-Language Evaluation” was published, focusing on the quality and linguistic fairness of vaccination information provided by ChatGPT and the Centers for Disease Control and Prevention (CDC) in both English and Spanish.
The study evaluated the accuracy, readability, and understandability of vaccination information from ChatGPT and the CDC in both languages. This research is crucial because past evaluations have largely concentrated on English, often neglecting non-English speakers' needs within the U.S. Responses to common vaccination-related questions were compared using both quantitative and qualitative methods.
Key findings from the study indicate that both ChatGPT and the CDC delivered highly accurate information, scoring over 95 out of 100, with similar trends observed in both languages. However, neither source met the American Medical Association’s recommended readability level of Grade 6. This issue was more pronounced in English than in Spanish, with average grade levels being 12.84 for English and 7.93 for Spanish against a recommended level of 6.0. Qualitative analysis revealed some Spanish responses included less common words like “patógeno” instead of “infección,” while some sentence structures appeared as direct translations from English, leading to unnatural phrasing.
Both sources offered understandable information; however, some Spanish texts included English terms such as "herd immunity" and "fake" within quotations.
These findings hold significant implications for public health policy in New Jersey's diverse population:
Understanding AI tools' strengths and weaknesses can enhance health information access for non-English speakers.
Linguistic equity is crucial so all residents have access to accurate health information irrespective of their language proficiency.
Contextual evaluation is important since readability scores showed issues like unnatural phrasing due to direct translations; engaging stakeholders is essential for effective health communication.
To leverage AI's potential in disseminating health information effectively, addressing these challenges is critical. Policymakers should consider investing in AI improvements through research to enhance readability and linguistic equity of AI-generated content. Fostering community engagement by collaborating with diverse communities can help tailor health information according to linguistic and cultural contexts. Additionally, implementing regulatory oversight can ensure AI tools used for health communication are accurate, fair, and accessible to all.
