Artificial intelligence is a tool to spread nutrition-related information, studies find

A recent study published in JAMA Open Network investigated the accuracy and reliability of nutritional information provided by two versions of Chatbots Generative Pre-trained Transformer (ChatGPT).

Their findings show that while chatbots cannot take the place of nurses, they can improve communication between health professionals and patients if they are further refined and strengthened.

Study: Consistency and Accuracy of Artificial Intelligence for Providing Nutritional Information. Image Credit: Iryna Imago/Shutterstock.com

Background

Many people today rely on the internet to access health, medicine, food and nutrition information. However, studies have shown that almost half of the nutrition information online is of low quality or inaccurate.

Artificial intelligence (AI) chatbots have the potential to streamline how users digest the vast array of publicly available scientific information by providing easy-to-understand, conversational explanations of complex topics.

Previous research has assessed how well chatbots can disseminate medical information, but their reliability in providing nutritional information remains unexplored.

About the study

In this cross-sectional study, researchers followed the Strengthening Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. They evaluated the accuracy of the information provided by ChatGPT-3.5 and ChatGPT-4 on macronutrients (proteins, carbohydrates, and fats) and energy content of 222 foods in two languages ​​- Traditional Chinese and English.

They provided a prompt that asked the chatbot to generate a table containing the nutritional profile of each food in its uncooked form. This search was conducted in September-October 2023.

Each search was performed five times to assess consistency; the coefficient of variation (CV) was calculated across these five measurements for each food.

The accuracy of the chatbot’s responses was evaluated by cross-referencing its reactions with nutritionist recommendations according to the food composition database maintained by the Food and Drug Administration of Taiwan.

An answer was considered accurate if the chatbot’s estimate of energy (in kilocalories) or macronutrients (in grams) was within 10% to 20% of what the nutrients provided.

The researchers also calculated whether there was a significant difference between the chatbots’ responses to the maintainers’ suggestions and between the two versions of ChatGPT.

Results

There were no significant differences between the estimates provided by the chatbots and the nutritionists regarding the fat, carbohydrate and energy levels of eight adult menus. However, the researchers found that protein estimates varied widely. The chatbot responses were estimated to be accurate for energy content in 35-48% of the 222 foods included and CV was lower than 10%. ChatGPT-4, the latest version, performed better than ChatGPT-3.5 overall but tended to overestimate protein levels.

Conclusions

The study shows that chatbot responses compare well with nutritionist recommendations in some respects but can overestimate protein levels and show high levels of inaccuracy.

As they become widely available, they have the potential to be a handy tool for people who want to look up macronutrient and energy information about common foods and don’t know which resources to turn to. consult them.

However, the authors stress that chatbots are not a substitute for nurturers; they can improve communication between patients and public health professionals by providing additional resources and simplifying complex medical language into easy-to-follow conversational terms.

They also note that the foods they included in the search may not be eaten frequently, which has implications for the relevance of their results.

AI chatbots cannot provide users with personalized nutritional advice or precise portion sizes, nor can they generate specific nutrition and nutrition guidelines. In addition, chatbots may not be able to adapt their responses to the region where the user lives.

Portion sizes and consumption units differ greatly from country to country, and according to the type of food and how it is prepared. Chatbots cannot take into account critical cultural and geographic differences or provide the relevant household units for each consumer.

Arguably, the most important limitation is that ChatGPT is a general chatbot – not one specifically trained in dietetics and nutrition.

The cutoff for the training data set was September 2021, so later research would not have been included. Users should not mistake a search error for a search engine, as their responses are a product of their training data set as well as the wording of the prompts.

However, given the huge demand for chatbots and other forms of generational AI, future products will overcome these limitations and provide increasingly accurate, up-to-date, relevant and practical information on diet and nutrition