CIO OPINION understanding how to identify and prevent these failures is not just a technical necessity , but a matter of business safety and sustainability .
Cláudio Lúcio , Founder , A3Data
When AI deceives us
Cláudio Lúcio , data specialist and founder of A3Data , on identifying and preventing AI ‘ hallucinations ’.
Would you trust a machine that , in the blink of an eye , can fabricate a convincing lie ? In the current scenario , where Generative Artificial Intelligence ( GenAI ) and Large-Scale Language Models ( LLMs ) are on the rise , so-called ‘ hallucinations ’ – moments when AI generates false or disconnected information – are becoming an urgent problem that results in the question : how far can we trust these creations that shape the digital future ?
Recent studies published by Vectara , a startup founded by former Google employees , point out that AI hallucinations occur with a frequency ranging from 3 % to 27 % in simple tasks , such as summarizing news articles . This means that most of the metrics used to measure the quality of automatically generated writing , such as text summarization and translation , are not adequate to quantify the level of hallucination of that same text .
The relevance of this topic cannot be underestimated , given that the reliability of these technologies is critical for their application in relevant industries such as medicine , finance , and law .
In the healthcare industry , for example , misdiagnoses can result in inappropriate treatments . In the financial sector , wrong forecasts can cause huge losses . And in the legal field , inaccurate information can compromise important court cases . Thus , understanding how to identify and prevent these failures is not just a technical necessity , but a matter of business safety and sustainability .
This is because these metrics ( such as ROUGE , BLEU , and METEOR ) evaluate textual quality based on similarity to human references , but do not consider the factual accuracy or veracity of the information contained in the text . Therefore , there are active research efforts to define effective metrics to quantify hallucination .
In this sense , there are several ways to minimize and avoid this behavior , including proactively . For this , it is necessary to have knowledge in the business area in question and in the type of task that the model is performing , and thus have governance , creating robust development methods and monitoring in real time .
www . intelligentcio . com INTELLIGENTCIO LATAM 39