AIMIR: assistente para recuperação de imagens médicas baseado em LLMs

Abstract

This work describes the development and evaluation of a medical image retrieval as- sistant, named AIMIR (Artificial Intelligence-Based Medical Image Retriever), which uses Large Language Models (LLMs) to optimize healthcare professionals’ interaction with patient image exam data and contribute to building annotated datasets for AI. The application aims to facilitate the management and querying of this data, addressing the scarcity of annotated medical data, a significant challenge in developing AI models for healthcare. The system’s architecture integrates the ChromaDB vector database with the Ope- nAI Embeddings API (for semantic representation), the OpenAI Whisper API (for audio transcription), a hybrid query routing system (GPT models), and an intuitive Streamlit interface. The system operates in two main flows: user image addition (via text or audio descriptions) and user queries (by ID, semantic, description, or general). The system’s evaluation used 24 chest X-ray images from a public database, comple- mented by detailed audio descriptions generated by a medical imaging specialist. The tool’s responsiveness tests involved evaluating the entire application pipeline, from data insertion to responses to pre-designed user queries, comparing the system’s output with the expected correct answer. The results showed the effectiveness of data insertion and the importance of accuracy in classifying user intent. The GPT-4o model demonstrated 100% accuracy in query classification, outperforming GPT-3.5 Turbo, which achieved 70% accuracy. The quality of responses varied according to the query type and the LLM used. Re- sults for semantic query types showed lower accuracy rates, which can be explained by very similar image descriptions, suggesting the need for more robust embedding models. Description-based queries, however, benefited from the GPT-4o model. General queries yielded reasonable results with both models, while queries about the database demonstra- ted a notable performance difference, with GPT-4o responding with superior precision. The implementation of the Cache-Augmented Generation (CAG) strategy was crucial for optimizing resource usage and improving efficiency. The study concludes that the image retrieval assistant proposed in this work is a promising application and suggests future tests with other datasets and the possibility of fine-tuning for local execution.