Comparative Assessment of LLMs for Classifying Cognitive Theory Parameters in Patient-Chatbot Interactions

Abstract

This study presents a comparative evaluation of leading Large Language Models (LLMs) for classifying cognitive triad parameters—self-negative, world-negative, and future-negative—in patient–chatbot interactions based on Beck’s Cognitive Theory. A dataset of patient statements extracted from chatbot conversations was analyzed using GPT-4, GPT-4o, GPT-4o-mini, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Structured prompt-engineering techniques were employed to identify and categorize negative cognitive patterns associated with depression. The proposed framework uses a modular prompt-based architecture that independently detects self-directed, world-directed, and future-oriented negative thoughts. Experimental results demonstrate that all evaluated models achieved high classification performance, with GPT-4o obtaining the highest overall accuracy and macro F1-score. The findings highlight the potential of LLMs to support automated psychological assessment and early detection of depressive thought patterns. This work contributes toward the development of AI-assisted mental health systems capable of enhancing clinical decision-making and scalable mental health screening.