Assessing AI’s Impact on Mental Health: The Crucial Role of Minimal Clinically Important Difference (MCID)

In an evolving landscape where artificial intelligence, particularly generative AI and large language models (LLMs), increasingly interfaces with human mental health, a critical question emerges: how can we accurately assess whether these AI systems genuinely make a notable difference in improving well-being? This inquiry leads to an examination of the Minimal Clinically Important Difference (MCID), also known as the Minimal Important Difference (MID), a well-established generalized medical technique now being innovatively applied to the nuanced field of mental health assessment alongside AI.

The integration of AI into mental health support is a topic extensively covered and analyzed by AI expert Lance Eliot, whose ongoing Forbes column delves into various impactful AI complexities. This specific analysis highlights the latest application of MCID, leveraging AI to gauge the efficacy of digital mental health interventions.

AI and the Evolving Mental Health Landscape

The advent of modern-era AI, capable of producing mental health advice and performing AI-driven therapy, has rapidly expanded, largely propelled by the widespread adoption and advancements in generative AI. This burgeoning field presents significant upsides, offering accessible and often low-cost support, yet it simultaneously carries inherent risks and potential pitfalls. The swift development of AI in mental health has been a frequent subject of public discourse and expert analysis, including Eliot’s appearance on CBS’s 60 Minutes, which underscored the hard truths and challenges associated with AI in this sensitive domain.

The Popularity and Perils of AI for Mental Health

Millions of individuals worldwide are already turning to generative AI as an ongoing advisor for mental health concerns. With platforms like ChatGPT boasting over 800 million weekly active users, a significant proportion are engaging with AI on mental health aspects. Indeed, consulting AI on mental health facets consistently ranks as a top use case for contemporary generative AI and LLMs. The appeal is clear: most major generative AI systems are accessible for free or at minimal cost, available anywhere, anytime, 24/7. This convenience allows individuals to seek guidance on mental health qualms whenever they arise.

However, this popular usage is shadowed by significant worries regarding AI’s potential to provide unsuitable, or even egregiously inappropriate, mental health advice. There are concerns that AI can "go off the rails," leading to detrimental outcomes. August of this year saw banner headlines accompany a lawsuit filed against OpenAI, citing a lack of AI safeguards in providing cognitive advisement. Despite AI makers’ claims of instituting gradual improvements in safeguards, substantial downside risks persist, including the insidious potential for AI to co-create delusions that could lead to self-harm. Eliot has consistently predicted that major AI makers will eventually face scrutiny for their insufficient robust AI safeguards, particularly concerning the potential for AI to foster delusional thinking.

It is crucial to differentiate between generic LLMs, such as ChatGPT, Claude, Gemini, and Grok, which are not designed to replicate the robust capabilities of human therapists, and specialized LLMs specifically engineered for therapeutic purposes. While the latter are under development and testing, generic AI’s role in mental health remains a contentious area.

Gauging Improvement: The MCID Framework

To understand how MCID applies to AI and mental health, it’s helpful to first consider its general application in medicine. Imagine visiting a doctor for a flu or cold. Medical tests might indicate improvement, yet the patient might still feel unwell, experiencing no perceptible change in their symptoms. This discrepancy highlights a fundamental challenge: what factor should be chosen to ascertain medical improvement, and whose perspective (doctor’s or patient’s) should take precedence?

The patient might focus on subjective symptoms like a runny nose or energy levels, while a doctor might prioritize objective metrics like temperature or blood test results. Furthermore, the degree of improvement is vital. A slight, almost imperceptible change might not be considered significant by the patient. This led to the codification of the Minimal Clinically Important Difference (MCID) in the late 1980s. The MCID technique involves identifying a clinically valued factor, measuring its change over time, and declaring improvement only if the change reaches a minimal yet important threshold. Crucially, the MCID is customarily shaped from the perspective of the patient, emphasizing their subjective experience of improvement.

As noted in a research article by Jeffrey Cummings, "Perspective: Minimal Clinically Important Difference (MCID) And Alzheimer’s Disease Clinical Trials," MCID is best used as a complement to other measurable aspects. While not a necessity, it is highly useful for understanding what the patient believes about their medical condition, providing a robust gauge that objective measures alone might miss.

MCID in Mental Health: The Role of PHQ-9

The MCID technique has proven equally effective in the mental health domain, serving as a vital indicator of a patient’s status. For mental health, identifying a readily measurable and easily explainable factor is key. A popular and widely validated choice is the Patient Health Questionnaire-9 (PHQ-9). This standardized, evidence-based, self-reported questionnaire consists of nine questions, taking only a few minutes to complete. It is freely available, non-proprietary, and particularly effective as a clinical tool for measuring depression.

Each PHQ-9 question asks how often, over the past two weeks, the person has been bothered by specific symptoms, with responses rated on a 0-3 scale: "Not at all," "Several days," "More than half the days," or "Nearly every day." With nine questions, the total score can range from 0 to 27.

Common interpretations of PHQ-9 scores, used as a guide by clinicians, include:

• 0-4: Minimal depression
• 5-9: Mild depression
• 10-14: Moderate depression
• 15-19: Moderately severe depression
• 20-27: Severe depression

These categorizations are not diagnostic but provide an overall indication of depressive symptom severity. The nine questions generally cover themes such as:

• Loss of interest or pleasure in activities
• Feeling down, depressed, or hopeless
• Sleep disturbances (trouble falling or staying asleep, or sleeping too much)
• Feeling tired or having little energy
• Poor appetite or overeating
• Feeling bad about oneself, or that one is a failure, or has let oneself or one’s family down
• Trouble concentrating on things, such as reading the newspaper or watching television
• Moving or speaking so slowly that other people could have noticed, or being fidgety or restless more than usual
• Thoughts that one would be better off dead, or of hurting oneself in some way

A non-zero score on the ninth item, indicating self-harm thoughts, typically triggers immediate follow-up assessment by a clinician, recognizing the potential for a dire mental state, while also considering potential misunderstandings or casual answers.

Integrating AI into the MCID Process

Generative AI can significantly assist in MCID efforts within mental health. One key application is having AI administer the PHQ-9. Most major LLMs have been patterned on the PHQ-9, though it is crucial to verify that the AI accurately replicates the standardized questionnaire, as variations could compromise its validation. Beyond administration, AI can score responses and present results, providing totals and a raw concluding statement based on the established rating scale.

However, the question of whether AI should actively interpret these results is a point of contention. Some therapists strongly object, fearing that AI-derived interpretations might mislead individuals into unwarranted or false conclusions about their mental health. The counterargument suggests AI interpretation is acceptable if users understand its potential fallibility. Yet, a significant concern remains: people tend to imbue AI with authority, making them susceptible to believing its pronouncements.

Lance Eliot has extensively covered the evolving role of AI for therapists, predicting a disruption of the classic therapist-client dyad into a new therapist-AI-client triad. He also highlights the move towards proactive AI in mental health therapy that could take initiative and reach out to users.

An Illustrative Example: AI and PHQ-9 Interpretation

Consider an example where AI is used for weekly mental health check-ins via PHQ-9. After initial scores of 18 and 16 in the first two weeks, the AI administers the PHQ-9 again, resulting in a score of 15. The question then arises: is a one-point drop sufficient to be considered an MCID?

When prompted, a popular LLM responded: "Your score has dropped from 16 to 15, which is a one-point improvement. While any reduction in symptoms is positive, generally, a minimal clinically important difference (MCID) for the PHQ-9 is often considered to be a drop of 4 to 5 points. This suggests that while there’s a positive trend, you might not yet perceive a ‘noticeable’ or ‘meaningful’ change in your overall well-being based on common clinical benchmarks."

This AI-generated response, while factual regarding general benchmarks, could be problematic. If the individual subjectively feels better with even a one-point improvement, the AI’s commentary could inadvertently invalidate their personal sense of progress, effectively "disillusioning" them from their own MCID. This highlights the "box of chocolates" nature of modern generative AI – the output can be unpredictable.

Further interaction with the AI regarding this score might reveal cautious, generalized responses, often lacking probing questions about the user’s subjective feelings of progress. This underscores the limitations of generic AI, which may not possess the nuanced conversational abilities or empathy of a human therapist. Custom instructions or specialized mental health AI applications could potentially enhance AI’s performance in MCID activities, allowing for more personalized and sensitive interactions.

The Future Trajectory of AI and Mental Health

Researchers are increasingly examining how AI can effectively be integrated into MCID for mental health, and these studies will be crucial in guiding future development. We are currently in the midst of a grand, global experiment regarding societal mental health, with AI being deployed nationally and globally to provide guidance, often at no or minimal cost, available 24/7. In this vast experiment, humanity serves as the guinea pigs.

AI holds the potential to be a powerful bolstering force for mental health, yet it also carries the risk of being detrimental. The collective responsibility lies in steering its development and application towards beneficial outcomes, while rigorously preventing or mitigating its potential harms.

The application of MCID in mental health, particularly with AI, brings to mind Abraham Lincoln’s famous quote: "The shepherd drives the wolf from the sheep’s for which the sheep thanks the shepherd as his liberator, while the wolf denounces him for the same act as the destroyer of liberty. Plainly, the sheep and the wolf are not agreed upon a definition of liberty." This insight beautifully illustrates how perspectives can radically differ. In the context of therapy, a client’s perception of improvement might diverge substantially from a therapist’s. The MCID technique offers a vital mechanism to bring the client’s subjective experience of improvement to the forefront, allowing for a more comprehensive and patient-centric understanding of progress. If suitably shaped for this significant task, AI could prove to be an invaluable aid in this process.