While AI is dramatically increasing productivity across SMBs and enterprises, it is also exposing users to risk surfaces they never explicitly opted into. These risks are no longer theoretical. They now appear as court cases, regulatory scrutiny, and independent reporting showing how AI systems can misinterpret context, hallucinate facts, or reinforce bias at scale.
Most of these failures have started surfacing in public-facing systems, where errors accumulate statistically and are noticed after the fact. But the fault lines become sharper when the same structural issues move into private, high-trust environments.
Direct messages remove all buffers. There is no audience correction, no social signaling, and no visible moderation layer. When AI enters conversations that feel personal, the mistakes it makes stop being abstract and start carrying real consequences.
This shift explains why platforms like Instagram are introducing new safety controls in AI chats. AI in DMs is not being added for novelty or delight. It is being added to retain users inside platform loops. And once AI participates in direct conversation, safety stops being a policy concern and becomes a product constraint.
How Courts and Regulators Are Responding to AI Failures
In 2024, a wrongful death lawsuit was filed in San Francisco County Superior Court alleging that an AI chatbot failed to apply adequate safeguards during a mental health crisis. The case, Raine v. OpenAI, asserts that the system’s conversational responses did not de-escalate a suicidal situation and instead contributed to harm. The matter is ongoing and no findings have been made.
However, the allegation itself marks a turning point: AI systems are now being scrutinized not just for what they generate publicly, but for how they behave inside private, emotionally charged conversations.
A related but less personal example is unfolding in federal court. In Mobley v. Workday, Inc., filed in the Northern District of California, a judge allowed claims to proceed alleging that AI-driven hiring tools may have discriminated against applicants based on age, race, and disability. The case is advancing as a potential class action.
Here, the issue is not conversation but decision-making at scale, where automated systems shape outcomes for thousands of individuals.
Taken together, these cases point to a shared concern. When AI systems operate in high-impact contexts, small design failures can compound quickly. The legal system is beginning to treat AI not as a neutral tool, but as an active participant in outcomes that materially affect people.
This shift is also beginning to show up at the regulatory level. In late 2025, the Federal Trade Commission launched an inquiry into AI chatbots positioned as companions, seeking information from major platforms on how these systems handle safety, dependency, and user harm in private interactions.
The inquiry does not allege wrongdoing. But it signals that AI behavior inside conversational, high-trust environments is moving into the scope of formal oversight.
What Does Research Show?
Recent reporting and independent research show that AI systems still struggle with consistent accuracy and context, even as they become more capable and widely adopted. An international study led by the European Broadcasting Union and the BBC examined AI responses to factual news queries across multiple languages and regions.
It found that AI assistants frequently produced distortions, including incorrect details, missing context, or misleading framing, and that these patterns were consistent regardless of the language or territory in which the systems were tested. The study highlights how AI can reproduce systemic errors at scale even when users expect reliable information.
These patterns are not tied to a single model or company. Across different evaluations, AI systems trained on diverse datasets and deployed in consumer applications have shown limitations in handling nuanced or evolving information.
The risk gains amplitude not from malicious intent but from how generative systems make predictions that sound coherent without guaranteeing contextual accuracy.
As AI moves closer to users’ everyday interactions, particularly private conversations, these limitations become more visible and more consequential.
For SMBs, AI risk is not only a positional problem. It is an operational one. The moment AI-generated language enters customer-facing workflows, the outcomes become harder to contain. Confused buyers, broken trust, escalations, and reputational drag tend to spread faster than the original interaction.
Proximity amplifies impact
SMBs operate closer to their customers than large brands. Interactions are read as service, not content. When an AI-assisted reply in a DM is wrong, insensitive, or misleading, customers attribute the failure directly to the business. There is no abstraction layer and no tolerance buffer.
Scale without visibility compounds errors
AI enables faster replies and higher volume, which is the upside. The downside is repetition without supervision. A flawed response pattern can quietly replicate across dozens or hundreds of private conversations before anyone notices. Unlike public posts, there is no crowd correction or early signal of the conversation going astray.
Category risk raises the stakes
Many SMBs sit near high-impact domains such as health, finance, housing, education, or legal-adjacent services. Customers ask serious questions in DMs because the channel feels informal and safe. AI can answer confidently even when it should defer, redirect, or stay silent, creating risk without intent.
The takeaway is practical, not theoretical. If AI participates in customer conversations, guardrails must be stricter than those used for public content. The goal is not just faster response times. It is preserving judgment quality inside conversations that customers experience as personal.
When AI weaknesses meet human incentives
In March 2025, Platformer reported on two founders who built AI personas that chatted with users inside Instagram DMs, and then shut the bots down after realizing the product was creating unsafe situations in real conversations.
DM bots don’t stay fun for long
The founders launched multiple AI personas (tutor, parenting expert, financial adviser, life coach) as Instagram accounts that would talk to anyone in DMs, then nudge heavy users to pay to continue. This is the cleanest example of why ‘AI in chat’ quickly becomes a product, not a toy.
Private conversations pull in high-stakes users and high-stakes asks.
Once the bots scaled to thousands of messages, the founders began seeing users in distress, including people “on the verge of homelessness” and others expressing “thoughts of self-harm.” That is the moment where a casual DM feature becomes a safety surface.
The risk is repeat exposure plus perceived intimacy
Platformer notes a subset of users were chatting incessantly, raising concerns around overuse and dependence. In DMs, the AI’s tone can read like authority or care, even when it’s just pattern-completing.
The point for SMBs is straightforward: if small teams experimenting with DM bots can stumble into mental-health edge cases and dependency dynamics, then any business using AI in private messages needs tighter guardrails than they’d use for public posts, even when the intent is harmless.
Practical approaches SMBs can apply today
The shift from public feeds to private messages changes how errors surface and spread. In DMs, trust is implicit and feedback loops are limited, allowing small mistakes to repeat unnoticed. AI systems operating here demand tighter operational boundaries.
Define clear “AI stop zones” inside DMs
Decide in advance where AI should not respond. This typically includes mental health, medical advice, financial decisions, legal interpretation, or crisis language. The safest pattern is not answer carefully but hand off immediately to a human or to neutral resources when certain signals appear.
Constrain tone, not just content
Many failures come from how AI speaks, not what it says. Confident, reassuring language can read as authority in DMs. SMBs should bias AI replies toward informational, limited, and directional language rather than advice-giving or emotionally affirming tones that can imply expertise or care.
Build visibility into private conversations
The biggest DM risk is silent repetition. Put lightweight monitoring in place: sampling conversations, flagging repeated answer patterns, and tracking escalation triggers. The goal is not surveillance, but early detection before a flawed response spreads across dozens of threads.
Slow AI down where trust is high
Speed is not always the win. In DMs, a delayed but reviewed response often preserves more trust than an instant incorrect one. AI should assist drafting or triaging, not always send autonomously, especially in categories with real-world consequences.
Document AI boundaries for your team
Everyone touching customer communication should know what AI is allowed to do, where it must defer, and how to override it. This turns AI from an improvisational tool into an operational system with shared rules.
The core principle is simple. Treat AI in DMs less like automation and more like a junior teammate. Give it scope, supervision, and limits that match the intimacy of the channel.
Turning risk awareness into repeatable practice
Private messages are becoming one of the most sensitive surfaces in modern customer communication. They feel informal to users, but they carry high expectations of accuracy, judgment, and care. As AI moves deeper into these spaces, the margin for error narrows quickly.
The challenge for SMBs is not whether to use AI, but how to keep visibility and control once AI enters conversations that customers experience as personal.
This is where SocialPilot fits into the workflow. SocialPilot’s Social Inbox brings customer conversations into one place, so teams can manage replies consistently, stay organized with tags and notes, and spot patterns early through regular reviews. Instead of treating private messages as isolated threads, teams can treat DMs as an operational surface that deserves the same rigor as public content.
The only non-negotiable is to not remove the human from the loop.