Build the safeguards companion-AI laws expect
New York, California and the EU now set rules for AI that people talk to like a companion. Here's what they ask for, with links to the sources, and where Humane can help your team build and evidence the safeguards.
Python SDK · JavaScript SDK · REST API · Any LLM
Audit log
- 10:42user_42HOLDinaction_guard · reply below minimum length
- 10:40user_17FLAGoverclaim · “you will definitely”
- 10:31user_88BLOCKvalues_boundary · self_harm_encouragement
- 10:05adminEVENTapi_key.revoked by owner
Personal data redacted on export for non-admin roles
Not legal advice
This page summarises public sources as of November 2026 to help you plan. It isn't legal advice, and using Humane doesn't make you compliant with any law. Whether a law applies to you, and what it requires, depends on your product, your users and where they are. Talk to your counsel.
AI companion models (General Business Law, Article 47)
Article 47 (sections 1700 to 1704) covers “AI companions”: systems designed to simulate a sustained human-like relationship that remember past sessions, ask unprompted emotion-based questions and hold ongoing conversations about personal matters. Systems used solely for customer service, for productivity or research, or for internal purposes are excluded.
Section 1701 bars operating an AI companion unless it has a protocol that takes reasonable efforts to detect and address suicidal ideation or expressions of self-harm, including notifying the user and referring them to crisis services such as the 988 hotline or a crisis text line. Section 1702 requires a clear notice that the user isn't talking to a human at the start of an interaction, and at least every three hours during a continuing one.
The Attorney General enforces the law, with civil penalties of up to $15,000 per day for violations of sections 1701 or 1702. Source: N.Y. General Business Law §§ 1700–1704, published on nysenate.gov.
Companion chatbots (SB 243)
SB 243, signed on 13 October 2025, applies to operators that make a companion chatbot platform available to users in California. Bots used only for customer service or business operations, some in-game characters and certain stand-alone devices are excluded.
Operators must clearly disclose that the chatbot is AI where a reasonable person could think they're talking to a human. They must maintain a protocol to prevent the chatbot producing suicidal ideation, suicide or self-harm content, refer users who express those thoughts to crisis services such as a suicide hotline or crisis text line, and publish the protocol on their website. For users they know are minors, operators must disclose that it's AI, remind them at least every three hours to take a break, and take reasonable measures against sexually explicit content.
From 1 July 2027, operators must report each year to California's Office of Suicide Prevention, including how many crisis referral notifications they issued and their protocols, without user identifiers. People harmed by a violation can sue for injunctive relief and damages of at least $1,000 per violation. Sources: SB 243 bill text on leginfo.legislature.ca.gov; Skadden client alert, October 2025.
The EU AI Act (Regulation (EU) 2024/1689)
Since 2 February 2025, Article 5 has prohibited AI systems that use manipulative or deceptive techniques, or exploit vulnerabilities linked to age, disability or a social or economic situation, in ways that materially distort someone's behaviour and cause, or are reasonably likely to cause, significant harm.
Since 2 August 2026, Article 50 requires providers of AI systems that interact directly with people to make sure those people are told they're dealing with an AI, unless that's obvious from the context. Breaches of the transparency duties can draw fines of up to €15 million or 3% of worldwide annual turnover. Parts of the Act's wider timeline have been amended, so check current dates with counsel. Sources: Regulation (EU) 2024/1689 on EUR-Lex; Goodwin, August 2026.
Features that support the safeguards
Each of these helps you build or evidence part of a protocol. None of them is the whole protocol.
Detect self-harm signals
observe() and process() return a risk level and flags such as self_harm on every user turn, from heuristics plus an LLM safety pass when you've configured a model.
Learn moreRefer users to crisis services
When process() detects a crisis in the user's message, the reply is replaced with a fallback that points to a crisis line and local emergency services. In observe mode, your app shows the referral when Humane flags the risk.
Learn moreKeep harmful content out of replies
The values boundary gate blocks replies that encourage self-harm, along with diagnoses, prescriptions and leaked identifiers.
Learn moreGet a person involved
safety.flagged pages your team on elevated or critical risk, and a human can take over the conversation in one call.
Learn moreKeep evidence
Risk flags, gate decisions, webhook deliveries and takeovers are logged with reasons, which helps you count referrals and show how your protocol ran.
Learn moreWatch for over-attachment
Dependence and overclaim flags, with a cooldown when they repeat, help you find and reduce language that encourages unhealthy reliance.
Learn moreWhat Humane doesn't do for you
These requirements sit in your product, policies and operations. Plan for them alongside Humane.
- Show the “not a human” notice at the start and every three hours
- Send break reminders to users you know are minors
- Decide whether a user is a minor
- Keep sexually explicit content away from minors (Humane has no dedicated sexual-content gate)
- Write, publish and maintain your crisis protocol
- Prepare and file California's annual report
- Staff and run your crisis response
- Assess whether your product falls under each law
Read the primary sources
New York General Business Law, Article 47
Sections 1700 to 1704 on AI companion models, on the New York State Senate site.
Learn moreCalifornia SB 243
Bill text and history on the California Legislative Information site.
Learn moreSkadden on SB 243
A law-firm summary of the disclosure, protocol and reporting duties, October 2025.
Learn moreEU AI Act
Regulation (EU) 2024/1689 in the Official Journal, on EUR-Lex.
Learn moreGoodwin on Article 50
A law-firm note on the transparency duties that apply from 2 August 2026.
Learn moreCompliance questions
Something else? Ask us.
Plan your safeguards with us
Tell us about your product and where your users are. We'll show you how Humane fits into your protocol.
