Proposed New York law would bar AI chatbots from posing as lawyers, allow duped users to sue

By Karen Sloan

March 5 (Reuters) - A proposed law working its way through New York's legislature would bar artificial intelligence chatbots from impersonating lawyers and other licensed professionals in the state, opening up AI platforms to lawsuits by users.

The bill, whose sponsor called it the first of its kind in the country, would bar AI chatbots from giving substantive responses and offering advice that “if taken by a natural person” would constitute the unauthorized practice of law.

“Today, there is no law that says that a large language model cannot tell you that it is a lawyer, that it is a licensed therapist, and then give you legal advice or therapy accordingly,” New York State Senator and bill sponsor Kristen Gonzalez told Reuters Thursday. “I think that's really concerning.”

Chatbot users should be able to sue if they rely on erroneous legal information provided by a platform that represents itself as a lawyer, Gonzalez said.

AI platforms under the bill would not be able to avoid liability by notifying users that they are interacting with a “non-human chatbot,” and users could seek damages in court against companies that violate the law.

New York, like all U.S. states, prohibits people from representing themselves as lawyers or offering legal services without being licensed to practice law.

The bill would apply to law and other licensed professions such as doctors and mental health providers. It is part of a larger suite of New York bills seeking to regulate AI, including one that would protect minors from unsafe AI chatbot features and one that would require AI platforms to "conspicuously" display a notice that outputs may be inaccurate.

OpenAI and Anthropic, which operate two of the most popular AI chatbots, did not immediately respond to requests on Thursday for comment on the proposed professional impersonation law.

The bill, which advanced out of the New York Senate’s Internet and Technology Committee late last month, comes as AI platforms face mounting scrutiny over the impacts and ethics of the rapidly expanding technology. ChatGPT maker OpenAI, Google’s Gemini, and Character.AI are each facing lawsuits alleging that the tools led to users' suicides. The companies have denied wrongdoing but settled some cases.

Debates are also growing over the technology’s use in law. Nippon Life Insurance Company of America sued OpenAI on Wednesday, accusing ChatGPT of practicing law without a license and helping a former disability claimant breach a settlement and flood a federal court docket with meritless filings. OpenAI said the case lacks merit.

A growing number of lawyers have separately faced court sanctions for submitting briefs with AI-generated fictitious case citations and other hallucinated material, with some judges imposing fines.

Read more:

US appeals court orders lawyer to pay $2,500 over AI hallucinations in brief

OpenAI hit with lawsuit claiming ChatGPT acted as an unlicensed lawyer