Good direction, but very futile considering that guardrails on LLMs are about as real as LLMs are intelligent
The nice thing about laws is that they don't give a shit. If they say you can't do something, it's your problem to make sure you don't do it.
What about engineering? Food safety? Education? So many other fields could be added
Anything that requires a license is a good start.
Some legislation is to lay the ground work to expand it later.
Easier to lay a sub floor and put carpet over it later, than to just put carpet over the studs. When you're changing a massive thing like this, it's better to start small and work outwards. Canada introduced universal Dental and Phamacare and it was limited to certain demographics. People bitched about "Well what about me?"
You're coming. Start with what/who is most important, make sure the system works, and carry on from there.
New York State not New York City.
I don't know why Mamdani is pictured. I think it is because the proposal was introduced in the State Senate by Kisten Gonzalez who is a DSA member.
New York State not New York City.
I don’t know why Mamdani is pictured
the text in this image is also just flat-out wrong: the proposed legislation does not actually "ban AI from answering questions" on these subjects at all, as this post claims it does.
https://www.nysenate.gov/newsroom/press-releases/2026/kristen-gonzalez/ny-state-senator-kristen-gonzalez-her-bill-address-ai explains that her bill is strictly about impersonating licensed professionals:
We have documented instances of AI chatbots providing false medical license numbers to users and documented cases of real harm from AI chatbots providing misinformation. This is what my common-sense legislation addresses—protecting users from misinformation, scams, and fraud. This legislation does not prohibit a user from asking a chatbot questions or receiving general information and advice, as long as the chatbot is not presenting that information as a licensed professional. It would also not prohibit licensed professionals from consulting AI technology in the course of their work.
If those AI techbros could form a coherent thought without AI, they'd be very upset!
Counterpoint:
I diagnosed myself with something pretty significant using the internet, and then I went to get evaluated, and my suspicions were confirmed.
This would prevent a lot of good, in addition to preventing some bad.
I diagnosed myself
using the internet
I'm going to assume you researched medical documents, studies, and journals to link things together, and not ask a chat bot "My foot hurts, what is wrong?"
You're smart enough to self diagnose yourself with readily available information to the public, but not smart enough to not know the difference between that and a chat bot. That's remarkable.
I diagnosed myself
Congrats, glad your life is hopefully changing for the better because of this.
I hate taking the wind out of your sails but this is in the area of anecdotal evidence. You are a single person saying this for your specific case. I am sure we will get another few people confirming the same here.
But what about how many people tried this and got nothing useful out? Or even a misdiagnose? Anyone enthusiastic about this possibility of AI diagnosing people wouldn't really mention this. People like to mention the successes but it needs to be weighed against the failures too, and I don't think there is a good record of that.
If there is, its probably a medical study and I'd venture a guess that AI did not perform as well as a trained doctor.
What i don't like about this, is the ignorance towards every other field.
The ai is not allowed to tell you that you might have some illness but the LLM is allowed to vibe code the system that holds your medical data, or the software of the medical device that analyses your e.g. blood and provide the data to the doctor.
Systems and professions are interconnected. Chatgtp telling me that I have cancer is not as bad as chatgtp making mistakes causing my medical data to be stolen and sold.
Can we add to this the engineering of any physical object that could threaten human safety? If this keeps up much longer we're going to have buildings falling on top of us.
Incredibly dumb approach.
You should ban using AI for the purpose of legal disputes, medicine, etc, profesionally.
Its not about the output, its about what you use it for.
I am also 100% okay with legally requiring a clearly visible disclaimer that AI shouldnt be used for x/y/z and its output should alwats be fact checked or whatever.
But trying to outright ban specific types of output is pointless and trivial to circumvent.
I agree. You're spot on. This isn't a technical problem. We need to think less like engineers and more like lawyers on these things.
You don't require AI companies to have their models refuse to give out legal advice. As you note, that's a fool's errand. Instead, you make it a felony to use any LLM to write legal advice. Currently lawyers are already facing court sanctions for using AI to write their briefs. If you really want to get serious, you make it a felony. Apply the same penalty to a doctor using an LLM to diagnose their patients as you would if I decided to open a clinic in my house and start practicing medicine without a license. Charge people operating AI therapists with practicing psychology without a license.
This isn't a technical problem, it's a legal one. You don't try to regulate the LLMs themselves. You just make it a severe felony to use that technology in certain highly sensitive applications and professions.
And sure, one could say that we don't have a reliable way of screening AI from human made work. But again, that would be thinking like an engineer, not a lawyer. You don't enforce these laws by screening thousands of legal briefs or medical records with an AI detection filter. You offer huge cash rewards to whistle blowers, so that a disgruntled employee can rat out their bastard boss that's forcing everyone to illegally use AI. Apply massive fines and give half of whatever is collected to the person who blows the whistle.
Are you a structural engineering company owner forcing your employees to use AI to engineer buildings? Now you have to worry about one of those employees, an employee who probably really dislikes using AI, ratting you out. And their whistleblower reward will be so large, they'll end up owning the company. They won't have to worry about being blacklisted from future employers, as they'll be getting enough money to retire.
Don't look for technical solutions for the problem of AI abuse. Look for human solutions.
I feel like finding and citing appropriate law cases would be a great way for AI to be used, especially because the bullshit legal-ese that surrounds everything in court cases is so spectacularly opaque and trivial that you have to pay out the nose just for someone to parse through all the bullshit... Like, lawyers shouldn't even be a thing you need unless it's for some really complicated case. The courtroom should be a place where an average citizen could speak for themselves.
Also, as someone who studied medicine... Do you know how often doctors will take a patients history, step outside the exam room, and then google shit? I don't want to imply that they don't know what they're doing, like, you need to study medicine to know what to google, BUT... they still be googling shit. I think AI could be trained to ask the same patient history questions that a nurse asks. Then using the answers to those questions, then they can provide some feedback, with a big ol disclaimer that it's NOT to be taken seriously, and you should consult with a physician.
When people say "AI" these days, they mean the LLM-derived generative AI such as ChatGPT, Google Gemini, Anthropic Claude, and Grok.
These AI systems are not capable of finding authoritative search results. They are good at generating fluent, plausible, context-appropriate text. In legal and medical situations, that's the opposite of what you'd want.
Do you know how often doctors will take a patients history, step outside the exam room, and then google shit?
Think about how often the doctors land on some pseudo-science result. Now imagine that instead of a doctor it's a patient and instead of asking "could these be related" the patient is told "these are related".
a big ol disclaimer that it’s NOT to be taken seriously
This has worked very well for Fox News, which is not a news agency and is for entertainment purpose only. Also for dietary supplements, which are not medicine.
I'd agree with you to some extent if people weren't so god damn stupid. So many lazy fucks just take the AI's output at face value, believing it to be some all knowledgeable god, rather than a compilation of all human biases expressed on the internet with a bunch of hallucinations thrown in for good measure.
Should ban it giving finance advice as well. Conservatives and Neo-Liberals already want to give the economy over to Ai companies, what's stopping them from going full circle and making the Ai do all the trading as well?
Maybe I'm just ignorant about this, but it feels a bit "gatekeepy" to outlaw answering questions about the law itself. Like... so I can't use an LLM to help do taxes? Or if I'm served a confusing legal document, I have to pay an expensive attorney to understand the legalese? I can't even just get a rough idea from an LLM first?
EDIT: Also I've literally had a professional accountant send me a ChatGPT conversation they had because they didn't know tax law well enough. I feel like if anything that should be outlawed. Professionals should actually have to know shit and know how to research primary sources.
Those things should only be available to people who can pay hundreds of dollars an hour!
Cost is a reasonable concern. How about we pass universal healthcare so this isn't a fucking problem to being with???
I understand the sentiment but I think this is too far of a pushback. Restricting access to information even though it may be inaccurate or misleading isn't the right answer. AI is a tool. I'd wager, force a prompt to specifically be visible whenever subjects break that. Essentially the cigerette equivelant "THIS PRODUCT KILLS YOU" but something like "AI RESPONSES TO MEDICAL INFORMATION SHOULD NOT BE TREATED AS ACCURATE."
I feel like this logic should of banned search engines since they could provide links to sites with misinformation.
I really think this is a bad idea these should really just be flagged, there's many situations where asking these questions in necessary and there's few if any other ways to get an answer
Engineering and design.
You know that there are MBAs thinking right now they can get an LLM to design a mine, building or bridge for them.
Add edibility of wild plants to the list please
Trying to save white collar jobs with obstructive law won’t have the effect they think it will.
I'm guessing this isnt about saving jobs, it's about reducing the harm caused by people taking advice from a misinformation machine for complex issues like the law and their physical and mental health.
Why is Mamdani pictured here? He's the Mayor of NYC, he has no direct role in statewide legislation
I disagree, knowledge should be easily accessed by everyone, they should only append it with a warning.
Some people can't afford seeing many lawyers and doctors, so it is cheaper to cross refrence with ai and do some online search.
This doesn't really work though. If it were simply directing you to actual sources that could help out (articles, and lawyer authored blogs or something), that would be one thing, but LLMs are just probability machines. It'll write out what looks like good sound reasoning and logic, but in reality it's simply the most probable word followed by the next most probable word, over and over again. It's using pattern matching, not knowledge
Except AI is an over engineered autocomplete operating completely off of statistical models. Knowledge is not statistical or random. I should get the same outputs to the same inputs.
Modern capitalism derives it's scarcity via artificial means such as legislation.
The only thing this does is restrict the means of information to the people of NYC. Have you used traditional web search for health/legal/etc. advice? Even pre-2015 if you didn't know what you're doing you would get the most unhinged, predatory, incorrect advice imaginable, but legislating away access wasn't going to make health information more correct, accurate, or equitable. It would've just further entrenched the predatory health/legal/etc. industry as the gatekeepers of that knowledge.
This is no different.
Absolutely not.
Ai has saved me from suicide multiple times. Without it I'd 100% be dead.
However, it's a recursion engine and I have more objective self awareness than most so I understand I am a minority.
"AI's" are fast, for sure, but they are no substitute for human intelligence and knowledge. Mostly a desperate cash grab, because they had no where else to go after cellphones.
I love this kind of law because it's likely impossible for ai companies to comply.
Could be there'll be some exemptions if the user employs workarounds, because at the moment it's pretty easy to get llms to tell you things they aren't supposed to.
But if the laws are written right with actual penalties that may force the companies to create a product that's capable of having safety parameters - something llms don't currently seem capable of.
Cue the first amendment arguments from the "corporations are people" crowd
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
