New York State not New York City.
I don’t know why Mamdani is pictured. I think it is because the proposal was introduced in the State Senate by Kisten Gonzalez who is a DSA member.
New York State not New York City.
I don’t know why Mamdani is pictured
the text in this image is also just flat-out wrong: the proposed legislation does not actually “ban AI from answering questions” on these subjects at all, as this post claims it does.
https://www.nysenate.gov/newsroom/press-releases/2026/kristen-gonzalez/ny-state-senator-kristen-gonzalez-her-bill-address-ai explains that her bill is strictly about impersonating licensed professionals:
We have documented instances of AI chatbots providing false medical license numbers to users and documented cases of real harm from AI chatbots providing misinformation. This is what my common-sense legislation addresses—protecting users from misinformation, scams, and fraud. This legislation does not prohibit a user from asking a chatbot questions or receiving general information and advice, as long as the chatbot is not presenting that information as a licensed professional. It would also not prohibit licensed professionals from consulting AI technology in the course of their work.
Fake news? On my social media??
Fake news? On my social media??
I’m surprised that this post is still up so many hours after i (and i’m sure others) reported it, especially given the large number of moderators that this community has.
This makes way more sense.
Removed by mod
Maybe I’m just ignorant about this, but it feels a bit “gatekeepy” to outlaw answering questions about the law itself. Like… so I can’t use an LLM to help do taxes? Or if I’m served a confusing legal document, I have to pay an expensive attorney to understand the legalese? I can’t even just get a rough idea from an LLM first?
EDIT: Also I’ve literally had a professional accountant send me a ChatGPT conversation they had because they didn’t know tax law well enough. I feel like if anything that should be outlawed. Professionals should actually have to know shit and know how to research primary sources.
I guess it’s like banning loan sharks. Sure some people could only.get loans through them because banks were racist or whatever. But it’s objectively bad for everyone using it mainly because when it’s bad its really really bad.
I would say its to protect people, if you ask a LLM for a legal answer and trust it, then you could get into legal trouble when it turns out to be wrong.
Not only are LLMs wrong like ~1/10th of the time (for the more advanced ones) but law is also an area where there is black and white rules, but also there is interpretation of some aspects.
Some things are set in stone and some are about understanding the intention of the law and being able to back that up in a court if it was challenged.Do you want the public trusting a LLM that says “oh yes you can claim a tax deduction for an Xbox because its for stress relief related to your job” and then you get whacked with a fine for incorrectly filing your tax.
Using a LLM to help your draft an email to your uncle = low risk. Using a LLM to help you decide if you actually need a medical procedure your doctor has reccomended = high risk.
If that was the logic we were using then Wikipedia shouldn’t be allowed to have articles on health and law either. WebMD shouldn’t have ever been legal either.
Obviously, the benefits to having that information available outweigh the hypothetical negatives. Even with them being incorrect, partial and easy to draw invalid conclusions from, having the information be freely and readily accessible is better for everyone.
I think the distinction is that looking up wikipedia or any other website that appears to have topics of information you can learn from is different from asking a tailored answer to your specific query.
One feels like more like research to improve your understanding, but you know you would have to go to a professional to ask specific questions around your circumstances.
And the other feels like your own ‘personal professional’ that answers all your little specific questions, but is really just a complex algorithm trying to make you happy and doesnt know if is telling you right or wrong, only that it shouldnt say it doesnt know.A program that can be very confidently wrong while giving people advice that can mess up their life (healthcare, finiancial, legal advice, etc) could benefit some, but will ruin others make it not worth the risk?
A program that can be very confidently wrong while giving people advice that can mess up their life
Literally WebMD.
I think there’s an assumption that professionals are always correct. They are not. Professionals are also heavily trained to “[try] to make you happy and doesnt know if [they are] telling you right or wrong, only that [they] shouldnt say [they don’t] know.”
You’re also neglecting how bad health care access, and legal etc., is. Especially in countries like the USA where access to healthcare is financially gated and increasing extraction makes “go[ing] to a professional” not possible unless you wait >6 months to go somewhere that’s a >3hr commute from where you live. WebMD, Wikipedia and chatbots can help to decide whether the >$1,000 and 7hr professional consultation is worth it, and if so be more informed going in so that the time is better spent.
That being said, poor healthcare can absolutely have worse outcomes than no healthcare. I do believe that is a minority of situations, however I also lack any hard evidence to really back that up. Do you have any?
WebMD is just a website with information pages on health topics, not a program or custom response, is it not?
I think that there is a future that it may be a reasonable first step for people with restricted access barriers, but at this stage its too much of the wild west to be using it for these higher level cases.
But regardless i dont have specific information to present you, i woild argue the amount of people that have committ suicide after interacting with LLMs rather than seeing a mental health professional would be a fairly relevant example though.
I thought the issue you had was with the information itself. Why does it “being a program” matter? That said, it is; “symptom checker” specifically and “page rank” more generally.
its too much of the wild west
And? There seems to be this assumption that “wild West” == bad/dangerous/scary/harmful. The modern Internet is certainly far less of the “wild West” it used to be. I would also argue it is also far more dangerous, harmful and nefarious than the “wild West” web.
the amount of people that have committ suicide after interacting with LLMs rather than seeing a mental health professional would be a fairly relevant example though.
Anecdotes aren’t useless, but they need to be contextualized. Is it higher than the baseline? Is it greater then those who didn’t commit suicide because when the mental health institutions failed them they at least had some sort of backup?
Navigating healthcare while dealing with a mental health issues is a paradox. Is the damage caused greater than the damage prevented from allowing access to a tool that can help lower that barrier to entry?
Looking at general trends I find that the harms outweighing the benefits very hard to believe. Do you have anything non-anecdotal, on that specific point, to back it up?
Look mate, all i am saying is that cutting edge technology shouldnt be used for critical areas until it is proven safe, i really am not invested enough in the cenversation to continue this thread, i support the descision being made to restrict LLM use for these areas at this point in time.
LLMs have been linked to mental health impacts, specifically demonstrably in suicide cases (see: https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots) and i do believe in a lot of these cases the LLMs caused greater harm than would have been presented without the LLM interference, and definitely less harm than if these people saw a human professional to talk about their issues rather than a program. Yes there is a cost of seeing a health professional but this is not a case of ‘somethings better than nothing’.
law in itself is very gatekeepy - you have to learn the “correct” way to interpret the words and then get certified for it.
More like you as an average person have no idea what it’s hallucinating and what is the truth and may end up in jail or the hospital like the cases that are starting to get reported on who did just that.
with a group of people that has a bias towards a certain delusion, and/or high acceptance of directly scientifically contradictory beliefs, you can get a bubble and affirm themselves out of reality in the same way. i also wouldn’t suggest banning groups of people, but i believe with groups or with LLMs, a good grounding in scientific thinking is the best way to keep ‘stable’ away from delusion.
a shame that grants are being filtered by a project 2025 author right now, and a bunch of important science got cut by this admin.
also see it like mobile phones, which instead of being useful mobile personal computers, we got data collection bricks that has a skeleton of a few practical tools built into the soft edges of a giant gambling and addiction pipeline.
there’s a whole rant about the high-level psychological manipulation employed against average people just to make them more comfortable losing their money, which does not mean that portable personal computers are bad, but the failure of anti-trust and consumer protection has allowed all technological and social development to be hindered purely for the extractive demands of the people who actually get to decide what happens.
as long as you keep the general public too tired to think about it, you just have to convince them that some other small group is their tribal enemy and that’s where your collective focus and energy should be going, rather than the more simple answer of rich assholes being opportunistic and stupid/evil.
I understand the sentiment but I think this is too far of a pushback. Restricting access to information even though it may be inaccurate or misleading isn’t the right answer. AI is a tool. I’d wager, force a prompt to specifically be visible whenever subjects break that. Essentially the cigerette equivelant “THIS PRODUCT KILLS YOU” but something like “AI RESPONSES TO MEDICAL INFORMATION SHOULD NOT BE TREATED AS ACCURATE.”
I feel like this logic should of banned search engines since they could provide links to sites with misinformation.
AI shouldn’t be an excuse to take a job, it can be a tool but that’s it
In what way is this restricting information? AI usually just summarizes info that was already trivially searchable
I feel like finding and citing appropriate law cases would be a great way for AI to be used, especially because the bullshit legal-ese that surrounds everything in court cases is so spectacularly opaque and trivial that you have to pay out the nose just for someone to parse through all the bullshit… Like, lawyers shouldn’t even be a thing you need unless it’s for some really complicated case. The courtroom should be a place where an average citizen could speak for themselves.
Also, as someone who studied medicine… Do you know how often doctors will take a patients history, step outside the exam room, and then google shit? I don’t want to imply that they don’t know what they’re doing, like, you need to study medicine to know what to google, BUT… they still be googling shit. I think AI could be trained to ask the same patient history questions that a nurse asks. Then using the answers to those questions, then they can provide some feedback, with a big ol disclaimer that it’s NOT to be taken seriously, and you should consult with a physician.
When people say “AI” these days, they mean the LLM-derived generative AI such as ChatGPT, Google Gemini, Anthropic Claude, and Grok.
These AI systems are not capable of finding authoritative search results. They are good at generating fluent, plausible, context-appropriate text. In legal and medical situations, that’s the opposite of what you’d want.
Do you know how often doctors will take a patients history, step outside the exam room, and then google shit?
Think about how often the doctors land on some pseudo-science result. Now imagine that instead of a doctor it’s a patient and instead of asking “could these be related” the patient is told “these are related”.
a big ol disclaimer that it’s NOT to be taken seriously
This has worked very well for Fox News, which is not a news agency and is for entertainment purpose only. Also for dietary supplements, which are not medicine.
Specifically tuned LLMs that are trained on medical databases and less sycophant (E.g. OpenEvidence) are one of the killer applications of AI. Tons of frivolous AI nonsense but this could actually save lives. Access to broad knowledge and connections that a doctor can make use of.
We’re demanding and regulating the complete wrong shit about AI.
And answering general health questions isn’t how AI is being used in the medical field.
In that we aren’t regulating any of it?
What would be your demands for regulation?
Companies should be liable for what their LLM says.
That seems technically not possible.
How would that be technically impossible? Or difficult? Companies are currently liable for the actions of the human agents, and I don’t think the implementation needs to be different.
Maybe you mean socially impossible? Like, the major providers have a lot of money behind them and so why would they allow themselves to be liable for anything?
I’d agree with you to some extent if people weren’t so god damn stupid. So many lazy fucks just take the AI’s output at face value, believing it to be some all knowledgeable god, rather than a compilation of all human biases expressed on the internet with a bunch of hallucinations thrown in for good measure.
It’s a great search engine for case law, but yeah most people are too stupid to use it.
Yeah, I don’t disagree, it can be a decent starting point, emphasis important lol
I really think this is a bad idea these should really just be flagged, there’s many situations where asking these questions in necessary and there’s few if any other ways to get an answer
Sorta agree. Maybe force them to have a very obvious sign that says AI is routinely wrong about these topics specifically and are not a source to consider reliable, and then also offer a helpline where one can get reasonable answers even if the answer is “unfortunately you need to seek expert opinion on this”
But googling basic labor laws and simple healthcare tricks should be acceptable like stretches for wrist or knee pain, skin/hair care or what level of diarrhea is hospital worthy should be fine.
Same, especially for medical advices.
I am currently going through a very complex diagnosis of combination of SCDS and migraines, and neck joint / facial nerve issues through multiple doctors - ENT, neurologists and so on.
Having AI backing me up with providing a structural understanding from multiple disconnected sources is a huge blessing.
Especially when you die from migraine, and trying to understand are symptoms worth to go to emergency ward or not.
AI can’t back you up better than you can with your own research. Why are you even considering AI credible? What gives it credibility?
It’s not credible, but you can ask it for a credible source and read that, which is still incredibly helpful.
I guess the question is, why ask it for the source? What gives it credibility to be the best at doing such a thing?
It’s not, but a source can be validated by a human much easier than finding the source in the first place.
Use it like a fancy search engine.
The problem with medical questions like these is LLM would spew out info confidently that doctors/specialist don’t even know the answer to. LLM doesn’t know when it’s guessing.
Ask it for a source and verify it, you can even give the source to a doctor if you can’t
Modern capitalism derives it’s scarcity via artificial means such as legislation.
The only thing this does is restrict the means of information to the people of NYC. Have you used traditional web search for health/legal/etc. advice? Even pre-2015 if you didn’t know what you’re doing you would get the most unhinged, predatory, incorrect advice imaginable, but legislating away access wasn’t going to make health information more correct, accurate, or equitable. It would’ve just further entrenched the predatory health/legal/etc. industry as the gatekeepers of that knowledge.
This is no different.
Add edibility of wild plants to the list please
I’d rather have AI as the CEO of my company, than as my doctor. But this feels a bit like overreach.
Trying to save white collar jobs with obstructive law won’t have the effect they think it will.
I’m guessing this isnt about saving jobs, it’s about reducing the harm caused by people taking advice from a misinformation machine for complex issues like the law and their physical and mental health.
As long as they don’t solve all the other problems where more people die we can be sure that it is about protecting jobs.
The working class saw this coming decades ago, only to be mocked and told to retrain by the same credentialized individuals who’re now crying about AI. Yesterday’s push to offshore jobs and today’s rush toward digital automation are simply different phases of the exact same economic displacement. It cannot be stopped with legislation in a globalized world where capital flows freely.
So now you’ll have to pay $200 every time you have an important question you need answered? My PCP just tells me to use over the counter products anyways. Also is this NY state or city? Why is there a picture of Mamdani?
Removed by mod
No you will just need to sign into a VPN
I disagree, knowledge should be easily accessed by everyone, they should only append it with a warning.
Some people can’t afford seeing many lawyers and doctors, so it is cheaper to cross refrence with ai and do some online search.
This doesn’t really work though. If it were simply directing you to actual sources that could help out (articles, and lawyer authored blogs or something), that would be one thing, but LLMs are just probability machines. It’ll write out what looks like good sound reasoning and logic, but in reality it’s simply the most probable word followed by the next most probable word, over and over again. It’s using pattern matching, not knowledge
Have you used the newest LLM models? They certainly do fact checks, research info and provide online sources - if they are instructed to do that.
They output responses that read like fact checks, and link things that are semantically similar to their output.
LLMs don’t understand concepts like “fact” and “source” though, so it’s a stretch to say they “do fact checks” and “provide online sources”.
Always remember, LLMs don’t actually understand anything other than what order words typically go in. They’re really good at it, sure, but don’t let that trick you into thinking they have any deeper understanding than that.
I’ve tried on several occasions to try and explain this to people, and it’s like they WANT it to be a sentient thing, and refuse all evidence to the otherwise.
They output responses that read like fact checks, and link things that are semantically similar to their output.
So basically they do what humans do. What exactly qualifies as deeper understanding to you?
Being able to say “I don’t know” rather than making shit up to fill in what it doesn’t know. That would be a good start.
They do say that. I’ve seen Claude realise some programming package must be internal / non public and it therefore can’t provide any information on that.
It’s nothing more than surface level though. Their “knowledge” is limited to whatever is present in their training data, and any claims it makes outside of that are manufactured based on statistics and it’s knowledge of word order, not actual understanding of actual facts.
Just because it gets it right sometimes doesn’t mean it knows it’s right. Because, again, the only thing LLMs actually understand is word order.
No it’s not limited to their current training data. Like I said, they can search the internet, get information from websites or available studies, summarize and use that in their answers.
Just because it gets it right sometimes doesn’t mean it knows it’s right.
Again, how do you know?
how do you know?
Because I’ve studied LLMs and the underlying concepts they’re based on.
LLMs don’t understand concepts like “fact” and “source” though
Do you?
Yes
How do you know that you don’t just guess words, too?
Except AI is an over engineered autocomplete operating completely off of statistical models. Knowledge is not statistical or random. I should get the same outputs to the same inputs.
Knowledge is not statistical or random. I should get the same outputs to the same inputs.
Where does the thought come from? If you write five articles about the topic or any topic, will they be the same?
They won’t look the same but they’ll convey the same information consistently.
Even about topics that you barely are familiar with?
If the prompt includes the older articles, or if the NN is updated once an article is written, wouldn’t AI be consistent, too?
Then I research from known references starting with Wikipedia and using the sources at the bottom of the page to expand my research out further.
The entirety of how LLMs work is token prediction based on the entire collective of human creation, that is not research, that is not understanding. that is an over engineered autocomplete. Guessing what the next token is in a string of characters is not the same knowledge of the topic. How much does the Epic of Gilgamesh and the complete works of Shakespeare and the writings of Adolf Hitler and the unhinged Tumblr posts of every Susie, Harry, and Jillian and all of the statistical models born from vectorizing all of that into a billion dimensional vectorspace have any impact on the final output, how does any of that have anything to do with any medical or legal inquiries?
Ive already explained multiple times how a glorified autocomplete predicting words is not knowledge. If you’re going to keep sealioning I’m done with this conversation.
Ive already explained multiple times how a glorified autocomplete predicting words is not knowledge.
Not on your last two comment pages. Do you have a link?
Guessing what the next token is in a string of characters is not the same knowledge of the topic
How do you write? Do you know your entire sentences and paragraphs before you write them down?
So you’re just gonna keep sealioning, exactly as I predicted. Got it, glad to establish this conversation is going to go nowhere. You can repeat my last handful of comments in this thread. I dont have time to waste entertaining your asinine questioning. You’ve been nothing but fake polite in order to waste all of our time.
We need to both ban AI from this and make sure people have reasonable and affordable ways of accessing that info from human professionals
“AI’s” are fast, for sure, but they are no substitute for human intelligence and knowledge. Mostly a desperate cash grab, because they had no where else to go after cellphones.
This is super dumb. People will just Google their questions instead, and get worse results.
Cue the first amendment arguments from the “corporations are people” crowd
I doubt the companies will go that route, it would infer that the AI’s output represent the companies and that would make them liable.
