This is what I call a “wild goose chase”. LLMs like to do that sort of thing, and it’s your job to notice that and pull the plug as soon as possible. If you’re not qualified to evaluate output, the risk of an aimless never ending chase increases dramatically. The good thing is, that you’ll learn to be more careful next time. Take those half hallucinated answers with a spoon of salt.
LLMs can be good for troubleshooting, but if the first suggestion doesn’t work, the 10th won’t either. Never do any LLM “fixes” that you don’t know are reversible.
It’s been a mixed bag for me. The case has to be a relatively simple one for it to work. LLMs need a lot of hand holding to be useful.
I find I need to either/both:
- tell the LLM all the context I can think of, excessively
- remind it to ask me questions about the problem before suggesting solutions.
I have made a few agents in Mistral, and I think I also added a line about asking clarifying questions. Should probably make another one just for troubleshooting. There should be many lines about critical thinking, ruling out the obvious, avoiding assumptions, and asking questions.
There should be many lines about critical thinking
You people do realize that this isn’t possible, right? LLMs do not “think”, let alone think critically.
Well, that was just a quick summary of an idea. You can’t just tell someone (or a an LLM) to think critically and expect it to work magically. However, you can follow certain guidelines to take a few steps in the right direction. There is a method to it you know. It starts with simple things like investigating whether or not a claim is supported by the evidence, are there unstated assumptions and so on. If you follow rules like this, you can avoid some of the simplest reasoning mistakes. Putting all of that into a prompt takes some time and effort. Doing it properly will probably result in a 500 line long manual of critical thinking.
I’ve done something similar with programming, and the LLM usually follows most of my style guide instructions reasonably well. It’s not perfect though, and it will deviate from some rules no matter how hard you try. Also, it requires constant hand holding, because it’s an LLM. Anyway, I don’t expect a critical thinking agent to be a fool-proof solution. As long as it can avoid some of the stupidest reasoning mistakes, it could be a bit more useful than the default version, and might actually be helpful in troubleshooting.
By the time you have added all of these you have just googled the solution yourself?
Googling the solution yourself can honestly be such a pain in the ass for some problems. Search engines have become so shite. You’ll find some stack overflow question from 8 years ago that’s no longer relevant, and in the end you might have to consult documentation directly which can take many minutes or hours. I reaaaally hate to say it but LLMs are actually useful for this usecase
Oh and the man pages of terminal commands. Usually it feels like those were written for the people who already know all about the command, but simply forgot which flag does what. Like, was that -n or -t again? Oh, let’s check the man pages. Got it, it’s -n, so let’s go with that.
What about those who have never used that command before? I feel like the man pages are the wrong place for people like that. You could spend 20 minutes reading and get very little value out of it. Instead, you could spend 2 minutes reading a tutorial blog, adapt the command to your use and get on with your life.
Alternatively, you could ask an LLM to generate a command that does what you want and… hope for the best. Hallucinations do exist. However, I’ve also discovered some awesome commands this way. In simple cases, it’s surprisingly fast. In complex cases, you’ll find yourself on a wild goose chase again.
Google’s quality has been nosediving for years. I don’t use it for searches any more because it’s practically useless. I hear Kagi is good, but it is a paid service.
If you have a list of commands to type in the terminal, but you need them only once, you aren’t going to make a bash script for that. If you think you’ll need to type that thing every week, making a script suddenly begins to make sense.
Same thing with agents. Do you think you’ll be asking these kinds of questions again in the future? If so, making an agent for it could make sense.
Makes me wonder if there are people out there right now on some crazy AI goose chase. Just taking them all over the place doing random nonsensical tasks.
The chances are very high. People always have technical problems, and they are also impatient enough to use an LLM. Why spend hours reading stackoverflow, and try solutions that aren’t exactly what you’re currently facing? That takes time, effort and improvisation skills.
Meanwhile, you could type a question to an LLM, and get an answer in a few seconds. In the best case scenario, you’ll get something useful out of it, but you could also start another wild goose chase. Humans are lazy, so they’ll fall into this trap very easily. It’s a gamble.
I’m sure it happens all the times because it doesn’t always have all the expectations/understanding of what the end result may look like when completed. Add that to the fact that it isn’t actually knowing to check for limiting factors you will run into later, because it is mostly just regurgitating ideas that worked in specific scenarios
The good thing is, that you’ll learn to be more careful next time.
Unfortunately, most clankers don’t think this way. It’s always either a “rare” fluke, or just because they didn’t use the most expensive model
LOL. Just throw more money at the problem until it goes away. Sure… that worked great last time.
True, I wonder what the AI was chasing. I would imagine you could download the updated bios for that board to a USB using another machine and maybe get the bios working again, then they may be able to salvage things. Dells bios recovery can usually bypass any USB boot restrictions that were in the previous bios settings
The AI probably forgot what the initial prompt was halfway through the conversation due to context length lol
It’s usually some random thought that is in the right neighborhood, but not quite spot on. A human troubleshooter would straight up say that it’s impossible to tell you what the problem is, so you would need to narrow it down by testing a few things.
An LLM just says that you need to update drivers or whatever. If the problem is caused by something obscure (like this one), an LLM will never be able to figure it out. This kind of stuff apparently just doesn’t exist in the training data, so there’s no way for the model to extrapolate and reach the right conclusion. Instead, it will continuously interpolate with the data it has, and you’ll end up with an infinite list of wrong answers.
Sycophancy really doesn’t help either. If you have any ideas what might cause the problem, the LLM will cling to those, no matter how wrong you might be. Troubleshooting requires critical thinking and LLMs don’t seem to be very good at that.
Can we all agree that giving a bunch of shit advice followed by: welp, that’s all i got, good luck. Is kinda funny
Removed by mod
I’m also a medical professional on the Internet and can safely say injecting bleach can cure many ailments, including migraines and curing covid.
Also permanently cures hemorrhoids
Funny, I’m a dog on the internet and I eat chocolate all the time, it’s not toxic at all.
Thanks doc
deleted by creator
Train an AI on 4chan, get 4chan level tech support.
It’s like all those memes about finding information in old internet correspondence.
1 person has your exact problem and it’s five pages of people offering general or unhelpful advice until you finally get to them marking it solved with: "got it, thanks everyone!
At this point, we could start setting tires on fire while conducting informal polls without pen and paper and probably cut down on the hallucinating.
Backup file corrupts, program still loads and operates normally Oh shit Better destroy my laptop
Found the linux user lmao 🐧
I’ve tried to use it before for troubleshooting but even with as much detail as I can manage, it almost always gives me irrelevant boilerplate advice. I get it, it’s a predictive model and the most common answer to the question. The problem is, tech issues are specific and general advice isn’t going to solve it (plus man, of course I rebooted, duh).
I found that instead of AI, it’s always an obscure forum post that saves me. Like, cold boot after a 30 second power off saved me so much trouble after going Windows to Linux, and AI didn’t even have that in the 60 bullets of advice it was giving. Ugh.
Yeah, I once had ChatGPT lead me down a rabbit hole chasing a windows network share issue that I had absolutely no business troubleshooting. At least I knew enough to test one change at a time and verify functionality with the concerned parties before moving to the next change, so nothing broke catastrophically like this, but looking back on the experience scared the shit outta me. Never fixed the issue, but the slop machine had me believing I was just on the cusp of fixing everything the whole time.
I know for a fact that there are people way dumber than me, way more powerful than me, and way more willing to eat up this garbage psuedo-technical sycophancy all at the same time, and that sends a cold shiver down my spine.
This shit needs to die. The sooner the better.
I had some issues getting my MIDI setup working on Linux the first time, and made the mistake of asking an AI. It was just a disaster. Kept giving me more and more random shell commands, most of them didn’t even run, asking for outputs of shell commands, “yes now I see the issue here” (it didn’t see anything). I eventually gave up and found my own dodgy workaround
This is one of the comments 💀
“Hopefully” 💀💀💀
Goddamn clankers
“Three hours of following Goggles suggestions”
Say no more
Haha yeahh, AI helps, but everything needs to be verified with experience
Put that on a shirt and sell me 20
To be fair, bad advice from random internet people was already a thing long before big babble machines.
Y’all - This post is actually a full level more sinister and all about shareholder value than it seems.
There is a huge difference between the paid versions of LLMs and the unpaid/free tier. I’ve heard some estimations of about a year lag time.
Why is this about shareholder value? Because these companies only tell you about the pie in the sky good stuff that the corporate level paid versions are doing.
Case in point, I’ve had 2 really excellent times troubleshooting issues with Claude, just like @paranoia@feddit.dk which only worked because it was the paid version. One was trying to re-format and re-use an external SSD in container to use as a bootable Win 11 drive. The paid version managed to troubleshoot the issue, which was an MBR label for the boot sector. Didn’t even take long, maybe 20 minutes on a side quest. Similarly, paid Claude helped me get a home network up and running in minutes.
I’ve tried unpaid Claude for even rudimentary troubleshooting, and it throws boilerplate “have you tried turning off and then on again?” level troubleshooting that might only help grandmas and dipshits. Useless. Which is the point. "Oh, hey, you want the thing that isn’t soft in the head? Pay us $20.
There’s some problems with this take of “the unpaid level is about a year behind”.
-
This is absolutely idiotic marketing. “Here, try our free service to see what you’re getting. Oh, it’s absolute shit, completely useless, and in fact damaging? Give us money!” The whole point of free trials is to show off your strengths. If your free trial is a piece of shit, it’s not going to convert into paid subscriptions.
-
If we use this logic, well, let’s rewind a year. The current free level is the same as what the pro level was a year ago. A year ago they were selling their pro levels as the greatest things since sliced bread. As “Ph. D. equivalent assistants” in fact. And here we are a year later, so the public level should be at the “Ph. D. equivalent assistants” that were advertised a year ago. And they’re basically Ph. D. assistants with a diploma from a mill, wearing clown shoes. So … colour me suspicious of claims that the commercial grades are any better.
Now for point 1, I wouldn’t put it past the techbroaidudes to be absolute shit at marketing. A lot of their marketing, in fact, seems like it was dreamed up by an LLMbecile. But point 2 is a harder one to counter. They were saying the same thing a year ago that they’re saying now, and the free tier now is supposed to be what the paid tier was a year ago. This doesn’t add up.
-
At Google, AI does Reinforcement learning on You!
a GParted backup file was corrupted
I don’t know much about Linux, but is a single backup that important? Like it feels like you could just delete that backup and make another one
time better spent rtfm and thinking
I’ve had pretty good success with Claude doing this even two years ago.
I have an extremely dodgy home server where I needed to move the OS from the eMMC to an SSD connected to the WiFi card port. Obviously had a lot of problems with partitions and cloning but it got there in the end.
