I'm a data analyst and primary authority on the data model of a particular source system. Most questions for figures from that system that can't be answered directly and easily in the frontend end up with me.

I had a manager show me how some new LLM they were developing (which I had contributed some information about the model to) could quickly answer some questions that usually I have to answer manually, as part of a pitch to make me switch to his department so I can apply my expertise for improving this fancy AI instead of answering questions manually.

He entered a prompt, got a figure that I knew wasn't correct and I queried my data model for the same info, with a significantly different answer. Given how much said manager leaned on my expertise in the first place, he couldn't very well challenge my results and got all sheepish about how the AI still in development and all.

I don't know how that model arrived at that figure. I don't know if it generated and ran a query against the data I'd provided. I don't know if it just invented the number. I don't know how the devs would figure out the error and how to fix it. But I do know how to explain my own queries, how to investigate errors and (usually) how to find a solution.

Anyone who relies on a random text generator - no matter how complex that generation method to make it sound human - to generate facts is dangerously inept.

When you delegate, to a person, a tool or a process, you check the result. You make sure that the delegated tasks get done and correctly and that the results are what is expected.

Finding that it is not the case after months by luck shows incompetence. Look for the incompetent.

I raised this as a concern at the corporate role I work in when an AI tool that was being distributed and encouraged for usage showed two hallucinated data points that were cited in a large group setting. I happened to know my area well, the data was not just marginally wrong but way off, and I was able to quickly check the figures. I corrected it in the room after verifying on my laptop and the reaction in the room was sort of a harmless whoops. The rest of the presentation continued without a seeming acknowledgement that the rest of the figures should be checked.

When I approached the head of the team that constructed the tool after the meeting and shared the inaccuracies and my concerns, he told me that he'd rather have more data fluency through the ease of the tool and that inaccuracies were acceptable because of the convenience and widespread usage.

I suspect stories like this are happening across my industry. Meanwhile, the company put out a press release about our AI efforts (literally using Gemini's Gem tool and custom ChatGPTs seeded with Google Drive) as something investors should be very excited about.

I work in a regulated sector and our higher ups are pushing AI so much. And there response to AI hallucinations is to just put a banner on all internal AI tools to cross verify and have some quarterly stupid "trainings" but almost everyone I know never checks and verifies the output. And I know of atleast 2 instances where because AI hallucinated some numbers we sent out extra money to a third party.

Apparently that reddit post itself was generated with AI. Using AI to bash AI is an interesting flex.

As an unemployed data analyst / econometrician:

lol, rofl, perhaps even... lmao.

Nah though, its really fine, my quality of life is enormously superior barely surviving off of SSDI and not having to explain data analytics to thumb sucking morons (VPs, 90% of other team leads), and either fix or cover all their mistakes.

Yeah, sure, just have the AI do it, go nuts.

I am enjoying my unexpected early retirement.

To everyone I've talked to about AI, I've suggested a test. Take a subject that they know they are an expert at. Then ask AI questions that they already know the answers to. See what percentage AI gets right, if any. Often they find that plausible sounding answers are produced however, if you know the subject, you know that it isn't quite fact that is produced. A recovery from an injury might be listed as 3 weeks when it is average 6-8 or similar. Someone who did not already know the correct information, could be damaged by the "guessed" response of AI. AI can have uses but it needs to be heavily scrutinized before passing on anything it generates. If you are good at something, that usually means you have to waste time in order to use AI.

Before anything else: whether the specific story in the linked post is literally true doesn’t actually matter. The following observation about AI holds either way. If this example were wrong, ten others just like it would still make the same point.

What keeps jumping out at me in these AI threads is how consistently the conversation skips over the real constraint.

We keep hearing that AI will “increase productivity” or “accelerate thinking.” But in most large organizations, thinking is not the scarce resource. Permission to think is. Demand for thought is. The bottleneck was never how fast someone could draft an email or summarize a document. It was whether anyone actually wanted a careful answer in the first place.

A lot of companies mistook faster output for more value. They ran a pilot, saw emails go out quicker, reports get longer, slide decks look more polished, and assumed that meant something important had been solved. But scaling speed only helps if the organization needs more thinking. Most don’t. They already operate at the minimum level of reflection they’re willing to tolerate.

So what AI mostly does in practice is amplify performative cognition. It makes things look smarter without requiring anyone to be smarter. You get confident prose, plausible explanations, and lots of words where a short “yes,” “no,” or “we don’t know yet” would have been more honest and cheaper.

That’s why so many deployments feel disappointing once the novelty wears off. The technology didn’t fail. The assumption did. If an institution doesn’t value judgment, uncertainty, or dissent, no amount of machine assistance will conjure those qualities into existence. You can’t automate curiosity into a system that actively suppresses it.

Which leaves us with a technology in search of a problem that isn’t already constrained elsewhere. It’s very good at accelerating surfaces. It’s much less effective at deepening decisions, because depth was never in demand.

If you’re interested, I write more about this here: https://tover153.substack.com/

Not selling anything. Just thinking out loud, slowly, while that’s still allowed.

My broseph in Christ, what did you think a LLM was?

This would suggest the leadership positions aren't required for the function of the business.

I guarantee you this is how several, if not most, fortune 500 companies currently operate. The 50k DOW is not just propped up by the circlejerk spending on imaginary RAM. There are bullshit reports being generated and presented every day.

I patiently wait. There is a diligent bureaucrat sitting somewhere going through fiscal reports line by line. It won't add up.. receipts will be requested.. bubble goes pop

Ah yes, what a surprise. The random word generator gave you random numbers that aren't actually real.

It doesn't matter. Management wants this and will not stop until they run against a wall at full speed. 🤷

Tbf at this point corporate economy is made up anyway so as long as investors are gambling their endless generational wealth does it matter?

Jesus Christ, you have to have a human validate the data.

Dumbasses. Mmm, that's good schadenfreude.

My workplace, the senior management, is going all in on Copilot. So much so that at the end of last year to told us to use Copilot for year end reviews! Even provided a prompt to use, told us to link it to Outlook (not sure why, since our email retention isn't very long)... but whatever.

I tried it, out of curiosity because I had no faith. It started printing out stats for things that never happened. It provided a 35% increase here, a 20% decress there, blah blah blah. It didn't actually highlight anything I do or did. And I'm banking that a human will partially read my review, not just use AI.

If someone read it, I'm good. If AI reads it, I do wonder if I screwed myself. Since senior mgmt is just offloading to AI...

Surely this is just fraud right? Seeing they have a board directors they have shareholders probably? I feel they should at least all get fired, if not prosecuted. This lack of competency is just criminal to me.

I-want-to-believe.jpg

The output from tools infected with LLMs can intrinsically only ever be imprecise, and should never be trusted.

I hope they sue whoever sold it to them. it's not artificial intelligence, it's a machine learning chat bot. they may as well be running their company with a magic eight ball.

I fucking love this. It's amazing.

This is why I hate search engines promoting AI results when you are researching for something. It is confidently giving incorrect responses. I asked for sources on one LLM model before while using Duckduckgo, and it just told me that there are no sources and the information is based on broad knowledge. At one point, I challenged the AI that it is wrong, but it insisted it doesn't. It turns out that it is citing a years old source written by a different bot long ago. But on the one hand, most of you are probably familiar that on occasions that the AI is incorrect and you challenge it, it will relent, although it will be a sycophant even though you yourself are actually incorrect. This is Schrödinger's AI.