953
muskrat's data eng expert's hard drive overheats while processing 60k rows
(lemmy.dbzer0.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 16 Mar 2025
953 points (100.0% liked)
Programmer Humor
23247 readers
546 users here now
Welcome to Programmer Humor!
This is a place where you can post jokes, memes, humor, etc. related to programming!
For sharing awful code theres also Programming Horror.
Rules
- Keep content in english
- No advertisements
- Posts must be related to programming or programmer topics
founded 2 years ago
MODERATORS
Wow.
I've been processing a couple of billion rows of data on my machine, the fans didn't even come on. WTF are they teaching "experts" these days, or has Elmo only hired people who claim that they can "wrangle data" and say "yes" ?
Even if querying data was processing-heavy and even if somehow the ‘hard drive’ got warm during this, then there still would need to be a hardware defect in order for the drive to overheat.
Yes, but this may be a symptom of an issue I've been seeing with younger programmers; they've siloed themselves so specifically into whatever programming they "specialize" in, that they become absolutely useless at dealing with absolutely anything else related to their job. And exasperating this issue is the fact that they've grown up with systems that "just work". Windows, iOS, and android are all at the point where fucking around with hardware issues is very uncommon for the average person.
Asking this guy to solve a hardware problem is like asking hime to tune a carburetor. He likely has not the slightest clue how to start.
In my experience, a lot of software dev degree paths basically don't even have relevant classes on hardware at all. Classes on hardware are all in IT Helpdesk and Network Admin degree paths whereas the software dev students are dumped straight into Visual Studio right off the bat with no relevant understanding of the underlying hardware or OS.
My experience does not reflect yours. Computer Architecture, Discrete Math (logic gate math), and Operating System Concepts were all required classes in my CS degree from just a few years ago.
Honestly that's good to hear. I've run into some devs who are completely mystified on how to connect to a remote database and couldn't tell a socket from sandwich.
In my degree, we had to write kernel mods and device drivers
My CS degree had a hardware/IT support class, but A) it was entirely simulation based. We never touched any actual hardware. We "built" PC's or identified physical issues in 3d sim software, set up RAID arrays in software, etc. B) it was super hand holdy and you only ever go over a problem once, so nothing on the class has stuck. I know much more from having built, troubleshot and maintained my own computers and network than I ever learned from that class, then learned more by doing in an actual IT support position before becoming an engineer.
I mean to be fair the sheer amount of material most university engineering programs require these days makes spending significant time on specific problems almost impossible. They try to shove so much theory into your head they lose track of practical implementation. Basically everyone I went to school with complained about the lack of practical application relative to theory, and I studied mechanical engineering which is theoretically and literally chiefly concerned with hardware.
That’s the price of specialization. Don’t ask a software engineer to troubleshoot hardware. Don’t ask a backend dev to write a frontend. Don’t ask a proctologist to look at your cough.
You simply cannot be proficient at every sub-sub-specialty. That’s why we collaborate and hand the ‘my computer gets hot’ problems to the hardware people. The alternative would be only moderately useful generalist.
I'm not asking everyone to be able to become a hardware specialist, but if you can't even figure out "my computer gets hot" I'm not going to be able to trust anything you do. Identifying a heat issue does not take a rocket surgeon.
He hired a bunch of 19-25 year old. Not experts
Hey! Thats offensive to 19-25 year olds, there are many who just finished college/university and are more than aware.
They’re just role playing like in movies, with no idea of the consequences.
How on earth is it offensive to say they're "not experts"? They're not prodigies with PhDs. These specific young men are just technical enough and ideologically aligned.
Except they’re not, as you will know their tweet would be false after your first year of any technical (IT oriented) education.
First year? That shit is like A+ cert level knowledge or below, and A+ is damn near worthless. They would know that in the first few hours of a study guide
I was being generous when you consider the people in school who somehow pass, even when they don’t know a thing 🥲
Technical enough to be hired, is all I meant. 🙄
Even then, no. These were all obviously nepotism hires who would not have otherwise qualified.
Meh, some of them won some hackathons and scholarships, it's pretty clear they're otherwise at least somewhat bright but they don't have any relevant domain knowledge.
In other words, the type of person most likely to be prone to hubris and catastrophic failures.
Apologies, if I came over as hostile. I did not get your meaning through text.
Your original comment was ambiguous as to if being an "expert" and "being 19-25" are mutually exclusive.
There is nothing wrong with being 19-25. There's something wrong with being wholly incompetent.
There's not really anything wrong with being incompetent, so long as you have the humility to admit it and learn from people who know better, and try not to cause harm. That's not Musk's minions though.
I think it's important to differentiate incompetence from ignorance. Ignorance is not knowing. Incompetence is not being able to fulfill the requirements for your assigned task. If you cannot fulfill the requirements for your given task, then you should not be given said task.
Bunch of 1337 hax0rs script kiddies who don't understand anything but they suck elon's balls or something idk.
These are the type of people that have deleted the French language from their GNU/Linux system.
There's two issues going on:
I've read a story on the forbidden website where a "database" was a single table with a single column holding a single row that contained the actual data as a CSV blob. I'm willing to bet the muskies are not beyond such acts of genius.
It's terrifying that this is plausible.
60k rows is generally very usable with even wide tables in row formats.
I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.
After 1M rows move on to something better like Dask, polars, spark, or literally any DB.
The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.
My go-to tool of late is
duckdb, comes with binaries for most platforms, works out of the box, loads any number of database formats and is FAST.Yes, his Boy Harem.