582
you are viewing a single comment's thread
view the rest of the comments
[-] Jason2357@lemmy.ca 1 points 3 weeks ago

The delimiter isn't really the issue. Its that there are lots and lots of weird edge cases that break reading csvs. If you use commas, at minimum, you need to escape commas in the data, or quote strings that might contain commas... But now you have to deal with the possibility of a quote character or your escape character in the data.

Then you have the fact that csvs can be written with so many different character encodings, mangling special characters where they occur.

Aaand then you have all the issues that come with lack of metadata - good formats will at least tell you the type of data in each column so you dont have to guess them.

Lets see, its also really annoying to include any binary data in a csv, theres no redundancy or parity checks to catch currupted data, and they arent compressed so you need to tack on compression if you want efficient storage, but that means you always have to read the whole csv file for any task.

Oh, that brings me to the joys of modern columnar formats where you can read selected columns super fast without reading the whole file.

Oh god, I really kept going there. Sorry. Its been a year.

[-] espurr@sopuli.xyz 1 points 3 weeks ago

No that was really cool, what's an exaple of modern format? CSV sounds kinda not based

this post was submitted on 04 Apr 2026
582 points (100.0% liked)

Science Memes

20074 readers
2776 users here now

Welcome to c/science_memes @ Mander.xyz!

A place for majestic STEMLORD peacocking, as well as memes about the realities of working in a lab.



Rules

  1. Don't throw mud. Behave like an intellectual and remember the human.
  2. Keep it rooted (on topic).
  3. No spam.
  4. Infographics welcome, get schooled.

This is a science community. We use the Dawkins definition of meme.



Research Committee

Other Mander Communities

Science and Research

Biology and Life Sciences

Physical Sciences

Humanities and Social Sciences

Practical and Applied Sciences

Memes

Miscellaneous

founded 3 years ago
MODERATORS