Turns out we're already doing a lot of compression at the file and filesystem level.
Not necessarily. For example, you can't really compress encrypted files. You can certainly try but the result will likely be what the meme portrays.
Turns out pseudo random byte streams don't really repeat that often.
We just need to develop an algorithm to compress random byte streams, easy
Media files are always thoroughly compressed (except in certain settings like professional video and audio work).
Media files can can benefit from a codec change. Going from h264 to h265/hevc can net a 30-50% reduction in size for almost no quality loss.
The only trade off is increased cpu usage if the client doesnt have hardware h265 support and the time to do the transcoding.
I don't see how that relates to encryption.
It’s more relevant to the previous comment as an example of how we are doing a lot of compression at the filesystem level.
The files that are typically largest are already quite thoroughly compressed.
Plot twist: Your file manager reports directories in GB and files in GB converted to GiB with GB as units
~/j~
Well, yeah. The real advantage is only having a single file to transfer, makes eg. SFTP a lot less annoying at the command line.
Lossless compression works by storing redundant information more efficiently. If you've got 50 GB in a directory, it's going to be mostly pictures and videos, because that would be an incredible amount of text or source code. Those are already stored with lossy compression, so there's just not much more you can squeeze out.
I suppose you might have 50 GB of logs, especially if you've a logserver for your network? But most modern logging stores in a binary format, since it's quicker to search and manipulate, and doesn't use up such a crazy amount of disk space.
I actually just switched a backup script of mine from using a tar.gz to just a regular tar file. It's a little bigger but overall, the process is so much faster I don't even care about the tiny extra bit of compression (100gb vs 120gb transferred over a 1gbit connection). The entire reason I do this is, like you said, transferring files over the Internet is a billion times faster as one file, BUT you don't need the gzip step just for that
You probably know of this already, but you might consider using Borg Backup instead. It only sends the changed bits which is even faster.
I'll have to take a look. I was trying to keep it simple but I'm not opposed to giving that tool a shot. Thanks. :)
Meh, that barely fits an empty MS Word doc...
/s
Can we transcribe it to a QR code?
Efficiency
haha. I spent so much time looking for an efficient compression algorithm for audio and movies ... until I finally understood there are already compressed X) However it allow me to discover zst, wich is uncredible with text :
35M test.log
801K test.7z
32 test.log.7z
46 test.log.tar.bz2
45 test.log.tar.gz
108 test.log.tar.xz
22 test.log.tar.zst
2,1M test.tar.bz2
8,1M test.tar.gz
724K test.tar.xz
1,1M test.tar.zst
8,1M test.zip
Am I reading that correctly that test.log.7z compressed 35 mB of text into 32 bytes?
And .tar.zst into 22!?
What was that log file? Millions of identical characters?
Email written only with the letter 'P'
I shall have check. No for some reason some archives failed. Not 7z, but it size is 800kB :
35M test.log
804K test.7z
8,1M test.tar.gz
8,1M test.zip
724K test.tar.xz
2,1M test.tar.bz2
1,1M test.tar.zst
Yeah, I found zst recently as well. Quite a bit faster than xz/7z which is what I previously used (or gz when I just needed something fast).
"Look what they need to mimic a fraction of our compression ratio." - tar cJf said to tar cjf
I dont remember him crushing a tank, is this an edit or did they release another watchmen movie?
It’s from the 2009 movie, during the montage where he is on Mars and is recounting to himself how he got his powers and his relationship with Janey. Timestamp is 1:10:08.
Ah, thanks. Figured it was just an edit of the climax of the movie. You know the one
I believe it was a short clip shown when they went into Dr. Manhattans back story.
compress ze file
Me compressing binary data
you know the computer thing is it plugged in?
A community for memes and posts about tech and IT related rage.