66

About a year ago I switched to ZFS for Proxmox so that I wouldn't be running technology preview.

Btrfs gave me no issues for years and I even replaced a dying disk with no issues. I use raid 1 for my Proxmox machines. Anyway I moved to ZFS and it has been a less that ideal experience. The separate kernel modules mean that I can't downgrade the kernel plus the performance on my hardware is abysmal. I get only like 50-100mb/s vs the several hundred I would get with btrfs.

Any reason I shouldn't go back to btrfs? There seems to be a community fear of btrfs eating data or having unexplainable errors. That is sad to hear as btrfs has had lots of time to mature in the last 8 years. I would never have considered it 5-6 years ago but now it seems like a solid choice.

Anyone else pondering or using btrfs? It seems like a solid choice.

top 50 comments
sorted by: hot top controversial new old
[-] sem 6 points 6 hours ago

Btrfs came default with my new Synology, where I have it in Synology's raid config (similar to raid 1 I think) and I haven't had any problems.

I don't recommend the btrfs drivers for windows 10. I had a drive using this and it would often become unreachable under load, but this is more a Windows problem than a problem with btrfs

[-] fmstrat@lemmy.nowsci.com 4 points 6 hours ago

What kind of disks, and how is your ZFS set up? Something seems amis here.

[-] bruhduh@lemmy.world 2 points 6 hours ago

Raid 5/6, only bcachefs will solve it

[-] possiblylinux127@lemmy.zip 1 points 2 hours ago

Btrfs Raid 5 and raid 6 are unstable and dangerous

Bcachefs is cool but it is way to new and isn't even part of the kernel as of yet.

[-] bruhduh@lemmy.world 2 points 2 hours ago* (last edited 2 hours ago)

https://en.wikipedia.org/wiki/Bcachefs it was added as of Linux 6.7

Edit: and I've said raid 5/6 as what troubles btrfs have so you proved my point while trying to explain to me that I'm not right

[-] possiblylinux127@lemmy.zip 2 points 2 hours ago

I though was then removed later as there was a disagreement between Linus and the bcachefs dev

[-] bruhduh@lemmy.world 1 points 2 hours ago

Yeah, i remember something like that, i don't remember exactly which kernel version it was when they removed it

[-] exu@feditown.com 11 points 10 hours ago

Did you set the correct block size for your disk? Especially modern SSDs like to pretend they have 512B sectors for some compatibility reason, while the hardware can only do 4k sectors. Make sure to set ashift=12.

Proxmox also uses a very small volblocksize by default. This mostly applies to RAIDz, but try using a higher value like 64k. (Default on Proxmox is 8k or 16k on newer versions)

https://discourse.practicalzfs.com/t/psa-raidz2-proxmox-efficiency-performance/1694

[-] cmnybo@discuss.tchncs.de 31 points 14 hours ago

Don't use btrfs if you need RAID 5 or 6.

The RAID56 feature provides striping and parity over several devices, same as the traditional RAID5/6. There are some implementation and design deficiencies that make it unreliable for some corner cases and the feature should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%.

https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices

[-] Anonymouse@lemmy.world 1 points 3 hours ago

I've got raid 6 at the base level and LVM for partitioning and ext4 filesystem for a k8s setup. Based on this, btrfs doesn't provide me with any advantages that I don't already have at a lower level.

Additionaly, for my system, btrfs uses more bits per file or something such that I was running out of disk space vs ext4. Yeah, I can go buy more disks, but I like to think that I'm running at peak efficiency, using all the bits, with no waste.

[-] Eideen@lemmy.world 2 points 4 hours ago

I have no problem running it with raid 5/6. The important thing is to have a UPS.

[-] dogma11@lemmy.world 1 points 3 hours ago

I've been running a btrfs storage array with data on raid5 and metadata I believe raid1 for the last 5 or so years and have yet to have a problem because of it. I did unfortunately learn not to fully trust the windows btrfs driver but was fortunately able to restore from backups and redownloading.

I wouldn't hesitate to set it up again for myself or anybody else, and adding a UPS would be icing on the cake. (I added UPS to my setup this last summer)

[-] lurklurk@lemmy.world 6 points 8 hours ago

Or run the raid 5 or 6 separately, with hardware raid or mdadm

Even for simple mirroring there's an argument to be made for running it separately from btrfs using mdadm. You do lose the benefit of btrfs being able to automatically pick the valid copy on localised corruption, but the admin tools are easier to use and more proven in a case of full disk failure, and if you run an encrypted block device you need to encrypt half as much stuff.

[-] avidamoeba@lemmy.ca 18 points 13 hours ago

You shouldn't have abysmal performance with ZFS. Something must be up.

[-] possiblylinux127@lemmy.zip 6 points 13 hours ago* (last edited 13 hours ago)

What's up is ZFS. It is solid but the architecture is very dated at this point.

There are about a hundred different settings I could try to change but at some point it is easier to go btrfs where it works out of the box.

[-] avidamoeba@lemmy.ca 1 points 2 hours ago

What seems dated in its architecture? Last time I looked at it, it struck me as pretty modern compared to what's in use today.

[-] possiblylinux127@lemmy.zip 2 points 1 hour ago* (last edited 1 hour ago)

It doesn't share well. Anytime anything IO heavy happens the system completely locks up.

That doesn't happen on other systems

[-] avidamoeba@lemmy.ca 1 points 1 hour ago* (last edited 1 hour ago)

That doesn't speak much of the architecture. Also it's really odd. Not denying what you're seeing is happening, just that it seems odd based on the setups I run with ZFS. My main server is in fact a shared machine that I use as a workstation and games along as a server. All works in parallel. I used to have a mirror, then a 4-disk RAIDz and now an 8-disk RAIDz2. I have multiple applications constantly using the pool. I don't notice any performance slowdowns on the desktop, or in-game when IO goes high. The only time I notice anything is when something like multiple Plex transcoders hit the CPU hard. Sequential performance is around 1.3GB/s which is limited by the data bus speeds (USB DAS boxes). Random performance is very good although I don't have any numbers out of my head. I'm using mostly WD Elements shucked disks and a couple of IronWolfs. No enterprise grade disks on this system.

I'm also not saying that you have to keep fucking around with it instead of going Btrfs. Simply adding another anecdote to the picture. If I had a serious problem like that and couldn't figure it out I'd be on LVMRAID+Ext4 which is what used prior to ZFS.

[-] possiblylinux127@lemmy.zip 1 points 1 hour ago

Yeah maybe my machines are cursed

[-] avidamoeba@lemmy.ca 1 points 1 hour ago

That is totally possible. I spent a month changing boards and CPUs to fix a curse on my main, unrelated to storage. In case you're curious.

[-] Andres4NY@social.ridetrans.it 0 points 1 hour ago
[-] avidamoeba@lemmy.ca 1 points 42 minutes ago

I feel like this one flew right over my head. 🥹

[-] prenatal_confusion@feddit.org 15 points 11 hours ago

Since most people with decently simple setups don't have the described problem likely somethings up with your setup.

Yes ifta old and yes it's complicated but it doesn't have to be to get a decent performance.

[-] avidamoeba@lemmy.ca 1 points 1 hour ago

I used to run a mirror for a while with WD USB disks. Didn't notice any performance problems. Used Ubuntu LTS which has a built-in ZFS module, not DKMS, although I doubt there's performance problems stemming from DKMS.

[-] possiblylinux127@lemmy.zip 1 points 2 hours ago

I have been trying to get ZFS working well for months. Also I am not the only one having issues as I have seen lots of other posts about similar problems.

[-] poVoq@slrpnk.net 1 points 7 hours ago

I am using btrfs on raid1 for a few years now and no major issue.

It's a bit annoying that a system with a degraded raid doesn't boot up without manual intervention though.

Also, not sure why but I recently broke a system installation on btrfs by taking out the drive and accessing it (and writing to it) from another PC via an USB adapter. But I guess that is not a common scenario.

[-] Bookmeat@lemmy.world 42 points 16 hours ago

A bit of topic; am I the only one that pronounces it "butterface"?

[-] prole 5 points 5 hours ago

Isn't it meant to be like "better FS"? So you're not too far off.

[-] Asparagus0098@sh.itjust.works 5 points 5 hours ago

i call it "butter FS"

[-] combatfrog@sopuli.xyz 2 points 6 hours ago

Similarly, I read bcachefs as BCA Chefs 😅

[-] wrekone@lemmy.dbzer0.com 42 points 15 hours ago
[-] myersguy@lemmy.simpl.website 30 points 15 hours ago

You son of a bitch, I'm in.

[-] uhmbah@lemmy.ca 17 points 15 hours ago

Ah feck. Not any more.

[-] vividspecter@lemm.ee 29 points 17 hours ago

No reason not to. Old reputations die hard, but it's been many many years since I've had an issue.

I like also that btrfs is a lot more flexible than ZFS which is pretty strict about the size and number of disks, whereas you can upgrade a btrfs array ad hoc.

I'll add to avoid RAID5/6 as that is still not considered safe, but you mentioned RAID1 which has no issues.

load more comments (3 replies)
[-] Moonrise2473@feddit.it 6 points 12 hours ago

One day I had a power outage and I wasn't able to mount the btrfs system disk anymore. I could mount it in another Linux but I wasn't able to boot from it anymore. I was very pissed, lost a whole day of work

[-] Philippe23@lemmy.ca 3 points 8 hours ago
[-] Moonrise2473@feddit.it 2 points 5 hours ago

I think 5 years ago, on Ubuntu

load more comments (1 replies)
[-] zarenki@lemmy.ml 8 points 14 hours ago

I've been using single-disk btrfs for my rootfs on every system for almost a decade. Great for snapshots while still being an in-tree driver. I also like being able to use subvolumes to treat / and /home (maybe others) similar to separate filesystems without actually being different partitions.

I had used it for my NAS array too, with btrfs raid1 (on top of luks), but migrated that over to ZFS a couple years ago because I wanted to get more usable storage space for the same money. btrfs raid5 is widely reported to be flawed and seemed to be in purgatory of never being fixed, so I moved to raidz1 instead.

One thing I miss is heterogenous arrays: with btrfs I can gradually upgrade my storage one disk at a time (without rewriting the filesystem) and it uses all of my space. For example, two 12TB drives, two 8TB drives, and one 4TB drive adds up to 44TB and raid1 cuts that in half to 22TB effective space. ZFS doesn't do that. Before I could migrate to ZFS I had to commit to buying a bunch of new drives (5x12TB not counting the backup array) so that every drive is the same size and I felt confident it would be enough space to last me a long time since growing it after the fact is a burden.

load more comments (2 replies)
[-] suzune@ani.social 9 points 14 hours ago

The question is how do you get a bad performance with ZFS?

I just tried to read a large file and it gave me uncached 280 MB/s from two mirrored HDDs.

The fourth run (obviously cached) gave me over 3.8 GB/s.

load more comments (10 replies)
load more comments
view more: next ›
this post was submitted on 23 Nov 2024
66 points (100.0% liked)

Selfhosted

40330 readers
339 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS