66
Whats the best way to archive data long term while on a budget ?
(lemmings.world)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 15 Nov 2023
66 points (100.0% liked)
Asklemmy
43831 readers
775 users here now
A loosely moderated place to ask open-ended questions
If your post meets the following criteria, it's welcome here!
- Open-ended question
- Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
- Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
- Not ad nauseam inducing: please make sure it is a question that would be new to most members
- An actual topic of discussion
Looking for support?
Looking for a community?
- Lemmyverse: community search
- sub.rehab: maps old subreddits to fediverse options, marks official as such
- !lemmy411@lemmy.ca: a community for finding communities
~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~
founded 5 years ago
MODERATORS
Self hosting principals aside, is this data actually important? If so, then don't fuck around with self hosting it. Are you looking for lowest cost? Then don't waste a bunch of money spinning your own disks.
Amazon glacier to guarantee availability and your own encryption to guarantee privacy.
It's currently running me about $4/month for around 10tb that I don't want to lose but just don't want to deal with. An equivalent HDD solution would be around $500, that's 10 years to break even assuming zero disk failures and zero personal maintenance time.
Plus it's guaranteed. Inherent multiple copies, has SLA, and there's no worry about the service just disappearing. It's they decide to shut down or raise prices or whatever, you can reevaluate and move.
Edit: Glacier and similar services are meant for archival which is the term OP used. You never expect to need it again, but can't get rid of it. Retrieval cost is mostly irrelevant, but yes much more expensive. (I'd wager still less expensive than a home RAID array.)
What would it cost to retrieve though? You probably still have the appropriate cost-effective solution but it's an important consideration for newcomers to have complete math.
Retrieving from S3 glacier is approximately ~~10 times the monthly cost of storing the data~~ 100 times actually. Didn't realize retrieval from Glacier isn't actually downloading it onto your local, but rather just moving it into a frequent access tier S3 bucket from which you can then download, and this download is the expensive part.
OP said "archive", not "backup". Glacier is for days you need to keep but rarely touch.
How is it to get the data back?
Can I do it in real time so I could mount it as a media storage or would I need to rent one of the faster S3 tiers?
I think you can technically do it, but it's expensive to retrieve. But that isn't the point of an archive.
So instead of "fucking around" with putting it on a long lasting storage device to keep in a wardrobe, he should give up control of the data, hand it to a company and risk forgetting to inform them about an adress change, so everything is lost, when the bills arent paid?
How is that more secure?
Guess it depends on how much you trust that Amazon is going to steal your data instead of doing the thing you're paying them for, vs a house fire or media failure or whatever.
There's also pretty clear rules about unpaid bills, the data doesn't just vaporize.
This is what we call a "risk assessment", and imo if I must have that data available long-term, then a single copy on DVDs in a closet isn't good enough.
I'd argue for most consumer use cases having one or better two physical back ups is more reliable, because it is simple and straightforward. Also the risk mitigation is already in place, as you wouldn't want your place to burn down either way.
Are you sure it's $4/month and not $40/month? If so, which region is this in?
Us-East. Look specifically at glacier, which is long term, near free to store, expensive to remove.
Is it Glacier Deep Archive? I just realized I was looking at the Glacier flexible retrieval prices earlier. US-East lists it as $0.00099/GB (about $1/TB), which is still higher than what you're getting.
Last months bill for my entire Amazon account was $4.72. most of that was the glacier storage.
Move at how many hundred $ per TB?
Still less than an equivalent RAID array. Particularly if you consider that archives are very rarely extracted as a complete bulk, vs pulling the specific records needed.