22
Downloading all archive.org metadata
(lemmy.dbzer0.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 09 Nov 2024
22 points (100.0% liked)
datahoarder
6736 readers
1 users here now
Who are we?
We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.
We are one. We are legion. And we're trying really hard not to forget.
-- 5-4-3-2-1-bang from this thread
founded 4 years ago
MODERATORS
I use the CLI tool, even right now waiting to finish some downloads. The CLI tool can actually give you a list of all items with
ia list {ID}(replace {ID} with the actual id of the stuff you want to download). But you don't even need to list the items, because you can download with a glob (in example*.torrentlike your shell has. Or if you have the ID anyway, you can specify the filenames too with{ID}_archive.torrentHere is an example how to do this with my own upload https://archive.org/details/gamefaqs_txt where the id becomes
gamefaqs_txtor use a variable to set id and download all files that start with the id, which should be all the meta data
Thank you for the tips. I am actually interested in enumerating metadata for all the "items" as defined by the API page ever uploaded. For example, one item = one ID:
You did cause me to look at the API docs again, though, and I think I found something that does enumerate all item names, and as a bonus, it will keep you updated when changes are made: https://archive.org/developers/changes.html
We'll see how much progress I can make. It might take a while to get through all the millions of them.
Isn't "item" and "id" basically the same thing? Because every item has a unique id. So in my example
gamefaqs_txtwould be the item and id.Yes, I think so. I'll definitely use the example for downloading some of the files (.torrent, metadata file) once I have some items. But first I need to find all the items ever uploaded.