6
Tools and methods to save a webpage from the CLI (preferrably not single-file-cli) (lemmy.sdf.org)
submitted 1 week ago* (last edited 1 week ago) by evenwicht@lemmy.sdf.org to c/text_ui@lemmy.sdf.org
11 comments fedilink hide all child comments

I could really use a commandline mechanism for saving a webpage locally for offline viewing later. Saving the page in a single file is ideal.

“singlefile”, the browser extension

For GUI users there is a quite useful extension (single-file). That can be run from the CLI by installing single-file-cli. But I am not enthusiastic about the bloaty option. You must first install some obscure package manager called “Deno”, which does not exist in the official Debian repositories (thus has not proven to be worthy for mainstream public use, and also complicates every Debian upgrade). Then it uses JavaScript to instrument a bloaty GUI browser. Not great.

Worth noting that there is Org-mode integration but this does not get around the the non-Debian dependency on Deno.

wget

A page can be fetched using this command:

$ wget -U 'stopDiscriminatingAgainstWg3t' -c -P "$dir" -E -H -k -K --xattr -p "$url"
$ printf '%s' "$url" > "$dir"/url.txt

The printf is needed because wget just dumps a big tree of files with no indication of which file must be opened to render the whole page. Indeed it sucks. Saving the URL gives us a fighting chance at locating the file but it’s still a pain. And from there, the files are still scattered on your hard drive.

There is a non-Debian tool called HTMLArk. Again, I am not thrilled about non-Debian stuff. But it claims to be able to produce a single HTML file from a scattered collection of files. I have not tried it but in principle it can be used to tidy up the mess dumped by wget.

ArchiveBox and Grunt-inline

ArchiveBox looks interesting. But it’s non-Debian so not exactly spot-on. Same issue with Grunt-inline. There is some grunt stuff in the Debian repos but not this tool specifically.

webpages2html

This is a python script but the script and its dependencies are non-debian. Requires using pip which is a disaster of a tool that is barely suitable for developers but hardly suitable for the end users who end up getting pushed into it b/c there is no proper pkg manager.

What else?

Any decent option I have overlooked?

top 11 comments
sorted by: hot top controversial new old
[-] pmjv@lemmy.sdf.org 2 points 1 week ago

do you care about it looking like the webpage, or do you just want the text ala reader-mode?

[-] evenwicht@lemmy.sdf.org 1 points 1 week ago

Probably rare that I would opt for a lossy text version, but I’ll consider it in some cases along the same lines as my response to the w3m suggestion.

[-] Freyi@lemmy.sdf.org 2 points 1 week ago

I put links to pages I want to read in a local HTML file and run PyPlucker on it, which scrapes up to a specified depth (usually 1). Then I send the resulting PDB file to my old Palm T3 over Bluetooth and read it all offline. There's a desktop viewer but I haven't tested that yet.

Before that I used to use a Chromium extension "EpubPress" that dumps all open tabs into an EPUB, but that relied on a 3rd party service and no longer works. There's a project called EpubPressX which tries to do it fully local but it didn't work for me. There's also offpunk which is a text-only local CLI offline reader.

[-] mp3@lemmy.ca 1 points 1 week ago

kage

[-] evenwicht@lemmy.sdf.org 1 points 1 week ago

Looks great! It’s a shame it’s not in the Debian official repos. I wonder why. I have to say I’m tempted to try it because at least there is an unofficial Debian pkg so we need not use an obscure or language-specific pkg manager.

[-] Zachariah@lemmy.world 1 points 1 week ago

https://www.httrack.com/

Command-Line Guide
https://www.httrack.com/html/cmdguide.html

[-] evenwicht@lemmy.sdf.org 1 points 1 week ago* (last edited 6 days ago)

Glad to see it’s in the official Debian repos. That encourages me to try it and fiddle around with it.

The landing page says: “It allows you to download a World Wide Web site from the Internet”, which gives the feeling of a 1990s project. Which isn’t bad.. It suggests the dev is probably not keen to use silly modern day bloaty garbage like JS, npm, electron, python, and awkward pkg managers. And indeed it’s in C.

Although I had to wonder how it notably differs from wget -c -E -H -k -K … Then I saw the --single-file option. So that pretty much settles it -- I must check it out. This may be the perfect tool. Thanks for the tip!

(update) I installed it. Looks promising.

(update 2) Superficially i love the design.. the way it has a server and remembers sessions. A lot of thought went into the design. But I can’t get it to work. There is a config for “proxy”; does not say whether it’s an HTTP proxy or SOCKS. It does not indicate if an URL is expected or just a hostname. My proxy setting does not stick. And no data is fetched in the end. Hard to know if respect for robots.txt is hindering it.

[-] dadarobot@lemmy.ml 1 points 1 week ago

w3m -dump

[-] evenwicht@lemmy.sdf.org 1 points 1 week ago* (last edited 6 days ago)

I usually want to archive a page that can be faithfully reproduced, sometimes for borderline forensic purposes. I usually don’t have time during a browsing session to inspect the results of a saved page for quality. If I pop into a public hotspot I want to save and go.

But later on when I’m reading and managing the offline content, I might decide the graphics on a page are unwanted. So perhaps it would be useful to do w3m -dump "$my_local_GUI_page.html" > lean_capture.txt && rm $my_local_GUI_page.html". I’ll have to try that.

(update) I works to run w3m -dump "$my_local_GUI_page.html" > lean_capture.txt but strangely the file:// scheme is not understood. E.g. this does not work: w3m -dump file://./"$my_local_GUI_page.html".

(update 2) If the webpage is MIME-formatted, w3m does not handle it and just dumps the base64 blobs.

[-] graycube@lemmy.world 0 points 1 week ago

curl ?

[-] evenwicht@lemmy.sdf.org 1 points 1 week ago

I did not think that curl could interpret HTML and fetch all elements needed to render a webpage. What would be a sample commandline for curl?

this post was submitted on 13 Aug 2026
6 points (100.0% liked)

Text-Based User Interfaces (TUI; CLI) 💻🖮

123 readers
1 users here now

Forum for advanced users who grok the power of text-based apps, the advantage of tmux/GNU screen, the keyboard and who often find the mouse a hinderance to a fast workflow. A text-based UI is also a decent escape from enshitified resources.

This forum broadly covers tools, hacks, and advocacy of text-based environments.

Slightly marginally kind of related:

founded 2 years ago
MODERATORS
evenwicht@lemmy.sdf.org