Wholeheartedly agree. It's sad to slowly see the bitrot in my own bookmarks, trying to pull up important resources or media from only a few years ago only to hit a 404 or domain placeholder. And many times there is nothing on Archive.org or similar.
That ArchiveWeb extension looks excellent for custom captures. Picking the pages and elements you want during capture is a great feature.
For quicker one-offs, SingleFile gives you a single HTML with all the media of a page: https://www.getsinglefile.com
If you're making archiving a more involved part of your workflow, I highly recommend ArchiveBox. It can save pretty much every type of file from everywhere, in almost every format you can imagine, while building a searchable index of the archives. Been using it successfully for many years: https://archivebox.io
I used ArchiveBox for years. It's really good. The self-hosting and maintenance gets annoying after a while though.
Link rot really bothers me and I want people to get into archiving web pages so that eventually there will be a decentralized mirror of the unslopped web sitting on people's hard-drives, ready to share whenever someone needs it. So, I made a worse version of ArchiveBox ( https://memoryhole.app ). My philosophy is that, if people see how it is easy to start archiving and how useful it is, then they'll keep it up with SingleFile, or ArchiveBox or anything else.
SingleFile is great for single pages. All the tweaking I do is enable or disable saving scripts, and it works for >90% of webpages. Output being a single file (heh) simplifies things enormously, there’s no hierarchy or folders to deal with - hard to imagine your saved files not loading or breaking in some way as long as the original capture works.
I've been using Archivebox for years also, but it recent years it started to have multiple issues. My queues are full of pages that failed to scan, and it's been several versions since I actually had anything new there.
I did small amount of troubleshooting a few times with no resolution, but now I'm thinking of switching my thought process. Maybe something like a "web clipper" is much better, as complex SPA pages often fail to archive well anyway. And I should focus on content itself, not the page. That SingleFile extension fits well here as well.
It's funny - I actually went through a similar transition and also realized it was a little too heavy-duty for my personal workflow.
At this point I'm using Linkding (self-hosted bookmarks, which has an archive function) paired with Singlefile (which integrates with Linkding) which locally captures and uploads. For larger, or multi-page, captures I still fire off an Archivebox command. I run into blocks occasionally, but it also just feels like the open web is becoming less and less open in general.
Archivebox annoys me because (as far as I remember) it doesn't have any options for capturing a webpage on your end and uploading it to your self-hosted Archivebox instance. With numerous sites moving to Cloudflare to mitigate AI agent crawling, it makes it really hard (or outright impossible) to capture a page because it wants to do it server side.
This is basically the main reason that I started building https://github.com/mfinelli/recueil. It _only_ works in your browser so you can bypass captchas or paywalls or whatever and then it uses singlefile (https://www.getsinglefile.com) to upload the result to your self-hosted archive.
I learned this the hard way by getting too casual with Angelfire keeping an old website running for 20 years after I stopped updating it.
It had lots of photos that I hand-scanned from printed film, back in the day when it took a good bit of time to scan photos manually.
The site was a running joke, where I would open it to show people that it still existed... until it didn't, and I couldn't find my backup of all of the photos. The archive sites have the 'bones' of the website, but not the photos. It was easily avoidable, which is why it is painful.
In 5 years, cloud LLMs may have largely replaced the open Internet as a source of information, with centralized control and ability to rewrite the past.
Please, please buy books and keep them in good condition, and please archive websites with important information (assuming it's legal in your jurisdiction). There are still plenty of excellent sites and blogs around from the 2000s and even earlier, but they won't be there forever. Ironically, coding agents have made archival much easier -- I've always found problems with using pre-written tools, but Claude or Qwen make it trivial to download, test, and archive sites on your local.
Bonus suggestion: get that internet archive extension for your browser that automatically sends every page you visit to the IA (if you're ok with the privacy side).
You don't have to worry about having your copy easily accessible, and as a bonus it's also available for everyone else in the world.
This is why I started using Obsidian Web Clipper. There is SOME work to learn it and set it up, but the small amount of upfront work will pay for years to come. And the folder itself is backed by Dropbox, so I don't have to worry about losing it all forever.
I also regularly use the web clipper specifically for this reason - most of my conversations can be around articles I read online, and instead of googling for the article I have it referenced right in my journal. I find myself regularly upset that I didn't save old articles from years ago because I didn't have this system
I recently spent a few days trying to figure out and discover my old websites and things from the early 2000s on the web archive. I hadn't realised my memory had gone quite so bad, but it was revitalising to find some of what I'd forgotten. So much of what I do remember is long dead and missing, sadly.
I thought it might be rose-tinted glasses, thinking that the earlier internet was better, but spending time retracing my teenage steps made me even more certain that it was indeed better. So many derpy little websites made by nerds, so many fun games and websites that weren't preying on vulnerable people, and everything spread out across many places - not centralised in the hands of large corporations. I've been using LLMs to have fun recreating concepts from that time, like my own forum software, as some sort of anti-progress. It's been fun.
I did actually. I took a whole day to go to each of my individual saved bookmarks and save the articles as pdf using edge reader mode, sometimes hunting down articles on the way back machine
the last source of truth are forums.. but not reddit... if it has a CEO it will rot and enshitify.
I'm talking about forums... the bmw forum for your old car... the motorcycle forum for the enthusiast of the SV650... those places are full of anecdotes, people talking, lending a hand to another person... being humans and care...
Which is what social media is/should/was about, is what internet is about before the enshitification happened... I would vote to have this ID associated to you via check of your local goverment issued id and ONLY UPPON VERIFICATION of you being using that you can log and read and use those websites without tracking you (Yes 10,000 ifs where this is secure, handled in a confidential matter, the whole tech is not to track that you are doing something... but that a human is behind that, no api, no robot, no adds, no nothing... just the old school forum where you can go, talk, submit a photo... where someone who wanted to spend time to write something did it, SO WE CAN ENSURE TWO THINGS, A HUMAN IS THERE, AND NO FVK!NG CEO WILL ENSHITIFY IT, no to track people, but to know that a person is interacting there)
But because "empathy is the weakness" and shareholder value...
There are far better tools that are not built by this person’s company. Dig around and you’ll find an incredible array. Not everything has to be a sales pitch.
Yes, and with AI you can even back up the logic behind a website (the back-end), and even generate fake but plausible data to fill it if crawling is too much work.
Ages ago (like 2001 or so?) I started building something to automatically archive everything I browse. I used tcpflow (https://www.circlemud.org/jelson/software/tcpflow/). I've seen a fair bit of varions of this idea over the years
Then SSL/TLS started getting widespread and I lost focus. Nowadays, I guess you would build it as an extension in the browser.
i'm saddened that http://macsecret.com/ along with /files directory seems to have all be been lost. I has waiting on some storage for the NAS to archive it and it went away before I could back it up.
Have a look at Hister [1], a 'personal web history search engine' which by way of extensions for browsers captures your web history and stores actual page content on your own instance where it can be browsed and searched. It does not store images so if that's what you're mostly interested in it won't work for you. For text is works quite well, maybe sometimes too well since it also stores ephemeral things like HN comment pages unless you tell it not to.
Wholeheartedly agree. It's sad to slowly see the bitrot in my own bookmarks, trying to pull up important resources or media from only a few years ago only to hit a 404 or domain placeholder. And many times there is nothing on Archive.org or similar.
That ArchiveWeb extension looks excellent for custom captures. Picking the pages and elements you want during capture is a great feature.
For quicker one-offs, SingleFile gives you a single HTML with all the media of a page: https://www.getsinglefile.com
If you're making archiving a more involved part of your workflow, I highly recommend ArchiveBox. It can save pretty much every type of file from everywhere, in almost every format you can imagine, while building a searchable index of the archives. Been using it successfully for many years: https://archivebox.io
I used ArchiveBox for years. It's really good. The self-hosting and maintenance gets annoying after a while though.
Link rot really bothers me and I want people to get into archiving web pages so that eventually there will be a decentralized mirror of the unslopped web sitting on people's hard-drives, ready to share whenever someone needs it. So, I made a worse version of ArchiveBox ( https://memoryhole.app ). My philosophy is that, if people see how it is easy to start archiving and how useful it is, then they'll keep it up with SingleFile, or ArchiveBox or anything else.
I'm using Linkding hooked up with SingleFile to bookmark+archive all important pages. Highly recommend! https://linkding.link/archiving/#using-the-singlefile-browse...
SingleFile is great for single pages. All the tweaking I do is enable or disable saving scripts, and it works for >90% of webpages. Output being a single file (heh) simplifies things enormously, there’s no hierarchy or folders to deal with - hard to imagine your saved files not loading or breaking in some way as long as the original capture works.
SingleFile has a nice batch download feature as well. It's a little rough around the edges, but useful when needed.
I've been using Archivebox for years also, but it recent years it started to have multiple issues. My queues are full of pages that failed to scan, and it's been several versions since I actually had anything new there.
I did small amount of troubleshooting a few times with no resolution, but now I'm thinking of switching my thought process. Maybe something like a "web clipper" is much better, as complex SPA pages often fail to archive well anyway. And I should focus on content itself, not the page. That SingleFile extension fits well here as well.
Did you run into similar dilemmas at some point?
It's funny - I actually went through a similar transition and also realized it was a little too heavy-duty for my personal workflow.
At this point I'm using Linkding (self-hosted bookmarks, which has an archive function) paired with Singlefile (which integrates with Linkding) which locally captures and uploads. For larger, or multi-page, captures I still fire off an Archivebox command. I run into blocks occasionally, but it also just feels like the open web is becoming less and less open in general.
Archivebox annoys me because (as far as I remember) it doesn't have any options for capturing a webpage on your end and uploading it to your self-hosted Archivebox instance. With numerous sites moving to Cloudflare to mitigate AI agent crawling, it makes it really hard (or outright impossible) to capture a page because it wants to do it server side.
This is basically the main reason that I started building https://github.com/mfinelli/recueil. It _only_ works in your browser so you can bypass captchas or paywalls or whatever and then it uses singlefile (https://www.getsinglefile.com) to upload the result to your self-hosted archive.
I learned this the hard way by getting too casual with Angelfire keeping an old website running for 20 years after I stopped updating it.
It had lots of photos that I hand-scanned from printed film, back in the day when it took a good bit of time to scan photos manually.
The site was a running joke, where I would open it to show people that it still existed... until it didn't, and I couldn't find my backup of all of the photos. The archive sites have the 'bones' of the website, but not the photos. It was easily avoidable, which is why it is painful.
In 5 years, cloud LLMs may have largely replaced the open Internet as a source of information, with centralized control and ability to rewrite the past.
Please, please buy books and keep them in good condition, and please archive websites with important information (assuming it's legal in your jurisdiction). There are still plenty of excellent sites and blogs around from the 2000s and even earlier, but they won't be there forever. Ironically, coding agents have made archival much easier -- I've always found problems with using pre-written tools, but Claude or Qwen make it trivial to download, test, and archive sites on your local.
This is hyperbole. Do you really think an LLM will try to rewrite the past? Come on…
It's the premise of Terminator, after all.
[delayed]
Bonus suggestion: get that internet archive extension for your browser that automatically sends every page you visit to the IA (if you're ok with the privacy side).
You don't have to worry about having your copy easily accessible, and as a bonus it's also available for everyone else in the world.
This is why I started using Obsidian Web Clipper. There is SOME work to learn it and set it up, but the small amount of upfront work will pay for years to come. And the folder itself is backed by Dropbox, so I don't have to worry about losing it all forever.
I also regularly use the web clipper specifically for this reason - most of my conversations can be around articles I read online, and instead of googling for the article I have it referenced right in my journal. I find myself regularly upset that I didn't save old articles from years ago because I didn't have this system
I recently spent a few days trying to figure out and discover my old websites and things from the early 2000s on the web archive. I hadn't realised my memory had gone quite so bad, but it was revitalising to find some of what I'd forgotten. So much of what I do remember is long dead and missing, sadly.
I thought it might be rose-tinted glasses, thinking that the earlier internet was better, but spending time retracing my teenage steps made me even more certain that it was indeed better. So many derpy little websites made by nerds, so many fun games and websites that weren't preying on vulnerable people, and everything spread out across many places - not centralised in the hands of large corporations. I've been using LLMs to have fun recreating concepts from that time, like my own forum software, as some sort of anti-progress. It's been fun.
one of the great tragedies of my life was when https://buthowdoitknow.com/ went offline, taking the visual emulator with it
i managed to reconstruct (and then improve) it with an LLM:
https://scott-cpu.cs.montana.edu/
I did actually. I took a whole day to go to each of my individual saved bookmarks and save the articles as pdf using edge reader mode, sometimes hunting down articles on the way back machine
I tend to save things I read as PDFs because it feels the most robust archival formal for the future.
the last source of truth are forums.. but not reddit... if it has a CEO it will rot and enshitify.
I'm talking about forums... the bmw forum for your old car... the motorcycle forum for the enthusiast of the SV650... those places are full of anecdotes, people talking, lending a hand to another person... being humans and care...
Which is what social media is/should/was about, is what internet is about before the enshitification happened... I would vote to have this ID associated to you via check of your local goverment issued id and ONLY UPPON VERIFICATION of you being using that you can log and read and use those websites without tracking you (Yes 10,000 ifs where this is secure, handled in a confidential matter, the whole tech is not to track that you are doing something... but that a human is behind that, no api, no robot, no adds, no nothing... just the old school forum where you can go, talk, submit a photo... where someone who wanted to spend time to write something did it, SO WE CAN ENSURE TWO THINGS, A HUMAN IS THERE, AND NO FVK!NG CEO WILL ENSHITIFY IT, no to track people, but to know that a person is interacting there)
But because "empathy is the weakness" and shareholder value...
Thanks for the guide ,I never new about this
Same goes for youtube vids
Over the summer, I had the urge to do this with a few favorite blogs. wget did the trick for static content
There are far better tools that are not built by this person’s company. Dig around and you’ll find an incredible array. Not everything has to be a sales pitch.
Yes, and with AI you can even back up the logic behind a website (the back-end), and even generate fake but plausible data to fill it if crawling is too much work.
Another great writeup about backing up websites: https://gwern.net/archiving
Ages ago (like 2001 or so?) I started building something to automatically archive everything I browse. I used tcpflow (https://www.circlemud.org/jelson/software/tcpflow/). I've seen a fair bit of varions of this idea over the years
Then SSL/TLS started getting widespread and I lost focus. Nowadays, I guess you would build it as an extension in the browser.
i'm saddened that http://macsecret.com/ along with /files directory seems to have all be been lost. I has waiting on some storage for the NAS to archive it and it went away before I could back it up.
Have a look at Hister [1], a 'personal web history search engine' which by way of extensions for browsers captures your web history and stores actual page content on your own instance where it can be browsed and searched. It does not store images so if that's what you're mostly interested in it won't work for you. For text is works quite well, maybe sometimes too well since it also stores ephemeral things like HN comment pages unless you tell it not to.
[1] https://github.com/asciimoo/hister/
https://hister.org/ is a neat website indexing and archival tool.