journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.
It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).
If you're interested, here are my findings this time so far:
- workrave: 60 second stat sync https://github.com/rcaelers/workrave/pull/717
- kde klipper: saves to disk on every copy, even if permanent storage is disabled https://bugs.kde.org/show_bug.cgi?id=501030
- kde plasmashell: saves qt shader cache each time notification popup disappears https://bugs.kde.org/show_bug.cgi?id=523805
- bitwarden firefox extension: tries to connect to desktop application every 10 seconds, writes about every failure to browser's WebStorage 14+ KB https://github.com/bitwarden/clients/issues/22192
- firefox datareporting/glean: very chatty .mozilla/firefox/xxx/datareporting/glean/db/data.safe
- ipfs: writes every received DHT announce to disk, 20 GB in 3 hours https://discuss.ipfs.tech/t/constant-writes-to-datastore-log/20316
- mailcow: redis saves data every 5 minutes https://github.com/mailcow/mailcow-dockerized/pull/7405
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.
I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
journald is IMO the worst part of the systemd ecosystem. You're better off using it only as a router and not storing any logs in it. The indexing system it uses is slow and provides no control over chatty subsystems - you cannot truncate the logs for just a single identifier. For all the use indexing is doing you will get better performance out of a modern grep like ag or rg. Structure is worth something but it's better off somewhere other than journald.
How do you try to copy Windows NT's Event Log — which is essentially unchanged from the 1990s when systems ran on 32MB of RAM or less — and fail so spectacularly?
The first thing I do on a Linux system is install a proper syslog daemon.
I recently looked into disk usage of journald and was also shocked. My next step towards peace of mind is https://www.devuan.org/os/init-freedom
Will try it out as next distro for my Debian system, longtime experience with Void Linux (runit) on another box is great.
Many applications hammer the disk even if the developers don't believe this is an issue, not only journald, unfortunately.
It's my third attempt to make my regular Linux desktop less disk-chatty. This is a huge issue for btrfs and for COW FS in general, because they have massive write amplification for small and frequent writes (38,7 TB written to my idle desktop SSD in 2 years).
If you're interested, here are my findings this time so far:
The two I most often see in Ubuntu's dmesg are:
audit - appears to be some sort of AppArmor logging?
br[] - bridge interface docker uses consistently rebuilds itself? May be related to docker compose networking.
systemd-journald has one of the most deranged log file formats I have ever dealt with, and one of the worse user interfaces, too.
I am not again binary logs, or logs in a database. It's just yet another time I deal with good ideas implemented horribly, horribly badly when it comes to systemd.
Cool to see @ValdikSS here as well. The guy never sleeps or he is AI in disguise ;-)
Ooh, mmapped writes. I make that mistake once, years ago. :) I posted a comment in that GH issue.
https://github.com/systemd/systemd/issues/40262#issuecomment...
journald has never been of great quality. It somehow manages to be visibly slower than grepping gzipped text logs.
hello ValdikSS! nice to see you alive