Centos7 log learning -- Linux system / var/log/journal / garbage log cleaning

There are two log services in CentOS system: traditional rsyslog and SYSTEMd journal

SYSTEMd journal is an improved log management service, which can collect logs from the kernel, the early startup phase of the system, the standard output and error information of the system daemon during startup and operation, and the logs of syslog.

The log service only saves the logs in a single structure log file / run/log. Because the logs are binary data that has been compressed and formatted, they can be viewed and located quickly.

By default, the log will not be persisted and saved, and only one month's log will be retained. In addition, some logs that rsyslog cannot collect will also be recorded by journal.

As a traditional system log service, rsyslog records all the collected logs into various log files in the / var/log / directory.

Common log files are as follows:

/var/log/messages most of the system logs are recorded in this file
/var/log/secure all logs related to security, authentication and authorization will be recorded in this file
/var/log/maillog log log of mail service
/var/log/cron crond log of scheduled tasks
/var/log/boot.log relevant logs of system startup

Someone once said: Linux has no garbage files, Windows has garbage files. In fact, this is not the case. Both will have garbage files.

An operating system is a system that operates various files. It cannot be without log files, let alone without generating temporary files. Just like paper cutting, it is natural to generate some waste materials temporarily.

Whether Linux has garbage files that occupy space depends on how to judge. For example, log files and system files several years and months ago are basically useless. Are they garbage files?

# ls -lhm --full-time /var/log/journal/f9d400c5e1e8c3a8209e990d887d4ac1_bk_20190122/ | sort -k6 | head -n30
total 3.5G
-rw-r-x---+ 1 root systemd-journal  64M 2018-03-28 01:36:01.010275802 +0800 system@00000000000000000000000000000000-00000000000002ca-000567d28f35cca7.journal
-rw-r-x---+ 1 root systemd-journal 8.0M 2018-03-28 01:36:01.100275802 +0800 user-65534@4ee96b2fbd8b4a82beee0402402fee03-0000000000005419-000567f7fd08bd2f.journal
-rw-r-x---+ 1 root systemd-journal  72M 2018-04-02 19:16:41.644934707 +0800 system@00000000000000000000000000000000-0000000000010a47-0005686852f561be.journal
-rw-r-x---+ 1 root systemd-journal 8.0M 2018-04-02 19:16:41.714934707 +0800 user-65534@4ee96b2fbd8b4a82beee0402402fee03-0000000000011f98-00056872cab77761.journal
-rw-r-x---+ 1 root systemd-journal  72M 2018-04-08 05:48:01.673026304 +0800 system@00000000000000000000000000000000-0000000000021c40-000568dbb97116ae.journal
-rw-r-x---+ 1 root systemd-journal  72M 2018-04-13 18:25:01.967846109 +0800 system@00000000000000000000000000000000-0000000000033800-00056949207ae8a1.journal
-rw-r-x---+ 1 root systemd-journal  72M 2018-04-18 04:12:35.385621922 +0800 system@00000000000000000000000000000000-0000000000045c3e-000569b848f6f86c.journal

How to view garbage files

Before cleaning, it was found that the space of the hard disk root partition was urgent. Check it with du -t 100M /var or journalctl -- disk usage command. It was found that the / var/log/journal log files occupied nearly 3G space, and the volume of each log file was as high as 8-128M. These log files recorded the systemd situation for a long time, which was worthless. After cleaning it with journalctl -- Vacuum size = 10m command, 2.7G space was vacated. Use the df command to check that the / root partition is really spacious.

View and sort the file size of a directory in MB

# du -hm --max-depth=1 /var/ | sort -n
1       /var/adm
1       /var/crash
1       /var/db
1       /var/empty
1       /var/games
1       /var/gopher
1       /var/kerberos
1       /var/local
1       /var/nis
1       /var/opt
1       /var/preserve
1       /var/spool
1       /var/tmp
1       /var/yp
131     /var/www
198     /var/lib
486     /var/cache
3695    /var/log
8513    /var/

Method of emptying / var/log/journal file

1. Use the echo command to redirect the empty string content to the specified file

echo "" > system.journal

Note: this method will only empty once. It is troublesome to empty manually again after a period of time. Here, you can use the following command to make journalctl automatically maintain the space

2. The journalctl command automatically maintains the file size

1) Only keep the log for nearly a week

journalctl --vacuum-time=1w

2) Keep only 500MB logs

journalctl --vacuum-size=500M

3) Directly delete the log files in the /var/log/journal/ directory

rm -rf /var/log/journal/f9d400c5e1e8c3a8209e990d887d4ac1

Problem analysis and solution

Error was encoded while opening journal files: input / output error

# journalctl --vacuum-time=1w
Error was encountered while opening journal files: Input/output error

Problem analysis: log file corruption

Solution: delete the previous log and restart the journalctl service

mv journal/f9d400c5e1e8c3a8209e990d887d4ac1 journal/f9d400c5e1e8c3a8209e990d887d4ac1_bk_20190122
systemctl restart systemd-journald.service

View the / var/log/journal / log directory as follows:

# ll /var/log/journal/
drwxr-sr-x  2 root systemd-journal  4096 Jan 22 11:26 f9d400c5e1e8c3a8209e990d887d4ac1
drwxr-sr-x+ 2 root systemd-journal 12288 Jan 14 15:37 f9d400c5e1e8c3a8209e990d887d4ac1_bk_20190122

Then, execute the command of journalctl to restrict logs:

# journalctl --vacuum-time=1w
Vacuuming done, freed 0B of archived journals on disk.

# journalctl --vacuum-size=500M
Vacuuming done, freed 0B of archived journals on disk.

Tags: Linux

Posted by ErnesTo on Sun, 17 Apr 2022 11:36:39 +0930