Centralized Logging
No stranger to controversy, the systemd system and service manager introduced a significant change in the way system logs are gathered. Logs used to be located at different places in the file system according to the service or daemon that was creating them. But they all had one thing in common. They were plain text files.
With systemd all the system, boot, and kernel log files are collected and managed by a central, dedicated logging solution. The format they are stored in is a binary one. One thing this facilitates is being able to extract the data in different formats, such as JSON, as we shall see.
it can also make it easier to cross-reference related information that would have previously been recorded in separate log files. Because the data is now held in a single journal, the data from several sources of interest can be selected and displayed in a single interwoven list of entries.
journalctl is the tool used to work with the journal.
journalctl With No Frills
You can invoke journalctl with no command line parameters:
journalctl displays the entire journal, with the oldest entries at the top of the list. The list is displayed in less, allowing you to page and search using the usual navigation features of less. You can also use the Left Arrow and Right Arrow keys to scroll sideways to read wide log entries.
Pressing the End key will hop straight to the bottom of the list, and the newest log entries.
Press Ctrl+C to exit.
RELATED: How to Use the less Command on Linux
Although journalctl can be called without using sudo, you will ensure you see all the detail within the log if you do use sudo.
If you need to, you can make journalctl send its output to the terminal window instead of to less, by using the –no-pager option.
The output scrolls quickly through the terminal window, and you are returned to the command prompt.
To limit the number of lines that journalctl returns, use the -n (lines) option. Let’s ask for ten lines of output:
Following Journal Updates
To make journalctl display the newest entries as they arrive in the journal, use the -f (follow) option.
The newest entry has a timestamp of 07:09:07. As new activity takes place, the new entries are appended to the bottom of the display. Near real-time updates—cool!
At 07:09:59 an application called geek-app injected a log entry into the journal that said, “New Message from HTG.”
Changing the Display Format
Because the journal is a binary file, the data in it needs to be translated or parsed into text before it can be displayed to you. With different parsers, different output formats can be created from the same binary source data. There are several different formats that journalctl can use.
The default output is the short format, which is very similar to the classic system log format. To explicitly request the short format, use the -o (output) option with the short modifier.
From left to right, the fields are:
The time the message was created, in local time. The hostname. The process name. This is the process that generated the message. The log message.
To obtain a complete date and time stamp use the short-full modifier:
The date and time formats in this output are the format in which you need to provide dates and times when you are selecting log messages by period, as we shall see shortly.
To see all the metadata that accompanies each log message, use the verbose modifier.
There are many possible fields, but it is rare for all fields to be present in a message.
One field worth discussing is the Priority field. In this example, it has a value of 6. The value represents the importance of the message:
0: Emergency. The system is unusable. 1: Alert. A condition has been flagged that should be corrected immediately. 2: Critical. This covers crashes, coredumps, and significant failures in primary applications. 3: Error. An error has been reported, but it is not considered severe. 4: Warning. Brings a condition to your attention that, if ignored, may become an error. 5: Notice. Used to report events that are unusual, but not errors. 6: Information. Regular operational messages. These do not require action. 7: Debug. Messages put into applications to make it easier for them to debug them.
If you want the output to be presented as properly formed JavaScript Object Notation (JSON) objects, use the json modifier:
Each message is properly wrapped as a well-formed JSON object, and displayed one message per line of output.
To have the JSON output pretty-printed, use the json-pretty modifier.
Each JSON object is split across multiple lines, with each name-value pair on a new line.
To only see the log entry messages, without time stamps or other metadata, use the cat modifier:
This display format can make it difficult to identify which process raised the log event, although some messages do contain a clue.
Selecting Log Messages By Time Period
To limit the output from journalctl to a time period you’re interested in, use the -S (since) and -U (until) options.
To see the log entries since a particular time and date, use this command:
The display contains only messages that arrived after the date and time in the command.
To define a time period you wish to report on, use both the -S (since) and -U (until) options together. This command looks at log messages from a 15 minute time period.:
This is a great combination use if you know something odd happened on your system, and roughly when it happened.
Using Relative Time Periods
You can use relative addressing when you select your time periods. That means you can say things like “show me all events from one day ago up until now.” This is just what this command means. The “d” stands for “day”, and the “-1” means one day in the past.
The log messages are listed from 00:00:00 yesterday, up until “now.”
If you want to investigate something that happened in the recent past, you can specify a relative time period measured in hours. Here we’re reviewing log messages from the last hour:
The messages from the last hour are displayed for you. You can also use “m” to set relative time periods measured in minutes, and “w” for weeks.
journalctl understands today, yesterday, and tomorrow. These modifiers provide a handy way to specify common time periods. To see all the events that happened yesterday, use this command:
All journal log events that happened yesterday, up to midnight 00:00:00, are retrieved and displayed for you.
To see all the log messages received today so far, use this command:
Everything from 00:00:00 up until the time the command is issued, are displayed.
You’re able to mix the different time period modifiers. To see everything from two days ago up until the start of today, use this command:
Everything since the day before yesterday up until today is retrieved and displayed.
Selecting Log Messages By Data Fields
You can search for log messages that match a wide range of journal fields. These searches try to find matches in the metadata attached to each message. It is recommended that you refer to the list of fields and choose the ones that will be most useful to you.
Bear in mind, whether an application completes every field or not is entirely up to the authors of the application. You can’t guarantee every field will be populated.
All of the journal field modifiers are used in the same way. We’ll use a few in our examples below. To look for log messages from a specific application, use the _COMM (command) modifier. If you also use the -f (follow) option, journalctl will track new messages from this application as they arrive.
You can search for log entries using the process ID of the process that generated the log message. Use the ps command to find the process id of the daemon or application you’re going to search for.
On the machine used to research this article, the SSH daemon is process 751.
You can also search by user Id. This is the user ID fo the person who launched the application or command, or who owns the process.
All messages associated with any other user ID’s are filtered out. Only messages related to user 1000 are shown:
Another way to search for log messages related to a specific application is to provide the path to the executable.
All of the anacron scheduler log messages are retrieved and displayed.
To make searching easier, we can ask journalctl to list all the values it holds, for any of the journal fields.
To see the user ID’s that journalctl has recorded log messages for, use the -F (fields) option, and pass the _UID field identifier.
Let’s do that again and look at the group IDs (GID’s):
You can do this with any of the journal field identifiers.
Listing Kernel Messages
There’s a built-in way to isolate kernel messages quickly. You don’t need to search and isolate them yourself. The -k (kernel) option removes all other messages and gives you an instant view of the kernel log entries.
The highlighting reflects the importance of the message, according to the values in the Priority field.
Reviewing Boot Messages
If you’ve got an issue related to booting that you wish to investigate, journalctl has you covered. Perhaps you’ve added new hardware, and it isn’t responding, or a previously working hardware component no longer works after your last system upgrade.
To see the log entries related to your last boot, use the -b (boot) option:
The log entries for the last boot are shown for you.
When we say “last boot,” we mean the boot process that brought your computer to life for your current logged-in session. To see previous boots, you can use a number to tell journalctl which boot you’re interested in. To see the third previous boot, use this command:
Generally, if you’ve had a problem and had to reboot your machine, it is a previous boot sequence you’re interested in. So this is a common command form.
It is easy to get mixed up with the sequence of boots. To help, we can ask journalctl to list the boots that it has recorded in its journal, using the –list-boots option.
You can identify the boot you wish to see messages for from the date and time stamp, and then use the number in the left-hand column to obtain the log messages for that boot sequence. You can also pick the 32-bit boot identifier, and pass that to journalctl.
The log messages from the boot sequence we requested are retrieved and displayed.
Managing Journal Hard Drive Space
Of course, the journal and all of its log messages are stored on your hard drive. That means they’ll be taking up hard drive space. To see how much space has been taken by the journal, use the –disk-usage option.
With today’s hard drives, 152 MB isn’t much space at all, but for demonstration purposes, we’ll still trim it back. There’s two way we can do this. The first is to set a size limit that you want the journal reduced back to. It’ll grow again, of course, but we can prune it now ready for that new growth.
We’ll use the wonderfully titled –vacuum-size option, and pass in the size we’d like the journal reduced to. We’ll ask for 100 MB. The way to think of this is we’re asking journalctl to “throw away whatever you can, but don’t go lower than 100 MB.”
The other way to trim back the journal size is to use the –vacuum-time option. This option tells journalctl to discard messages that are older than the period you provide on the command line. You can use days, weeks, months, and years in the time period.
Let’s weed out all messages older than one week:
Data vs. Information
Data isn’t useful unless you can get at it and make use of it. Then it becomes useful information. The journalctl command is a flexible and sophisticated tool that allows you to get to the information of interest in a variety of ways.
You can use just about any snippet of information you have to home in on the log messages you need.
