I was watching Chris Jenks' Intro to Linux system hardening talk from Derbycon 2012 yesterday, when he mentioned Logcheck as an additional tool that can help you harden your Linux system. Technically Logcheck isn't going to protect you against attacks, but it will be one of the first to notify you when something goes wrong. I have been meaning to play with it for some time, so I quickly set it up on a test server.
How Logcheck Protects Your Server
It turns out that Logcheck is a pretty simple tool. All it does is periodically check the log files you specify. It filters out the uninteresting parts and emails you what is left. It is smart enough to avoid mailing you the same information multiple times by keeping a "bookmark" for each log file. This way it can only email you newer entries -- those have been written after the last log check.
The really valuable part in the Logcheck distribution is its rule database that distinguishes between the interesting and uninteresting log entries. On Ubuntu it is provided in a separate package:
logcheck-database and it contains rules for many daemons you could have running on your system: cron, bind, postfix, and whatnot. I like that a lot and I can think about several projects where I can install and use the database in a separate program that analyzes log entries and sends out alerts.
Installing Logcheck on the Server
I'm using Ubuntu and I think all Debian-derived machines have this pretty simple:
sudo apt-get install logcheck
The above will make sure you have a mail server installed too since Logcheck needs a way to send those log summaries to you.
A piece of cake, really. At minimum, you need to specify your admin email address and the report level in the
# Controls the level of filtering: # Can be Set to "workstation", "server" or "paranoid" for different # levels of filtering. Defaults to server if not set. REPORTLEVEL="server" # Controls the address mail goes to: # *NOTE* the script does not set a default value for this variable! # Should be set to an offsite "firstname.lastname@example.org" SENDMAILTO="hristoNOSPAM@deshev.com"
There are three report levels:
- paranod - this one will ignore as little log entries as possible. Warning: emails may get a bit spammy.
- server - the default level. I recommend you select that.
- workstation - contains additional ignore rules that are suitable for non-critical machines.
In addition you need to tell Logcheck which log files it should monitor. You list them in
/etc/logcheck/logcheck.logfiles. By default it contains only two files:
# these files will be checked by logcheck # This has been tuned towards a default syslog install /var/log/syslog /var/log/auth.log
Add other files as needed and go have a coffee or three. You can relax now that your server is being watched and you will get notified as something interesting happens. For example, soon after I installed Logcheck on my test machine, I started getting notifications about SSH login attempts from Chinese IP addresses like this one:
Oct 15 22:44:16 test sshd: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key Oct 15 22:44:18 test sshd: input_userauth_request: invalid user aaron [preauth] Oct 15 22:44:21 test sshd: Received disconnect from 126.96.36.199: 11: Bye Bye [preauth] Oct 15 22:44:21 test sshd: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key Oct 15 22:44:24 test sshd: input_userauth_request: invalid user aarti [preauth] Oct 15 22:44:26 test sshd: Received disconnect from 188.8.131.52: 11: Bye Bye [preauth] Oct 15 22:44:26 test sshd: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key Oct 15 22:44:33 test sshd: input_userauth_request: invalid user abdenace [preauth] Oct 15 22:44:35 test sshd: Received disconnect from 184.108.40.206: 11: Bye Bye [preauth] Oct 15 22:44:36 test sshd: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key Oct 15 22:44:41 test sshd: input_userauth_request: invalid user abdol [preauth] Oct 15 22:44:44 test sshd: Received disconnect from 220.127.116.11: 11: Bye Bye [preauth] Oct 15 22:44:44 test sshd: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key Oct 15 22:44:46 test sshd: input_userauth_request: invalid user abdul [preauth] Oct 15 22:44:49 test sshd: Received disconnect from 18.104.22.168: 11: Bye Bye [preauth] Oct 15 22:44:49 test sshd: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key Oct 15 22:44:52 test sshd: input_userauth_request: invalid user abdulkaf [preauth] Oct 15 22:44:54 test sshd: Received disconnect from 22.214.171.124: 11: Bye Bye [preauth]
Hurrah for script kiddies trying to brute force their way into my server. To stop them dead in their tracks, remember to harden your server's SSH configuration. We'll also cover how to get rid of them using tools like Fail2ban or DenyHosts in a future post.
Monitoring Your Application's Log Files
Being a developer that likes to tinker with servers, I wanted to set up Logcheck so that it monitors my application's log files. Ideally I want to be able to get a summary of important application activity as soon as it happens. Let us try to do just that.
For our little experiment, we'll assume our app logs its events to
/var/log/myapp.log. First we need to add it to
If our app logs relatively few events, then we are done. We will get all of them soon after they get logged -- by default Logcheck runs every hour. What if we want to filter some of the entries though? Suppose that we have a web application that logs every request like so:
web: request from x.x.x.x
And it also logs user authentications like:
auth: user login from y.y.y.y
We are interested in user authentications only, so we want to ignore lines starting with
web:. Let's add the rule for that inside a new file:
/etc/logcheck/ignore.d.server/myapp. Note that I placed my rule file inside the folder containing the rules for the "server" report level. Here is my rule:
Yes, that's all it takes -- a single regular expression that gets evaluated for all log entries. We save the file and we are done. No more web request entries in our report emails!
Logcheck is an amazingly simple yet effective sysadmin tool that can watch your back and notify you of problems on your servers. It can be a great help to a developer as it can monitor custom application logs and send alerts about almost anything.