As a security consultant I often have to deal with machines that are already compromised. The ‘official’ standpoint is always to wipe the machine alltogether, reinstall your OS, and restore your data and configurations from the backups that you obviously have.
The above not always being possible, and as a second best alternative, you’ll have to recover the machine.
The first thing to do is compare each command line utility to that of a known good identical system before using it, so you can rely on the results that it returns. A hacker will often drop a modified ‘ls’, ‘lsmod’, ‘ps’ and various other tools onto your system to hide the various other things that he may have installed.
You’ll need to use md5sum and ls to check the size and checksum of each utility before you use it, although of course, md5sum and ls themselves could be hardcoded with predefined responses. You could also use ‘strings’ to check the ASCII contents of those tools, although the ‘strings’ could just as easily be rigged. If you’re that paranoid, you’ve got no choice but to wipe the machine alltogether.
So firstly, check the integrity, of each of your core utilities. If your Debian 5.0 with the latest updates installed system was compromised, you’ll need to check against another Debian 5.0 system with the same updates and tools installed. Or, if you can find a listing online somewhere of what binaries should be what sizes and have what MD5s then you should be fine.
Once you have confirmed your ‘md5sum’ utility, you should be able to just start comparing MD5s and not worrying about file sizes and strings. Check your package management utilities and check that you’re happy with them, then apt-get install rkhunter this will check a number of issues. There are other ‘root kit hunters’ that you can use as well if you wish. Once this has been run, check your ps utility and ensure that it is as you expect. Then once done just run ps auxw and check each running process in the same way. Assuming that all of that is done and has not shown up anything, all is good so far. If something has been found and one of your binaries is compromised. Assuming your package manager is in good order, dpkg -P <package> and reinstall. If it is a core package that can not be removed/purged without affecting the rest of the system, then just scp over a new binary. Check again that the libc6 version and package version is IDENTICAL, and check of course that scp itself is in good order.
At this point, we can assume that your binaries themselves are in good order. Check for any new SUID utilities with find / -perm +4000 and once done, firstly make sure that everything on that list is as expected, and secondly, double check your md5sums of each and everyone of those.
This all being OK, continue to check by looking at your /etc/passwd, /etc/group and /etc/shadow files checking for user accounts that you don’t recognise. Then check syslog, wtmp, lastlog, etc, and check the IPs and last logins of each account. Also check directories such as /tmp/ especially with ls -al to check for directories beginning with a ‘.’ which would otherwise be hidden.
If everything above returns success, then it’s unlikely that your system was directly compromised. There is always the chance that your web application or database was compromised, but then that’s outside of the scope of this article. In short though, check your webserver log files as that should give you the information on what was compromised, and how it was done. Obviously ensure that any 3rd party software that you may be using such as wordpress, vBulletin, etc, etc are always up to the latest version.
In response to a reader’s comments, I would add that should you be able to remove the network connection to the compromised machine and still access it, then do. Your login and anything you type could be being sent to an attacker without you even realising it.
Additionally, there is no point in simply recovering a hacked server without knowing how it was compromised in the first place. Arguably you should have worked it out though by following the steps above.