There are a number of warning signs that a system has been compromised. The cases below warrant further investigation. Of course, they aren’t all guarantees that your system has been compromised, however they can be strong indicators.
1. Your welcome banner shows the last log in from an unknown/foreign IP address:
Last login: Tue Dec 2 16:08:41 2014 from 18.104.22.168 root@mt:~#
2. The load on a usually idle system is suspiciously high:
root@mt:~# w 17:06:39 up 62 days, 22:37, 1 user, load average: 8.12, 8.14, 8.11 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root pts/0 pwn 17:03 7.00s 0.00s 0.00s w
This could indicate that unknown processes are running.
3. Unknown processes, specifically scripts running:
root@mt:~# ps auxw USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 10648 472 ? Ss Oct07 0:29 init  [...] root 10613 0.0 0.0 57344 1148 ? Ss Oct22 0:13 perl /root/.root/31337.pl
Note that the user account that the process is running under can often give a lot away about the source of compromise. The most common account to see malicious scripts running under (on Debian) is www-data indicating that the web application has been compromised.
4. High network usage and strange usage patterns on a usually quiet system:
Here’s a simple script for gathering the current transmit and receive bytes per second on eth0:
rx_1=$(cat /sys/class/net/eth0/statistics/rx_bytes) tx_1=$(cat /sys/class/net/eth0/statistics/tx_bytes) sleep 1 rx_2=$(cat /sys/class/net/eth0/statistics/rx_bytes) tx_2=$(cat /sys/class/net/eth0/statistics/tx_bytes) echo "Received: $(expr $rx_2 - $rx_1)bps Transmitted: $(expr $tx_2 - $tx_1)bps"
An idle machine showing a huge transmit rate is cause for concern!
5. Abuse reports start coming in:
Dear Mr. Box Owner, We have received abuse reports...
6. Unknown user accounts present on the system:
root@mt:~# cat /etc/passwd [...] fr3d:x:1002:1002::/home/fr3d:/bin/bash
7. Unknown hidden directories present:
root@mt:~# ls -al total 1768 drwx------ 9 root root 4096 Nov 5 18:07 . drwxr-xr-x 24 root root 4096 Oct 7 19:31 .. -rw------- 1 root root 12625 Dec 9 17:02 .bash_history drwxr-xr-x 6 root root 4096 Oct 31 16:13 .c0d3z
8. Unknown files present within the www directory:
root@mt:~# ls /var/www default index.php webshell.php
Note that the creation time and date of webshell.php can give further information on the compromise.
9. Strange PHP code appearing in existing scripts:
<?php /**/ eval(base64_decode("aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JB TFNbJ21yX25vJ10pKXsgICAkR0xPQkFMU1snbXJfbm8nXT0xOyAgIGlmKCFmdW5jdGlvbl9leGlzdHMoJ21yb2JoJyk peyAgICAgIGlmKCFmdW5jdGlvbl9leGlzdHMoJ2dtbCcpKXsgICAgIGZ1bmN0aW9uIGdtbCgpeyAgICAgIGlmICghc3 RyaXN0cigkX1NFUlZFUlsiSFRUUF9VU0VSX0FHRU5UIl0sImdvb2dsZWJvdCIpJiYgKCFzdHJpc3RyKCRfU0VSVkV [...] yk7ICAgfSAgfQ=="));?>
This is a clever trick that’s been used across scripting languages for years. By looking at the first two function calls, eval and base64_decode, we can see that first, the large block of ASCII text is decoded using the base64 algorithm (into PHP code), and that PHP code is then executed with eval. The purpose is two-fold:
- To obfuscate and hide the intention of the malicious code from the user viewing the script.
- To allow this plain ASCII code to be injected through potential filters that would filter out other code characters.
To view the actual code is simple. Simply decode the ASCII text with a web based base64 decoder. Note: NEVER run the unknown code through PHP..
10. Unknown code appearing that references ‘googlebot’:
[...] (!stristr($_SERVER["HTTP_USER_AGENT"],"googlebot")&& [...]
This is another sneaky one, most likely intended to increase the rankings of a malicious 3rd party site. The most common case is where a different version of the page is presented to Google’s crawlers than it is to everyone else. You and your site visitors will never notice anything different about the site, however Google is being provided with a page version containing a load of malicious links. This will harm your site in the rankings.
All of the cases above warrant further investigation. One last piece of advice – if you have been compromised, disconnect the system from the network as soon as possible. If you ever intend to get to the root cause of the compromise, do not begin deleting or attempting to repair files. Malicious files contain valuable meta data – owner, group, created on, modified on, and so on. Once you start accessing, modifying and deleting, evidence is permanently destroyed.