There are a number of warning signs that a system has been compromised. The cases below warrant further investigation. Of course, they aren’t all guarantees that your system has been compromised, however they can be strong indicators.

1. Your welcome banner shows the last log in from an unknown/foreign IP address:

Last login: Tue Dec  2 16:08:41 2014 from

2. The load on a usually idle system is suspiciously high:

root@mt:~# w
 17:06:39 up 62 days, 22:37,  1 user,  load average: 8.12, 8.14, 8.11
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    pwn              17:03    7.00s  0.00s  0.00s w

This could indicate that unknown processes are running.

3. Unknown processes, specifically scripts running:

root@mt:~# ps auxw
root         1  0.0  0.0  10648   472 ?        Ss   Oct07   0:29 init [2]
root     10613  0.0  0.0  57344  1148 ?        Ss   Oct22   0:13 perl /root/.root/

Note that the user account that the process is running under can often give a lot away about the source of compromise. The most common account to see malicious scripts running under (on Debian) is www-data indicating that the web application has been compromised.

4. High network usage and strange usage patterns on a usually quiet system:
Here’s a simple script for gathering the current transmit and receive bytes per second on eth0:

rx_1=$(cat /sys/class/net/eth0/statistics/rx_bytes)
tx_1=$(cat /sys/class/net/eth0/statistics/tx_bytes)
sleep 1
rx_2=$(cat /sys/class/net/eth0/statistics/rx_bytes)
tx_2=$(cat /sys/class/net/eth0/statistics/tx_bytes)
echo "Received: $(expr $rx_2 - $rx_1)bps Transmitted: $(expr $tx_2 - $tx_1)bps"

An idle machine showing a huge transmit rate is cause for concern!

5. Abuse reports start coming in:

Dear Mr. Box Owner,

We have received abuse reports...

6. Unknown user accounts present on the system:

root@mt:~# cat /etc/passwd

7. Unknown hidden directories present:

root@mt:~# ls -al
total 1768
drwx------  9 root root    4096 Nov  5 18:07 .
drwxr-xr-x 24 root root    4096 Oct  7 19:31 ..
-rw-------  1 root root   12625 Dec  9 17:02 .bash_history
drwxr-xr-x  6 root root    4096 Oct 31 16:13 .c0d3z

8. Unknown files present within the www directory:

root@mt:~# ls /var/www
default  index.php  webshell.php

Note that the creation time and date of webshell.php can give further information on the compromise.

9. Strange PHP code appearing in existing scripts:

<?php /**/ eval(base64_decode("aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJEdMT0JB

This is a clever trick that’s been used across scripting languages for years. By looking at the first two function calls, eval and base64_decode, we can see that first, the large block of ASCII text is decoded using the base64 algorithm (into PHP code), and that PHP code is then executed with eval. The purpose is two-fold:

  1. To obfuscate and hide the intention of the malicious code from the user viewing the script.
  2. To allow this plain ASCII code to be injected through potential filters that would filter out other code characters.

To view the actual code is simple. Simply decode the ASCII text with a web based base64 decoder. Note: NEVER run the unknown code through PHP..

10. Unknown code appearing that references ‘googlebot’:


This is another sneaky one, most likely intended to increase the rankings of a malicious 3rd party site. The most common case is where a different version of the page is presented to Google’s crawlers than it is to everyone else. You and your site visitors will never notice anything different about the site, however Google is being provided with a page version containing a load of malicious links. This will harm your site in the rankings.

All of the cases above warrant further investigation. One last piece of advice – if you have been compromised, disconnect the system from the network as soon as possible. If you ever intend to get to the root cause of the compromise, do not begin deleting or attempting to repair files. Malicious files contain valuable meta data – owner, group, created on, modified on, and so on. Once you start accessing, modifying and deleting, evidence is permanently destroyed.