Detecting intrusions

Detecting Intrusions
After doing everything you can to protect your site from attackers, you still need to monitor your sites. One way to mitigate a hacked site is to detect the intrusion as early as possible so that corrective actions can be taken. Here are some simple scripts you can run as a daily cron job.

As of May 2013, DreamHost also offers StopTheHacker web security services with both free and paid options. Another Simple and Free way to check if your site has been compromised is by using SiteCheck.

Basic file
All of these scripts can be combined or run individually, but they will all need a common header:

MY_SITE=www.domain.com LOGS_PATH=$HOME/logs/$MY_SITE/http ERROR_LOG=$LOGS_PATH/error.log.0 ACCESS_LOG=$LOGS_PATH/access.log.0
 * 1) !/bin/bash
 * 1) Replace this line with your hosted domain

What IP addresses were used to log into your account
If you haven't logged in recently and you suddenly see a recent login and an IP address from a foreign country, then you've probably been hacked. The following won't tell you just the IP address because that on its own is not very useful. It will get the hostname of the IP address so that you can see what domain its coming from.

/usr/bin/awk -v USER=${USER:0:8} \ 'BEGIN{ cmd="/usr/bin/last -i && /usr/bin/last -if /var/log/wtmp.1" while( (cmd | getline) > 0 ){ if($1~USER){ user =$1; ip   =$3; day  =$4; month =$5; date =$6; login =$7; logout=$9; gethost="host "$3; while( (gethost | getline) > 0){ printf "%8s : %3s %3s %2s (%5s - %5s) : %15s : %s\n", user, day, month, date, login, logout, ip, $NF }          close(gethost) }    }     close(cmd) }'

Here's an alternative to the above that isn't quite as verbose, but gets the point across. If you don't recognize any extraneous IP's there's a possibility you've been hacked.

USER=${USER:0:8} ( last -i | grep $USER && last -if /var/log/wtmp.1 | grep $USER ) | awk '{ print $3 }' | sort | uniq -c | sort -rn
 * 1) !/bin/sh
 * 2) useriplist.sh
 * 3) List IP's from which the current user logged in. Sort by most popular IP.
 * 4) Define the user as the current user.
 * 1) If you see one or two logins from IP's you don't recognize, you may have been compromised.

Check by looking at modification times
If you haven't been working on your site in the past 24 hours, then why have some of your files been modified? Many apps will keep logs and modify files, but it's good to keep an eye on things. If one day all of your .php files have been modified, then you've probably been hacked! You can exclude directories and files which change often and you are sure do not represent a threat.

/usr/bin/find $HOME -mtime -2 -type f \ | /bin/sed -r 's|^(\/[^\/]+){2}||' \ | /usr/bin/awk ' # the following line skips any directories which change often and you are sure do not represent a threat /\/(\.git|\.svn|cronjobs|cache|objects|logs|tmp)\// || # the following line skips any file types which you are sure do not represent a threat /\.(txt|log|zip|prop|meta|gif|png|gz|po|mo|ico)$/ || /^\./ {            next; }        {                print; }'

The limitation of this method is that a sophisticated script might modify the modification times to be the same as what they originally were. Many scripts don't bother to do that, but it is a possibility and relatively easy to do.

Check by using git
git is a program which is used to track changes in software. It is typically used by programmers and web designers to keep a history of changes, but it can also be used to monitor changes in a website or user account. git is extremely fast at detecting any changes to files and can be called from a script to check the integrity of every file that its set up to track.

Taking this approach would be similar to looking at modification times, but is actually more robust and faster. It takes longer to set up though. A git repo needs to be created, and either:
 * add all files in a user's account added to the repo and ignore files which are expected to change frequently or
 * add just files which are perceived as vulnerable and critical.

Recent requests
It helps to see what requests are being made to your site. Generally patterns will emerge that you can ignore, but adversaries looking for vulnerabilities will run scripts which make dozens or even hundreds of requests for known vulnerabilities and then move on to another site if they don't find any. These will usually show up as a big increase in the number of unique requests for the day.

List and count all unique requests in the past 30 days
This actually just lists the first part of a request (www.domain.com/assets/picture.jpg). You can report everything by changing the part after awk. It's probably not necessary to run this as a cron job, but can be used to follow up on any suspicious activity.

MY_LOGS=$LOGS_PATH/access.log.*.gz /bin/gunzip -c $MY_LOGS \ | /usr/bin/awk '{ # to list the entire request, not just the base, # replace the following two lines with: a[$7]++ split($7,req,"/"); a[req[2]]++ }END{ for(i in a){ print a[i]"\t"i }    }' \     | /usr/bin/sort -nr

List and count all unique requests from yesterday
This is probably more suitable for a cronjob. However, if you are confident that your app handles requests well enough, you may want to just look for 404 requests (below).

/usr/bin/awk '{ # to list the entire request, not just the base, # replace the following two lines with: a[$7]++ split($7,req,"/"); a[req[2]]++ }END{ for(i in a){ print a[i]"\t"i } }' $ACCESS_LOG \ | /usr/bin/sort -nr

List and count all unique requests that created a 404 response
/usr/bin/awk '$9==404 { a[$7]++ }END{ for(i in a){ print a[i]"\t"i } }' $ACCESS_LOG \ | /usr/bin/sort -nr

Check disk usage
This is probably not very useful for detecting intrusions unless your site is hijacked and used to distribute something. It might be useful to add to your cronjob anyway to keep an eye on anything that might be using up more space than you realised. For example, some apps create logs which are never used for anything and could easily use up hundreds of MB for no purpose.

/usr/bin/find $HOME -maxdepth 1 -type d -exec du -s {} \; \ | /usr/bin/sort -nr \ | /usr/bin/awk -v HOME="$HOME/?" '{    $1 = $1/1000;     if ($1 > 1000) { $1 = $1/1000        UNIT="G" } else { UNIT="M" }    sub(HOME,"",$2); printf "%.2f%s\t%s\n", $1, UNIT, $2 }'

Ferret - a single script combining the above tools
All of the above tools have been cobbled together into a single scripted solution called ferret available on github. This bash script should be fairly self-explanatory for anyone who has a rudimentary understanding of secure shell (ssh). If you're not familiar with git, feel free to download the repository as a zip file, install it per the instructions in the readme and start using it to 'ferret' out vital information that may help you determine if your site has been hacked.