Detecting intrusions

From DreamHost
Jump to: navigation, search

Detecting Intrusions

After doing everything you can to protect your site from adversaries, you still need to monitor your sites. One way to mitigate a hacked site is to detect the intrusion as early as possible so that corrective actions can be taken. Here are some simple scripts you can run as a daily cron job.

Another approach is to use an external service such as Pingdom which can be configured to request a web-accessible script every n minutes. This script should be secured at least with a password. Even better would be to also check the user-agent string with .htaccess and deny access if the string does not match. Pingdom settings allow for custom user-agent strings, so a long string of random characters would be ideal.

As of May 2013, DreamHost also offers StopTheHacker web security services with both free and paid options.

Basic file

All of these scripts can be combined or run individually, but they will all need a common header:

#!/bin/bash

# Replace this line with your hosted domain
MY_SITE=www.domain.com

LOGS_PATH=$HOME/logs/$MY_SITE/http
ERROR_LOG=$LOGS_PATH/error.log.0
ACCESS_LOG=$LOGS_PATH/access.log.0

What IP addresses were used to log into your account

If you haven't logged in recently and you suddenly see a recent login and an IP address from a foreign country, then you've probably been hacked. The following won't tell you just the IP address because that on its own is not very useful. It will get the hostname of the IP address so that you can see what domain its coming from.

/usr/bin/awk -v USER=${USER:0:8} \
 'BEGIN{
    cmd="/usr/bin/last -i && /usr/bin/last -if /var/log/wtmp.1"
    while( (cmd | getline) > 0 ){
       if($1~USER){
          user  =$1;
          ip    =$3;
          day   =$4;
          month =$5;
          date  =$6;
          login =$7;
          logout=$9;
          gethost="host "$3;
          while( (gethost | getline) > 0){
             printf "%8s : %3s %3s %2s (%5s - %5s) : %15s : %s\n", user, day, month, date, login, logout, ip, $NF
          }
          close(gethost)
       }
    }
    close(cmd)
 }'

Here's an alternative to the above that isn't quite as verbose, but gets the point across. If you don't recognize any extraneous IP's there's a possibility you've been hacked.

#!/bin/sh
# useriplist.sh
# List IP's from which the current user logged in. Sort by most popular IP.
# Define the user as the current user.
USER=${USER:0:8} 
# If you see one or two logins from IP's you don't recognize, you may have been compromised.
( last -i | grep $USER && last -if /var/log/wtmp.1 | grep $USER ) | awk '{ print $3 }' | sort | uniq -c | sort -rn

Recently modified files

Check by looking at modification times

If you haven't been working on your site in the past 24 hours, then why have some of your files been modified? Many apps will keep logs and modify files, but it's good to keep an eye on things. If one day all of your .php files have been modified, then you've probably been hacked! You can exclude directories and files which change often and you are sure do not represent a threat.

/usr/bin/find $HOME -mtime -2 -type f \
   | /bin/sed -r 's|^(\/[^\/]+){2}||' \
   | /usr/bin/awk '
        # the following line skips any directories which change often and you are sure do not represent a threat
        /\/(\.git|\.svn|cronjobs|cache|objects|logs|tmp)\// ||
        # the following line skips any file types which you are sure do not represent a threat
        /\.(txt|log|zip|prop|meta|gif|png|gz|po|mo|ico)$/ ||
        /^\./ {
            next;
        }
        {   
            print;
        }'

The limitation of this method is that a sophisticated script might modify the modification times to be the same as what they originally were. Many scripts don't bother to do that, but it is a possibility and relatively easy to do.

Check by using git

git is a program which is used to track changes in software. It is typically used by programmers and web designers to keep a history of changes, but it can also be used to monitor changes in a website or user account. git is extremely fast at detecting any changes to files and can be called from a script to check the integrity of every file that its set up to track.

Taking this approach would be similar to looking at modification times, but is actually more robust and faster. It takes longer to set up though. A git repo needs to be created, and either:

  • add all files in a user's account added to the repo and ignore files which are expected to change frequently or
  • add just files which are perceived as vulnerable and critical.

Recent requests

It helps to see what requests are being made to your site. Generally patterns will emerge that you can ignore, but adversaries looking for vulnerabilities will run scripts which make dozens or even hundreds of requests for known vulnerabilities and then move on to another site if they don't find any. These will usually show up as a big increase in the number of unique requests for the day.

List and count all unique requests in the past 30 days

This actually just lists the first part of a request (www.domain.com/assets/picture.jpg). You can report everything by changing the part after awk. It's probably not necessary to run this as a cron job, but can be used to follow up on any suspicious activity.

MY_LOGS=$LOGS_PATH/access.log.*.gz
/bin/gunzip -c $MY_LOGS \
    | /usr/bin/awk '{
        # to list the entire request, not just the base,
        # replace the following two lines with: a[$7]++
        split($7,req,"/");
        a[req[2]]++
    }END{
        for(i in a){
            print a[i]"\t"i
        }
    }' \
    | /usr/bin/sort -nr

List and count all unique requests from yesterday

This is probably more suitable for a cronjob. However, if you are confident that your app handles requests well enough, you may want to just look for 404 requests (below).

/usr/bin/awk '{
    # to list the entire request, not just the base,
    # replace the following two lines with: a[$7]++
    split($7,req,"/");
    a[req[2]]++
}END{
    for(i in a){
        print a[i]"\t"i
    }
}' $ACCESS_LOG \
| /usr/bin/sort -nr

List and count all unique requests that created a 404 response

/usr/bin/awk '$9==404 {
   a[$7]++
}END{
   for(i in a){
       print a[i]"\t"i
   }
}' $ACCESS_LOG \
| /usr/bin/sort -nr

Check disk usage

This is probably not very useful for detecting intrusions unless your site is hijacked and used to distribute something. It might be useful to add to your cronjob anyway to keep an eye on anything that might be using up more space than you realised. For example, some apps create logs which are never used for anything and could easily use up hundreds of MB for no purpose.

/usr/bin/find $HOME -maxdepth 1 -type d -exec du -s {} \; \
   | /usr/bin/sort -nr \
   | /usr/bin/awk -v HOME="$HOME/?" '{
    $1 = $1/1000;
    if ($1 > 1000) {
        $1 = $1/1000
        UNIT="G"
    } else {
        UNIT="M"
    }
    sub(HOME,"",$2);
    printf "%.2f%s\t%s\n", $1, UNIT, $2
   }'

Ferret - a single script combining the above tools

All of the above tools have been cobbled together into a single scripted solution called ferret available on github. This bash script should be fairly self-explanatory for anyone who has a rudimentary understanding of secure shell (ssh). If you're not familiar with git, feel free to download the repository as a zip file, install it per the instructions in the readme and start using it to 'ferret' out vital information that may help you determine if your site has been hacked.