Troubleshooting Hacked Sites

From DreamHost
Jump to: navigation, search

Overview

What do you do when your sites have been hacked? In general, recovering from an exploit requires consideration of:

  1. locating the source
  2. closing the security hole
  3. removing backdoors
  4. cleaning up damage

This guide covers the process of clearing up any exploits on your account.

Who is responsible?

In short, you are. DreamHost may assist in some situations, but is not obligated to do so. The Terms of Service, to which all customers must agree, states the following:

DreamHost Web Hosting is an information provider connected to the Internet. DreamHost Web Hosting offers storage and transfer services over the Internet through access to its Web Server.

Basically, this means that DreamHost sells the use of its servers. Whilst DreamHost does manage the OS and utilities on the servers themselves, they do not provide any management of customer software on those servers. The liability for the software housed in a user's account is clearly stated.

DreamHost sells the use of servers and manages the OS and utilities on the server themselves, but DreamHost does not provide management of customer software which the customer chooses to put on the server. The liability for software housed in a user’s account is stated below:

Material products

  • Customer will provide DreamHost Web Hosting with material and data in a condition that is “server-ready”, which is in a form requiring no additional manipulation on the part of DreamHost Web Hosting. DreamHost Web Hosting shall make no effort to validate this information for content, correctness, or usability.
  • Use of DreamHost Web Hosting’s service requires a certain level of knowledge in the use of Internet languages, protocols, and software. This level of knowledge varies depending on the anticipated use and desired content of Customer’s Webspace by the Customer.
  • The following examples are offered:
  • Web Publishing: requires a knowledge of HTML, properly locating and linking documents, FTPing webspace contents, graphics, text, sound, imagemapping, and so on.
  • CGI-Scripts: requires a knowledge of the UNIX environment, TAR & GUNZIP commands, Perl, CShell scripts, permissions, and so on.
  • The Customer agrees that he or she has the necessary knowledge to create Customer’s Webspace. Customer agrees that it is not the responsibility of DreamHost Web Hosting to provide this knowledge or Customer Support outside of the defined service of DreamHost Web Hosting.

Determining how the account was hacked

The first step in getting things back to normal is to determine how the account was hacked. In general, almost all hacks occur through three methods ranked in order of likelihood:

  1. A CGI vulnerability in software you've uploaded to your website has been exploited and used to write/execute arbitrary code on the server.
  2. Your FTP/SSH password has been compromised.
  3. You have world-writable directory permissions on web-accessible folders.

Perhaps counter-intuitively, CGI hacks are more common than FTP/SSH password hacks primarily due to the sheer proliferation of pre-bundled software that people set up and frequently later forget to update. However, FTP/SSH hacks are the easiest to check up on. World-writable folders attacks are possible when the attacker is in the same machine that your account is. With world-writable permissions (o=w), anyone with an account on the hosting machine is able to put files there.

Looking for FTP/SSH hacks

Since recent FTP/SSH hacks are easiest to spot, start by eliminating this option. Log into your user via SSH and run the following commands.

Shows you your login history for the current month:

last -i | grep youruser

Shows you your login history for the prior month:

last -if /var/log/wtmp.1 | grep youruser
Note2 icon.png Note: The usernames printed by "last" truncate after 8 characters, so if you have a longer username you'll want to truncate yours in the grep string as well.


The output looks something like this:

youruser pts/4        99.139.XXX.XXX   Wed May 28 06:10 - 07:11  (01:00)    
youruser pts/5        66.33.XXX.XXX    Sun May 25 09:31 - 12:14  (02:42)    
youruser ftpd30715    66.33.XXX.XXX    Wed May 21 14:16 - 14:16  (00:00)    
youruser pts/2        66.33.XXX.XXX    Tue May 20 13:22 - 14:18  (00:56)    
youruser pts/2        66.33.XXX.XXX    Tue May 20 13:06 - 13:22  (00:15)    

You can simplify this data to print out only IP addresses and counts by adding pipes to a few simple commands:

last -if /var/log/wtmp.1 | grep youruser | awk '{print $3}' | sort | uniq -c

The output looks more like this:

4 66.33.XXX.XXX
1 99.139.XXX.XXX

You may find either method useful for determining who has logged into your FTP/SSH user.

Note2 icon.png Note: The wtmp logs only go back at most 1–2 months, so if the hack is older than that, DreamHost won’t have any records of it.


If you've determined that FTP/SSH is the source of the hack, you should:

  1. Change your password. Visit the Change password article for further details.
  2. Discontinue use of FTP which sends your password over the internet in plaintext, and then switch to SFTP or SSH. You can disable FTP for the account in the control panel on the same page that you changed your password.
  3. Ensure there is up-to-date virus/malware scanning on any computers on which you've used the password/user in question.

Once you've done so, please proceed to the Cleaning up after a hack section. Otherwise, if you didn't find any strange IPs in the logs, proceed to Looking for CGI hacks.

Looking for CGI hacks

Checking your logs on a specific date

If possible, start by trying to determine when the hack occurred. This can often be determined by noting the modification date of any files which have been modified by the hacker. Usually, they will all share a common modification time. If that time can be determined, check the logs for that day and time to see what was requested or posted to your site.

Careful examination of your log files often reveals exactly how your site was hacked. Every request and post to your site is logged and the log files can not be changed by the user, so the record of the attack is preserved provided that the exploit was discovered within the 30 days or so which are recorded in the logs.

Checking all of your logs

If the original time can not be ascertained, the log files can be scanned en masse although it's more difficult to uncover the evidence of the intrusion. You can run the following command on your web server after you log in via SSH:

gunzip -c ~/logs/example.com/http/access.log.* | gawk '{a[$7]++}END{for (i in a) {print a[i]"\t"i}}' | sort -n | more

This returns a report of all requests to your site from all available logs sorted in ascending order. Since the majority of requests should (hopefully) be legitimate, it's easiest to start by looking at the top of the report — the least frequent requests — for evidence of the exploit:

It's not uncommon to see many requests which should not be there, as this is the nature of a website since it is publicly accessible. These requests do not, in and of themselves, indicate a breach, but should be investigated to see if the request successfully revealed a vulnerability.

Looking world-writable directories

World-writable directories allow file writing by any user on the machine, and these directories can be mass-scanned so this attack has been surfacing quickly. It's recommended that you perform this step anytime you suspect a breach.

Even if you're sure you didn't make any permission mistakes, some less security-aware software vendors or plugin developers often use system commands or language-native permission-management functions to make some directories (usually ones used for caching and temporary files, session files, and so on) to ease installation and management.

To scan for directories with world-writable permissions, use the UNIX find tool. Run this command via SSH in your domain’s web directory.

This command searches all folders below the folder in which you run it:

find . -type d -perm -o=w

If no results are displayed, then no folders are world-writable.

Preventing future hacks

Updating software

Failure to keep software up-to-date almost guarantees that your site will eventually be compromised. Whilst the latest software is not immune to exploitation, there are publicly available databases of known vulnerabilities which hackers use to probe for weaknesses. Once an exploit is discovered and made publicly available, your site is vulnerable until a patch is issued and you use that patch to update your site.

Make sure that it is up-to-date with the most recent version offered by the vendor. "Pre-packaged software" effectively means any software package that you've placed in your domain directory such as a blog, gallery, forum, shopping cart, content management system, and so on. Out-of-date versions of such software frequently have well-known security holes that can be exploited via simple scripts that are bandied about freely amongst "hacker" and "script-kiddie" groups.

Updating plugins

Don't overlook plugins when updating software. If you have any non-standard plugins activated for your applications, try a search engine query for the plugin name + "vulnerability" to see if anything crops up in the version you're using. If there are known vulnerabilities for the plugin in the version you're using, make sure to apply any available patches; otherwise, deactivate the plugin.

Fixing world-writable directories

You can mass-change all your world-writable directories permission with the UINX find tool. Run this command via SSH:

find . -type d -perm -o=w -print -exec chmod 770 {} \;

When you run this command in your web directory, the following happens:

  • All directories that have 777 permissions are found.
  • Those directories with 777 permissions are changed to 770.
  • Other directories are not touched.

It's always better to enumerate all the world-writable directories and then deciding the proper permissions. Some folders require special attention.

Removing backdoors

The main purpose of exploiting insecure websites is to gain and maintain access for the benefit of the hacker. Therefore, after discovering a vulnerability, backdoors are almost always installed so that the hacker can return at a later time. Failure to identify and remove these backdoors will result in your site becoming exploited again.

Common methods or retaining access include:

  • installing a shell – a file which can be requested like any other page on your site, but gives the requester complete access. These are often written in PHP but can be in any language. They are often given innocent sounding names or contain obfuscated code. The may also be appended to legitimate files with extra code added so that the functionality is only available when an extra parameter is passed to the file during the request.
  • creating new users – These may be new database users or web app users.
  • storing authenticated cookies – Most apps have the option of remembering who is logged in by storing a cookie on the user's computer. If a hacker manages to modify/create a user and receive an authenticated cookie, then this cookie can often be reused until it expires even if the password for that user is changed. To force holders of authenticated cookies to log in again, the salt used to encrypt the cookies must be changed after passwords have been reset.
  • changing folder and file permissions to world-writable
  • adding a public key to .ssh/authorized_keys

Unfortunately, there is no single-step, foolproof way of finding and removing backdoors. It takes time and knowledge to investigate the mode of entry and the actions that occurred afterwards. There are many examples on the web of the types of steps that you can take.

Cleaning up after a hack

Removing a hacked file

Regardless of the mode of the intrusion, you'll want to clean up after a hack to ensure the integrity of your site and to ensure that it cannot be compromised again. To do this, you will need to go through all of the files under the compromised user account and delete anything which you did not place there. If you're using an FTP client, make sure to enable viewing "hidden" files – and the same for shell by using the -a option with ls.

It can sometimes help to first look for files with modification timestamps that occurred since you last modified your site or around the time the hack took place. If you have identified a file that was definitely modified in the hack (such as a defaced index page), you may be able to pinpoint the files used to modify the hacked file by searching for the file's timestamp in your HTTP logs via shell.

A useful command for doing this is:

find /home/yourusername/www.example.com/ ! -name "log" -mtime -3

where the "-3" is the number of days in the past to look for modified files. This command lists all files under /home/user/example.com that have been modified in the past 3 days.

Note2 icon.png Note: If the above command does not work, try without www, for example:
find /home/yourusername/example.com/ ! -name "log" -mtime -3


Recent HTTP logs are located in the following directory:

/home/yourusername/logs/example.com/http/

Cleaning up database hacks

Certain hacks, particularly SQL injection attacks against vulnerable Joomla! installations, may result in the database becoming altered with malicious code. Such a modification can allow the hacker back in even if you've updated to the latest version and cleaned off all foreign files. For this reason, it's a good idea after a hack to inspect the database in the same way you check your files to see if anything has been changed that should not have been. If you know when the hack occurred, you may even wish to revert the database back to a prior time via the backup feature available in the panel. View the Backup article for further details.

Restoring lost/modified files

Finally, to restore files that have been modified or deleted in the hack please see the article on restoring your domain. DreamHost offers database and domain restores from the panel, and the sooner you get to them the better as backups are only kept for a few days.

View the Backup article for further details and options.