Troubleshooting Hacked Sites

From DreamHost
Revision as of 19:39, 26 March 2012 by Myqlarson (Talk | contribs) (links to blogs detailing the processes of looking for backdoors)

Jump to: navigation, search

My Account Has Been Hacked - What Do I Do?

In general, recovering from an exploit, hack, or exploit will require consideration of:

  1. locating the source
  2. closing the security hole
  3. removing backdoors
  4. cleaning up damage

An example of this process can be seen here. Note this is only an example and does not cover all possibilities.

It also helps to have monitoring in place to detect intrusions so that they can be mitigated as soon as possible.

Who is responsible?

In short, you are. DreamHost may assist in some situations, but it they are not obliged to do so. The Terms of Service, to which all customers must agree, clearly states that:

DreamHost Web Hosting is an information provider connected to the Internet. DreamHost Web Hosting offers storage and transfer services over the Internet through access to its Web Server

Basically that means that DreamHost sells the use of servers. Whilst DreamHost does manage the OS and utilities on the servers themselves, they do not provide any management of customer software on those servers. The liability for the software housed in a user's account is clearly stated [emphasis added]:


  • Customer will provide DreamHost Web Hosting with material and data in a condition that is “server-ready”, which is in a form requiring no additional manipulation on the part of DreamHost Web Hosting. DreamHost Web Hosting shall make no effort to validate this information for content, correctness or usability.
  • Use of DreamHost Web Hosting’s service requires a certain level of knowledge in the use of Internet languages, protocols, and software. This level of knowledge varies depending on the anticipated use and desired content of Customer’s Webspace by the Customer.
  • The following examples are offered:
    • Web Publishing: requires a knowledge of HTML, properly locating and linking documents, FTPing Webspace contents, Graphics, text, Sound, imagemapping, etc.
    • CGI-Scripts: requires a knowledge of the UNIX environment, TAR & GUNZIP commands, Perl, CShell scripts, permissions, etc.
  • The Customer agrees that he or she has the necessary knowledge to create Customer’s Webspace. Customer agrees that it is not the responsibility of DreamHost Web Hosting to provide this knowledge or Customer Support outside of the defined service of DreamHost Web Hosting.

Determining the Hack Method

The first step in getting things back to normal is to determine how the account was hacked. In general, almost all hacks occur through three methods ranked in order of likelihood:

  1. A CGI vulnerability in software you've uploaded to your website has been exploited and used to write/execute arbitrary code on the server.
  2. Your FTP/SSH password has been compromised.
  3. You have world-writable directory permissions on web-accessible folders

Perhaps counter-intuitively, CGI hacks are more common than FTP/SSH password hacks primarily due to the sheer proliferation of pre-bundled software that people set up and frequently later forget to update. FTP/SSH hacks are the easiest to check up on, though. World-writable folders attacks are possible when the attacker is in the same machine that your account is. With world-writalble permissions (o=w) anyone with an account on the hosting machine will be able to put files there.

Looking For CGI Hacks

Careful examination of your log files will often reveal exactly how your site was hacked. Every request and post to your site is logged and the log files can not be changed by the user, so the record of the attack will be preserved provided that the exploit was discovered within the 30 days or so which are recorded in the logs.

If possible, start by trying to determine when the hack occurred. This can often be determined by noting the modification date of any files which have been modified by the hacker. Usually they will all share a common modification time. If that time can be determined, check the logs for that day and time to see what was requested or posted to your site.

If the original time can not be ascertained, the log files can be scanned en masse although it will be more difficult to uncover the evidence of the intrusion. The following command will return a report of all requests to your site from all available logs sorted in ascending order. Since the majority of requests should (hopefully) be legitimate, it's easiest to start by looking at the top of the report — the least frequent requests — for evidence of the exploit:

gunzip -c ~/logs/DOMAIN.COM/http/access.log.* | gawk '{a[$7]++}END{for (i in a) {print a[i]"\t"i}}' | sort -n | more

It is not uncommon to see many requests which should not be there. This is the nature of a website; it is publicly accessible so some passers-by will check the locks. These requests do not, in and of themselves, indicate a breach, but should be investigated to see if the request successfully revealed a vulnerability. Some requests that indicate someone is probing your site for known vulnerabilities include:

5       //admin/phpmyadmin/index.php
5       //admin/pma/index.php
5       //dbadmin/index.php
5       //index.php
5       //myadmin/index.php
5       //mysqladmin/index.php
5       //phpMyAdmin-2.2.3/index.php
5       //phpMyAdmin-2.2.6/index.php
5       //phpMyAdmin-2.5.1/index.php
5       //pma/index.php
5       //typo3/phpmyadmin/index.php
5       //web/index.php
5       //web/phpMyAdmin/index.php
5       //websql/index.php
5       /wplookup.php

This is not an exhaustive list.

Looking for FTP/SSH Hacks

Since recent FTP/SSH hacks are easiest to spot, start eliminating there. Log into your user via SSH and run the following command:

last -i | grep youruser
last -if /var/log/wtmp.1 | grep youruser

The first command will give you your login history for the current month, while the second will give you your login history for the prior month. Note that the usernames printed by "last" truncate after 8 characters, so if you have a longer username you'll want to truncate yours in the grep string as well.

The output will look something like this:

youruser pts/4        99.139.XXX.XXX   Wed May 28 06:10 - 07:11  (01:00)    
youruser pts/5        66.33.XXX.XXX    Sun May 25 09:31 - 12:14  (02:42)    
youruser ftpd30715    66.33.XXX.XXX    Wed May 21 14:16 - 14:16  (00:00)    
youruser pts/2        66.33.XXX.XXX    Tue May 20 13:22 - 14:18  (00:56)    
youruser pts/2        66.33.XXX.XXX    Tue May 20 13:06 - 13:22  (00:15)    

You can simplify this data to print out only IP addresses and counts by adding pipes to a few simple commands:

last -if /var/log/wtmp.1 | grep youruser | awk '{print $3}' | sort | uniq -c

The output will look more like this:

     4 66.33.XXX.XXX
     1 99.139.XXX.XXX

You may find either method useful for determining who has logged into your FTP/SSH user. Note that the wtmp logs only go back at most 1-2 months, so if the hack is older than that we won't have records of it.

If you've determined FTP/SSH to be the source of the hack, you should:

  1. Change your password via "Users" -> "Manage Users" -> "Edit" in the DreamHost control panel.
  2. It is strongly recommended that you discontinue use of FTP which sends your password over the internet in plaintext; switch to SFTP or SSH. You can disable FTP for the account in the control panel on the same page that you changed your password.
  3. Ensure there is up-to-date virus/malware scanning on any computers on which you've used the password/user in question.

Once you've done so, please proceed to the "Cleaning Up After A Hack" section. Otherwise, if you didn't find any strange IPs in the logs, proceed directly below to "Looking For CGI Hacks".

Looking world-writable directories

World-writable directories will allow file writing by any user on the machine. These directories can be mass-scanned so this attack has been surfacing quickly. Even after you've checked all the above options this step must be performed. Even if you're sure you didn't make any permission mistakes, some less security-aware software vendors or plugin developers often use system commands or language-native permission-management functions to make some directories (usually ones used for caching and temporary files, session files etc) to ease installation and management.

To scan for directories with world-writeable permissions use the UNIX find tool

find . -type d -perm -o=w

If no results are displayed, then no folders are world-writeable.

Preventing Future Hacks

Update software

Failure to keep software up to date almost guarantees that your site will eventually be compromised. Whilst the latest software is not immune to exploitation, there are publicly available databases of known vulnerabilities which hackers use to probe for weaknesses. Once an exploit is discovered and made publicly available, your site will be vulnerable until a patch is issued and you use that patch to update your site.

Make sure that it is up-to-date with the most recent version offered by the vendor. "Pre-packaged software" effectively means any software package that you've placed in your domain directory such as a blog, gallery, forum, shopping cart, content management system, etc. Out-of-date versions of such softwares frequently have well-known security holes that can be exploited via simple scripts that are bandied about freely amongst "hacker" and "script-kiddie" groups.

Don't overlook plugins when updating software -- if you have any non-standard plugins activated for your applications try a search engine query for the plugin name + "vulnerability" to see if anything crops up in the version you're using. If there are known vulnerabilities for the plugin in the version you're using make sure to apply any available patches, otherwise deactivate the plugin.

Fixing world-writable directories

You can mass-change all your world-writable directories permission with the UINX find tool

find . -type d -perm -o=w -print -exec chmod modeyouwant {} \;

For example

find . -type d -perm -o=w -print -exec chmod 770 {} \;

It's always better to enumerate all the world-writable directories and then deciding the proper permissions. Some folders require special attention.

Removing Backdoors

The main purpose of exploiting insecure websites is to gain and maintain access for the benefit of the hacker. Therefore, after discovering a vulnerability, backdoors will almost always be installed so that the hacker can return at a later time. Failure to identify and remove these backdoors will result in your site being exploited again.

Common methods or retaining access include:

  1. installing a shell - a file which can be requested like any other page on your site, but gives the requester complete access. These are often written in PHP but can be in any language. They are often given innocent sounding names or contain obfuscated code. The may also be appended to legitimate files with extra code added so that the functionality is only available when an extra parameter is passed to the file during the request.
  2. creating new users. These may be new database users or web app users.
  3. storing authenticated cookies. Most apps have the option of remembering who is logged in by storing a cookie on the user's computer. If a hacker manages to modify/create a user and receive an authenticated cookie, then this cookie can often be reused until it expires even if the password for that user is changed. To force holders of authenticated cookies to login again, the salt used to encrypt the cookies must be changed after passwords have been reset. Here is a WordPress example.
  4. changing folder and file permissions to world-writable
  5. adding a public key to .ssh/authorized_keys

Unfortunately, there is no single-step, foolproof way of finding and removing backdoors. It takes time and knowledge to investigate the mode of entry and the actions that occurred afterwards. There are many examples of the types of steps that need to be taken. This blog describes some of the steps involved and provides a PHP script to assist in the process. This blog discusses more technical aspects of shells which may give clues regarding what to look for.

Cleaning Up After A Hack

Hacked File Removal

Regardless of the mode of intrusion you'll want to clean up after a hack to ensure the integrity of your site and that it cannot be compromised again. To do this you will need to go through all of the files under the compromised user account and delete anything which you did not place there. If you're using an FTP client make sure to enable viewing "hidden" files, and the same goes for the shell and using the -a option with ls.

While we recommend going through ALL files, it can sometimes help to first look for files with modification timestamps that occurred since you last modified your site or around the time the hack took place. If you have identified a file that was definitely modified in the hack (such as a defaced index page) you may be able to pinpoint the files used to modify the hacked file by searching for the file's timestamp in your HTTP logs via the shell.

A useful command for doing this is:

find /home/yourusername/ ! -name "log" -mtime -3

where the "-3" is the number of days in the past to look for modified files This command will list all files under /home/user/ that have been modified in the past 3 days.

Note: If the above command does not work, try without www, i.e.

find /home/yourusername/ ! -name "log" -mtime -3

Recent HTTP logs are located in the following directory:


Cleaning Up Database Hacks

Certain hacks, particularly SQL injection attacks against vulnerable Joomla! installations, may result in the database being altered with malicious code. Such a modification can allow the hacker back in even if you've updated to the latest version and cleaned off all foreign files. For this reason it's a good idea after a hack to inspect the database in the same way you check your files to see if anything has been changed that should not be. If you know when the hack occurred you may even wish to revert the database back to a prior time via the backup feature available in the panel:

"Goodies" -> "Manage MySQL" -> click "Restore DB" next to the database in question.

Restoring Lost/Modified files

Finally, to restore files that have been modified or deleted in the hack please see the article on Domain_Restore. We also offer full database and domain restores from the web panel, on the Manage Domains and Manage Mysql pages. The sooner you get to them the better, as for example we only keep mysql backups for a few days. For future reference, you may use our full account backup (under Home/Backup Your Account) feature once a month to keep your data locally, as we don't guarantee backups.