Advanced Troubleshooting Techniques

From DreamHost
Jump to: navigation, search

Overview

In a shared hosting environment, there are many issues that can affect the performance of your site. They can range from problems with your database, compromised code, misconfigurations in your site, and hitting memory limits which causes your processes to be killed.

To make matters worse, these problems can often cause each other. For example, a compromised site could slow down the database with junk data, causing it to take more time and memory to load, which then causes the user for the site to hit memory limits.

This article lists a few common errors along with troubleshooting solutions.

Troubleshooting a single site

There are many reasons why your site cannot load. The first thing you will need to do is identify the symptoms that you are seeing. Common symptoms observed when a site does not load include:

Each of the above issues is examined below.

Immediate 500 Internal Server Error

There are two things that can potentially cause this to happen. If the error happens instantaneously, then most likely the cause is something to do with your site's .htaccess file. Some potential things to look for:

  • Syntax errors in the .htaccess file
  • Custom PHP setup that isn't working

To see if this is the cause, try renaming the .htaccess file in your site's home directory to something like ".htaccess.disabled". If the problem is in your .htaccess file, that will immediately solve the problem.

Keep in mind that .htaccess rules apply to all subdirectories – so it's possible for an .htaccess file outside of your site's web directory to affect your site. Make sure you also look higher up in the directory structure for .htaccess files that might affect things and try renaming any you find in order to see if it helps.

If that doesn’t solve the problem, you may be having processes killed due to exceeding your user's memory limit. One quick way to see if this might be affecting you is to simply check to see which processes you have running as your user:

  1. Log in to your server via SSH.
  2. Once you're in, run the top -c command like this:
    youruser@server:~$ top -c
    top - 14:37:35 up 10 days, 17:35,  3 users,  load average: 0.83, 0.89, 1.11
    Tasks:  16 total,   1 running,  15 sleeping,   0 stopped,   0 zombie
    Cpu(s): 34.7%us,  4.8%sy,  1.7%ni, 56.5%id,  0.9%wa,  0.2%hi,  1.1%si,  0.0%st
    Mem:  32966092k total, 32546460k used,   419632k free,  6369232k buffers
    Swap:  8000328k total,   228972k used,  7771356k free, 12650516k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                  
     8384 youruser  20   0 66984  11m 6852 S    1  0.0   0:03.62 php5.cgi                                                                                                                                  
     8385 youruser  20   0 66044  10m 6700 S    0  0.0   0:00.24 php5.cgi                                                                                                                                  
    10895 youruser  20   0 65940  10m 6848 S    0  0.0   0:00.92 php5.cgi                                                                                                                                  
    10917 youruser  20   0 65980  10m 6848 S    0  0.0   0:00.79 php5.cgi                                                                                                                                  
     7542 youruser  20   0 65956  10m 6860 S    0  0.0   0:00.51 php5.cgi                                                                                                                                  
     7818 youruser  20   0 65980  10m 6860 S    0  0.0   0:00.35 php5.cgi                                                                                                                                  
     7828 youruser  20   0 65988  10m 6860 S    0  0.0   0:00.33 php5.cgi                                                                                                                                  
     7917 youruser  20   0 66016  10m 6860 S    0  0.0   0:00.43 php5.cgi                                                                                                                                  
     8152 youruser  20   0 65976  10m 6856 S    0  0.0   0:04.21 php5.cgi                                                                                                                                  
     8380 youruser  20   0 65932  10m 6848 S    0  0.0   0:04.03 php5.cgi                                                                                                                                  
     8386 youruser  20   0 66020  10m 6860 S    0  0.0   0:00.32 php5.cgi                                                                                                                                  
    10896 youruser  20   0 65908  10m 6848 S    0  0.0   0:00.66 php5.cgi                                                                                                                                  
    10919 youruser  20   0 65948  10m 6848 S    0  0.0   0:00.24 php5.cgi
    

If it looks something like the above, then you're very likely running into this problem. Generally, if you're running more than 10 PHP processes at once and they hold pretty steady, then this is an indication of memory issues.

Site spins trying to load for a while and then displays a 500 Internal Server Error

This can be caused by a few different things. One cause can sometimes be running into memory limit issues as described in the 'Immediate 500 Internal Server Error' section above, and the most common cause is PHP timing out. If this only happens on a subset of pages (in particular admin pages for the software you're using), then it's very likely that this could be the cause. By default, the PHP timeout is 30 seconds. You can find out if this is the cause by creating a custom phprc for your site and adjusting the max_execution_time setting to 2-3 times what it is now. If this happens to all of your pages, then it can still potentially be a PHP timeout. You can check your site error logs in the logs/example.com/http/error.log file inside your user's home directory to see if there are any helpful error messages. If all you see is a "Premature end of headers" error, then that is generally not too helpful as it simply means the script exited before completing.

You can read more about creating a custom phprc in the following article:

Site spins trying to load forever

This is perhaps the most generic thing that can happen with your site. Most often this means that something is causing your PHP processes to hang. If you check the top -c command on the server you might notice <defunct> showing up next to some of those processes as well. This can be caused by a large number of things.

With WordPress, this often has to do with database tables your site is using having overhead. That shouldn't cause problems, but for some reason WordPress can get itself into a bad state if there is any overhead (particularly in the wp_options table) and it will oftentimes exhibit this symptom in those cases. Other major causes of this are software misconfigurations or third-party addons to the software you're using that have a compatibility issue or poor coding.

You can read more about reducing overhead and optimizing WordPress in the following articles:

Site loads immediately, but only a blank page displays

This issue is most often related to either a theme that's being used having problems or a caching addon behaving improperly. Depending upon what exactly is going on this one can be a little tricky to solve and will likely require some fiddling to get things working properly again.

Site loads immediately, but displays a database connection error

This can happen either because the database server is unavailable or because the database connection information is incorrect. Under some conditions servers may have trouble contacting MySQL servers due to networking issues (fairly rare). In many cases this is simply because the database login information was changed without updating the connection information or the MySQL hostname isn't working properly (e.g., domain being used expired, hostname was removed from the panel, and so on).

Log in to your site via SSH or FTP. Once logged in, open your site’s configuration file. For WordPress, it is called wp-config.php, for other apps, it may just be called config.php. That file contains the settings used to connect to MySQL. Generally, the parts you want to test are:

  • Hostname
  • Username
  • Password
  • Database Name

First, test the hostname by trying to access it with a web browser. If nothing loads, then you likely have an incorrect hostname and you'll need to replace it with a functioning one. Luckily, hostnames on the same server are interchangeable, so if you have another hostname that does work on the same server, you can just use the one that works. If you have no other hostnames for that server, you will need to create one and wait for it to propagate. View the PhpMyAdmin article for further details.

If the hostname loads, you will be prompted for a username and password. If the username and password from the config file are rejected, you may need to update them with the correct information. View the PhpMyAdmin article for details on how to obtain the correct credentials in your panel.

If you are able to log in, check to see which databases are available. If the database listed in the config file does not exist, you will need to update the config file if the database’s name has changed, or restore the missing database.

Site loads immediately, but displays a 403 Forbidden Error

The 403 Forbidden error is displayed when a deny rule is set for an IP in a site's .htaccess file or when file permissions keep the web server from serving up a page. In most cases this is file-permission related. To check permissions for your site:

  1. Log in via SSH.
  2. Get a directory listing:
    youruser@server:~$ ls -la
    drwxr-x--x   16 youruser pg123456    4096 2009-12-10 04:25 ./
    ...
    

    The first line should look like the above. Notice the permission string that reads "drwxr-x--x". The first letter stands for "directory", then there are three sets of three permissions. The first set are the owner permissions, which are set to read/write/execute. The second set are the group permissions which are set to read/execute. The third set are "other" permissions (or what all other users have), which is set to execute only. If you have Enhanced Security enabled for your user it would look like this instead:

    youruser@server:~$ ls -la
    drwxr-x---   16 youruser  adm    4096 2009-12-10 04:25 ./
    ...
    

    The above are correct permission settings. If instead they look like this:

    drw-r-----   16 youruser pg123456    4096 2009-12-10 04:25 ./
    

    or this:

    d---------   16 youruser pg123456    4096 2009-12-10 04:25 ./
    

    Then that means your user has been disabled and you should contact DreamHost. If this is the case you'll likely see errors like this when attempting to log in and won't be able to get the directory listing as described above:

    Could not chdir to home directory /home/youruser: Permission denied
    -bash: /home/youruser/.bash_profile: Permission denied
    

    If you see this, contact DreamHost support for assistance.

    If permissions look fine, but you're still getting a 403, then try renaming the .htaccess file for your affected domain from ".htaccess" to ".htaccess.disabled" like this:

    mv .htaccess .htaccess.disabled
    

    Then, try loading up your site. If the 403 is gone, then open up your .htaccess file and look for lines starting with "deny". If you find any, comment them out by putting a "#" before the line and saving the file. You can re-enable the .htaccess file you disabled above like this:

    mv .htaccess.disabled .htaccess
    
    Note2 icon.png Note: If you see the site directory has an addition to the end of its directory saying that it was disabled by DreamHost, check your email for a notice from support concerning why. If you cannot find one, write in for support immediately.


Visit the following page for further details on file permissions:

Site loads, but displays a 404 Error rather than what you expected

This happens most often with sites that use software such as WordPress that use .htaccess rules for their permalinks/pretty URLs. If those rules are removed or changed somehow then a 404 will appear rather than the content you expect. The easiest way to fix this is to download a fresh copy of the software you're using from its website (e.g., http://www.joomla.org/download.html) and then copy the contents of the default .htaccess file and paste it into your existing one (keep in mind that ".htaccess" files are invisible files, so you might need to enable viewing of invisible files in your client to find the file if it's there).

Not all software comes with an .htaccess file by default. For instance, WordPress generates one when you change your permalinks settings, so keep that in mind as well. If putting in the default .htaccess rules doesn't resolve the 404 issue, write in to support and request additional help.

For WordPress specifically, this can often be fixed by just resaving your permalink structure. To do this, log into your admin panel, go to ‘Settings’ > ‘Permalinks’, and then click the save permalinks button.

Troubleshooting when all sites are affected

If the problem isn't specific to a single site but all your sites are affected, this is almost always because one of your sites is causing problems which affects all the others. In cases like this, it can be hard to know which site is causing the trouble. Here are some tips on how to proceed in those cases.

Checking for active processes

This is the first step. Many times you can tell which site is causing trouble by simply checking your active processes. Log into your server via SSH and take a look at which processes are running. Let's say you see something like this:

youruser@server.dreamhost.com:~$ top -c
top - 14:37:35 up 10 days, 17:35,  3 users,  load average: 0.83, 0.89, 1.11
Tasks:  16 total,   1 running,  15 sleeping,   0 stopped,   0 zombie
Cpu(s): 34.7%us,  4.8%sy,  1.7%ni, 56.5%id,  0.9%wa,  0.2%hi,  1.1%si,  0.0%st
Mem:  32966092k total, 32546460k used,   419632k free,  6369232k buffers
Swap:  8000328k total,   228972k used,  7771356k free, 12650516k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                  
8384 youruser  20   0 66984  11m 6852 S    1  0.0   0:03.62 php5.cgi                                                                                                                                  
8385 youruser  20   0 66044  10m 6700 S    0  0.0   0:00.24 php5.cgi                                                                                                                                  
10895 youruser  20   0 65940  10m 6848 S    0  0.0   0:00.92 php5.cgi                                                                                                                                  
10917 youruser  20   0 65980  10m 6848 S    0  0.0   0:00.79 php5.cgi                                                                                                                                  
7542 youruser  20   0 65956  10m 6860 S    0  0.0   0:00.51 php5.cgi                                                                                                                                  
7818 youruser  20   0 65980  10m 6860 S    0  0.0   0:00.35 php5.cgi                                                                                                                                  
7828 youruser  20   0 65988  10m 6860 S    0  0.0   0:00.33 php5.cgi                                                                                                                                  
7917 youruser  20   0 66016  10m 6860 S    0  0.0   0:00.43 php5.cgi                                                                                                                                  
8152 youruser  20   0 65976  10m 6856 S    0  0.0   0:04.21 php5.cgi                                                                                                                                  
8380 youruser  20   0 65932  10m 6848 S    0  0.0   0:04.03 php5.cgi                                                                                                                                  
8386 youruser  20   0 66020  10m 6860 S    0  0.0   0:00.32 php5.cgi                                                                                                                                  
10896 youruser  20   0 65908  10m 6848 S    0  0.0   0:00.66 php5.cgi                                                                                                                                  
10919 youruser  20   0 65948  10m 6848 S    0  0.0   0:00.24 php5.cgi

In the above case, you can see that there are a lot of php5.cgi processes running under your user, but if you have more than one site that's less than helpful.

Assuming you’re running a PHP site, run the following command. If you’re not using PHP, just change the ‘grep php’ section to the language you’re using:

youruser@server.dreamhost.com:~$ lsof -u youruser | grep php | grep /home

Ignore any errors you see in the response. What you’re looking for is the list of open files for your PHP processes. You can then look closer into those files to see what the issue may be.

Finding your busiest sites

Sometimes the above method doesn't work well, which means you might need to investigate your busiest sites since those will likely be the ones causing the most trouble. The easiest way to find out which sites are your busiest is by checking the access.log file for each site – the larger it is, the more traffic it's getting (they're rotated daily). To do this, log into your account via SSH and run this command from your user's home directory:

for k in `ls -S logs/*/http/access.log`; do wc -l $k | sort -r -n; done

This command prints a list of your most active sites. Visit the following article for further information:

Investigating the site

Checking the version of your software

At this point, you've identified the site in question. The first thing you always want to do is check to make sure it's running the most recent version of whatever software it's running. In WordPress' case, you would do that by running this command inside the directory in which WordPress is installed:

wp core version

If the version number you get is not the current version, you will want to upgrade. Visit the following article for more commands you can run using wp-cli:

Checking for Database Table Overhead

Many web applications seem to have trouble if your database tables develop overhead. Database table overhead shouldn't cause problems, but it’s been demonstrated in many cases that it does. In fact, with WordPress 2.9 they introduced a new option to make your site automatically check for this. To manually check, browse to the MySQL hostname you're using in your site to get to the phpMyAdmin interface. Select the database your site is using from the dropdown menu at the top left of the page and you should see something like this:

Adv troubleshooting phpmyadmin.jpg

Notice the column on the far right titled "overhead". Rows with values are tables that have overhead. Just click the "Check tables having overhead" link at the bottom (this only shows up if some tables actually have overhead), then select "Optimize table" from the dropdown to the right of that link.

Checking your addons

Next, you'll want to make sure you have some kind of caching addon installed

If you do not have a caching plugin installed, you will want to install one, as they can dramatically reduce resource usage and improve performance.

Disabling an old and potentially hacked site

If the version of WordPress is really old, it may have been hacked. View the following article to troubleshoot:

Now, check to see if your sites are loading properly and monitor top to make sure PHP processes aren't building back up again.