- 1 Overview
- 2 The basics of memory usage on a DreamHost VPS
- 3 Checking your memory usage
- 4 Recovering from Memory Saturation
- 5 Simulating high traffic
- 6 How to optimize your PS
- 7 Deactivating a web server
- 8 See also
Optimizing your private server helps it to use less memory which saves you money as well as speeding up your websites. This articles describes the steps to pinpoint and mitigate the source of any memory issues on a DreamHost VPS.
The basics of memory usage on a DreamHost VPS
In a DreamHost virtual private server (PS), every PS uses roughly 100 MB of memory as a baseline without any user processes running. That's because there are a number of system processes that have to run in order for your PS to work (sshd, proftpd, and so on). It's important that you keep this in mind when allocating memory for your PS since that initial 100 MB of memory is largely unusable for your site processes.
After that initial 100 MB, most of the memory that is used by your PS consists of Apache and PHP processes (or in the case of a Rails applications, Passenger and Rails processes).
Example of how processes use memory
Whenever someone views your website, the PHP file for the page they view is parsed and executed by a PHP CGI process. At the same time, an Apache process will need to run, in order to serve the resulting HTML output from the script to the browser.
The number of these processes that run is proportional to the amount of traffic you have. For example, if you have a large influx of traffic, additional PHP and Apache processes will spawn to facilitate that traffic. In fact, Apache and PHP will continue to spawn processes as requested until your PS is completely out of memory.
You will need to be careful how you set things up to prevent such problems. This often leads to problems where websites on a PS will stop working due to memory saturation during peak hours and will then start working normally again once the traffic dies down to more manageable levels.
Raising the amount of Apache processes allowed to spawn
Another important consideration is that all PS' are configured to scale the number of Apache instances allowed to spawn with the amount of memory you allocate for it. In most cases, this works fine when your site isn't getting hit with abusive accesses. If your site is getting a significant amount of traffic, increasing the memory allocation basically allows more of that significant traffic to be served, where previously it would have been getting extremely slow load times or errors. So, until you hit the amount of memory needed to serve all of your viewers optimally, the memory usage for your site may scale upwards as you increase your memory limit.
Checking your memory usage
You can view a graph of your private server’s memory usage in your panel.
- Go to the (Panel > ‘VPS’ > ‘Dashboard’) page.
- To the right of your VPS, click the RAM graph icon under the ‘Usage’ column.
The memory usage graph on your web panel is helpful for seeing usage trends, but isn't really helpful for getting a good picture of what's actually going on in real time since these graphs are generally updated once a day around midnight PST.
The top, free, and ps aux commands
To obtain more information on exactly what's going on, you'll need to SSH into your PS.
|Tip:||Open two connections in separate windows so you can easily review the results.|
Once you're in, the primary tools you'll use are top -c, free -m, and ps aux.
Using the top command
The top command displays the current active processes, the % of the CPU they're using, how much memory, which user is running it, and so on. Once top is running, you can press '[Shift] + m' to sort processes by memory usage rather than by CPU usage.
While examining your top output, it's important to know what information you need to look at. The memory allocation you set in your panel relates to physical memory being used (as opposed to virtual memory). That memory corresponds to the RES column in top output. Below is what top output sorted by memory ('[Shift] + m') might look like (the actual processes will likely be different on your PS):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14871 exampleu 20 0 250m 14m 8340 S 5 1.3 0:04.12 php55.cgi 23591 exampleu 20 0 250m 12m 8256 S 6 1.3 0:00.36 php55.cgi 23618 exampleu 20 0 250m 13m 8256 S 4 1.3 0:00.19 php55.cgi 23684 exampleu 20 0 250m 12m 8256 S 7 1.3 0:00.10 php55.cgi 23652 exampleu 20 0 250m 13m 8256 S 6 1.3 0:00.18 php55.cgi 23654 exampleu 20 0 250m 15m 8252 S 6 1.3 0:00.17 php55.cgi
In the above example, you'll see a series of php55.cgi processes running which means the site spawning these processes is running under PHP 5.5. In the RES column, you can see how much memory each is using. In this case, they're all using between 12–15 MB of memory. On a busy PS, you'll likely have quite a few Apache processes running. Those average around 9-11 MB of memory each. Apache and PHP processes multiply as your sites start serving more requests, so you can see that memory usage can quite easily skyrocket if you begin to receive a lot of traffic.
Using the free command
In your other window, you can run free -m. This command displays the current memory usage on your PS and should look something like this:
total used free shared buffers cached Mem: 4049 3941 107 0 123 1639 -/+ buffers/cache: 2178 1870 Swap: 6165 42 6122
The -m option tells the command to display the memory usage in megabytes. In the above example, the total available memory is roughly 4GB, 2.1GB is used, and 1.8GB is free. The total you see should correspond to the amount of memory you allocated to your PS in your web panel.
For more details about what memory usage looks like, visit the following:
Using the ps aux command
The ps command supplies you with similar information as top, but simply takes a snapshot of the active processes and their usage and displays it on your screen. This is useful if you only want to see a particular process(es). For instance, if you only want to see running Apache processes, run the following command:
ps aux | grep apache
That will "pipe" (or pass) the output from ps to the grep command which will filter that output on a per line basis looking for the string "apache" in it. Any line with that output is displayed.
Another useful variation is:
ps aux | grep php
This allows you to observe all the PHP processes that are running.
Recovering from Memory Saturation
Memory saturation is when your PS is using up its entire allotment of memory. When this happens, a variety of things will start to happen. Your sites will likely stop responding in many cases, but beyond that other essential processes on your PS will stop responding as well such as the SSH server, FTP server, streaming media server, and so. It's entirely possible for your PS to enter a state where you can't even log in due to memory saturation. You can usually tell if this is happening to you by checking a few things:
- Check the VPS Dashboard. View the ‘Checking your memory usage’ above for details on how to view your usage graph.
- If the resource usage graph is showing your memory spiking high above the memory allocated, then this is likely what's happening. If you can still log in to your PS via SSH, then try running the free -m command. This will show you how much free memory you have available. The closer that is to zero, the worse you are. At zero, you've reached total memory saturation, and what happens next can be unpredictable.
- Once you've determined that there is no free memory available, the fastest way to resolve it is to temporarily increase the memory allocation for your PS until you have some breathing room (unutilized and freely available memory). At that point, you can either leave the memory allocation there until you optimize your site and server and get your memory usage under control. Otherwise, you can start disabling your active sites until your memory gets into a range you are more comfortable with. This gives you room to optimize without having to spend more money on memory.
- One easy way to disable a site is to simply rename its web directory. For example, if you had a domain titled example.com, you could rename its web directory from example.com to example.com_disabled. By doing this, Apache will start serving 404 Not Found error pages for that domain. While this will still require memory, it will keep new PHP processes from spawning and will likely reduce traffic significantly to the point where your server is no longer overloaded.
Simulating high traffic
If your problem has to do with your memory usage spiking at certain times of the day, it can be difficult to figure out what's happening since things are often running well when you're able to check on them. When that's the case, it's important to be able to simulate the environment that's causing the problem.
A benchmark tool you can use to simulate traffic is the ‘ab’ command. View the following link for further details:
To use, enter the following command:
ab -n 1000 -c 20 http://www.example.com/index.html
This command attempts to create 1000 connections to http://www.example.com/index.html limiting itself to 20 concurrent connections.
Viewing your processes while testing
To observe what's going on, you'll likely want three terminal windows open.
- In the first window, SSH to your server and run top -c. Then, sort by memory (SHIFT + m).
- In the second window, run free -m (or, run watch free -m).
- In the third window, run the ‘ab’ command to simulate traffic. This is effective for profiling your sites and checking to see if caching is working, and you can run this for each of your domains and see which ones affect your memory usage the most so you know where to optimize.
How to optimize your PS
Now that you have a basic idea of what's going on with your PS from the above information, you can start working on optimizing various components to reduce your overall memory usage.
Blocking abusive access
Common problems for sites are caused by search engine indexing bots or abusive viewers hammering them. When search engine bots hit a site, they usually hit it pretty hard. To minimize this issue, you should definitely set up a Robots.txt file. After that, it's possible that someone can crawl your site, systematically downloading every page, image, and file available. This could especially be the case if your site contains media (video, images, audio, and so on). Finding abusive accesses like this can often be more of an art than a science, but you can get some data on this by analyzing the access.log file for your websites.
View the following wiki for detailed information on how to check your access.log file and see who is hitting it the most:
Hotlinking is when a site other than your own includes an image in their site that is actually hosted on your server. They do this by directly referencing the image's URL from your site in their own. This can cause massive problems for you if some of your hosted images are hotlinked in another popular site. Even a relatively low traffic site can cause poor server performance if this is the case. To prevent this from happening, you can review the Preventing hotlinking article.
Caching your site
As you've probably seen by now, PHP processes can take up a significant amount of memory. If your site is getting hammered with traffic, the fewer hits that have to spawn a PHP CGI process, the better. This can be accomplished by enabling caching on your site. Essentially, caching renders a PHP script and then "caches" that HTML output as a file. The next time that page is accessed, the server outputs the static HTML page, rather than process the PHP script again. The task of serving a static file requires far less memory and generally serves more efficiently, than the processing and execution of a PHP script. So if you're having memory trouble on your PS, it's essential that you confirm that you're running software that has caching functionality and that you enable the use of this feature.
Also, if you expect a large volume of website visitors, you can enable CloudFlare. This is a service that DreamHost partners with to add additional caching options for your websites.
Enabling FastCGI and XCache
Another improvement you can make in order to reduce memory usage is to turn on FastCGI and XCache for your domains that run PHP-based sites. Without FastCGI enabled, an entire PHP binary process has to be executed individually for each PHP page that's viewed in order to parse and process it. This can add up dramatically as shown earlier in the top command section above.
Enabling FastCGI for all your PHP domains switches your sites over to using mod_fcgid (see Fcgid php). What this means is that when a PHP script is hit by a browser, the PHP interpreter process begins, processes the script, and then stays in memory, waiting for another request. When a new request hits, it is sent to this running interpreter for processing, rather than spawn a new one. If you have a lot of sites or a lot of traffic, this can greatly reduce your overall memory usage.
Enabling this also allows you to enable XCache which essentially caches compiled PHP code in memory to output as PHP scripts are requested. When configured properly, this can potentially increase the rate of page generation (speed) by up to five times. For more information on configuring this after it's enabled, view the XCache wiki page. If you'd like additional information about XCache, the introduction page of the official XCache wiki is a good place to start.
To enable these features:
- Navigate to the (Panel > ‘Domains’ > ‘Manage Domains’) page.
- To the right of your domain, click the Edit button under the ‘Web Hosting’ column.
- Scroll down to the ‘Web Options’ section and enter the following:
- Select which version of PHP you’d like to use in the dropdown menu.
- Check the box next to ‘PHP XCache Support’.
- Scroll down and click the Change settings button to save your changes.
- There is also an option on your PS configuration page (Panel > ‘VPS’ > ‘Dashboard’) to enable PHP cache for the entire VPS.
- Click the Configure button to the right of your server to view this option.
- Make the selections, and then click the Save settings... button to save your changes.
|Note:||If all domains are set to use FCGI or CGI for the PHP mode, then you may safely deactivate mod_php in the PS configuration to save a good deal of memory. All modern PHP applications will run fine exclusively on fcgi.|
If you're running multiple instances of the WordPress software on your PS, you should definitely review the Fine Tuning Your WordPress Install article.
Make sure to do the following with each installation:
- Always run the most recent version of WordPress.
- Install only essential plugins.
- Keep your plugins up-to-date.
- Remove/delete unused plugins as opposed to just deactivating them.
- Confirm your active theme isn't generating any errors in your error.log file (located in the same directory as the access.log).
- View Optimizing Page Load Time and make sure you have a popular WordPress caching plugin installed and configured. For example, W3TC or WP Super Cache.
- Please keep in mind that running a third-party WordPress caching plugin simultaneously with your panel’s XCache option can actually increase overall memory usage. It's best to pick either one or the other for your caching needs.
- Another useful tool to detect memory intensive third-party plugins is P3 (Plugin Performance Profiler) plugin for WordPress. Make sure to disable this plugin if you're not currently using it as it can increase your overall memory usage.
- Finally, compressing your image files improves loading time and reduces overall memory usage.
Deactivating a web server
If you do not need any DreamHost-supplied web server to run (Apache, nginx, and so on), you can disable it in your panel:
- Navigate to the (Panel > ‘VPS’ > ‘Dashboard’) page.
- Click the Configure button to the right of your VPS.
- From the ‘HTTP Server’ dropdown menu you’ll see the following:
- No web server
- From the ‘HTTP Server’ dropdown menu you’ll see the following:
- Select the ‘No web server’ option, and then save the changes.