Disk Usage Warning

Introduction

DreamHost enforces various disk related limitations (quotas) for user accounts on its servers. From time to time you (or the owner/administrator of the account) may receive "Disk Usage Warning" e-mail notices when a user account nears, reaches or exceeds those limits. They may also forward those messages on to the actual users so you know what's going on and can take the appropriate action to resolve the situation. This Wiki article should help you to understand what's going on and instruct you on how to resolve the situation.

You'll notice that there are many important notes in this Wiki article. Please pay close attention to them. That's usually where the most important information is!

Common Disk Related Error Messages

If you receive any of these common error messages, the most likely cause is that you've reached/exceeded one of your disk limits for the particular account in question. If that's the case then this article should provide information and assistance in dealing with the situation.

• "Disk quota exceeded"
• "IMAP connection dropped"
• "insufficient disk space"
• "Unable to create file..."
• "Unable to delete file..."
• "touch: cannot touch..."
• "disk quota reached"

Terminology

Before I continue some terminology must be clarified. The generic term "account" needs to be defined properly so our usage can be clearly understood.

• The term "user account", "FTP/shell" or "e-mail account" refers to an account with which you would log into one of our servers.
• The term "account" or more accurately "sub-account" usually refers to the entire "account" you have under your control via our control panel. When you log into the control panel and go to (BILLING > MANAGE ACCOUNTS) in the menu system you'll see your account information listed there. You'll see the account number and its name listed in large blue text near the top of that section. Note that you can change the name of the account by clicking on the "rename account" link. If you have several "sub-accounts" under your control under that WebID you'll see several more listed in the lower section of the menu column. You can select which one you want to administer by making it the "Active Account". If you do have access to more than one sub-account via your WebID, please be very careful to make sure you have the proper one set as the "Active Account" before making any changes!
• The term "account" in general is often used when the term "WebID" should be used. This is probably the most commonly abused term withing the DreamHost world! Your WebID is what you use to log into the control panel. We've simplified things a bit by allowing you to log into the control panel using the primary contact e-mail address defined for your WebID. To find out what your actual WebID is click on the "Edit profile" link at the top right corner of the control panel, at the top of the next page you'll see the phrase "Welcome, WebID" (next to the "Logout" button). That's your actual WebID there. It never changes. However you can change the primary contact e-mail address if you want.

Disk Limits (quotas)

There are basically two different types of disk related limits/quotas that we enforce for FTP/shell and mailbox accounts; size and files.

Size

• Size: the amount of disk space (usually in MB or GB) that can be used by a particular account.

In order to have user accounts, host domains, e-mail addresses, databases, etc. under your WebID you must have an active "Hosting Plan". You'll see the various hosting plans and other add-ons you have under your sub-account in the control panel under (BILLING > MANAGE ACCOUNTS) in the "Hosting Plans" section. A "Hosting Plan" provides a specific disk space allocation which can be used for your user accounts, e-mail accounts, databases, etc.. The resources that your hosting plan(s) provide is listed under the "Your Account Provides:" section (which is below the account number and name).

The amount of disk space that your hosting plan(s) provides is shared between all of the user accounts, e-mail accounts, databases, etc. under that particular sub-account.

Whenever you create a new user account (FTP/shell or e-mail) by default it has no disk limit set! If you intend to limit a particular user accounts disk usage I'd recommend that you set a specific limit when the account is created. You can also change/remove the disk limit at any time via the control panel under (USERS > MANAGE USERS) or (MAIL > MANAGE E-MAIL).

NOTE: It is recommended that you never set an e-mail account with a disk limit below 100MB! Most e-mail clients require some free space in order to operate (even to delete messages). If an e-mail account reaches/exceeds its disk limit it will give errors (ie; "IMAP connection dropped") when this happens. It will also bounce incoming messages! If that happens you'll need to increase/remove the disk limit in order to get it working again (information on how to do that will be provided at the end of this article). Remember that our mail servers allow incoming messages as large as 40MB. A couple of large messages could fill your account very quickly if its has a very low disk limit! For more information on incoming message limitations see this Wiki article.

NOTE: When changing/removing disk limits (quotas) it can sometimes take up to 24 hours for the changes to be pushed to the live servers! If you have run into a problem by an account reaching/exceeding its disk limit, and you've made a change via the control panel, you may want to go to (SUPPORT > CONTACT SUPPORT) and submit a support request asking if they can expedite the change. Sometimes we can and sometimes we can't (due to various technical issues). But it can't hurt to ask! Don't forget to specify which e-mail account you're referring to!

NOTE: If a user account has no disk limit set that means that it could potentially use all of the available disk space allocated to that sub-account (or more). The overall disk allocation provided by your hosting plan is a soft limit. If you exceed that limit the sub-account will be charged overage fees according to the schedule listed under (BILLING > MANAGE ACCOUNTS).

NOTE: User accounts that have a disk limit set that reach/exceed their disk limits (quotas) will no longer be able to modify or upload files and may have other problems as well. Additionally, e-mail accounts with bounce incoming messages and may not be able to delete messages! More information will be provided regarding e-mail account below in this article.

NOTE: E-mail accounts that use a FTP/shell account for their mailbox require special attention! If you set a disk limit that will apply to BOTH the E-mail account and the FTP/shell account!!! This means if you're hosting your domain(s) under this account their disk usage will be combined with the disk usage of the mailbox. If you reach/exceed the disk limit it could cause problems for the operation of your domain(s) as well as your e-mail address.

NOTE: Disk limits (quotas) are assigned and enforced on a user account by user account basis -- meaning each user account can have an independent disk limit. Please see the previous note for additional information regarding e-mail accounts that use a FTP/shell account for their mailbox!

Files

• Files: the number of files that can be used by a particular account.

The files limit is a relatively new limit that we had not been enforcing until recently (July 2008). We've found that when users have what we consider to be too many files under their account it tends to case severe performance problems for all customers on that particular file server.

NOTE: The files limit is NOT something you can set via the control panel like you can with disk space limits.

NOTE: The actual disk file limit varies with the disk limit set for that particular user account. The specific number is not cast in stone and DreamHost reserves the right to make changes to this at any time as may be necessary to maintain the stability and performance of our servers.

If you receive a "Disk Usage Warning" e-mail that indicates that your user account is near, reached or exceeded the file limit we recommend that you review your directory structure and see if there aren't any files/directories that could be deleted.

In order to determine which directories contain the most files you might want to execute this command string from the shell in your users home directory (I would recommend using copy/paste for this);

for i in `find . -type d | sed 's/ /_space_/g'`; do name=`echo $i | sed 's/_space_/ /g'`; fc=`find "$name" | wc -l`; echo $fc $name ; done | sort -n > disk_usage_report

NOTE: This command string is very long! Make sure that you select all of it when using copy/paste! The end part of the string is "disk_usage_report".

NOTE: This command string may take quite a while to run if you have many directories and files! It will not output any thing until it's completed so you have to be very patient. But the output is sorted so the largest directories will be listed last. It will output the data into the file called "disk_usage_report" which you can review with a text editor.

If you want to find out which are the largest files within a directory you can execute this command;

ls -lrS | tail

Unfortunately if your user account is configured for FTP access you won't be able to execute these commands. You may want to enable it for shell access temporarily to execute these commands. See this Wiki article for more information on how to enable shell access.

You may need to contact support for assistance if you've reached/exceeded the file limit for your user account. Our admins may need to investigate to see what's going on under your user account. It could be an indication that there's a problem with something running under that user account or there could be innumerable cron messages being sent to your mailbox that you're not aware of. Depending on the situation the admins may increase the file limit for that particular user account, but it is solely at their discretion.

Viewing Your Disk Quota Status

You can view the disk usage status for your user accounts several different ways;

Option #1 Via the control panel under (STATUS > DISK USAGE).

NOTE: The control panel stats only update once per night so any changes you make during the day won't be reflected in the control panel until the next day!

NOTE: The control panel status shows both the disk usage (size and number of files). It shows the current disk limit that the control panel has configured, but that may not necessarily reflect that the servers are actually using. Those types of changes often take up to 24 hours to be pushed to the live servers.

NOTE: The control panel does not show the files limit. That information can only be obtained by executing the "quota" command from the shell. More information on how to do that follows.

Option #2 If you have shell access enabled for a particular user account you can execute the "quota" command to see the current quota status for that user account.

Unfortunately if your user account is configured for FTP access you won't be able to execute these commands. You may want to enable it for shell access temporarily to execute these commands. See this Wiki article for more information on how to enable shell access.

NOTE: Sometimes the output can come out scrambled in your terminal window! You may have to "reset" it afterwards to get things working again. If the output comes out scrambled you might try to copy/paste it into a text editor which may convert it so you can read it better. But the numbers are the most important part, they almost always come out readable.

First you need to know which "filesystem" your data is mounted on. If you execute the command "pwd -P" it will show you the hidden directory where YOUR data is actually mounted.

$ pwd -P
/home/.delmonte/sampleuser

In this example the hidden directory is ".delmonte" so that's the output line we're interested in.

Here's an example of what the "quota" command output looks like;

$ quota 
quota: ’¡@/: error while getting quota from 10.3.41.11:/vol/navyblue/spunky/fxdwgcb for 593723: Connection refused
quota: ’¡@/: error while getting quota from 10.3.41.11:/vol/navyblue.narc/spunky/fxdwgcb for 593723: Connection refused
Disk quotas for user sampleuser (uid 999999): 
    Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
10.174.0.114:/vol/boot/spunky/cascarilla
               17344712  785592320 785592320          151947  512000  512000        
10.174.0.114:/vol/boot/spunky/delmonte
               17344712  785592320 785592320          151947  512000  512000        
10.174.0.133:/vol/boot/spunky.crontabs/alameda
                     4  17179869180 17179869180               1  4294967295 4294967295

You can usually ignore the errors (like the 2 listed first in this output).

This is the "filesystem" at we're interested in (as determined by the "pwd -P" command). You can ignore the other filesystems that may be listed by the "quota" command.

10.174.0.114:/vol/boot/spunky/delmonte
               17344712  785592320 785592320          151947  512000  512000

Let's first look at the disk space portion of this output (which is the 3 leftmost set of numbers).

This output indicates that this user is currently using 17,344,712 blocks (approximately 17GB) of disk space. As they do NOT have a specific disk limit set for this user account the total available disk space for their sub-account 785,592,320 blocks (approximately 749GB) is available to them to use. The number is duplicated because the "limit" and "grace" amounts are the same.

Now let's look at the number of files portion of the output (which is the 3 rightmost sets of numbers).

The output shows that this account is currently using 151,947 files of its current 512,000 limit. Again the limit number is duplicated because the "limit" and "grace" amounts are the same.

NOTE: If you ever see and asterisk (*) next to one of those amounts that indicates that it's at or above the limit. That indicates that you'll have problems with this user account until the situation is remedied.

E-mail Accounts

When e-mail accounts reach/exceed their disk limits (size or number of files) they usually stop working and give various errors when you access the mailbox (ie: "IMAP connection dropped") and bounce incoming messages.

The first step to getting an e-mail account working again is to increase/remove its disk limit. Instructions are provided below.

Once the disk limit issue has been resolved you'll want to see what caused the account to reach/exceed its disk limit in the first place.

Three of the most common problems are;

• You may have the Inbox Archiver configured to move messages from your inbox to the "old-messages" folder. Checkout its Wiki article for more information on how to deal with that issue. There may be so many messages in this folder that you may not be able to index it to find out how many are in there. If this is the case please contact support for assistance.
• The "Trash" folder is not emptied or purged. Disk space is NOT necessarily recovered when messages are deleted. Messages are often moved to the "Trash" folder (or whatever folder your e-mail client is configured to use for this purpose) so you can recover them if necessary.
• You may have one or more Cron jobs that are sending informative messages to this account every minute of the day. Though those messages are usually small they can easily overwhelm your mailbox! You might want to review your Cron jobs and possibly reconfigure them to stop e-mailing.

NOTE: Unless your e-mail account is using a FTP/shell account for its mailbox you won't be able to determine its actual files limit. However if you contact support we can find out for you and let you know what it is.

How to Change or Remove Disk Limits

For "FTP/shell" user accounts log into the control panel and go to (USERS > MANAGE USERS) and click on the "Edit" link for the user account in question.

For "E-mail" accounts log into the control panel and go to (MAIL > MANAGE E-MAIL) and click on the "Edit" link for the address in question.

• Either increase the "Limit total disk usage" or un-check that option.
  • If you increase the disk limit make sure that the "Email me daily warnings" option is checked so they are notified!
• Click on the "Save changes" button to push the changes into place. NOTE: It can often take up to 24 hours for these types of changes to be pushed to the live servers!
• Contact support by going to (SUPPORT > CONTACT SUPPORT) and ask them if they can expedite the disk quota change manually. Don't forget to specify which user account or e-mail account you've just changed!