Process Accounting
From DreamHost
Process Accounting (to find and remove the cause of slowness)
Unix Process Accounting is a tool that literally takes account of every single process on your server, recording how much CPU (processor) and memory (RAM) each process consumes. When summarized, this information can be very useful for tracking down overzealous processes and hogs of resources so that you may restore your server to it's rightful speed and performance!
DreamHost keeps a daily accounting of your processes, which can be found in /var/log/sa/ (note: you may have to request DreamHost to provide you with access to those files). For example, /var/log/sa/sa.itemized.0 is an itemized report of all the processes run from yesterday, sa.itemized.1 is the itemized report for the day before yesterday, sa.itemized.2 is the itemized report for two days before yesterday. Along those same lines is sa.summary.0, which is a summary of each user's consumption of resources.
You may have the daily reports of your processes in your home directory, in the logs/resources/ directory, by enabling CPU reporting for the user. See Where can I find resource reports? for more information.
What does all of this mean? The best way to use this information is to examine the CPU-minutes. A CPU-minute can be considered a unit of work for a server. No, CPU-minutes are not "real time", that is, there are not 60minutesX24hours=1440 CPU-minutes in a day. If a machine is idle, doing nothing at all, that doesn't count towards CPU time. Only when the processor is actually doing something does it count as CPU-minutes.
So knowing how many CPU-minutes per day your users and processes are consuming can help you to find the user or process which is causing problems for the entire machine. The best way to understand all of this is to breeze through the examples below, and then try it out on your own machine.
One final confusing note: Yes, there are CPU-seconds as well: 1 CPU-minute = 60 CPU-seconds
In the examples below: "cp" = CPU-minute "cpu" = CPU-second
# Give me a summary of the 10 "biggest" users # (Note that the entire machine utilized 599 CPU-minutes for the entire day yesterday) yourserver: yourserver:$ head /var/log/sa/sa.summary.0 133573 100.00% 46089720.15re 100.00% 599.89cp 100.00% 0avio 78478595k*sec bob 84064 62.93% 618688.32re 1.34% 349.68cp 58.29% 0avio 35499807k*sec mysql 2158 1.62% 29884017.65re 64.84% 147.47cp 24.58% 0avio 413986887k*sec dhapache 9201 6.89% 14111172.13re 30.62% 87.92cp 14.66% 0avio 17167069k*sec root 35136 26.30% 860049.95re 1.87% 14.14cp 2.36% 0avio 395749k*sec postfix 1267 0.95% 612272.05re 1.33% 0.44cp 0.07% 0avio 21756k*sec mailuser1 271 0.20% 542.82re 0.00% 0.03cp 0.01% 0avio 728k*sec mailuser2 271 0.20% 612.53re 0.00% 0.03cp 0.01% 0avio 673k*sec louanne 271 0.20% 539.83re 0.00% 0.03cp 0.01% 0avio 659k*sec sarahlee 271 0.20% 610.13re 0.00% 0.03cp 0.01% 0avio 655k*sec # Give me an analysis of all the processes run by all the users. yourserver:$ /dh/misc/bin/sa.analysis.pl /var/log/sa/sa.itemized.0 | head Process CPU seconds user machine count average php.cgi 20980.6200 58.291% 87.419% 84063 0.250 mysqld 8786.3600 24.411% 36.610% 2217 3.963 libhttpd.ep 5290.4900 14.699% 22.044% 8650 0.612 ps 286.3300 0.796% 1.193% 726 0.394 sshd 266.5500 0.741% 1.111% 1459 0.183 analog 83.8400 0.233% 0.349% 204 0.411 mysqld 62.3700 0.173% 0.260% 12 5.197 analyze_log.pl 37.9200 0.105% 0.158% 5 7.584 top 26.7400 0.074% 0.111% 2 13.370 gzip 18.5000 0.051% 0.077% 85 0.218 # That was the information that you really want to look at. # Notice that for this machine, php.cgi was run a whopping # 84,000 times in one day (about once a second), consuming # about 350 CPU-minutes that day (which is a lot!). But it only # consumed on average 0.25 CPU-minutes. It looks pretty efficient, # yet even a 15% increase in efficiency would drop the amount of # resources consumed by php.cgi to under 300. # Give me a summary of all the processes run by the user `bob' yesterday: yourserver:$ grep ^bob[[:blank:]] /var/log/sa/sa.itemized.0 > /tmp/asdf yourserver:$ /dh/misc/bin/sa.analysis.pl /tmp/asdf |head -10 Process CPU seconds user machine count average php.cgi 20980.6200 99.999% 87.419% 84063 0.250 dothis.pl 0.1300 0.001% 0.001% 1 0.130 # Give me some of the raw data for the fun of it yourserver: yourserver:$ tail -10 /var/log/sa/sa.itemized.0 root 0.01 cpu 311k mem 0 io chmod dhapache 0.04 cpu 3370k mem 0 io libhttpd.ep * root 0.00 cpu 314k mem 0 io date root 0.02 cpu 514k mem 0 io savelog bob 0.23 cpu 1723k mem 0 io php.cgi bob 0.11 cpu 1723k mem 0 io php.cgi dhapache 0.01 cpu 3370k mem 0 io libhttpd.ep * bob 0.03 cpu 1719k mem 0 io php.cgi root 0.01 cpu 310k mem 0 io uname dhapache 0.02 cpu 3370k mem 0 io libhttpd.ep * root 0.00 cpu 310k mem 0 io cut
