Sunday, March 23, 2008

dstat - Analyzing Linux System Performance

Knowing the effect of your application on the system and observing system performance while your application is going thru different scenarios can give you great insight about the bottlenecks and performance ceilings in your application. Today, lets look at some of the tools on Linux that will help us understand the system performance.

dstat is a tool that shows various system performance parameters in real time.

Install dstat on your Fedora system with "yum install -y dstat" or on your Debian based system with "apt-get install dstat"

 

Here's a screenshot of it working. Click on it to see the full image.

dstat in action

Here, you can see various useful information that can tell you if system performance is throttled due to one or more of the resources being maxed out or if there is scope to improve performance by balancing the usage of the under-utilized resources.

For example, on this specific system, you can see its resources are hardly utilized. It has capacity to handle to many times it's current load. Currently, there is barely some disk activity (maybe logging) and some network activity. Interrupts and context-switches are within decent range. Most of the memory is used by kernel for caching and hardly any swap usage.

Interesting column to watch is the first 3 columns under procs. It tells us the status of the current scheduler queue. We can see new processes are being spawned. If the system is suffering from too many processes being spawned and competing for CPU, these columns can indicate that. If we see too many processes in wait state and we see high disk or network I/O, it should indicate that the system is I/O bound. Also, memory column gives us the correct split of the memory actually in use v/s the memory used by kernel for caching and buffers.

These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • Furl
  • Reddit
  • Spurl
  • StumbleUpon
  • Technorati

Saturday, March 22, 2008

htop: more than top. pstree: ps with tree.

I'm a big fan of procexp and other sysinternals tools on Windows. Many people who use those and then come to Linux complain that there are no such tools in Linux. Most people are familiar with top and ps commands. I would like to introduce you to a bunch of utilities on Linux that are very useful while debugging issues. Lets start with htop, a better alternative to top.

You can install htop on a yum based system like Fedora, Redhat, CentOS etc with "yum -y install htop" or with "apt-get install htop" on systems that are debian apt based like Ubuntu.

htop is much better in displaying the process and resource information than top. It gives a more accurate and clearer picture of process memory usage than top.

A screenshot of htop in action:

image

It has many interesting features. It's display layout is refreshing different and nice. It can show tree view, a very important feature to understand the ancestry of a process. In the size column, the MB part is highlighted with a different color. The command name in the command line is highlighted. Other user names can be shadowed. You can do multiple process selection and apply nice values or kill them. With the setup menu, you can add or remove the columns, change the color scheme etc. Lots of useful features.

Knowing process hierarchy is very useful in understanding the system. I prefer to use pstree to ps for this reason.

For example:

[ root@zen ~]# pstree -nulap
init,1
  |-udevd,539 -d
  |-dhclient,1597 -1 -q -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.pid eth0
  |-rsyslogd,1677 -m 0
  |   `-{rsyslogd},25683
  |-rklogd,1681 -x
  |-named,1696,named -u named
  |   |-{named},1697
  |   |-{named},1698
  |   `-{named},1699
  |-ntpd,1747,ntp -u ntp:ntp -p /var/run/ntpd.pid -g
  |-dovecot,1755
  |   |-dovecot-auth,1758
  |   |-pop3-login,24204,dovecot
  |   |-imap-login,24334,dovecot
  |   |-pop3-login,24414,dovecot
  |   |-imap-login,24447,dovecot
  |   |-imap-login,24466,dovecot
  |   `-imap,24468,vinay
  |-master,1817
  |   |-qmgr,1841,postfix -l -t fifo -u
  |   |-tlsmgr,1923,postfix -l -t unix -u
  |   `-pickup,24418,postfix -l -t fifo -u -o content_filter dksign:[127.0.0.1]:10027
  |-crond,1825
  |   `-crond,24450
  |       `-freshclam-sleep,24453 /usr/share/clamav/freshclam-sleep
  |           `-sleep,24455 2829
  |-atd,1832
  |-amavisd,2245,amavis
  |   |-amavisd,2247
  |   `-amavisd,2248
  |-httpd,12379
  |   |-httpd,25708,apache
  |   |-httpd,25709,apache
  |   |-httpd,25710,apache
  |   |-httpd,25711,apache
  |   |-httpd,25712,apache
  |   |-httpd,25713,apache
  |   |-httpd,25714,apache
  |   `-httpd,25715,apache
  |-mysqld_safe,12423 /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log--pid-file=/var/run/mysqld/mysqld.pid
  |   `-mysqld,12483,mysql --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking--socket=/var/lib/mysql
  |       |-{mysqld},12484
  |       |-{mysqld},12485
  |       |-{mysqld},12486
  |       |-{mysqld},12487
  |       |-{mysqld},12489
  |       |-{mysqld},12490
  |       |-{mysqld},12491
  |       |-{mysqld},12492
  |       `-{mysqld},12493
  `-memcached,12526,memcached -d -p 11211 -u memcached -m 64 -c 1024 -P /var/run/memcached/memcached.pid
      |-{memcached},12527
      |-{memcached},12528
      `-{memcached},12529

As you can see from the above tree, it is much more clearer as to which process is spawned by which process and helps us to understand the relationships better. Many times, there can be issues due to orphaned child process which are preventing the clean restart of a service. This can be easily identified with pstree.

More tools in next entry.

These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • Furl
  • Reddit
  • Spurl
  • StumbleUpon
  • Technorati