AIXOptimizePagingWithIOO

Optimizing AIX paging with ioo

This is going to be short, sweet and to the point: what do you do when you have a powerful AIX server, constantly trashing and waiting on its virtual memory? Use ioo to optimize all this, of course!

Essentially, syncd the AIX virtual-memory daemon locks pages of memory that need to be swapped in or out of virtual memory, and prevents all access to it. This can cause applications to wait unnecessarily for the lock to be lifted before they can read or write to memory.

This can be seen very clearly when running topas: the Wait status consumes up to 90% of CPU time, when Kernel or User should be the most important consumers of CPU time:

Topas Monitor for host:    galactus             EVENTS/QUEUES    FILE/TTY
Thu Jan 28 17:48:08 2010   Interval:  2         Cswitch    2938  Readch   183.9K
                                                Syscall    8151  Writech   62459
Kernel    3.8   |##                          |  Reads       166  Rawin         0
User      2.0   |#                           |  Writes       92  Ttyout      162
Wait     94.2   |########################### |  Forks         0  Igets         0
Idle      0.1   |#                           |  Execs         0  Namei       126
Physc =  0.14                     %Entc=   7.0  Runqueue   11.5  Dirblk        0
                                                Waitqueue   0.0
Network  KBPS   I-Pack  O-Pack   KB-In  KB-Out
lo0      12.3     31.0    31.0     6.1     6.1  PAGING           MEMORY
en0       7.7     26.0    15.0     5.6     2.0  Faults      126  Real,MB    7680
                                                Steals        0  % Comp     78.6
Disk    Busy%     KBPS     TPS KB-Read KB-Writ  PgspIn        2  % Noncomp  21.3
hdisk18   0.0      0.0     0.0     0.0     0.0  PgspOut       0  % Client    6.9
hdisk14   0.0      0.0     0.0     0.0     0.0  PageIn        2
hdisk7    0.0      0.0     0.0     0.0     0.0  PageOut      21  PAGING SPACE
hdisk16   0.0      8.0     2.0     8.0     0.0  Sios         19  Size,MB   28128
                                                                 % Used     34.3
Name            PID  CPU%  PgSp Owner           NFS (calls/sec)  % Free     65.6
topas       1954020   0.5   5.5 andre           ServerV2       0
oracle      3231798   0.4   4.1 oracle          ClientV2       0   Press:
secldapc    3199154   0.2   4.3 root            ServerV3       0   "h" for help
java        3215434   0.1 113.3 pamc            ClientV3       0   "q" to quit

The above, slightly edited screenshot, show clearly that Wait is consuming almost all CPU, even though the virtual memory Paging Space is not overloaded. This happens on machines with a lot of memory, a lot of paging space, and a lot of virtual memory used.

Use the following ioo command to ask syncd not to lock memory pages:

# ioo -o sync_release_ilock=1

Command should be entered as root, of course, and is applied immediately. Performances can sometimes be dramatically enhanced using this simple command.

See Also: