This is going to be short, sweet and to the point: what do you do when you have a powerful AIX server, constantly trashing and waiting on its virtual memory? Use ioo
to optimize all this, of course!
Essentially, syncd
the AIX virtual-memory daemon locks pages of memory that need to be swapped in or out of virtual memory, and prevents all access to it. This can cause applications to wait unnecessarily for the lock to be lifted before they can read or write to memory.
This can be seen very clearly when running topas
: the Wait
status consumes up to 90% of CPU time, when Kernel
or User
should be the most important consumers of CPU time:
Topas Monitor for host: galactus EVENTS/QUEUES FILE/TTY Thu Jan 28 17:48:08 2010 Interval: 2 Cswitch 2938 Readch 183.9K Syscall 8151 Writech 62459 Kernel 3.8 |## | Reads 166 Rawin 0 User 2.0 |# | Writes 92 Ttyout 162 Wait 94.2 |########################### | Forks 0 Igets 0 Idle 0.1 |# | Execs 0 Namei 126 Physc = 0.14 %Entc= 7.0 Runqueue 11.5 Dirblk 0 Waitqueue 0.0 Network KBPS I-Pack O-Pack KB-In KB-Out lo0 12.3 31.0 31.0 6.1 6.1 PAGING MEMORY en0 7.7 26.0 15.0 5.6 2.0 Faults 126 Real,MB 7680 Steals 0 % Comp 78.6 Disk Busy% KBPS TPS KB-Read KB-Writ PgspIn 2 % Noncomp 21.3 hdisk18 0.0 0.0 0.0 0.0 0.0 PgspOut 0 % Client 6.9 hdisk14 0.0 0.0 0.0 0.0 0.0 PageIn 2 hdisk7 0.0 0.0 0.0 0.0 0.0 PageOut 21 PAGING SPACE hdisk16 0.0 8.0 2.0 8.0 0.0 Sios 19 Size,MB 28128 % Used 34.3 Name PID CPU% PgSp Owner NFS (calls/sec) % Free 65.6 topas 1954020 0.5 5.5 andre ServerV2 0 oracle 3231798 0.4 4.1 oracle ClientV2 0 Press: secldapc 3199154 0.2 4.3 root ServerV3 0 "h" for help java 3215434 0.1 113.3 pamc ClientV3 0 "q" to quit
The above, slightly edited screenshot, show clearly that Wait
is consuming almost all CPU, even though the virtual memory Paging Space
is not overloaded. This happens on machines with a lot of memory, a lot of paging space, and a lot of virtual memory used.
Use the following ioo
command to ask syncd
not to lock memory pages:
# ioo -o sync_release_ilock=1
Command should be entered as root, of course, and is applied immediately. Performances can sometimes be dramatically enhanced using this simple command.