AIXNetworkPerformance

== How to solve network performance problems ==

Network performance is a very complicated subject, so these are just quick notes, and links to more detailed discussions.

I may update the information on this page in the future.

==== 1. Display summary information on one, or more, network card(s) ====

Use your good friend {{{ifconfig}}} for this:

{{{ andre@galactus$ ifconfig -a en0: flags=1e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN> inet 10.128.100.74 netmask 0xffffff00 broadcast 10.128.100.255 tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1 lo0: flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT> inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255 inet6 ::1/0 tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1 }}}

So far, so good. Also interesting is the {{{netstat -in}}} command, which displays (among other things) the MTU of the network cards:

{{{ andre@galactus$ netstat -in Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll en0 1500 link#2 52.bf.10.0.30.2 596504316 0 3000586619 0 0 en0 1500 10.128.100 10.128.100.74 596504316 0 3000586619 0 0 lo0 16896 link#1 3651446917 0 3593076357 0 0 lo0 16896 127 127.0.0.1 3651446917 0 3593076357 0 0 lo0 16896 ::1 3651446917 0 3593076357 0 0 }}}

Finally, you can use {{{lsattr}}} to display a bit more information on the network options of a network adapter:

{{{ andre@galactus$ lsattr -E -l en0 alias4 IPv4 Alias including Subnet Mask True alias6 IPv6 Alias including Prefix Length True arp on Address Resolution Protocol (ARP) True authority Authorized Users True broadcast Broadcast Address True mtu 1500 Maximum IP Packet Size for This Device True netaddr 10.128.100.74 Internet Address True netaddr6 IPv6 Internet Address True netmask 255.255.255.0 Subnet Mask True prefixlen Prefix Length for IPv6 Internet Address True remmtu 576 Maximum IP Packet Size for REMOTE Networks True rfc1323 Enable/Disable TCP RFC 1323 Window Scaling True security none Security Level True state up Current Interface Status True tcp_mssdflt Set TCP Maximum Segment Size True tcp_nodelay Enable/Disable TCP_NODELAY Option True tcp_recvspace Set Socket Buffer Space for Receiving True tcp_sendspace Set Socket Buffer Space for Sending True }}}

==== 2. Display detailed information on a network card ====

Use the {{{entstat}}} utility to display more information you ever wanted on a network interface. I recommend piping the result to {{{more}}} to be able to manage the results more easily.

Here is a quick example:

{{{ andre@galactus$ entstat -d en0 ------------------------------------------------------------- ETHERNET STATISTICS (en0) : Device Type: Virtual I/O Ethernet Adapter (l-lan) Hardware Address: 52:bf:10:00:30:02 Elapsed Time: 308 days 3 hours 23 minutes 43 seconds

Transmit Statistics: Receive Statistics: -------------------- ------------------- Packets: 28770714563 Packets: 17776370338 Bytes: 37790718802056 Bytes: 2635416664507 Interrupts: 0 Interrupts: 16924116843 Transmit Errors: 0 Receive Errors: 0 Packets Dropped: 0 Packets Dropped: 0 Bad Packets: 0 Max Packets on S/W Transmit Queue: 0 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0

Broadcast Packets: 24061 Broadcast Packets: 34408873 Multicast Packets: 24916478 Multicast Packets: 85733620 No Carrier Sense: 0 CRC Errors: 0 DMA Underrun: 0 DMA Overrun: 0 Lost CTS Errors: 0 Alignment Errors: 0 Max Collision Errors: 0 No Resource Errors: 0 Late Collision Errors: 0 Receive Collision Errors: 0 Deferred: 0 Packet Too Short Errors: 0 SQE Test: 0 Packet Too Long Errors: 0 Timeout Errors: 0 Packets Discarded by Adapter: 0 Single Collision Count: 0 Receiver Start Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0

General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 20000 Driver Flags: Up Broadcast Running Simplex 64BitSupport? ChecksumOffload? DataRateSet?

Virtual I/O Ethernet Adapter (l-lan) Specific Statistics: --------------------------------------------------------- RQ Length: 4481 No Copy Buffers: 0 Trunk Adapter: False Filter MCast Mode: False Filters: 255 Enabled: 1 Queued: 0 Overflow: 0 LAN State: Operational

Hypervisor Send Failures: 0 Receiver Failures: 0 Send Errors: 0

Hypervisor Receive Failures: 0

ILLAN Attributes: 0000000000003002 [0000000000002000]

Buffers Reg Alloc Min Max MaxA? LowReg? tiny 512 547 512 2048 606 502 small 512 715 512 2048 990 502 medium 128 128 128 256 128 128 large 24 24 24 64 24 24 huge 24 24 24 64 24 24 }}}

== Most common network bottlenecks ==

The most common pitfalls and network bottlenecks are the following:

Jumbo Frames are requested, for instance by TSM (Tivoli Storage Manager), but should only be used if two machines are on the same physical network.

The "Large Send" and "Checksum Offload" are actually supposed to optimize network card performance (!), but, too often can be the source of hard-to-diagnose problems. Activating both these options can be a source of performance degradation.

//**PLEASE NOTE**//

Most of the commands shown below actually modify the network cards - you can be disconnected doing this. Also, most of the commands shown **require a reboot of the machine** in order for the changes to be applied permanently. Make sure you take this into account when modifying your machines!!

==== 3. Force a card to 100 Mbps Full Duplex mode and deactivate Auto-negotiation ====

Use the following commands to force {{{en0}}} to use 100 Mbps Full Duplex mode instead of trying to auto-negotiate its transmission speed and duplex:

{{{ # ifconfig en0 down detach # chdev -l en0 -a media_speed=100_Full_Duplex # ifconfig en0 up }}}

In a slightly more detailed way:

==== 4. Changing the MTU value ====

Almost the same as the example shown above:

{{{ # ifconfig en0 down detach # chdev -l en0 -a mtu=9000 # ifconfig en0 up }}}

In the example shown above, the MTU of {{{en0}}} has been increased to 9000. The default MTU value under AIX is 1500 (and should probably stay there, unless you have a very good reason to modify it).

==== 5. Disabling jumbo frames ====

To disable jumbo frames:

{{{ # ifconfig en0 down detach # chdev -l en0 -a jumbo_frames=no # ifconfig en0 up }}}

==== 6. Disabling "Large Send" and "Checksum Offload" options ====

Here is a short script that goes over all the interface in an AIX machine and:

# Disables auto-negociation by forcing 100 Mbps Full Duplex. # Disables "Large Send" and "Checksum Offload".

Of course, this works, but needs to be adapted to your site:

{{{ #!/usr/bin/bash

# reconfigure all interfaces and reboot

# ######################################################################## # # # # AIX SPECIFIC # # # ######################################################################## # # # # This script modifies all network interfaces to optimize performance. # # # # - It forces the network interface to 100 Mbps Full Duplex # # - It removes "Large Send" and "Checksum Offload" options, # # since these can impact network performance. # # - As a precaution, it also reboots the machine at the end, # # to make sure the changes 'stick'. # # # # Make sure you have clearance when running the script since it will # # reboot the machine, this could be problematic on a production # # machine. # # # ########################################################################

chdev -l ent0 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent1 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent2 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent3 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent4 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent5 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent6 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent7 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent8 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P chdev -l ent9 -a media_speed=100_Full_Duplex -a large_send=no -a chksum_offload=no -P shutdown -Fr

# end of script }}}

Not much to say about this script, except that to note the {{{-a}}} (=attribute) option of the {{{chdev}}} command. Also, the {{{-P}}} option makes the changes permanent, so that the devices will be configured exactly in the same way the next time the machine is rebooted.

Improving this script is left as an exercice to the reader... :-)

Hope this helps!

== See Also: ==