Linux Tuning for Ultra Low Latency

These are some of my notes from BIOS and Linux tuning for Ultra Low Latency with SubMicroTrading.

Don’t blindly copy ANY settings here, test each one for impact and pick the values that suit your system. I include them as possible points of interest.

I have spent many weeks simply doing  this, its tedious but necessary to determine best settings for your system.

BIOS settings

Research every option, remember try to change one item at a time and run benchtest to ascertain impact

Disable hyper-threading

Disable turbo mode if overclocking

Disable all options related to power saving (eg CPU C State support)

Set SATA Configuration to ENHANCED

ACPI Power Management Features :  APIC ACPI SCI IRQ and  High Precision Timer ENABLED                                     

If using PCI3 NetworkCard the slot with the card HAS PCI3 enabled .. default may be PCI2

Operating System Tuning for Ultra Low Latency

Its over 20 years since I worked at a low level with Unix/Linux and the truth is I have forgotten more about the kernal than I  now know. I am NOT an O/S specialist. For SubMicroTrading I didn’t have the luxury to pay someone to configure linux for me so I had to do it myself. I still have my trusty Stevens UNIX Network Programming book which helped and ofcause today we have Google !  David Riddoch from Solarflare was also helpful at answering questions regarding Solarflare and OpenOnload tuning.

I started with Redhat and dismissed the Realtime variant as it was slower for my benchtest, I currently recommend CentOS 5.10 (I am somewhat behind the later versions but honestly what do they have that helps with low latency ?).

Read the Solarflare optimisation document, only wish it had existed when I started !   INSTALL OpenOnload !! I don’t understand why people working in microsecond level latency still don’t use kernal bypass !  OpenOnload is great as its non intrusive and requires ZERO application code changes.

INSTALL SOLARFLARE (requires linux install to have dev env) … note these notes are OLD so versions will be well out of date

copy the SolarFlare drivers to /INSTALL/SOLARFLARE

s1) rpmbuild --rebuild /INSTALL/SOLARFLARE/sfc-3.0.6.2199-1.src.rpm

==> CREATES  /usr/src/redhat/RPMS/x86_64/kernel-module-sfc-RHEL5-2.6.18-194.el5-3.0.6.2199-1.x86_64.rpm

s2) Install the RPM

rpm -ivh /usr/src/redhat/RPMS/x86_64/kernel-module-sfc-RHEL5-2.6.18-194.el5-3.0.6.2199-1.x86_64.rpm

==> eth2 and eth3 are now available

==> use rpm -e if old version around

s3) install OpenOnLoad

    tar -xvf openonload-20100923.tar

    ./scripts/onload_install

    modprobe -r sfc

    modprobe sfc

    openonload now ready to use

s4) install BIOS update tools

   

    gunzip SF-104451-LS-4_Solarstorm_Linux_and_VMware_ESX_Utilities_RPM.tgz

    tar -xvf SF-104451-LS-4_Solarstorm_Linux_and_VMware_ESX_Utilities_RPM.tar

    ==> creates ==> sfutils-3.0.8.2216-1.rpm

    rpm -ivh /INSTALL/SOLARFLARE/sfutils-3.0.8.2216-1.rpm

   

    (if get clash with  previous version use rpm -e {oldRpm}

   

s5) check is BIOS update is required and update

   

    sfupdate

    sfupdate --write

   

    onload_tool disable_cstates persist

s6) setup and env var with your required profile eg latency :-

export PRERUN="onload --profile=latency "

s7) simply add $PRERUN to the start of your application invocation command

$PRERUN java …….

Beware O/S upgrades, Linux 6 has some extra horrid latency which after several days tweaking I still hadnt eradicated … I went back to Centos 5.10

Here are my notes from CentOS / RedHat installation

Deselect virtualisation

Disable firewall

Disable SELinux

Delete SWAP partition (you don’t want swapping so ensure you have enough memory!)

Obviously disabling SELinux and firewall is for benchtesting in controlled environment. For colocation running you need determine an appropriate security level for your org. If you must have a firewall between you and the exchange then use a hardware one.  Benchtest without security then with security on so you know the impact.

Protect cores against unwanted intrusion (will be discussed when I blog on using thread affinity)

Avoid millisecond latency impact by using discrete threading model with core affinity via kernal param isolcpus. The O/S wont share these cores via the scheduler so you will need to use thread affinity to bind threads to the protected cores (code to follow in later blog).

Edit the /boot/grub/grub.conf

kernel /vmlinuz-2.6.18-194.el5 ro root=LABEL=RH_ROOT    nohz=off  isolcpus=6,7,8,9,10,11    rhgb

Kernal Params

There are many ! Here are some to look at :-

transparent_hugepages=never

intel_idle.max_cstate=0

nohz=off

nosoftlockup

idle=poll

Disable unwanted services

/sbin/chkconfig --list | grep "5:on"

 

chkconfig irqbalance off

chkconfig anacron off

chkconfig atd off

chkconfig avahi-daemon off

chkconfig bluetooth off

chkconfig cups off      

chkconfig hidd off      

chkconfig isdn off      

chkconfig pand off      

chkconfig rhnsd off     

chkconfig sendmail off  

chkconfig cpuspeed off  

chkconfig NetworkManager off

chkconfig iptables off

chkconfig ip6tables off

chkconfig libvirt-guests off

These are the services I disabled obviously you need to ensure you don’t need a service before you disabled it.

IRQBalance and CpuSpeed were the main services that I wished to disable … at the risk of sounding like a broken record disable single service, bench test rinse, repeat. Don’t disable ANY service without checking if YOU need it first !

System Scripts : rc.local

Edit /etc/rc.local

ethtool -C eth2 adaptive-rx off

ethtool -C eth2 rx-usecs 0 rx-frames 0 rx-usecs-high 0 rx-usecs-low 0 pkt-rate-low 0 pkt-rate-high 0

ethtool -C eth2 rx-usecs-irq 60

ethtool -A eth2 rx off tx off

ethtool -C eth3 adaptive-rx off

ethtool -C eth3 rx-usecs 0 rx-frames 0 rx-usecs-high 0 rx-usecs-low 0 pkt-rate-low 0 pkt-rate-high 0

ethtool -C eth2 rx-usecs-irq 60

ethtool -A eth3 rx off tx off

echo 0 > /sys/class/net/eth2/device/lro

echo 0 > /sys/class/net/eth3/device/lro

Only use rx irq 60  IF using openOnload …. You can experiment with this setting, I use spin reading so should never require an IRQ … but I found if I set lower or higher I could get nasty jitter.

System Scripts : sysctl.conf

Edit /etc/sysctl.conf

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_low_latency=1

# Controls the maximum size of a message, in bytes

kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue

kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes

kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages

kernel.shmall = 4294967296

kernal.isolcpus=6,7,8,9,10,11

kernel.vsyscall64 = 2

Some other useful CentOS "stuff"

Set Run Level

change /etc/inittab default run lvl from 5 to 3   

id:3:initdefault

Network config

/etc/sysconfig/network-scripts/ifcfg-eth*

After changing run

/etc/init.d/network restart

Label Root Partition

Label the root partition eg to RH_ROOT  … so its not confused with any other later O/S installs

SET DEVICE LABEL READY FOR GRUB  (** DONT FORGET UPDATE fstab OR IT WONT BOOT **)

e2label /dev/sda7 RH_ROOT

   edit /etc/fstab

LABEL=RH_ROOT          /                       ext3    defaults        1 1

Edit the /boot/grub/grub.conf

         kernel /vmlinuz-2.6.18-194.el5 ro root=LABEL=RH_ROOT    nohz=off  isolcpus=6,7,8,9,10,11    rhgb

         

Check Kernal CPU Params

     cat /sys/devices/system/cpu/cpuidle/current_driver

    

         intel_idle.max_cstate=0   idle=poll   transparent_hugepage=never processor.max_cstate=0

Click here for my list of Ultra Low Latency Blogs and Future Topics