These are some of my notes from BIOS and Linux tuning for Ultra Low Latency with SubMicroTrading.
Don’t blindly copy ANY settings here, test each one for impact and pick the values that suit your system. I include them as possible points of interest.
Don’t blindly copy ANY settings here, test each one for impact and pick the values that suit your system. I include them as possible points of interest.
I have spent many
weeks simply doing this, its tedious but
necessary to determine best settings for your system.
BIOS settings
Research every
option, remember try to change one item at a time and run benchtest to
ascertain impact
Disable
hyper-threading
Disable
turbo mode if overclocking
Disable
all options related to power saving (eg CPU C State support)
Set
SATA Configuration to ENHANCED
ACPI
Power Management Features : APIC ACPI
SCI IRQ and High Precision Timer
ENABLED
If
using PCI3 NetworkCard the slot with the card HAS PCI3 enabled .. default may
be PCI2
Operating System Tuning for Ultra Low Latency
Its over 20 years
since I worked at a low level with Unix/Linux and the truth is I have forgotten
more about the kernal than I now know. I
am NOT an O/S specialist. For SubMicroTrading I didn’t have the luxury to pay
someone to configure linux for me so I had to do it myself. I still have my
trusty Stevens UNIX Network Programming book which helped and ofcause today we
have Google ! David Riddoch from
Solarflare was also helpful at answering questions regarding Solarflare and
OpenOnload tuning.
I started with
Redhat and dismissed the Realtime variant as it was slower for my benchtest, I
currently recommend CentOS 5.10 (I am somewhat behind the later versions but
honestly what do they have that helps with low latency ?).
Read the Solarflare
optimisation document, only wish it had existed when I started ! INSTALL OpenOnload !! I don’t understand why
people working in microsecond level latency still don’t use kernal bypass ! OpenOnload is great as its non intrusive and
requires ZERO application code changes.
INSTALL
SOLARFLARE (requires linux install to have dev env) … note these notes are OLD
so versions will be well out of date
copy
the SolarFlare drivers to /INSTALL/SOLARFLARE
s1)
rpmbuild --rebuild /INSTALL/SOLARFLARE/sfc-3.0.6.2199-1.src.rpm
==>
CREATES
/usr/src/redhat/RPMS/x86_64/kernel-module-sfc-RHEL5-2.6.18-194.el5-3.0.6.2199-1.x86_64.rpm
s2)
Install the RPM
rpm
-ivh
/usr/src/redhat/RPMS/x86_64/kernel-module-sfc-RHEL5-2.6.18-194.el5-3.0.6.2199-1.x86_64.rpm
==>
eth2 and eth3 are now available
==>
use rpm -e if old version around
s3)
install OpenOnLoad
tar -xvf openonload-20100923.tar
./scripts/onload_install
modprobe -r sfc
modprobe sfc
openonload now ready to use
s4)
install BIOS update tools
gunzip
SF-104451-LS-4_Solarstorm_Linux_and_VMware_ESX_Utilities_RPM.tgz
tar -xvf
SF-104451-LS-4_Solarstorm_Linux_and_VMware_ESX_Utilities_RPM.tar
==> creates ==>
sfutils-3.0.8.2216-1.rpm
rpm -ivh
/INSTALL/SOLARFLARE/sfutils-3.0.8.2216-1.rpm
(if get clash with previous version use rpm -e {oldRpm}
s5)
check is BIOS update is required and update
sfupdate
sfupdate --write
onload_tool disable_cstates persist
s6)
setup and env var with your required profile eg latency :-
export
PRERUN="onload --profile=latency "
s7)
simply add $PRERUN to the start of your application invocation command
$PRERUN
java …….
Beware O/S upgrades,
Linux 6 has some extra horrid latency which after several days tweaking I still
hadnt eradicated … I went back to Centos 5.10
Here are my notes
from CentOS / RedHat installation
Deselect
virtualisation
Disable
firewall
Disable
SELinux
Delete
SWAP partition (you don’t want swapping so ensure you have enough memory!)
Obviously disabling
SELinux and firewall is for benchtesting in controlled environment. For
colocation running you need determine an
appropriate security level for your org. If you must have a firewall
between you and the exchange then use a hardware one. Benchtest without security then with security
on so you know the impact.
Protect cores against unwanted intrusion (will be
discussed when I blog on using thread affinity)
Avoid millisecond
latency impact by using discrete threading model with core affinity via kernal
param isolcpus. The O/S wont share these
cores via the scheduler so you will need to use thread affinity to bind threads
to the protected cores (code to follow in later blog).
Edit
the /boot/grub/grub.conf
kernel
/vmlinuz-2.6.18-194.el5 ro root=LABEL=RH_ROOT
nohz=off
isolcpus=6,7,8,9,10,11 rhgb
Kernal Params
There are many !
Here are some to look at :-
transparent_hugepages=never
intel_idle.max_cstate=0
nohz=off
nosoftlockup
idle=poll
Disable unwanted services
/sbin/chkconfig
--list | grep "5:on"
chkconfig
irqbalance off
chkconfig
anacron off
chkconfig
atd off
chkconfig
avahi-daemon off
chkconfig
bluetooth off
chkconfig
cups off
chkconfig
hidd off
chkconfig
isdn off
chkconfig
pand off
chkconfig
rhnsd off
chkconfig
sendmail off
chkconfig
cpuspeed off
chkconfig
NetworkManager off
chkconfig
iptables off
chkconfig
ip6tables off
chkconfig
libvirt-guests off
These are the
services I disabled obviously you need to ensure you don’t need a service
before you disabled it.
IRQBalance and
CpuSpeed were the main services that I wished to disable … at the risk of
sounding like a broken record disable single service, bench test rinse, repeat.
Don’t disable ANY service without checking if YOU need it first !
System Scripts : rc.local
Edit /etc/rc.local
ethtool
-C eth2 adaptive-rx off
ethtool
-C eth2 rx-usecs 0 rx-frames 0 rx-usecs-high 0 rx-usecs-low 0 pkt-rate-low 0
pkt-rate-high 0
ethtool
-C eth2 rx-usecs-irq 60
ethtool
-A eth2 rx off tx off
ethtool
-C eth3 adaptive-rx off
ethtool
-C eth3 rx-usecs 0 rx-frames 0 rx-usecs-high 0 rx-usecs-low 0 pkt-rate-low 0
pkt-rate-high 0
ethtool
-C eth2 rx-usecs-irq 60
ethtool
-A eth3 rx off tx off
echo
0 > /sys/class/net/eth2/device/lro
echo
0 > /sys/class/net/eth3/device/lro
Only use rx irq
60 IF using openOnload …. You can experiment with this setting, I use
spin reading so should never require an IRQ … but I found if I set lower or
higher I could get nasty jitter.
System Scripts : sysctl.conf
Edit
/etc/sysctl.conf
kernel.sysrq
= 0
kernel.core_uses_pid
= 1
net.ipv4.tcp_low_latency=1
#
Controls the maximum size of a message, in bytes
kernel.msgmnb
= 65536
#
Controls the default maxmimum size of a mesage queue
kernel.msgmax
= 65536
#
Controls the maximum shared segment size, in bytes
kernel.shmmax
= 68719476736
#
Controls the maximum number of shared memory segments, in pages
kernel.shmall
= 4294967296
kernal.isolcpus=6,7,8,9,10,11
kernel.vsyscall64
= 2
Some other useful CentOS "stuff"
Set Run Level
change
/etc/inittab default run lvl from 5 to 3
id:3:initdefault
Network config
/etc/sysconfig/network-scripts/ifcfg-eth*
After changing run
/etc/init.d/network
restart
Label Root Partition
Label the root
partition eg to RH_ROOT … so its not
confused with any other later O/S installs
SET
DEVICE LABEL READY FOR GRUB (** DONT
FORGET UPDATE fstab OR IT WONT BOOT **)
e2label
/dev/sda7 RH_ROOT
edit /etc/fstab
LABEL=RH_ROOT / ext3 defaults 1 1
Edit the
/boot/grub/grub.conf
kernel /vmlinuz-2.6.18-194.el5 ro
root=LABEL=RH_ROOT nohz=off isolcpus=6,7,8,9,10,11 rhgb
Check Kernal CPU Params
cat
/sys/devices/system/cpu/cpuidle/current_driver
intel_idle.max_cstate=0 idle=poll
transparent_hugepage=never processor.max_cstate=0
