We have the new 40Gb Ethernet Adapters and here is how to configure them great performance.  Most of the Default settings are okay and provided you set the adapters to use Jumbo Frame you should be able to acheive at least 30Gb/s.

Why not 40Gb/s you ask, well extensive testing on my Power8 E880 has shown that the crosss CEC/Node latency can cause throughput speeds to drop from 40Gb/s to between 30 and 40Gb/s.

I noticed that when you DLPAR the device in to a new LPAR or VIO Server, it does not create all the required interfaces, so here are the steps for creating the 40Gb adapters after a DLPAR operation.

 

Check that you have the 40GbE RoCE Adapter.

lsdev | grep RoC

hba0             Defined     PCIe3 40GbE RoCE Converged Host Bus Adapter (b31503101410b504)

 

Configure the 40GbE RoCE Adapter.

cfgdev -dev hba0

 

Check you now have both Etherenet Ports/Adapters Configured.

lsdev | grep RoC

ent39            Available   RoCE Converged Network Adapter

ent40            Available   RoCE Converged Network Adapter

hba0             Available   PCIe3 40GbE RoCE Converged Host Bus Adapter (b31503101410b504)

 

By Default, Jumbo Frames are NOT enabled on the Adapter Ports.

lsdev -dev ent40 -attr | grep jumbo

jumbo_frames    no                 Request jumbo frames                             True

jumbo_size      9014               Requested jumbo frame size                       True

 

Enable Jumbo Frames on the 1st Port.

chdev -dev ent39 -attr jumbo_frames=yes

ent39 changed

 

Enable Jumbo Frames on the 2nd Port.

chdev -dev ent40 -attr jumbo_frames=yes

ent40 changed

 

If you have used DLPAR to add the adapter to the VIO Server or LPAR, no en or et devices are created.

You can run cfgmgr or use oem_setup_env to add them.  Here is how to do it with oem_setup_env, as it can be quicker if your VIO Server has a lot of disk devices and you use cfgdev.

oem_setup_env

/usr/lib/methods/defif

en39

en40

et39

et40

 

Again, by Default, Jumbo Frames are NOT enabled on the en devices. 

lsdev -dev en39 -attr mtu

value

1500

 

lsdev -dev en40 -attr mtu

value

1500

 

Set the adpaters IP address, netmask and increase theMTU to 9000 for best performance.

chdev -dev en39 -attr netaddr=172.10.40.11 netmask=255.255.255.0 mtu=9000 state=up

chdev -dev en40 -attr netaddr=172.20.40.11 netmask=255.255.255.0 mtu=9000 state=up