tags:
Contents
1. What are jumbo frames?
Some network devices allow the standard size of an Ethernet packet to be changed. By default each network packet can carry 1500 bytes of data (also referred to as the packet’s payload). Any payload larger than 1500 bytes sent over the network will be split into more than one packet. If we enable jumbo frames we reduce the number of packets sent over the network when sending large amounts of data.
The most common application of jumbo frames is on storage networks such as iSCSI, NFS, and vMotion networks. These networks are generally closed and device configuration can be governed.
When we enable “jumbo frames” we are telling our network devices that we want to send more th an 1500 bytes in each packet. Most commonly, jumbo frames means setting the MTU (maximum transmission unit) to enable a payload of 9000 bytes.
To do this two things are required. First, devices that will talk to each other need to their MTU set to 9000.
Second, all switches and routers between the devices must be able to process these packets without fragmenting them (see below). When setting a server’s MTU to 9000 we are telling that server that each packet it sends on the network can carry 9000 bytes of data. What that means in practice, is that routers and switches must actually have ports set to an MTU that is greater than 9000 (packet header + packet data). Check the manufacturer’s recommendation when changes these settings, but on a router or switch 9216 is a common MTU size that accommodates an Ethernet packet with a 9000 byte payload. The additional 216 bytes are used for the Ethernet header.
2. Why use jumbo frames?
The obvious advantage of using jumbo frames is more data is transferred in less packets. This means less protocol overhead, e.g., if one ACK were sent every 8 packets, that ACK is now acknowledging 72,000 bytes of data vs 12,000 bytes of data. This results in less network chatter.
Let’s assume we need to transfer 20 gigabytes (21,474,836,480 bytes) of data as quickly as possible. With a standard 1500 byte MTU that will take 14,316,558 packets, but with an MTU of 9000 we are sending 2,386,093 packets. That’s a difference of 11,930,465 packets.
That’s our advantage. Speed when sending large amounts of data.
3. What are the drawbacks of jumbo frames?
The biggest drawback when considering jumbo frames is that they not defined by any Ethernet standard. Because of this each manufacturer plays by their own rules when implementing large frames. This has the potential to cause hard to troubleshoot issues.
Another major disadvantage to when using jumbo frames is that misconfigured devices usually only cause intermittent issues. It’s generally not recommended to enable large frames unless your organization have the ability to configure every device on the network.
Fragmentation. When using jumbo frames routers and switches can also get bogged down fragmenting packets of different sizes. If one device wants a 1500 byte payload but you’re sending 9000 bytes the packet will either get dropped or broken down into several smaller packets. This causes strain on the devices CPUs.
4. What is the best way to test jumbo frames?
After enabling jumbo frames it’s important to test connectivity between all devices. A tell-tale sign of jumbo frame issues is intermittent problems. Pings seem to work. Sometimes data and connections work perfectly. Sometimes the network is very very slow or doesn’t work at all.
The easiest way to test is using the ping command. But, the standard ping 10.0.0.1 won’t test jumbo frames. By default the ping command on Linux will send 56 bytes of data plus 8 bytes of header data for a total of 64 bytes.
Because a 64 byte packet easily fits inside a 1500 byte packet if you attempt to verify connectivity after enabling jumbo frames with a normal ping command you will get a response even if your network is not properly configured.
To properly test jumbo frames between devices you must specify a packet size larger than 1500.
But wait! There’s more!! We also have to tell our network not to fragment the packets or we’ll get a response even though our frame size is not correctly configured.
Windows Ping
ping -l 8000 -f 10.0.0.1
-l 8000tells ping to send an 8000 byte payload.-ftells ping to set the don’t fragment bit (DF).
Linux Ping
ping -s 8000 -M do 10.0.0.1
-s 8000tells ping to send an 8000 byte payload.-M dotells ping to set the don’t fragment bit (DF).
VMWare Ping
From ESXi you will probably want to use vmkping. This pings from a vmkernel port which is generally where you’ll have jumbo frames configured for iSCSI, vMotion, or NFS. On ESXi ping and vmkping use the same options.
vmkping -I vmk3 -s 8000 -d 10.0.0.1
-I vmk3tells vmkping to use the interface vmk3 as the source of the packets.-s 8000tells vmkping to send an 8000 byte payload.-dtells vmkping to set the don’t fragment bit (DF).
Make sure to your pings are going out the correct interface when you do your testing. Virtual machine hosts, switches, and routers all have multiple ports, networks, and VLANs configured. Be sure your tests are designed with every detail in mind.
Other systems such as FreeBSD probably have different command switches to enable the don’t fragment bit and specify the packet size. Be sure to know these options on whatever devices need tested.
Conclusion and Links
The highlights of this article are the basic principals of jumbo frames. They are most useful in situation where large amounts of data are being transferred and speed is important such as storage networks and vMotion networks. Small packet traffic like DNS queries do not benefit at all from larger MTU sizes.
It is easy to miss a device or configure devices incorrectly for jumbo frames. Generally, switches and routers should have an MTU of 9216 while nodes have the MTU set to 9000 bytes.
When devices are configured incorrectly problems will usually be intermittent. When a large packet is sent over a network with an MTU set lower than the packet size the packet will be fragmented into smaller packets unless the don’t fragment bit is set. If a packet is set to don’t fragment and it hits a device with a smaller MTU that packet will be discarded.
Different systems have different options on their ping commands to send large packets and set the don’t fragment bit. Be aware of these options when testing, and be sure you are testing from the correct network interfaces.
More information about jumbo frames can be found at the links below.