Why mtu 9000?
In computer networking, jumbo frames are Ethernet frames with more than 1500 bytes of payload, the limit set by the IEEE 802.3 standard. The payload limit for jumbo frames is variable: while 9000 bytes is the most commonly used limit, smaller and larger limits exist. Many Gigabit Ethernet switches and Gigabit Ethernet network interface controllers and some Fast Ethernet switches and Fast Ethernet network interface cards can support jumbo frames.
Each Ethernet frame must be processed as it passes through the network. Processing the contents of a single large frame is preferable to processing the same content broken up into smaller frames, as this makes better use of available CPU time by reducing interrupts. This also minimizes the overhead byte count and reduces the number of frames needing to be processed. This is analogous to physically mailing a packet of papers instead of several single envelopes with one sheet each, saving envelopes and cutting sorting time.
Jumbo frames gained initial prominence in 1998, when Alteon WebSystems introduced them in their ACEnic Gigabit Ethernet adapters. Many other vendors also adopted the size; however, jumbo frames are not part of the official IEEE 802.3 Ethernet standard.
Jumbo frames have the potential to reduce overheads and CPU cycles and have a positive effect on end-to-end TCP performance. The presence of jumbo frames may have an adverse effect on network latency, especially on low-bandwidth links. The frame size used by an end-to-end connection is typically limited by the lowest frame size in intermediate links. 802.5 Token Ring can support frames with a 4464-byte MTU, FDDI can transport 4352-byte, ATM 9180-byte and 802.11 can transport 7935-byte MTUs. The IEEE 802.3 Ethernet standard originally mandated support for 1500-byte MTU frames, 1518 byte total frame size (1522 byte with the optional IEEE 802.1Q VLAN/QoS tag). The IEEE 802.3as update grandfathered in multiple common headers, trailers, and encapsulations by creating the concept of an envelope where up to 482 bytes of header and trailer could be included, and the largest IEEE 802.3 supported Ethernet frame became 2000 bytes.
The use of 9000 bytes as preferred payload size for jumbo frames arose from discussions within the Joint Engineering Team of Internet2 and the U.S. federal government networks. Their recommendation has been adopted by all other national research and education networks. Manufacturers have in turn adopted 9000 bytes as the conventional MTU size, with a total jumbo frame size of between 9014 and 9022 bytes with ethernet headers included. Most Ethernet equipment can support jumbo frames up to 9216 bytes.
IEEE 802.1AB-2009 and IEEE 802.3bc-2009 added LLDP discovery to standard Ethernet for maximum frame length (TLV subtype 4). It allows frame length detection on a port by a two-octet field. As of IEEE 802.3-2015, allowed values are 1518 (only basic frames), 1522 (802.1Q-tagged frames), and 2000 (multi-tagged, envelope frames).
Errors in jumbo frames are more likely to go undetected by the simple CRC32 error detection of Ethernet and the simple additive checksums of UDP and TCP: as packet size increases, it becomes more likely that multiple errors cancel each other out.
One IETF approach for adopting jumbo frames avoids data integrity reduction of the service data unit by performing an extra CRC at the next network protocol layer above Ethernet. Stream Control Transmission Protocol (SCTP) transport (RFC 4960) and iSCSI (RFC 7143) use the Castagnoli CRC polynomial. The Castagnoli polynomial 0x1EDC6F41 achieves the Hamming distance HD=6 beyond one Ethernet MTU (to a 16,360-bit data word length) and HD=4 to 114,663 bits, which is more than 9 times the length of an Ethernet MTU. This gives two additional bits of error detection ability at MTU-sized data words compared to the Ethernet CRC standard polynomial while not sacrificing HD=4 capability for data word sizes up to and beyond 72 kbits. Support of Castagnoli CRC polynomial within a general-purpose transport designed to handle data chunks, and within a TCP transport designed to carry SCSI data, both provide improved error detection rates despite the use of jumbo frames where an increase of the Ethernet MTU would otherwise have resulted in a significant reduction in error detection.
In networking equipment, maximum jumbo frame size may be specified using either maximum frame size (maximum layer 2 packet size, includes frame headers) or maximum transmission unit (maximum layer 3 packet size, excludes frame headers), depending on the equipment's configuration interface.
A network that has a mixture of devices configured for jumbo frames and devices not configured for jumbo frames may have performance issues.
Jumbo frames can increase the efficiency of Ethernet and network processing in hosts by reducing the protocol overhead, as shown in the following example with TCP over IPv4. The processing overhead of the hosts can potentially decrease by the ratio of the payload sizes (approximately six times improvement in this example). Whether this is significant depends on how packets are processed in the host. A host that uses its network interface controller's TCP offload engine with already reduced overhead receives less benefit than a host that processes frames with its CPU. The throughput by bandwidth efficiency can increase by 4.4%.
The relative scalability of network data throughput as a function of packet transfer rates is related in a complex manner to payload size per packet. Theoretically, as line bit rate increases, the packet payload size should increase in direct proportion to maintain equivalent timing parameters. This however implies the scaling of numerous intermediating logic circuits along the network path to accommodate the maximum frame size required.
Baby giant or baby jumbo frames are Ethernet frames that are only slightly larger than allowed by the IEEE Ethernet standards. Baby giant frames are, for example, required for IP/MPLS over Ethernet to deliver Ethernet services with standard 1500 byte payloads. Most implementations will require non-jumbo user frames to be encapsulated into MPLS frame format which in turn may be encapsulated into a proper Ethernet frame format with EtherType values of 0x8847 and 0x8848. The increased overhead of extra MPLS and Ethernet headers means that the support for frames up to 1600 bytes is required in Carrier Ethernet networks.
Jumbo frames for PPPoE is defined in RFC 4638, with the purpose of removing the old 1492-byte limit (originally defined because PPP needs 8 more bytes of overhead), so that normal 1500-byte Ethernet can run without fragmentation. The "PPP-Max-Payload" tag can still accommodate much larger, non-baby jumbo frames.
Super jumbo frames (SJFs) are frames that have a payload size over 9000 bytes. As it has been a relatively difficult, and somewhat lengthy, process to increase the path MTU of high-performance national research and education networks from 1500 bytes to 9000 bytes or so, a subsequent increase, possibly to 64,000 bytes, is under consideration. The main factor involved is an increase in the available memory buffer size in every intervening persistence mechanism along the path. Another important factor to consider is the further reduction of CRC32's effectiveness in detecting errors within even larger frame sizes.
The Total Length field of IPv4 and the Payload Length field of IPv6 each have a size of 16 bits, thus allowing data of up to 65535octets. IPv6's jumbo payload option allows for up to 4 GiB (232-1 bytes) payload. These theoretical limits for the Internet Protocol (IP) MTU, however, are reached only on networks that have a suitable link-layer infrastructure.
Large send offload and large receive offload offload per-frame processing making CPU load largely independent of frame size. It is another way to eliminate the per-packet overhead that jumbo frames were designed to reduce. Jumbo frames are still useful from a bandwidth perspective, as they reduce the amount of bandwidth used for non-data overhead.
There are a number of reasons for this: MTU of 9000 has become the industry standard and is the most widely supported on various pieces of network equipment. Data Doman OS 5.4 is limited to 9000. Any performance gain using higher MTU values is minimal.
In a marathon of Bryan Cantrill interviews, Bryan wondered out loud why “jumbo” ethernet frames are 9000 bytes. These days, that’s not a particularly jumbo number, and it’s also a bizarre number. Why not a power of two, or at least a nice round 10000?
Thankfully, Wikipedia has a link to source material. It was a discussion between Internet2 and the US federal government, and they settled on 9k for 4-ish reasons.
First, they wanted to send 8192 byte TCP frames, so “something a bit larger than 8k” was desirable, to account for the various protocol headers. The memo doesn’t specify specifically why they wanted 8k payloads, but one possible reason is that the default NFSv2 packet size is 8192 bytes. There’s certainly a performance incentive to not fragment NFS traffic.
Second, at the time of this discussion, it turns out most routers could handle 9k ethernet frames. However, above that number the universe of working vendors started to drop off, so 9k was a sensible upper bound for rapid adoption.
Third, it was thought that PMTU discovery would work better if everyone agreed on common MTU values, so there was a desire to standardize on something, rather than just encourage people to max out the MTU capabilities of their current hardware. In hindsight, it’s not clear to me that standardizing MTU sizes did help PMTU discovery (i.e. it’s still broken for most people with a PPPoE hop in their network path), but I don’t have more context on the thought process there.
And finally: “9000 is an easy number to remember.” Presumably 10000 would have been even easier, but that ran afoul of the previous point about what existing network hardware supported.
And so, Internet2 and the US government agreed on a 9k MTU, and it became a mandatory requirement for equipment sold to the government. So, all vendors made sure they could do 9k jumbo frames to be able to get the juicy contracts.
Since then, it seems that nobody’s seriously pushed for a standardized increase. Perhaps understandable given how hilariously little adoption there’s been of jumbo frames on the wider internet. We’re still sending mostly 1500 bytes, and that seems to be going mostly fine, so why bother?
Still, it’s an interesting detour into a niche part of our industry’s history. Turns out the answer to “why 9000” is a bunch of random situational factors, and “9000 sounds nice.” It’s the kind of random “eh, that’s probably fine” decision that turns seamlessly into the unchanging bedrock of computing. Just like how /usr only exists because early Unix couldn’t fit on a single hard drive, so a second one got mounted at /usr.