Fragmentation after routing traffic through GRE tunnels in path.net

Symptoms

After initial implementation of the Path.net protection service routed mode with GRE Tunnels, traffic flow feels slower and/or some connections do not work properly or fail to establish, such as:

• some SSL sessions fail to establish
  • some slowness is observed when loading some sites
  • some sites respond with a connection timeout or error

Technical Issue

Maximum Transmission Unit (MTU) is the size of the largest Protocol Data Unit (PDU) that can be communicated in a single network-layer transaction. Essentially, if a packet is bigger than what a router can deliver and fragmentation is not performed, the packet will be dropped.

As a result, customer routed IPSEC traffic will have issues with a default Path.net tunnel MTU and fragmentation disabled. In order to successfully deliver customer routed IPSEC traffic, we need to account for the additional overhead it introduces.

Background

In a conventional network, there is no additional overhead encapsulation accounting from tunneling protocols  IPsec, GRE, or others. The typical(most common) maximum payload length of 1460 bytes goes without any risk of packet fragmentation. This is negotiated during an initial TCP three-way handshake between the source and destination host.

• The Maximum Transmission Unit (MTU) is the maximum length of data that can be transmitted by a protocol in one instance (at L3) for packets. An Ethernet interface has an MTU 1518 bytes by default, excluding 14-byte MAC-header, and 4-byte Frame Check Sequence.
  • The Ethernet Interface cannot carry any frame larger than 1500 bytes. If we look inside the frame, we have a 40(IPv6) or 20(IPv4) byte IP header dependent on IP versioning + 20 byte TCP header, leaving a 1460 byte of the payload that can be transmitted in one frame.
  • Maximum Segment Size (MSS) refers to the maximum bytes of payload data able to be sent in a single TCP packet. In other words, the MSS is the largest amount of TCP data (in bytes) that can be transported over IP. This is negotiated (at L4) during the TCP 3-way handshake of the initial SYN packet. The MSS is defined in RFC 879 for IPv4 and in RFC 2460 for IPv6. The MSS does not include the TCP header (20 bytes) or IPv4 header (20 bytes, if IPv6, the header would need 40 bytes).
• Protocols encapsulation overhead(recap).
  • GRE (IP Protocol 47) (RFC 2784) adds 24 bytes (20 byte IPv4 header, 4 byte GRE header)
    • IPsec encryption may add a maximum of 74 bytes for overhead (overhead depends on transport or tunnel mode and the encryption/authentication algorithm and HMAC)

Environment

• Path.net
  • Routed

Cause

Additional overhead from IPSEC protocol

• IPSEC Headers & Trailers

  • Tunnel Mode: 20 Byte header regardless of the protocol used
    • Transport Mode: No additional data, headers, or trailers
    • AH: 24 Byte header
    • ESP: 40 Bytes, (8 bytes header SPI and Sequence Number, 16 Byte IV and 16 Byte trailer)
    • AH and ESP both add headers to the TCP/IP packet itself, ESP also adds an Initialization Vector (IV) and a trailer. The size of this additional data depends on the IPsec protocol and mode used, as follows;
  • The overhead of a VPN concentrator varies by the encryption algorithm and hashing applicable. The basic value can be computed with the following formula, and with an overhead average value of 74 bytes (for only DTLS) and up to 94 bytes if using SHA-512 for authentication data encryption. This is without adding an additional 24 bytes for GRE encapsulation.
    • DTLS(IPSEC) = [ [ 8 [ UDP ] + [ 13 [DTLS header]] + [ 8 or 16 [IV DES / AES]] + [ 1 [CDTP] ] + [ 1 - 32 bytes [Padding for DES / AES] + 20 [MAC]] = 94 bytes
      • IP MTU = [ Ethernet MTU [1500] – [IP header [20]] – [ GRE Labels [ 4 ]]] = 1476 94(IPSEC)] = IP MTU vlan of 1382 bytes
      • MSS Adjust = IP MTU - 20 TCP provides a recommended value of 1362 for TCP MSS.

Resolution

Proper implementation of any of the following on customer server:

• Path-MTU Discovery (PMTUD) - See Issue / Solution: ICMP Exceptions For Path-MTU Discovery Allowing ICMP Code 3 Type 4 Messages
• TCP-MSS Clamping or clamping all IP protocols.
• Configure MSS for transient packets that egress a customer network. This will clamp only the TCP segments with the SYN bit set.

  • Cisco

  • ip tcp adjust-mss 1362 SUMMARY STEP

    Router1(config)#int ethernet 1/0
    Router1(config-if)# ! interface MTU accounting for outbound GRE overhead
    Router1(config-if)# mtu 1476
    Router1(config-if)# ! set ip to account for "all" internet protocols
    Router1(config-if)# ip mtu 1382
    Router1(config-if)# ! tcp-mss clamping of 1362 "tcp only"
    Router1(config-if)# ip tcp adjust-mss 1362
    Router1(config-if)# wr mem
    Router1(config-if)#
    

  • Juniper

    [edit interfaces]
     interfaces interface-name {
     unit 0 {
     family inet | inet6 {
     mtu {
     1382;
     tcp-mss {
     1362;
     }
     }
     }
     }
     }
    

Testing

ping [-f not mandatory] -g [1200] -G [1476] -h 5 -D [host destination advertised by AS396998/Path.net]

-D Set the Don't Fragment bit
-f Flood ping - Outputs packets as fast as they come back or one hundred times per second, whichever is more.  
-G sweepmaxsize - Specify the maximum size of ICMP payload when sending sweeping pings.  This option is required for ping sweeps.
-g sweepminsize - Specify the size of ICMP payload to start with when sending sweeping pings.  The default value is 0.
-h sweepincrsize - Specify the number of bytes to increment the size of ICMP payload after each sweep when sending sweeping pings.  The default value is 1.