{"id":53,"date":"2011-08-27T00:10:13","date_gmt":"2011-08-26T22:10:13","guid":{"rendered":"http:\/\/blog.copcea.ro\/?p=53"},"modified":"2011-09-29T20:49:51","modified_gmt":"2011-09-29T18:49:51","slug":"beware-ethernet-flow-control","status":"publish","type":"post","link":"http:\/\/blog.copcea.ro\/?p=53","title":{"rendered":"Beware Ethernet Flow Control (EN only)"},"content":{"rendered":"<div class=\"02a7dab7e6fe5a0fbf433999ff85f046\" data-index=\"1\" style=\"float: none; margin:10px 0 10px 0; text-align:center;\">\n<script type=\"text\/javascript\"><!--\r\ngoogle_ad_client = \"ca-pub-2343103762362018\";\r\n\/* 336x280 *\/\r\ngoogle_ad_slot = \"8815738048\";\r\ngoogle_ad_width = 336;\r\ngoogle_ad_height = 280;\r\n\/\/-->\r\n<\/script>\r\n<script type=\"text\/javascript\"\r\nsrc=\"http:\/\/pagead2.googlesyndication.com\/pagead\/show_ads.js\">\r\n<\/script>\n<\/div>\n<p>After having been recently bitten by Ethernet\u2019s flow control  mechanism, I decided to learn about this somewhat obscure but commonly  used facet of modern networks. This post is a summary of what I  discovered about it and its associated benefits and dangers.<\/p>\n<p><strong>What is flow control?<\/strong><\/p>\n<p>Ethernet flow control, or 802.3x, is a way for a network device to  tell its immediate neighbor that it is overloaded with data, such as  when a device is receiving data faster than it can process it. It allows  for an overloaded device to send out a special Ethernet frame, called a  pause frame, that asks the device on the other end of the wire to stop  sending data temporarily. If the receiving device honors the pause frame  then the sending device has time to catch up on the stack of received  data that it hasn\u2019t had time to process yet.<\/p>\n<p>There also exists an older method for flow control called \u201cback  pressure\u201d that is used in half-duplex environments (i.e. non-switched  Ethernet). It consists of the overloaded device \u201cjamming\u201d the medium  temporarily until it has the ability to accept more data. I don\u2019t know  much about half-duplex flow control, and thus I won\u2019t mention it again;  everything here applies solely to full-duplex flow control via 802.3x.  Also, TCP has a mechanism for performing its own flow control that is  entirely different from Ethernet\u2019s flow control; I will not be fully  explaining TCP\u2019s flow control method here, as it would merit a lengthy  discussion itself.<\/p>\n<p><strong>Rules of the game<\/strong><\/p>\n<p>When thinking about Ethernet flow control, it is important to keep several things in mind:<\/p>\n<ol>\n<li>Flow control operates at a lower layer than TCP or IP, and thus is  independent of them. Put another way, flow control is capable of being  used regardless of what higher-level protocols are put on top of it. An  important side-effect of this is that neither TCP nor IP know what  Ethernet\u2019s flow control is doing; they operate under the assumption that  there is no flow control other than what they may or may not provide  themselves.<\/li>\n<li>Flow control functions between two directly connected network  devices, and flow control frames are never forwarded between links.  Thus, two computers that are connected via a switch will never send  pause frames to each other, but could send pause frames to the switch  itself (and vice versa: the switch can send pause frames to the two  computers).<\/li>\n<li>Pause frames have a limited duration; they will automatically  \u201cexpire\u201d after a certain amount of time. The expiration time is set by  the device that transmits the pause frame.<\/li>\n<li>A paused link is not a discriminator of protocols; it will prevent <strong>any<\/strong> data from being passed across the link other than more pause frames.<\/li>\n<\/ol>\n<p>Perhaps you have begun to see some issues with flow control in light of some of the above points.  Let\u2019s start looking at them.<\/p>\n<p><strong>TCP breakage<\/strong><\/p>\n<p>Okay, it isn\u2019t true, TCP doesn\u2019t stop working when flow control is  enabled. However, an important part of it does stop working correctly:  its own flow control mechanism. TCP flow control uses a more complex  mechanism of timeouts and acknowledgement segments to determine when a  remote device is overloaded. It basically sends at a faster and faster  pace until it sees that some of its sent data isn\u2019t getting to the  remote device and then slows down. This allows TCP to utilize network  links in a somewhat intelligent manner, as an overloaded network or  device will cause some TCP segments to be lost and thus cause the sender  to send data at a slower rate.<\/p>\n<p>Now consider what happens when Ethernet flow control is mixed with  TCP flow control. Let\u2019s assume that we have two directly connected  computers, one of which is much slower than the other. The faster  sending computer starts sending lots of data to the slower receiving  computer. The receiver eventually notices that it is getting overloaded  with data and sends a pause frame to the sender. The sender sees the  pause frame and stops sending temporarily. Once the pause frame expires,  the sender will resume sending its flood of data to the other computer.  Unfortunately, the TCP engine on the sender will not recognize that the  receiver is overloaded, as there was no lost data \u2014 the receiver will  typically stop the sender before it loses any data. Thus, the sender  will continue to speed up at an exponential rate; because it didn\u2019t see  any lost data, it will send data twice as fast as before! Because the  receiver has a permanent speed disadvantage, this will require the  receiver to send out pause frames twice as often. Things start  snowballing until the receiver pauses the sender so often that the  sender starts dropping its own data before it sends it, and thus finally  sees some data being lost and slows down.<\/p>\n<p>Is this a problem? In some ways it isn\u2019t. Because TCP is a reliable  protocol, nothing is ever really \u201clost\u201d; it is simply retransmitted and  life goes on. Ethernet flow control accomplishes the same thing as TCP  flow control in this situation, as they both slow down the data  transmission to the speed that the slower device can handle. There are  some arguments to be made for there being an awkward overlap between the  two flow control mechanisms, but it could be worse.<\/p>\n<p>Unfortunately, it does get worse.<\/p>\n<p><strong>Head-of-line blocking<\/strong><\/p>\n<p>In the last example, I considered the case where two computers were  directly connected to each other. This example is too simplistic to be  of much use \u2014 when was the last time you saw two directly connected  computers? It is a bit of a rarity. Let\u2019s now look at what happens when  you introduce a switch into the mix. For our purposes, let us assume  that the switch fully supports Ethernet flow control and that it is  willing to use it. Our new setup will consist of two desktop computers  and one file server, all of which are attached to the switch. It isn\u2019t  any fun to make everything perfect, so let\u2019s also say that one of the  desktops has a 10 Mbps connection to the switch while the other desktop  and the server have 100 Mbps connections.<\/p>\n<p>This setup is usually fine \u2014 the 10 Mbps connection will be slower  than the others, but it doesn\u2019t cause too many problems, just slower  service to the one desktop. Things could get ugly, though, if Ethernet  flow control is enabled on the switch. Imagine that the 10 Mbps desktop  requests a large file from the file server. The file server begins to  send the file to the desktop initially at a slow rate, but quickly picks  up steam. Eventually, the file server will start to send data to the  desktop at 11 Mbps, which is more than the poor 10 Mbps connection can  handle. Without flow control enabled on the switch, the switch would  start to simply drop data segments destined to the desktop, which the  file server would notice and start to throttle back its sending rate.<\/p>\n<p>With flow control enabled on the switch, though, the switch takes a  very different approach; it will send out its own pause frames to any  port that is sending data to the now-overloaded 10 Mbps port. This means  that the file server will receive a pause frame from the switch,  requesting it to cease all transmissions for a certain amount of time.  Is this a problem? Yes! Because pause frames cease all transmissions on  the link, any other data that the file server is sending will be paused  as well, including data that may be destined to the 100 Mbps desktop  computer. Eventually the pause will expire and the file server will  continue sending out data. Unfortunately, the TCP mechanism on the file  server will not know that anything is wrong and will continue sending  out data at faster and faster speeds, thus overloading the 10 Mbps  desktop again. As before, the cycle will keep repeating itself until the  file server starts dropping its own data. Unlike the previous  situation, the innocent 100 Mbps desktop bystander is penalized and will  see its transfers from the file server drop to 10 Mbps speeds.<\/p>\n<p>This situation is called head-of-line blocking, and it is the major  reason why Ethernet flow control is somewhat dangerous to use. When  enabled on network switches, it can create situations where one slow  link in a network can bring the rest of the network to a crawl. It gets  especially bad if the backbones in your network have flow control  enabled; it should be obvious by this point just how bad that could get.<\/p>\n<p><strong>When to enable flow control<\/strong><\/p>\n<p>So what should you do? Should you completely disable flow control on  all computers and switches? Not necessarily. It is generally safe to  leave flow control enabled on computers. Switches, though, should either  have flow control disabled or configured such that they will honor  received pause frames but will never send out new pause frames. Some  Cisco switches are even permanently configured this way \u2014 they can  receive pause frames but never emit them. To be honest, the complete  answer to flow control is somewhat more complicated than this (e.g. you  could probably enable pause frame emission if a switch port is connected  to a slow backplane), but the safest bet is to disable flow control  when given the option.<\/p>\n<p>ref: http:\/\/virtualthreads.blogspot.com\/2006\/02\/beware-ethernet-flow-control.html<\/p>\n\n<div style=\"font-size: 0px; height: 0px; line-height: 0px; margin: 0; padding: 0; clear: both;\"><\/div>","protected":false},"excerpt":{"rendered":"<p>After having been recently bitten by Ethernet\u2019s flow control mechanism, I decided to learn about this somewhat obscure but commonly used facet of modern networks. This post is a summary of what I discovered about it and its associated benefits and dangers. What is flow control? Ethernet flow control, or 802.3x, is a way for <a href='http:\/\/blog.copcea.ro\/?p=53' class='excerpt-more'>[&#8230;]<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[6],"tags":[65,64,230,232,231,229],"_links":{"self":[{"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=\/wp\/v2\/posts\/53"}],"collection":[{"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=53"}],"version-history":[{"count":5,"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=\/wp\/v2\/posts\/53\/revisions"}],"predecessor-version":[{"id":424,"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=\/wp\/v2\/posts\/53\/revisions\/424"}],"wp:attachment":[{"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=53"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=53"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.copcea.ro\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=53"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}