Change your egress traffic queue discipline. fq_codel was mainlined in the last two years to solve buffer bloat and has the side effect of helping to alleviate ddos. It flags bad traffic flows (ie, ddos attacks) as bad queues and prioritizes other traffic around it.
Note, this doesn't help at the endpoint under attack. It requires a qos shift on the infrastructure leading to that thin last mile pipe, so the last mile packets they get are the the good flows.
Also, a warning: CoDel itself is pretty experimental still. It is only just being deployed as the default in proprietary routing hardware this year. Other queueing disciplines are an option, but the default (fifo) is a sack of garbage that causes a lot of these issues in its simplicity and inability to bias bad traffic from the good.
This is very exciting, and I can't wait to see some performance data from it. Bufferbloat is a huge problem so it's awesome to see work being done in this area. It's really cool also that it can improve things just by patching servers!
How does this interact sending traffic through routers using algorithms like fq_codel to reduce bufferbloat? Is it better to just have one or the other or do they work well together?
fq_codel alone doesn't solve buffer bloat. There are still huge buffers upstream. You need to combine it with local rate limiting so you can control the buffer in your local home router. fq_codel/cake work by looking at time a packet spent in a local queue. If you don't rate limit locally, then your local queue is always empty and everything queues upstream. Modern routers with things like "Dynamic QOS" that use fq_codel all require providing downstream/upstream values. This is way simpler than more traditional qos, but it's still a barrier to wide spread adoption.
You should have a look at traffic shaping and different networking queues. Fq_codel is great, but even SFQ works better than nothing. No need to turn off updates.
totally untrue. fq_codel manages normal torrents just fine in the presence of gamer-style traffic. We tested against torrents in fixing bufferbloat a lot! Pure aqm systems like pie (in docsis 3.1) do pretty well also. cake (per host fq) can make even insane amounts of torrenting (or slashdotting!) bearable.
CoDel doesn't do fair queuing so its performance on a mixed stream of traffic is much worse than fq_codel and other AQMs that have a fair queuing component. Please, read some of the literature on what's been developed in recent years. You don't seem to know anything about this subject other than having seen some of these terms show up in pfSense configuration pages and thinking it means it's at parity with Linux.
CoDeL is solution which has come out from all of this: http://www.bufferbloat.net/projects/codel/wiki - it can operate without any tuning, as it starts dropping packets when they start spending too much time in the queue.
To clarify, I meant that fq_codel solves bufferbloat. Its wide adoption will solve the problem globally and being present at Linux will help that, as Linux is the kernel in several home network appliances.
Other AQMs could solve it, if properly implemented, but fq_codel needs no tweaking.
CORRECTION:
fq_codel is not default in Linux, but is default in some distributions, like Fedora.
I'm sorry, but I have to disagree. The laws of physics (of the internet) dictate that the piece of equipment at a bottleneck is responsible for handling bloat/congestion on its own end. Your ISP's head end/DSLAM/etc. has an opportunity to prevent download bloat (with data being sent to the home), but it has no way to know what's happening on the consumer end of the link.
Your router at home needs to have the same logic: it needs to control the buffering of data being sent toward the ISP. fq_codel/cake actually measures the time packets dwell in the queues (their "sojourn time") and drops/marks with ECN some of the packets of flows that build up a queue.
A cool feature of fq_codel/cake qdisc in LEDE is that it can control bloat in both directions. It imposes a bottleneck that's slightly (a few percent) below the actual ISP download link speed so that traffic queues up within your home router (instead of at the DSLAM/head end). Consequently, it can do the fq_codel/cake algorithm on the download (as well as the upload) direction, keeping your link unbloated at a very small loss of link speed.
I enjoyed reading all the analogies on this thread. Stephen Hemminger and myself try to explain the latency problems involved in queueing with demos using water bottles in various configurations.
Dumb enough to me, includes basic fq and aqm technology at the router. FIFOs are the worst form of queuing.
FQ solves most of the problem thoroughly enabling delay based transports, Van and Kathie´s codel, most of the rest. Van was also on the team that produced BBR, which spends most of it´s time in delay based mode when fq_codel is on the path.
I do wish more folk were actively applying cake nowadays. It works wonders.
The "buffer bloat" guy has been useful. I've been arguing against big FIFO queues for decades, but he's more into PR and visibility than I am. Any place you have a major choke point, especially home router uplink connections, you need QoS, or fair queuing, or something, or queue delays will go through the roof. At least send the ACKs first.
> Common Applications Kept Enhanced (CAKE; sch_cake in Linux code) is a combined traffic shaper and AQM algorithm presented by the bufferbloat project in 2018. It builds on the experience of using fq_codel with the HTB (Hierarchy Token Bucket) traffic shaper. It improves over the Linux htb+fq_codel implementation by reducing hash collisions between flows, reducing CPU utilization in traffic shaping, and in a few other ways. [18]
Priority is the wrong way to think about it. Given all the sources of bursts on the internet today, fair queuing (or "flow queuing") has become the way to turn flows back into packets.
there's an awful lot of lit on FQ, what we do with fq_codel is to not only interleave packets better but apply congestion control signals at the right time so competing tcp flows don't overwhelm the link (with under 10ms of buffering (v seconds common on fifo ISP links)).
Of course, being perfectly fair to flows is sometimes undesirable, but making something strictly higher priority[1] is fraught with peril as you end up with a classification nightmare.
Having fq gives you the best shot at smaller flows completing sooner, and of big flows sharing better with each other.
Having vastly reduced buffering improves the responsiveness of competing TCP flows a lot, grabbing bandwidth whenever available, faster.
My take on folk that want "prioritization" is ask them to try some variant of sqm with just fq and codel and get back to us. being fair with well managed buffers works really well.[2]
[1] making something lower priority than best effort is actually a good idea.
[2] but if you really want some flows or devices prioritized, see the sch_cake work mentioned on this thread. I still tend to think per host FQ is what many want rather than attempting to raise the priority of certain flows from certain services.
> In either case, neither queue builds up in the home router -- that is exactly the problem, since we can't control the queue size in the modem or ISP! But by traffic shaping to 3.25/0.85 Mbps in the home router (either manually or automatically with fq-codel), we force the queues to build up there, thus giving us control.
YES! I understand what you're saying. (I had been treating the cable/DSL links and their terminating equipment as a black box; your description was being far more specific. Thanks for sticking with me to make yourself clear.)
And as you point out, the fix in all cases (at least, until our ISPs get serious about solving bufferbloat) is to force the home router to take control of the buffering by shaping the traffic a bit below the actual link speed. Thanks.
true, fq wont solve the problem completely.
but it will solve the elphant- vs. mice-flow problem
i.e. not make your interactive traffic suffer queueing latency from a simultaneous big transfer.
Notice the improvement in bufferbloat score from D to A+.
Subjectively, I have noticed the connection seems more responsive and there no longer seem to be latency spikes when utilising all the upload bandwidth.
It’s certainly worth considering bufferbloat, if you suffer from latency spikes when using all your upload bandwidth (I used to suffer from this a lot more, when I had an ADSL connection, with only 1 megabit upload).
Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too much data. It is a huge drag on Internet performance created, ironically, by previous attempts to make it work better. The one-sentence summary is “Bloated buffers lead to network-crippling latency spikes.”
The bad news is that bufferbloat is everywhere, in more devices and programs than you can shake a stick at. The good news is, bufferbloat is now, after 4 years of research, development and deployment, relatively easy to fix. See: fq_codel: wiki. The even better news is that fixing it may solve a lot of the service problems now addressed by bandwidth caps and metering, making the Internet faster and less expensive for both users and providers.
The introduction below to the problem is extremely old, and we recommend learning about bufferbloat via van jacobson’s fountain model instead. Although the traffic analogy is close to what actually happens… in the real world, you can’t evaporate the excessive cars on the road, which is what we actually do with systems like fq_codel wiki.
John, if you care to get back in touch with Jim Gettys and I, we can update you on where things stand on the whole bufferbloat effort.
FQ on modern cisco routers is generally quite crippled, and even that much is not available elsewhere.
However, on the whole, the fq + aqm situation along the edge of the internet is looking up, with nearly every third party home router firmware incorporating fq_codel (notably openwrt and derivatives), and it being integral to streamboost, and other QoS systems like those in the netgear X4.
Some examples of the kind of results we get now from these new algos:
I really liked the PCM paper (wanted code tho) for clearly identifying the parameter space modern tcps need to operate in. FQ everywhere, or nearly everywhere, would make the tcp's job far easier.
There is an aqm/packet scheduling working group in the ietf in the process of standardization, etc.
It's interesting how long we've been waiting for fair queuing to fix congestion problems in the Internet. The first proper research on this was published in 1989:
https://dl.acm.org/doi/10.1145/75246.75248
I'm a big fan of fair queuing, and have it enabled for my home network. But in core routers, the best approximation is likely to be WFQ, where you're likely to have each flow hashed to one of something like 256 queues. This means one badly behaved flow can't force well-behaved traffic out of the way and take over the whole link, but it can take over its WFQ queue, starving well behaved flows that hash to the same queue.
I'm not aware of any backbone router that implements true fair queuing. But even if all routers did, it's not a complete solution. Typically flows are mapped to queues based on the 5-tuple (src IP, dst IP, src port, dst port, proto). If you do this, then all Brutal-NG needs to do is use many source ports so it gets many queues, thus many times its fair share, and take over the link again. In fact, this would enable DoS attacks on router state, so no-one is going to do this.
An alternative would be to map to queues using just the source and destination IP addresses. But this has problems too. Brutal-NG could spoof the source address of most of the packets (but send ACKs back to the one unspoofed address), again taking over the link. And it could still cause DoS issues on router state.
The only thing you can't spoof if you want to actually exchange data (as opposed to DoSing the network) is the destination IP address. But now one Brutal flow can achieve the same fair share as all the traffic headed for a busy Google server or an entire ISP's CGNAT. Equally, one flow Brutal flow sending to a host behind the CGNAT can deny service to everyone else sending to the same CGNAT IP address.
So in the end, while I really like what fair queuing does for my VoIP latency on my home network, it is unlikely to ever be a complete solution for constraining misbehaving flows.
See : https://en.wikipedia.org/wiki/CoDel
Note, this doesn't help at the endpoint under attack. It requires a qos shift on the infrastructure leading to that thin last mile pipe, so the last mile packets they get are the the good flows.
Also, a warning: CoDel itself is pretty experimental still. It is only just being deployed as the default in proprietary routing hardware this year. Other queueing disciplines are an option, but the default (fifo) is a sack of garbage that causes a lot of these issues in its simplicity and inability to bias bad traffic from the good.
reply