Adding experimental per-edge bandwidth limiting to Shadow - #3695

From a quick look this mostly seems plausible, though I'm not sure how one would go about finding the real-world edge-link capacities, or how plausible it is that they would become the bottleneck. Are you saying this is intended to help model the case where tor relays communciate with each-other over some other overlay network?

This one probably needs review from @robgjansen , at least for the high-level questions of whether this is a sensible extension to the network model

We're working on some measurements to see if there are cases where link-level, rather than node-level, congestion dominates the performance of links between two Tor nodes. We wanted to extend Shadow with link-level constraints so that we can study how such link-level congestion would manifest itself.

We are a little ways from having a realistic model of link-level congestion, so it's not a priority to mainline this patch, but any comments you have about our approach would be most welcome!

Thanks,
Nikita

There is a lot here and I haven't reviewed in detail yet, but sharing some initial thoughts.

I'm happy to see that the new parts are added as experimental options, are optional, and are off by default. And it is great that the tests and the documentation components were added.

Most of the logic is extended from existing files in multi-level nested code that might be difficult to follow and maintain long term. I wonder if there is some module-based design that would be more intuitive and allow us to break up the new functionality into much smaller units of code that would be easier to test using unit tests.

More generally, we have resisted adding too many Internet-level components to Shadow in order to avoid it snowballing into an Internet/routing simulator, which is not something that we would be able to do well and maintain. This patch gets us a little closer to that, though just a little. I think we would need to tease apart if this feature is generally useful or is more like a research prototype that maybe isn't worth full integration quite yet.

I wonder if this patch would maybe not be needed at all if we implemented a design something like the one discussed in #3701:

Sender Interface -> TokenBucket1 -> Router(CoDel) -> TokenBucket2 ->
(== Unlimited Core Link ==)
-> TokenBucket3  -> Router(CoDel) -> TokenBucket4 -> Receiver Interface

If you wanted to test out link-level congestion effects, you could configure token buckets 2 and 3 to have lower rate limits than 1 and 4. Can you comment on this? It would be great if we could handle two related issues with the same design, as it would simplify code design and maintenance on our side.