Link Aggregates and Virtual LANs

A Link Aggregate (or Network Aggregate) is what you get when you push a single network link over several cables at once. In HP switches parlance, this is known as a trunk and in the Cisco dialect as an EtherChannel Bundle. On HP servers, it’s NIC Teaming and in other places it’s called Link/NIC bundling or bonding. In this post i’ll just call them Aggregates, and Aggregates are your friend.

Update: I’ve just been made aware that my assumptions on Link Aggregates were severely limited and have updated this post accordingly.

There are two reasons you’d want to use aggregated links: more capacity and redundancy. While a knee-jerk reaction is that moar cables is moar bandwidth and moar bandwidth is good, the true reason i love aggregated links is for the redundancy and the flexibility it gives.

One thing that confused me in the beginning was how to connect several switches together. For your regular conntect-two-switches-together kind of aggregated link, you’ll need one aggregate per link (or switch-pair). Thus, if you have one “main” switch connecting to two distribution switches, you’ll have two link aggregates on the main switch, one for one distribution switch and one for the other. If you connect half of such a link aggregate to one switch and half to the other, the link won’t work and Bad Things will likely happen.

Side note of epic proportion: This is, of course, not the whole truth. It’s possible to create a Distributed Trunk which not only can connect one logical link aggregate over several physical switches, but you can actually connect a link aggregate in square topology, meaning each switch in the Distributed Trunk are connecting the link to several switches at the same time. This means you can take down a whole switch and the link will still stay up. Distributed Trunks are a bit out of my league though, so this is all you’ll hear about them in this post. Thanks to Markus Åberg for pointing them out to me though!

I seem to have to reconfigure my switches a lot more often than i expect when i configure them (which, all things considered, is something i should have come to expect by now). When this happens, with an aggregated link, i can move the cables to another set of ports without disturbing or interrupting my colleagues who might be using that link. Being nice is, after all, one important value of a sysadmin.

Consider this situation: I have an aggregated link from Switch A, ports 9-12, to Switch B, ports 13-16. Say i want to move my cables on Switch A to 1-4 (for the cable feng shui). I’ll just add the ports 1-4 to the aggregate which contains the ports 9-12, then slowly and one cable at a time, move 9 to 1, 10 to 2, 11 to 3 and finally 12 to 4. The link will have enough time to rebuild itself and it’ll stay up during the move. When i’ve moved the cables, i’ll just remove the original ports 9-12 from the link aggregate.

It also means that there is a cable you (or somebody) can trip over without causing chaos and destruction.

With HP (HPE/Aruba) switches, at least in the 2530 series, there are two kinds of link aggregates, traditional “plain” trunks, and newer LACP trunks (or really, aggregates – LACP stands for Link Aggregation Control Protocol). If you are joining two HP switches, LACP is the better way to go. You can dig deep and get fairly lost with all LACP options, but truth is that you can just switch it on, plug it in, and be happy.

It’s also possible to connect a switch to a server running VMware ESXi with a link aggregate. For reasons beyond me, you’ll have to use the older “plain” trunk for this (Cisco switches can use LACP links). I suppose this is something that will be fixed in the future.

To make things neater, what you probably want to do when connecting switches (and ESXi hosts) together is push a number of Virtual LANs (VLANs) over a link aggregate. In this way, you just need a neat bundle of cables (say two, or four) to connect between two switches and you don’t need to worry about which of these go into which of the (two or four) ports. It really does bring peace of mind (which, as an admin, is something to strive to).

ESXi hosts also support VLANs, so for a typical server with four network ports, you might want to use one port for management and the remaining three for actual server payload, nicely mixed together in a VLAN/aggregate trunk.

Finally, and this is a bonus, you probably want to activate Spanning Tree on your switches once you start to toy around with VLANs over Link Aggregates and connecting your gear in exotic ways. Spanning Tree Protocol (STP) will shut down links if they create a loop, which would create a packet storm on your network and make several people much unhappy. So rather than taking down the whole network, STP will just shut down the bits which would break the net anyway (making only a subset of the people unhappy).

So this was just an introduction to happier networking. I’ll probably write a more practically oriented HOWTO in another post, though in all honesty, it’s all been written-up before. It’s a big Internet :)