The backup environment is typically one of these aspects and quite often we find that many customers focus on the primary storage and compute concerns, rather than the entire data centre architecture as an ‘integrated and fit for purpose’ solution.

 When this shift starts to happen with traditional (tape) backup, more often than not, we find that a customer’s backup solution was coping,  but as the environment grows, the traditional model quickly starts experiencing issues. Situations range from failed or very slow backups – resulting in poor SLA’s and risk of data loss – to excessive resource utilisation, which adds to infrastructure and environmental costs.

 This of course is just from moving to virtual from physical, add to that ‘year on year’ data growth and the common requirement to improve SLA’s and reduce operational costs, and the problem magnifies rapidly.  Very often when attempts are made to align tape solutions with business SLA’s for disaster recovery, they fall significantly short of what is expected and demanded – and this is all assuming that the data on tape can be successfully restored!

 We have clearly seen a shift in this approach over the past year, and tape as a medium for maintaining an effective backup strategy is no longer being considered the first or logical choice, and it’s being pushed further down the stack where it’s a great solution for long term archiving, but very poor for short term backup and recovery.

 Just as a primary storage solution is now commonly taking a tiered approach, so too is backup.  Disk-to-disk and disk-to-disk-to-tape when compared to traditional models has clearly reached a point where it’s not only commercially justifiable, but for most customers, it’s the only way they can meet and satisfy business requirements.

Based on the requirements of providing security separation between front-end webservers and back end database platforms they wished to maintain the same overall security design between the two by positioning firewall appliances sandwiched between two VDC’s. This presented a corner case scenario of potentially black hole traffic that we needed to assess the options for. Each of them had their own merits and drawbacks, and the chosen solution was one that I had not seen reference to in any documentation and I thought it would be worth sharing.

Please note that the remainder of this post has taken the real world situation and presented it in an example scenario – But all concepts remain the same.

The specific failure scenario being discussed here is the failure of the vPC peer-link. But before I go into the issue and the solutions, I’ll spend a little time to describe vPC’s general operation and behaviour.

In a vPC environment the peer-link is essentially the inter-switch link between two vPC devices that form the vPC domain. All downstream switches are dual homed to each vPC peer, and in the case of a downstream switch uplink failure the peer-link would carry traffic that needs to get to the downstream device that has experienced a failed uplink.

The number one rule of vPC is that all devices should be dual homed to both peers, and apart from being good practice, this ensures that all failures are dealt with appropriately. One of these failures is the failure of the peer-link.

The correct behaviour of vPC is that if the peer-link goes down one of the vPC peers will shut all of its member ports to avoid a dual active scenario (This is why it’s important to dual home devices to both peers). To determine which peer device does this, you have another mechanism called the peer keepalive.

The peer keepalive is a separate link that is typically established over the out-of-band management network between the two vPC peers. Its primary task is to deal with a peer-link failure. In a vPC domain you have a primary peer role and a secondary peer role associated. This is important from a control plane perspective but is slightly off topic. However in a peer-link failure the peers ‘role’ is quite significant.

If the peers see each other over the keepalive, the secondary vPC role will shut all of its vPC member ports (downlinks) and SVI interfaces; and if they don’t see each other, no further action will be taken by vPC. Describing the reasons for each these scenarios is again a little off topic. What’s important is that in most failure scenarios of the peer-link, it’s likely that one of the uplinks from a connecting device is going to have one of its links brought down. Again a key reason why you should dual always attach to the vPC domain.

But what if you have redundant devices deployed like in the example below:

Now let’s consider the possibility of losing the top vPC Peer-Link either by misconfiguration, or disconnected fibers etc. The secondary vPC peer will shut its member ports because the keepalive has identified that both are still reachable. For the load balancers they will both go active/active as they no longer see each other. This is perfectly fine as one load balancer is isolated entirely from the topology. If the load balancer on the right was the current active, the secondary one would assume primary and load balancing would function normally. But for the firewalls this is an entirely different scenario. If the firewall on the right hand side is the current active, the interface heartbeat will be broken on the front end, but the pair will still see each other via all other heartbeats. The firewalls interface to the front-end would remain up during a peer-link failure (as its not vPC’d), so it would stay as the primary since both firewalls recognize the same failure condition. The problem now is any northbound traffic will still hit the right hand firewall, resulting in a black hole for traffic. The diagram below shows how this happens:

It’s important to note that pre-emption on the firewalls to align with the primary vPC peer will not always work. vPC roles are not pre-emptive, so either peer could be the primary and remain primary until a failure happens. In addition, if using contexts, sometimes you will want to share between both firewalls.

So to deal with this kind scenario there are a number of possible options:

Option 1 – Dual home all devices
The simplest solution of course is to dual home the firewalls, and join them to the vPC domain. Reasons not to do this could be a port count limitation on either device to accommodate multiple DMZ’s, or the expense involved in adding the right port capacity (10Gbps).

Option 2 – Attach to a non vPC peer
We could consider attaching the single homed devices to a secondary switch that’s vPC dual homed to the vPC domain. This means the single homed device will always have connectivity during Peer-Link failures, as the secondary switch will only have one of its links brought down, thus maintaining a backup path. This is usually only applicable if the appropriate connectivity and features are available on the secondary switch.

Option 3 – Second ISL Trunk
In some environments a valid solution is to create a second trunk between the two peers for non-vPC VLANS. This method doesn’t compromise the design by introducing any loops providing you only have VLAN’s running across one of the inter-switch links. The down side is the use of additional ports and the administrative overhead of managing two inter-switch links.

Option 4 – Single member vPC
The three previous examples tried to avoid loss of connectivity for single homed devices. Single member vPC on the other hand can be used when maintaining connectivity for a single homed device is not a requirement, you just want to invoke failover. Because there are redundant firewalls, it would be acceptable if one of them goes down during the described failure. The way to do this is by creating a separate PortChannel on each peer for each of the firewalls interfaces and assigning them to a unique vPC. To the vPC peers it will simply look like a collection of ‘configured’ dual homed devices, and each one has one of their member ports down. When the Peer-Link fails, the keepalive will identify that both are up and it will shut all of its member ports. This will include the firewall member interfaces since the vPC mechanism isn’t concerned about single attached vPC devices. The firewall would now failover correctly and the topology continues to function.
The great thing is that the solution described here will also work with devices that do not support PortChannels, or 802.3ad link aggregation (LACP).