Any slowdown or interruption in packet forwarding has the potential to affect the performance of an application, particularly real-time sessions. For example, if packets are dropped from a voice session for more than 3 seconds in a row, the quality of the voice conversation will degrade to the point that most users will become frustrated enough to hang up.

To prevent data sessions from being dropped, it is advisable that a system be able to recover from a failure in less than 10 seconds, presuming the system supports a mix of Layer 3 routing protocols. All routing protocols specify minimum time intervals in which the peer device must receive an acknowledgment before a session is disconnected. So the router software must be able to recover at least as fast as the length of that interval to maintain uptime.

Intermediate system-intermediate system communication has the most stringent recovery requirements. Such communication times out somewhere between 6 and 15 seconds without a peer acknowledgment, and these intervals are not configurable. The default time-out of open shortest path first (OSPF) is about 15 seconds, and the default for Border Gateway Protocol (BGP) is about 120 seconds. Note, though, that OSPF and BGP time-out intervals are configurable.

Setting the routing-protocol timers is one example of the trade-offs that must be made in designing highly available IP networks. If the time-out interval is too big, the routers will not detect failures quickly, contributing to system downtime. If the time-out interval is too short, routing protocols could expect acknowledgments from one another so often that they could detect false outages and render the network unstable. Also, too-frequent communication consumes unnecessary system resources.

Availability Components
Without a highly available underlying hardware platform, a discussion of software recoverability design strategies is moot. In other words, if the basic system is not powered and running, the software will not be able to forward packets at all.

So the first step to creating a highly available system is to consider the degree to which backup hardware components should be built into the system. For example, redundant power supplies, line cards, fans and chassis are all critical to lowering MTTR. In addition, designing these components to be hot-swappable boosts uptime tremendously and enables components in a live system to be replaced without having to take the system down.

Dual route processors: For a highly available architecture, redundant route processors (RPs) are a necessity. The RP is the brain of the router. It calculates routing and forwarding tables, and communicates best-path information to peer routers. The degree to which primary and secondary RPs sync up their information is an important software consideration, because there must be some balance between the rate of recovery time and the consumption of system resources. These issues are explored further in Part 2.

Hardware system architecture choices: The hardware architecture selected for a router will play a role in the potential availability of the system. The primary router architecture choices are centralized and distributed, and within these architectures are additional design options. In all architectures, both active and standby RPs are required to maintain a low MTTR for the entire system.

In a centralized architecture, packet processing and forwarding are performed in a central, shared RP, and the individual network-interface line cards are relatively simple (Figure 2). In this configuration, each line card contains relatively few components. This lowers the probability of a component failure and reduces system cost. On the other hand, if the centralized RP fails, there is greater impact on the system overall, because all routing and packet forwarding will stop. This is where the dual-RP design becomes critical.

In a distributed system, a packet-forwarding capability is placed on each of several line cards (Figure 3). Moving the forwarding engines off the central RP and putting one on each line card lowers the impact of an RP failure, because the line cards can continue to forward packets during the outage.

In a distributed system, the probability of a single line card's failure is greater than in a centralized architecture, simply because each of the line cards is more complex and has a lower MTBF.

Centralized and distributed systems can both be designed using a traditional setup of line cards, each containing Layer 2 network interface connections and, in the case of a distributed system, forwarding engines that plug into a common bus. Another option is a midplane chassis architecture, which allows for some line card protection.

The midplane configuration divides the functionality of each line card in half, into a front and a back card. Each half-card plugs into a central chassis in the middle. By including some degree of line card redundancy in the design—one standby half-card serves as a backup to one or more primary half-cards—the network operator can hot-swap a half-card without disrupting its other half.

Most often, the front card, which is on the far side of the midplane from the system cabinet, terminates the network interfaces, while the back card supports most of the electronics, such as the packet-forwarding engine. Cable management, then, is more complex, as the back card with the electronics faces the cabinet, creating a challenge for connecting the router to other devices.

Software reliability: As noted, a baseline for any highly available system is hardware redundancy, particularly dual power supplies, buses, fans and RPs. Once a redundant hardware platform is in place, a number of software aspects contributes to the percentage of time that a router is available to process and forward the incoming packets.

There are several key objectives in designing highly available software:

• Minimize the system impact of a software failure.
• Detect and recover quickly from a software failure.
• Minimize the amount of time that new sessions will be blocked while the system recovers.

Detection of the actual hardware or software failure must happen very quickly to enable system recovery. Detecting a failure in most systems today takes 100 milliseconds to 3 seconds, depending on system design. The hardware and software interaction and failure detection must be thoroughly tested, because if the system does not accurately detect failures, all the system redundancy in the world will not matter. Fault insertion testing on the hardware, using probes, and online insertion and removal robotics testing can be used to ensure that the software detects any problems, including faulty hardware or software.

In the case of an RP failure—either hardware or software—the router will take some time to recover. This involves loading the new software image onto the standby, loading the configuration, initializing it, bringing up Layer 2 connections and Layer 3 routing protocols, and rebuilding routing tables. Of course, many of these tasks can happen in parallel or can be performed ahead of time to lower the system's MTTR.

Another option for minimizing the impact of software failures is to restart only the software process that has failed, rather than the whole system. This must be accomplished while preserving routes, forwarding tables and sessions. The main challenges with this approach are the difficulty of covering all processes in the system-for example, a software kernel failure-and the lack of protection against a hardware failure. The only way to recover from a hardware failure is with a resilient software architecture coupled with a redundant processor; restartable processes are not an option.

Unplanned Outages
Enabling a system to recover quickly and gracefully from an outage prevents failures from having a noticeable impact on application and network service performance. One design goal is to contain software failure impacts locally to the router so they do not have far-reaching effects on other network segments. Other primary goals include ensuring that physical links remain up during an outage; that adjacent routing nodes do not take a node out of the routing table and cause a network routing flap; that Layer 2 protocols, such as ATM, frame relay, high-level data link control and Point-to-Point Protocol, do not time out and cause a connection restart/teardown; and that packet forwarding will continue while the software recovers. To achieve these goals, the router must run active and standby RPs that can synchronize their information.

RP and software switchover: The degree to which the active and standby remain in sync with their information at any point in time is a key design consideration. This decision represents a balancing act between the speed of system recovery and cost. Options can be referred to as cold-, warm- and hot-standby backup systems (Figure 4).

With cold standby, no state information is shared between the active and standby RPs. A software failure results in a complete reset in which the standby RP automatically takes over for the failed active RP, but must then build its routing tables from scratch. In addition, all line cards are reset. In this configuration, there is a total user outage. Still, MTTR is lower than without the standby RP, because the system can automatically begin recovering by itself without the router having to be repaired or replaced.

In warm standby, both the router configuration and software image are already loaded on the backup. While the standby must rebuild its routing-table information, skipping the configuration and image reloads lowers the MTTR.

With hot standby, the standby is loaded with the router-configuration, software-image and network-state information, and it receives continual updates from the active RP. State information that is synced is specific to the protocols in use. Examples are local-management interface information for frame relay, SysUptime for Simple Network Management Protocol and specific sequence numbers that pertain to particular protocols. Enough state information must be transferred to the standby processor so that in the event of an active software or hardware RP failure, the standby processor can take over and recover the system without any loss of service.

The rate of synchronization between active and standby determines how up-to-date the standby processor is at any point in time and thus how quickly the system can recover. At any time, synchronization of data destined for the standby RP might not be completed due to a failure interrupting the communication flow between the two processors. This will result in an information mismatch. The software interacting between the new active RP and the line cards must be intelligent enough to recover from such mismatches to ensure system robustness.

The more that processors sync up, the less chance there is of a mismatch. However, more frequent synchronization increases the consumption of system resources. This could affect router throughput performance or require extra processing to be built into the system.

Line card backups: Similar to the switchover between active and standby RPs, the degree of line card redundancy contributes to a system's availability level. One option can be referred to as a 1+1 configuration, in which every line card has a backup to which it can switch over in hot-standby mode. As in the case of the redundant-RP design, a 1+1 line card configuration keeps the backup line card synchronized with Layer 2 and configuration information. While this is an expensive option, it can provide higher availability than the N+1 configuration, where for every "N" number of line cards in the system, there is one standby line card.

With N+1 line card redundancy, hot-standby switchover can be more difficult to achieve. This is because without a 1:1 correlation of a failed line card and a backup to pick up the load, it is difficult to keep the active line cards and standby synchronized. In other words, a single standby must have all configuration and state information for all N active line cards. Hence, the standby line card is usually run in cold mode.

Another consideration with the N+1 design is that an external switch might be required for switching between line cards during an active line card outage.

IETF Extensions
First, a high-availability system design requires a hardware platform with redundant, hot-swappable components to support fast-recovery software techniques. To achieve MTTR that is faster than using cold-standby designs, router software must have the intelligence to pass some amount of router configuration and state information to a backup component-or have the ability to recover from some other source. Among the considerations in Part Two, for example, will be extensions to routing protocols that have been completed by the IETF to minimize the duration and reach of an outage.

Related Articles