As the deployment of PCI Express (PCIe)-native systems becomes more prevalent, many of the commonly used communications system endpoint solutions are being redesigned for PCIe connectivity. These include system motherboards, network interface cards (NICs), network security processors (NSPs), storage adapters, and an array of other I/O functions previously available only with PCI and/or PCI-X interfaces. However, many of the endpoint chips have not been redesigned as PCIe-native ICs, and, in fact, for many there is no plan to ever do so.
This article examines the trade-offs associated with the deployment of PCIe endpoint solutions using PCIe-native silicon vs. using PCI (or PCI-X) silicon and adding a PCIe-to-PCI (or -PCI-X) bridge. These bridges have appeared on the market for some time and allow endpoint designers a quick migration path to PCIe, as well as create PCI (or PCI-X) slots on PCIe-native system boards. It was assumed from the outset that these bridges would only have a market until all endpoint solutions were available using PCIe-native silicon. However, even with PCIe-based systems on the market, there are still several endpoint solutions that have not been redesigned with native PCIe interfaces.
There are several trade-offs to factor-in when deciding whether to migrate an endpoint to become PCIe-native. They include the cost of the silicon implementation with PCIe-native vs. size of the opportunity, performance requirements of the endpoint, and the availability and compatibility that PCIe IP requires to make a PCIe-native solution. When you do the math, you find that in many cases it costs significantly less to deploy a bridge to create the PCIe solution than it does to implement a PCIe-native solution. This is because the additional cost of the chip on the board is difficult to recoup against the cost of the new ASIC and the lost market share associated with any delay in time to market.
Let's look at these trade-offs in the context of three representative case studies requiring decisions that have been taken along the following lines:
1) Develop a new PCIe-native CPU chip solution and create an NSP add-in card using that silicon.
2) Add a bridge to existing PCI-X native silicon to create a PCIe AMC module for a full-duplex ATM line card application.
3) Add a bridge to create PCI and PCI-X connectivity on a PCIe native network router system mother board.
To simplify the debate, assume that a PCI-X-native solution with sufficient features and bandwidth is already in production, and that the question under consideration is whether to employ a PCIe bridge or develop a completely new ASIC that replaces the PCI-X bus with a PCIe link.
Using a Bridge to Minimize Time to Market, Development Cost
The two most compelling reasons to use a bridge with an existing PCI/PCI-X solution are time to market (TTM) and development cost. The TTM advantage is fairly obvious: using two existing proven solutions allows the designer to go right into board layout, with the bulk of the development cycle being in the qualification stages. The TTM advantage is achieved from the reduced time it takes to get to market compared with the time it takes to execute a new chip design, create the new mask set and validate and qualify the new silicon. This would ordinarily require a year or more, which is added directly to the development time of the card. A TTM delay can result in significant lost revenues, as competing solutions are selected and designs lost while the new silicon is being developed.
The development cost of the PCIe-native solution can be significant. This approach can be overwhelming for lower-volume programs unable absorb the millions of dollars required for a new PCIe-native chip, which includes, among other challenges, implementing 2.5Gb/s PCIe links.
Going Native to Minimize Manufacturing Cost, Board Space
The need to add a bridge is eliminated when a new ASIC is developed with a PCIe link. In addition, the board space associated with the bridge is eliminated, as well as the pins required to support the PCI or PCI-X bus. The bus interface requires over 100 pins but this number is reduced to as few four pins (for a PCIe x1 link, or 16 pins for a x4 link). This pin reduction can also reduce both the cost of the ASIC and the size of its footprint.
Performance Considerations
The performance of the solution may be increased or reduced by the addition of a bridge. The bridge itself introduces latency, which can degrade the throughput for bursty traffic. However, the use of prefetch can increase the throughout, assuming the PCIe devices can specify the prefetchable address space. This is typically only available in embedded systems, since in an add-in card product one cannot assume the host system can take advantage of the bridge's prefetch capability.
Case 1: Creating a Network Security Processor Card Using PCIe Native Silicon
In Figure 1, the arrows plot the course for the migration that has taken place in the Ethernet communications space. NIC solutions emerged a few years ago, initially with 32-bit PCI interfaces and then, as performance demands increased, 64-bit PCI-X interfaces became commonplace as 10/100 Ethernet gave way to Gigabit Ethernet.
When PCIe hit the scene, servers and workstations converted over to PCIe quickly, driving a need for a rapid development of PCIe versions of the same network controllers. Some Ethernet card designers inserted PCI-X-to-PCIe bridges to provide the PCIe connection, while others went to work on PCIe-native ASICs to deploy on their new Ethernet and NSPs. Another did both and phased in the PCIe-native solution after winning some early adopters with bridge-based designs. Today most Gigabit Ethernet NIC solutions use PCIe native silicon. However, the transition has not been so quick for NSPs. Let's look at the latter case and understand the factors driving their choice.
In this case, the PCI and PCI-X native silicon-based security processor cards were just hitting the market when the need for a PCIe version was realized. Seeing that this move to PCIe was a long-term trend, and that there were sufficiently high unit volumes projected for NSPs that also needed extended functionality, ASIC developments were chosen -- taking 14 months of design and a process technology change to implement a PCIe-native solution (see Figure 1). The process technology change was required, since there were no PCIe physical layer (PHY) cells available in the 0.18 and 0.25 micron CMOS technology that the PCI-X native processors were using. Because of the new PHY cell requirement and the migration to the new process technology (0.13 micron CMOS), the ASIC implementation was seen as somewhat risky. In an effort to minimize TTM and mitigate the risks of the chip development, two parallel programs were initiated. This allowed the quick deployment of a bridge-based solution to satisfy the early adopters, followed up by an extended feature-set and higher performance PCIe-native solution.
