Fast Pc Routers

Fast PC Routers

What's this all about?

We're building IP routers out of PC's as tools for research and experimental network development. The goal of this work is to come up with an IP router platform which is completely open to bending, twisting, reprogramming, and the like, and yet has sufficient performance to be useful for experiments in the 1990's.

Open routing platforms are an important tool for researchers developing new protocols and architectures. The DARTnet testbed, a precursor of this work, used routers built from Sun Sparcstations to catalyze the development of IP Multicast, RSVP, and the MBONE conferencing tools. We hope our project will help do the same for Mobile IP, Scalable Reliable Multicast, and the as yet unknown technologies of tomorrow's Internet.

We're using these routers to support our own work on IP Integrated Services QoS management, new security models, and Internet service discrimination and pricing. We also expect this or a derivative design to become the base IP router for the nationwide CAIRN testbed, currently being deployed.

Hardware

Platforms

If you want decent performance from a PC router, you must plan to use current high-end hardware. After years of stagnation, the demands of multimedia and high-speed peripheral devices are finally driving PC overall performance up, in contrast with the marketing hype which previously put ever-faster processors on the same old memory and I/O subsystems. Even better, the performance emphasis is moving away from large-block (disk) I/O and towards short-transaction (graphics, networks, audio and video) I/O. The downside is that this development is happening today, and yesterday's (almost literally!) machines are noticeably

off the pace. Our blurb on PC Hardware for Network Researchers will give you some information about the equipment we're using at MIT.

Design

Here's the basic PC design with a processor, memory, and, in this case, two PCI I/O buses. A router constructed from this hardware exhibits several properties. First, the processor must control all aspects of the router's operation, both executing the forwarding loop and performing overhead functions such as routing and management protocols. This introduces two performance slow-downs. Not only must the CPU break away from the forwarding loop to execute overhead code, but executing that code will almost certainly remove the loop instructions and routing table from the CPU's cache. On a typical PC this is a crucial problem, because the main memory subsystem is not terribly fast.

Another point of interest is that with today's PC designs there is adequate main memory bandwidth to operate two PCI buses and a processor at full rate simultaneously. This is encouraging. The reason for this is that it appears the primary limit to PC router performance is not CPU function or I/O bandwidth, but PCI bus arbitration time. Having two buses available reduces this arbitration bottleneck by a factor of two.

The use of two buses does introduce one drawback. Since PCI is a multiple-master bus, it is possible for appropriately designed interface cards and router software to DMA packet data directly from one interface to another on the same bus, without ever touching system memory, reducing the data transfer cost by a factor of two. This technique is of course impossible between two different PCI buses, but the router can still use it between interfaces on the same bus.

We use this configuration, with one or two PCI buses, for code development and where variable performance is acceptable

Many performance limitations of the basic design can be avoided by adding another processor. With the advent of Intel's Multiprocessor PC specification, two-processor machines are becoming common.

Originally designed for symmetric multiprocessing, these machines are easily subverted to our needs.

A simple first step is to separate the IP input and forwarding code from the rest of the system and run it on the second processor. With this design, packet routing performance is not subject to slowdown or variation when routing protocols or other user applications execute. This alone gives a large improvement.

Further improvements are possible. With separate, large Level 2 caches for each processor, and correct memory layout and cache control policy, it is possible to ensure that the forwarding processor's L2 cache is used solely for the forwarding code and a routing table cache. The design of the hardware interrupt controller in these machines allows the forwarding processor to handle network interface interrupts while the control processor handles all others. In rare cases it may be useful for the control processor to handle the initial interrupt at high priority, and have it present the packet to the forwarding processor in a simple, low-overhead way.

We're currently implementing these ideas, and hope to have measured performance results shortly.

A key aspect of these designs, particularly the multiprocessor versions, is that CPU instruction count does not appear to be the limiting performance bottleneck. In almost all circumstances, memory bandwidth or PCI bus arbitration overhead restrict performance first. This is important for our experimental goals, because it suggests that some additional complexity, such as different queueing algorithms, can be added to the forwarding loop without seriously affecting forwarding throughput.

Network Interfaces

Our routers currently support a number of interface types, including 100Mb Ethernet, 155MB ATM, and T1 long-lines. We wouldn't mind finding PCIbus raw (not ATM) DS3 and SONET /OC3 interface cards, if you happen to know of one. Again, you can see our discussion of PC Hardware for Network Researchers for some specifics.

Software

The intent of our work is to build routers which combine a general programming environment (for authors of routing protocols, network management tools, and the like) with a tuned low-overhead IP forwarding path (for folks interested in packet scheduling algorithms and related technology). Our current development work is based in FreeBSD, a publicly available BSD-Unix derivative with non-restrictive copyrights. Ultimately, we hope the community will create CAIRN Networking Snapshots; vendor-neutral code releases which encompass and make available current research results.

We're changing the standard FreeBSD networking code in several