Inside the DS8700 Part 2 – PCI Express

Welcome back to this behind-the-scenes look at the DS8700.  In Part 1, I examined the POWER6 based processor complexes that form the heart of the DS8700.  Today, I’m looking at the PCI Express gen2 I/O fabric that makes up the backbone of the most advanced version of IBM’s flagship enterprise disk product.

There are many design decisions made during the development of a storage subsystem.  One of the most fundamental is the interconnect topology used to connect all the components in the machine.  The DS8100 and DS8300 use a high-speed bus known as a RIO-G loop to connect the processor complexes and the PCI-X based I/O towers.  This has been a very successful design (as 1000s of our customers can attest), but for the DS8700 we wanted more.  To borrow a phrase from Spinal Tap, we wanted the DS8700 to go to ‘11‘.

We still have a RIO-G loop in the DS8700, but it is only used to connect the two POWER6 processor complexes together for synchronization and control purposes.  The big change is in how we connect the processor complexes to the I/O towers that contain the back-end disk adapters and the front-end host adapters.  For these connections, the DS8700 has replaced the RIO-G loops with a fabric of point-to-point PCI Express connections.  Each I/O tower has a dedicated 2 GB/s connection to each of the processor complexes.  This translates into a significant increase in the amount of data throughput we can sustain with the DS8700.

Making the move to PCI Express has brought more than increased performance to the DS8700.  It has also allowed us to further raise the bar in terms of reliability.  PCI Express adapters are intelligent devices.  Transient bit or CRC errors that can freeze other I/O technologies are caught in the PCI Express adapter and handled by the adapter itself.  Persistent errors can be dealt with by gracefully degrading the data transmission speed and notifying the processor complex of the problem.  By using smarter adapters that can self-heal, we add another layer of reliability to an already highly available system.

In my next article on DS8700 internals, I’ll be stepping away from the hardware and taking a look at some of the microcode enhancements in the R5 code that powers the DS8700.

Update 1:  Added link to YouTube clip that illustrates taking things to ‘11′.  Thanks David!