Four wrong theories and a scratched riser

A RAID controller stopped appearing to the host. The management processor still read its temperature, firmware version and drive states, and reported everything optimal. The operating system saw nothing at all: no device on the PCI bus, no block devices, no entry in the firmware’s boot list.

That split is the whole story. The management controller reaches the card over a sideband bus that has nothing to do with PCIe, so a card can be powered, healthy and completely unreachable at the same time. Health telemetry never once predicted enumeration.

Four theories held up long enough to waste time on:

A damaged slot. Two replacement controllers behaved identically. Reasonable, and wrong.

A firmware incompatibility. One controller’s firmware predated the server’s current BIOS by two major versions, which fit the evidence well enough that I downgraded the entire machine to the matching era to test it. Still invisible.

A dead CPU. The PCIe lanes for that slot come off the processor, and the firmware dump showed the whole root port missing rather than a device failing to respond. Plausible, expensive, wrong.

Two dead controllers. Improbable on its face, which should have been the clue.

The actual cause was a scratch across the edge connector of the riser card, shorting two adjacent PCIe fingers. Cleaning it fixed everything.

What made this take two days was a confound, not a shortage of ideas: the riser travelled with the card through every swap test. “The fault follows the card” and “the fault follows the riser” produce identical results when the two never separate. Every conclusion along the way was locally sound and globally useless.

The lesson I’d keep: when swapping parts to isolate a fault, make sure you are swapping one part.