When I worked for EMC, they released a product called DSSD. It was a storage array that produced 10 million 4K IOPS in 2016 and claimed an impressive 100GB/s bandwidth. Networking was via PCI Express made via 1, 2, 3, or 4M cables and custom PCI Express cards. You could access it as regular storage, or for maximum performance use the built in Flood API. You can read about DSSD in detail here.
The raw specs are still impressive today, although we have certainly moved on. EMC spent a billion dollars buying DSSD, millions more more bringing it to maturity and establishing a sales force, and ultimately sold almost none. On paper it looked like an amazing system and in terms of raw performance it was industry leading. I remember being told you could use it to map a human genome in about 15 minutes. Why were sales and production halted after less than a year?
There are a few obvious things to cover. First and foremost is the networking and API. While an alternative to the Flood API existed (you could run it as conventional block storage,) there was no alternative to the PCI Express interfacing. This made DSSD a “top of rack” solution where you would be limited to about three racks maximum. It didn’t integrate with blade systems at all because of this.
This ignored the larger issue which was customers had no idea how to use it. One of the things it was marketed as was an “Exadata Killer.” While this is always a bold claim, to my knowledge there was no reference architecture in order to back these claims up. Other possibilities included using clustered file systems on it. In the end however, there was no clearly laid out technical vision on how to actually use the system. At the extremely high cost of DSSD, this was unacceptable.
Pure Storage released FlashBlade about the time DSSD was being cancelled. Pure Storage has released multiple variants and the product is nearing a decade in production, with good sales. The performance wasn’t as good until you scale it out, the protocols offered aren’t as efficient, but they succeeded where EMC failed. Why?
Pure Storage chose a different path; they decided to stick to NFS, S3, (and later SMB) which while not as performant, was far more accessible across an entire data center and to a far wider variety of systems. Not only that, but the use cases were far more obvious to users and easier to integrate into automation stacks. There was also a significant price difference between the two. FlashBlade is not cheap storage by any means, but it was cheaper than DSSD.
In retrospect one could argue that EMC was ahead of their time. NVME/IP and NVME/OF carry the torch for PCI Express interfacing. The fact these protocols exist and have propagated illustrate a clear need for low latency, high bandwidth data access. Not only that but progressively faster Ethernet has become accessible as well. Whereas 10 gigabit was the standard in 2016 for data centers, 100 gigabit or even faster Ethernet is accessible now.
We can conclude by looking at this case study and seeing a few things:
- The fastest doesn’t always win the race in technology.
- Users and buyers don’t let perfect get in the way of adequate.
- Betting against Ethernet is a bad idea. Unless it’s Fibre Channel. That’s a future post.
- Familiarity and common standards can and do sell products in to environments.
- It is critically important to understand how new, unfamiliar technologies will be used.
- It is critically important to have ready made use cases to sell or implement new, unfamiliar technologies.
We can take this lesson in to our daily lives. Technology does not exist in a bubble. It is easy for a technologist to get enamored with the latest and greatest technology, but then for get the most important questions; Who will use it, why will they use it and how will they use it? If you can’t answer these, the game is already lost.