The data storage crisis is big and getting bigger

Posted: August 20, 2026

We have a data storage problem. Even before the AI boom, there was a supply shortage. Now it’s even worse.

The big picture is this: Storage systems are still utterly dependent on hard drives. Seagate and Western Digital, the two main manufacturers of HDDs, are forecasting demand growth of 25% a year. At the same time, both companies are resisting calls for investment in new manufacturing facilities, preferring to focus on increasing the amount of bytes a drive can store rather the number of drives they are producing.

But that technological approach to supply is not satisfying today’s surging demand. Prices of flash memory and hard drives are rising fast—a trend that is widely forecast to continue for the next few years.

If those forecasts prove true, data-heavy enterprises will soon be facing spiralling storage bills, be it cloud or on-premise. Then again, forecasts are simply that—forecasts. They look at the available data, make minor assumptions and extrapolate from there. They do not take into account the new innovations and unforeseeable developments that so often render the past a useless predictor of the future.

Which raises two obvious questions: (1) Is there a paradigm shift coming to the world of data storage? And (2), if there isn’t, what should all those enterprises that run on data, industrials included, do?

How different kinds of data are stored today

When data is stored, the first consideration is how accessible it needs to be. If data needs to be retrieved quickly—as in, within milliseconds—it is deemed “hot.” Data that might be required on a regular basis but doesn’t need to be read within milliseconds is “warm.” Archival materials, which are read rarely, or even never, but that need to be stored for legal or regulatory reasons, are “cold.”

These different temperatures of data tend to be stored on different hardware, with hot data stored on flash memory, warm data stored on hard drives and cold data stored on magnetic tape.

“In cloud storage today,” Matt Klusas of boutique research firm West Ox Advisors recently told Our Industrial Life, “the vast majority of bytes are stored on hard drives, let's say something like 70%, so it truly is the workhorse of the cloud. About 20% of the bytes are stored on magnetic tape—the same underlying technology as VCRs from the 1980s. And then the last kind of 10% is flash NAND, and that’s really used for what you would describe kind of your hot workloads.”

None of this hardware is particularly durable. Magnetic tape can last up to 20 years, depending on the conditions in which it is stored. Flash memory lasts about 10 years. Hard drives need to be replaced every 5-10 years.

In other words, data storage is not a one-off expense—it requires regular, increasing investment.

How AI is changing data storage—and how it isn’t

The current challenge besetting storage is twofold: not only are the volumes of data growing year-on-year; the "average temperature" of data is rising.

That is largely down to AI. Both the training and running of generative AI models requires access to large amounts of instantly accessible data. It is this need for hot data that is driving the shortage of flash memory and driving up the prices of consumer electronics. Since January 2025, median NAND prices have risen 800%, according to Bloomberg.

AI is also contributing to the increasing amount of data being produced. This data needs to stay warm before it is eventually archived. As a result, hard drives, too, are in short supply. A Morgan Stanley note in June predicted HDD shortages would endure into 2028, and that prices could rise from roughly $15/TB to $25-30/TB.

The other contributing factor to the current storage crisis is the market itself. Although massive, it is dominated by just a handful of buyers and suppliers. The power to set prices swings periodically from one group to another, but neither side ever has much incentive to innovate, according to a white paper by Matt Klusas and Peter Faulhaber from earlier this year.

“Suppliers won’t invest in technologies that give buyers negotiating leverage a decade from now,” they write. “Buyers won’t fund supplier R&D that could benefit competitors. The result: both parties rationally choose derivative improvements over breakthrough technologies, even when facing supply shortages and fundamental scaling limits.”

Potential storage breakthroughs

While the current market may not be conducive to innovation, novel technology is nonetheless in development. Here are three storage technologies that could one day shake up the market and render forecasts like IDC’s moot.

DNA data storage

The prospect of encoding data in DNA has attracted headlines and excitement for years, for understandable reasons. DNA can theoretically store 215 petabytes per gram, “a density level,” as the New Yorker’s Matthew Hutson so vividly put it, “that would make it possible to fit a shipping-container’s worth of tapes into the volume of a few sesame seeds.” DNA is also very durable. At room temperature, it can last thousands of years, without any power demands.

But major issues with the technology remain. The process of sequencing and synthesizing (or, in IT terms, the reading and writing) of DNA is well understood and falling dramatically in price—but it is still nowhere near as cheap as it would need to be for widespread adoption. It also still relies on lab technicians doing more or less manual work. How to automate DNA data storage and retrieval at the scale of a data center is an open question.

Project Silica

An entirely different method for storing archival data has, until recently, been under development by a Microsoft research lab in Cambridge, UK. The Project Silica team are using high-speed lasers to encode data in glass. Resistant to heat, water and dust, glass could theoretically store data for 10,000 years.

In a paper published in Nature earlier this year, the team unveiled some major advances. Breakthroughs include the ability to encode data in regular, kitchen-grade glass, dramatic improvements in write speeds, and the simplification of the reading and writing machines.

Glass-based storage is not as dense as DNA, but, speaking to Data Center Dynamics in 2024, Project Silica researcher Richard Black pooh-poohed density measures as irrelevant. “There’ll be a point where humanity needs to leave planet Earth and find somewhere else,” he is quoted as saying. “When that happens, the megabytes-per-gram metric is going to matter […]. That’s a few billion years off. It’s not clear to me that between now and then there’s a use case for DNA storage.”

Unfortunately, the team's recent Nature paper suggests that no further work on Project Silica is planned.

Racetrack memory

A third novel approach to storage was born out of IBM. Unlike the potential cold storage solutions of DNA and glass, so-called racetrack memory promises to one day supplant existing flash and HDD storage.

Racetrack memory relies on the quantum property of “spin,” or the magnetic orientation of subatomic particles. These particles have a spin of either up or down, a property that lends itself to the encoding of binary data.

Racetrack memory works by manipulating the spin of subatomic particles and storing them in magnetic nanowires. An electrical current can then push the particles along the wire, allowing for the high-speed reading of information.

Although racetrack memory is still very much an academic pursuit, one of its leading figures has already revolutionized the world of data storage once before. In the 1990s, physicist Stuart Parkin developed a vastly more sensitive hard drive reader, which unlocked the massive growth in the storage density of HDD of the early 2000s.

Three approaches for the here and now

Exciting as these storage technologies are, it seems unlikely any of them will be deployed at any kind of meaningful scale within the next 10 years. Given a paradigm shift is not imminent, how then should data-heavy enterprises approach storage in the years to come? There are three approaches open to us: hope, audit and ponder.

We can hope for some kind of price correction, as the suppliers of storage lose their leverage over buyers—something that might already be happening in the NAND flash market, according to research firm Trend Force.

We can audit what data we currently have stored, and how it’s stored. Maybe a lot of data can be archived. Maybe it can even be archived on magnetic tape.

And then we can ponder. There is, after all, a non-technological solution to the storage problem, as outlined by Rob Carlson in IEEE Spectrum: “The discussion thus far has assumed that we’ll want to keep all the data we produce, and that we’ll pay to do so. We should entertain the counterhypothesis: that we will instead engage in systematic forgetting on a global scale.”