Server With Full Hard Drive Shuts Down Every Toyota Plant in Japan

Modern automotive factories typically run 24 hours a day, 7 days a week, with the loss from even a short disruption typically measured in thousands or even millions of dollars. Last week, Toyota faced a huge and uncharacteristic shutdown across multiple Japanese factories, all thanks to an embarrassingly simple IT problem.

As covered by The Guardian, all 14 Toyota assembly plants in Japan faced an unexpected stoppage on August 29. It was all thanks to a breakdown of a key computer system responsible for ordering vehicle parts for Toyota factories. The cause of the problem? A full hard drive.

Toyota's just-in-time production system sees parts and components only delivered to the production line as needed. It's a boon to efficiency, but it can quickly lead to problems when things aren't operating perfectly. Thanks to this, the factories were unable to operate when the crucial system went down.

Toyota stated that during routine database maintenance the day before the incident, "data that had accumulated in the database was deleted and organized, and an error occurred due to insufficient disk space." Like any diligent organization, Toyota had a backup in place, but as it was running "on the same system," a similar failure occurred and it could not be switched into operation. This failure brought all of Toyota's Japanese factories screeching to a halt in short order. Eventually, the system was transferred to a server with excess storage capacity, and the assembly plants were able to resume operations.

Toyota's Japanese plants are responsible for around a third of its total vehicle output. Back-of-the-envelope maths from Reuters suggests the 14 factories' average output is around 13,500 vehicles a day in total. Based on current average Toyota sales prices and production rates, the loss of a full day's production could be as steep as $356 million in revenue. Take that with a grain of salt, but it's a guide to how high the stakes are when a country's worth of factories all go offline at the same time.

It's not the first time Toyota has seen a surprise problem knock its Japanese operations offline, either. Last year, plants were stopped for a day when one of Toyota's suppliers reported its own servers had been infected with a virus. The issue raised questions about the cybersecurity of the production chain, though it bears noting that Toyota has clearly stated there was no cyberattack involved in this recent incident.

It's likely that some of Toyota's IT staff were in very deep trouble for this incident. If there's one thing to be said in their defense, it's that the vast majority of people stopped worrying about hard drive space shortly after multi-terabyte hard drives hit the market.

Regardless, if you're in charge of a crucial server that supports 14 factories, it pays to make sure that it's not about to choke on its own data. Oh, and make sure the backup server actually works, too. That last one? That's kind of unforgivable.

Got a tip? Let the author know: lewin@thedrive.com