Carmakers Have Another Screen in Their Sights: Your Windshield

Augmented reality in vehicles is nothing new. In fact, it’s currently installed in millions of vehicles equipped with Head Up Display (HUD). But major car and tech companies think projection technology is about to become the next big thing as it expands beyond the driver’s cockpit. After all, what else are you supposed to look at, stuck sitting in your new electric car while it gasses up at a charging station?

BMW, which has championed HUD as safety technology in vehicles for the past two decades, demonstrated at Consumer Electronics Show 2023 a “mixed-reality slider” that will extend augmented reality across the bottom length of the windscreen. In addition to taking up more visual real estate, the next-generation technology aims to “blend the real and virtual worlds." It'd potentially culminate in a full-screen display with the capability of turning the entire windshield into a big screen TV.

Chairman Oliver Zipse said (during the weirdest media event that no one is talking about) that the technology isn’t just a concept–it’s headed to series production in two years and will appear in the upcoming Neue Klasse in 2025. But just how far the German manufacturers can go with increasingly immersive AR effects in the cabin may ultimately be up to the United Nations.

The big picture

The BMW Vision DEE, which stands for Digital Emotional Experience, is an ambitious follow up to the iX Flow that debuted the previous year at CES. Rather than be painted in monochrome, the Dee is wrapped with hundreds of color-changing E Ink panels. It also features a Zuckerberg-esque avatar that greets you when you approach the vehicle, and uses “shy tech” controls woven into the fabric, hidden behind the seamless dash that illuminated and revealed the mixed-reality slider buttons when a finger passes over it. Suffice to say, there are a lot of details that need to be worked out before it makes it to the dealership, but full-length augmented reality is a sure thing in the next generation of BMWs, according to the company.

The CES presentation laid out four levels of mixed-reality AR that, despite the confusing naming convention, are not tied to the self-driving levels established by the Society of Automotive Engineers. The data demonstrated in BMW’s Level 1 of the mixed-reality slider is pretty much what’s already found in the instrument panel, but rather than toggling through different screens to see the speed, audio, vehicle settings and other details about the surroundings, the information is strung along across the bottom length of the windshield.

Selecting Level 2 adds vehicle telemetry and advanced driver assistance system functions, such as obstacle avoidance and navigation, which is already part of most HUDs. “Brand communication,” and social media “visualization” are introduced with Level 3 of the mixed-reality slider. Level 4 is where things begin to get weird.

In this level, AR overtakes the windshield and goes “way beyond reality,” BMW’s CES presenter, Alice, explains during the keynote event, and hints at the vehicle transforming into a metaverse filled with friends, “the entire virtual world right at your fingertips everything you can imagine right here right now.”

Details are nebulous–and it’s not clear who, if anyone, is doing the driving at this point–but in the cabin becomes “an endless virtual world you can meet, play, talk, love, hate–you can even go sightseeing together right inside your car,” according to BMW. In short, as the driver selects higher levels of AR, the mix of virtual overtakes the physical, and the windshield can become a digital environment that blocks out the harshness of the real world. And these levels of increasing windshield takeover are technically possible, but would it be allowed?

Pushing boundaries

At a separate event held adjacent to CES 2023, Harman Automotive demonstrated its next generation ReadyVision technology, which is a platform that handles data from multiple sensors, such as navigation, ADAS and microphone, and casts those graphics onto a more traditional layout on the windshield of up to 15-degree by five-degree field of view in front of the driver. Powering the AR visuals are artificial intelligence and machine learning systems that turn the forward view outside the windshield into a mapped canvas with directional arrows, contextual points of interest, street names, and traffic data overlaid–Level 3 AR by BMW standards.

But there was no Level 4 full-windshield display. There would be no social media, no avatars, or videos cast on the window–even if the vehicle was parked. In fact, Rhita Boufelliga, director of Harman’s digital cockpit product, explained that there are regulations that hadn’t yet gone into effect, but would eventually limit the type of information that the company could display in front of the driver above a certain speed. For example, while turn-by-turn directions were allowed, Point of Interest information likely wouldn’t be shown because it would be considered distracting.

But that was news to several policy experts, who were not aware of any specific rules governing HUDs.

“NHTSA has issued some visual-manual driver distraction guidelines that might apply to some of the augmented reality stuff as it pertains to the driver, but those are essentially recommendations with no force of law,” explains Michael Brooks, Executive Director for the Center for Auto Safety. Experts at the Insurance Institute for Highway Safety also could not find anything pertaining to the technology.

NHTSA itself, when reached for comment, clarified that "[c]urrently there are no direct regulations on heads-up displays, however, the agency has released research and information regarding the subject: 1. Equipment Design Features (Impacts on Safety) and 2. Human Factors Design Guidance For Driver-Vehicle Interfaces." NHTSA does have guidelines for what "interferes with driver attention" and officially counts as distracting, the agency also noted.