Until cars achieve level 5 driving automation, there will be moments where drivers have to intervene. Designing these so-called take-over moments is one of the biggest challenges in semi-autonomous cars.

Why It Is a Challenge Today

The cars with the highest level of autonomy today are level 2. A level 2 system can accelerate,decelerate, and stay inside a lane. At all times, the driver must monitor the environment and expect to take over at any time. Making sure the driver does this sounds easier than it is.

Setting the Right Expectations

Before even using any driver assistance system, drivers must be aware of its capabilities. To perform the right action during a take-over moment, they should have the correct expectations of the system's behavior.

Today, this is a challenge in designing the take-over moment. Each car company is at a different stage in the development of autonomous driving technology. Every brand also uses other terms to describe their systems and different ways to interact with them. This makes it difficult to set the right expectations. As an example, let's compare two systems: Tesla's Autopilot and GM's Super Cruise.

First of all, the language a car company uses to describe its systems is essential for setting expectations. With a name like Autopilot, you may expect it to be self-driving, but it is a level 2 system at most. It is an example of 'autonowashing', blurring the lines between the different autonomous driving levels with fancy marketing terms. It contributes to the problem of overestimating what driver assistance systems can do.

A company that is aware of this is Waymo. A recent blog post announced that they replaced the term 'self-driving' with 'autonomous-driving'. It more closely resembles what their vehicles do. Autonomous cars don't drive themselves; they automate the task of driving.

Next to the naming conventions, the interactions and behavior are also different per system. Autopilot engages by pulling a stalk behind the steering wheel towards you twice. While using Autopilot, drivers must hold the steering wheel at all times. 

Super Cruise, on the other hand, engages by pressing a button on the steering wheel. Unlike Autopilot, it is only available on specific certified stretches of highway. And unlike Autopilot, drivers can take their hands off the wheel. Instead, Super Cruise uses head movement and eye gaze detection to track the attention level.

Facilitating a take-over moment does not begin while driving; it starts before even entering the car. It begins by using appropriate naming conventions, clear interaction models, and standards. It is the reason why most of the essential controls for driving are the same in each car. If the expectations are out of line with reality, the system has to bridge this gap. Which is difficult while the driver is already engaging with it.

Are You Paying Attention?

Even though the capabilities of advanced driver assistance systems today are impressive, they are far from perfect. They might overlook road works or misinterpret a piece of trash for a car during everyday highway driving. In those cases, the driver has to intervene and perform the right action in a split second. That is why car companies have to ensure that the drivers are constantly monitoring the road and are aware of oncoming dangers at all times. This is the hardest challenge of level 2 systems.

A take-over moment can be stressful and requires a lot of concentration. The interesting paradox is that drivers pay more attention, respond faster, and perform better when they have low cognitive load. Adding to that, angry drivers also perform worse take-over maneuvers. The difficulty is to warn drivers of an oncoming dangerous situation without causing high cognitive load and stress. As an added challenge, research has shown that when using a driver assistance system like Autopilot, drivers monitor the road less than when they drive themselves.

Additionally, these systems can easily create a false sense of trust. Let's say you use a driver assistance system on a specific stretch of road every day without any problems. Each time your confidence in the system increases even though it is not performing better. As your confidence increases, you allow yourself to relax a bit more. Eventually, the system misses a potential danger, and you will not be paying as much attention as you did at first.

There is currently no strict legislation or standardization for car companies aimed at the design of these systems. They have a lot of freedom to design this and go to different lengths to ensure the driver pays attention. The acceptance criteria for these driver assistance systems differ per car company, falling on a scale between ensuring safety and allowing for more driver freedom. 

Let's compare Autopilot and Super Cruise again, as they represent two different ways to achieve this. Tesla's Autopilot uses a simple system with sensors in the steering wheel to detect if the driver is firmly holding the wheel. If the driver fails to do this, it uses icons and visual flashes in the interface, combined with beeps to warn the driver. Tesla allows 60 seconds of warnings before the car starts slowing down on its own. In that time, the vehicle can have traveled one mile without a driver paying attention. Detecting if the hands are holding the wheel is a limited way of verifying if the driver is actually paying attention. Even if drivers are holding the wheel, they might be totally distracted. It leaves a lot of responsibility to the driver and, therefore, more freedom as well.

On the other hand, Super Cruise uses a more advanced system of tracking head movement and eye gaze to make sure drivers are watching the road. If drivers stop paying attention, it provides different modalities of feedback to alert them. These techniques are all more rooted in academic research.

If a driver is not paying attention, there will be visual feedback in the form of a flashing light strip. This is located at the top of the steering wheel, impossible to miss by the driver. The auditory feedback starts with beeps, just like Autopilot. But as the criticality increases, it changes to a voice command. This has shown to have higher responsiveness in drivers. On top of that, the driver receives alerts via haptic feedback in the seat. It is another way of feedback that has been shown to positively impact take-over readiness and disruption.

Super Cruise is a more foolproof system with a higher focus on ensuring the driver is paying attention. But to drivers, it is more restricting than Autopilot, and therefore, perception may not be as favorable.

These feedback systems must ensure that drivers can take over the wheel on time when they are completely distracted. Even though they shouldn't be. But ultimately, a designer can only do so much to make sure drivers are aware of their surroundings. It is complicated to ensure drivers have the same level of attention when using these driver assistance systems compared to when they are actually driving. Even in the most advanced system, you can never be 100% sure that drivers pay enough attention. 

All of this makes this take-over moment in level 2 autonomous cars such a challenging but exciting topic! 

Why It Is a Challenge in the Future

The jump from level 2 to level 3 will have a significant impact on the take-over moment. In level 3 and higher systems, drivers will be allowed to turn their attention away from the road in specific scenarios. But until the car reaches level 5 autonomy, there will be situations where they will still have to take over.

Predicting Behavior

The main challenge of the take-over moment shifts from making sure the driver pays attention to making sure the driver takes over in time with the right action. The scope also broadens as it will not only include moments when the systems fail to execute correctly. Take-over moments will also come into play when a vehicle transitions from a road where level 3 and higher is possible to one where it isn't. At that point, the driver will have to take over again.

In moving to level 3 and higher, car companies enter uncharted territory when designing the take-over moment. Only a few car companies have prototypes that are capable of this level of autonomy. This makes it hard to do good experiments on how drivers will behave. In today's academic experiments, simulators are the preferred method to do tests. The problem is that hard to predict how level 3 autonomous vehicles will behave and look like. Although it is possible to approximate real-world scenarios in a simulator, they are based on predictions on the future.

One reason that it is hard to predict the future is the change in the car's physical design. As drivers are not required to pay attention all the time anymore, it gives more freedom to interior designers. Think of steering wheels that retract into the dashboards or even chairs that can swivel around to create a living room on wheels.

With new spaces come new activities. Even though most activities are either idle or interacting with a smartphone in today's experiments, they may include working, socializing, and napping in the future. This will come with a new set of problems like sleep inertia. Reaction times are 3 seconds longer when drivers experience sleep inertia, so the car will have to anticipate this. This leads to another big question. How can the vehicle make the driver perform the correct action in time? In level 2 systems, the car will stay in its lane and warns drivers instantly when a problem occurs or if they are not paying attention. In level 3 and higher systems, the driver does not have to pay attention. The car has to know that human intervention is needed before an event even happens.

Possible Solutions for the Future

In thinking about how to answer these questions, a significant variable is when a take-over moment will be necessary. Level 3 vehicles will have two situations: accident avoidance and transitioning from level 3 to lower levels. The challenges that this brings might be too risky to take on. For that reason, some car companies, like Ford, Volvo, and others, have decided to skip level 3 autonomy and develop only level 4 cars and higher. The big advantage of level 4 vehicles will be that they are probably not helpless when a problem occurs. The driver would not have to take full control but instead, give instructions to the car on how to deal with specific scenarios. One direction to take is to step away from the traditional binary solution of either the driver being in control or the system. Instead, one can imagine a system of shared control between car and driver. The driver can give the vehicle instructions like confirming possible routes to take, evasive maneuvers to perform, or more. Let's say the car approaches a traffic accident or another event that it fails to comprehend with enough accuracy. It can offer the driver choices like 'should I wait, should I drive around it via path x, or via path y?' In the simplest form, this can be done through a conversational voice interface. But more complex touch interfaces are also possible. It will be important that the driver has enough awareness of the situation and information to make a correct decision and that the car can provide logical options. What is interesting is that the car company has to decide which problems to present to the driver.

It would open many ethical discussions when a driver is presented with a version of the trolley problem. Ultimately, they may even decide that drivers should not be the ones doing the take-over moment at all. Perhaps they will discover that they can't design for the right take-over performance by drivers. Or it may just be too much of a change. You can imagine that it may be too tricky, tiring, or risky for drivers to make these decisions. In that case, there are other solutions like remote control rooms where trained drivers make these decisions whenever a car in their fleet reports a take-over moment.

The take-over moment is the most exciting design challenge of semi-autonomous cars. Designers and engineers will have a lot of responsibility to research and develop the right solutions to the many open questions. The way we interact with vehicles may change completely. It will be fascinating to see what concepts they will come up with!