Remember when you were learning to drive? It seemed like so many things were happening at once, literally enough to overwhelm your brain. Over time it becomes almost second nature, at least until there is an emergency, a surprise, or a crisis. And that is when accidents happen. So is it really possible to make vehicles that can do all of this like a human? If so, how? 

To replace a human, an autonomous vehicle has to "see" like a human, "react" like one, "move" like one, and most importantly, "think" like one of us. It has its advantages. Starting with never panicking, never getting tired, getting angry, distracted, or drunk.

In this conversation with Lei Wang, Bot Auto's Chief Technology Officer, and Xiaoling Han, Head of Hardware, they answer questions about just how difficult it is to completely remove the human driver from a big rig and the ins and outs of our autonomous driving system.

(Xiaoling Han & Lei Wang)

Q: Ultimately, to get to humanless operation, even at the validation stage, the human has to be replaced by something as good or better. What is that?   

(Lei) It is a pipeline that senses the world, understands the scene, carries out reasoning and executes the decisions. Coordination of all the functions. It should not miss anything and it must do that with redundancy. Every system is duplicated, in many cases much more than that. If one single piece fails another can step in. 

Q: What is the biggest challenge of removing the backup human safety driver?  

(Lei) When operating with the safety driver in fully autonomous mode, we can think of the human driver as the safety net for our autonomy algorithm. Taking the driver out removes our backup. So what do you do when your backup is gone? You need another backup, and when you make yourself more capable and less reliant on the backup, that's even better. We have to do both. We actively expand our Operational Design Domain (ODD) to make ourselves more capable, while simultaneously building up additional redundancy to find ourselves another backup.

System Safety by Design

Q: So let's start at the beginning. How does the truck "See"?  

(Xiaoling) The truck "sees" the world through sensors including cameras, LiDAR, and radar. Our truck has 15 cameras, 8 LiDARs and 3 radars all giving a complete "picture" that has advantages over any human driver. The system "sees" nearly 360 degrees at all times and does more than our eyes are capable of.

(Lei) Our eyes are the main "sensors" for humans and the eyes are similar to cameras. In this sense, our truck actually can see more things than a human can: the LiDAR gives the truck accurate measurements in object distances and the radar gives accurate measurements in object speeds.  

Q: So if the sensors are the eyes, what are the brains of the system?  

(Lei) The thinking part of the brain is our autonomy algorithm that provides an understanding of the world around the vehicle, and carries out driving tasks including object detection and tracking, motion planning and control. The algorithm is how the brain learns, whether from each and every run it makes or through validation scenarios.

Our servers and high performance GPU are actually the "Grey Matter" of the system. The brain itself. 

Q: So the steering and braking are similar to the arms and legs of the system? 

(Xiaoling) Humans rely on two arms and two legs, which provide both redundancy and enhanced performance. Human drivers are all taught to "drive at 10 and 2" because you have more control and it is safer. But if one arm is injured, you can still drive a car. But with both arms working together, you gain power, precision, and control.

Q: Is redundancy simply having a backup?

(Xiaoling) Redundancy is prioritized in the most critical areas—sensing, computation, actuation, and power—while shared components and software help keep the design simple, efficient, and economical without sacrificing safety. There may be a number of physical hardware redundancies; 2, 3 even 8, but we employ diversified redundancy designed for robustness.

(Lei) I can give you a concrete example. We use two braking controllers—one for the front brakes and another for the rear. If one fails, the truck can still brake safely. With both working, braking performance is significantly better. So during a traffic jam we have to apply a brake frequently and we apply the redundant brake alternatively so that none of the pneumatic subsystems get too hot (an issue that could happen in prolonged stop-and-go traffic for human driving). That is using the redundancy, not just placing it as a backup.

Continuous Observation: Immediate Reaction

Q: Redundancy is a good start, but how is that put to use?    

(Lei) To put redundancy to use, we need to be able to detect potential system faults, identify the right back-up component, decide the proper reaction and execute the corresponding decision to handle the fault gracefully. Fault detection and diagnostics run in real time across all systems, all the way down to the tire pressure.

Q: That sounds extremely complicated. How does it do that?   

(Xiaoling) We have built monitoring mechanisms for all of our components. Therefore, the system can quickly pinpoint the malfunction component and trigger the right backup actions. 

These are actually upgrades from a human driver. For example, we don't only have two eyes. We have a dozen that are always cross-validating each other. There are monitors for power draw and communication integrity—just like the nerves of a human—and they feed the brain of the system too. All of the elements of the system work together. 

Q: That allows the system to react?  

(Lei) Not only to react. Reacting is what human drivers do. What is much harder for a human is to anticipate what could happen in any given instant. The system anticipates because it is not only learning from the integrated parts that feed it data, it is able to draw from the experience of every other time it has been on this route and even simulations well beyond human comprehension. 

Q: It is bound to happen that something goes wrong. What happens if a sensor does fail? 

[Xiaoling] The system is designed to adapt its response to the nature of the failure, ensuring that safety is never compromised. Keep in mind that first and foremost, our design must ensure that no single point of failure can compromise the system. In practical terms, if a sensor fails, the vehicle must still retain full or partial control, and the truck must always be able to transition into a fail-safe state.

To achieve this, the system continuously monitors and detects different types of sensor failures. The response depends on both the sensor involved and the severity of the failure. A severe failure requires the truck to perform an immediate controlled stop to preserve safety. Minor failures may still allow the truck to continue, or the truck may safely pull over for inspection and recovery.

Q: Does that work the same way for both hardware and software?

[Xiaoling] Actually on the software side there are three levels of failure handling in the algorithm. Let's take LiDAR as an example. Low level would be the LiDAR sensor reporting internal issues, like overheating or a self-diagnosed malfunction. That is like when your eye tells your brain that there is a speck of dust in it. Mid level is the server software monitoring for issues and reacting accordingly. Like the brain forcing the eye to water to help push out the speck of dust. In a high level issue for LiDAR, the algorithm flags errors if there is critical data missing or if data seems suspect or irrational, suggesting results are not to be trusted. Like when the brain forces you to close your eye because it knows it is not functioning correctly. 

Q: Let's look at a worst case scenario. What happens to a human if they have a seizure while driving? There is a complete collapse of all systems, most often leading to a crash. Isn’t that true for a computer system too? 

(Lei) Our system has two brains, so for us the redundant computer, the backup "brain", instantly assumes control with no interruption to steering, braking or perception and begins the process of safely pulling over, executing a minimal risk condition. 

Sensing the "Real" World

Q: What about the "thinking" enhancements that get us to humanless operations?  

(Lei) Creating an Operational Design Domain in autonomy is like a child becoming an adult. If you are going to have the system be its own backup you have to know so much more. That growth comes both from education, or teaching, and experience or learning. When we had a backup human safety driver we could rely on that for many significant complexities of driving in the real world. 

Q: So the idea is to ensure the system can be its own backup?   

(Xiaoling) Exactly. The entire point of having the backup human driver is as a point of last resort if there is something that the system cannot understand, has yet to learn or be taught.   

Q: What became apparent during the autonomous runs with a safety driver that was still needed?  

(Lei) For example, how to handle a construction zone was not in our Operational Design Domain so the human safety driver would just override and adjust, but it is not something we can bypass during humanless operation. That creates a significant number of challenges to implement. On Interstate 10 where we operate, there are a number of construction zones that are ever-changing and at times very narrow. We had to add to our algorithm various ways to react to those changes, how it interprets information to "see" them and then how it should react, from changing lanes, to adjusting speed, and most importantly ensuring safe interaction with traffic. 

Q: Did that also require evolution from just "seeing" other vehicles?   

(Lei) Without a human as a back-up the system really needs to "see" everything and anticipate and react to what it encounters. We do share our streets and sidewalks with much more than cars and trucks. Our vehicle does not only drive on the highway. That means there are a range of intersections and sidewalks that lead us to encounter pedestrians or cyclists. We all know from our own driving experience that pedestrians can be even more difficult than other vehicles to see and predict. That required much more complex details to be added to the overall "vision" to allow for any possible scenarios. 

Q: How do you take emergency vehicles into account? 

(Xiaoling) That is something that is extremely important which does happen often in real life on our continuous validation runs, but it is best accomplished on the test track. Like with construction zones early on, our human backup drivers would automatically take over when encountering emergency vehicles. We partnered with the Texas Department of Public Safety, Fire, and EMS departments to replicate a range of emergency vehicle interactions on the closed course. That experience is added to the (ODD) and expanded upon when those interactions then occur in real world human backup driver runs.

Q: What is the impact of these adjustments overall?   

(Lei) The system has even more conservative behavior. Essentially a driver that does not ever take chances. Keeping a safe distance, and reducing speed if there is possible interaction not only from vehicles, but from pedestrians or cyclists. Adjusting for gradual speed changes before a construction zone because we are able to see it clearly from a greater distance. 

Turning Thoughts Into Actions

Q: So what happens when something does go wrong?   

(Lei) The truck does not just drive. It knows when not to drive and how to stop safely. Depending on the severity of the issue, we would take different actions. If it is some transient failure, we would give the system some chance to recover itself by entering a degraded operation mode with speed reduction and limited lateral movement; if it is a severe issue, we will decide to move to a Minimal Risk Condition, or MRC. In the real world that means locating the safest location, turning on hazard lights and pulling over. For example, if one of our servers is down, we would use the other server to carry out MRC. 

Q: Talk me through a real life example of that situation. 

(Lei) I was in the truck validating our algorithm. All of a sudden the truck notified me and the test driver of the "MRC" and the truck started pulling over to the emergency lane. We did not notice anything wrong. The system immediately switched to the backup server and pulled over safely. After examination of the system I found there was a significant delay in the data that determines location. MRC was a wise decision because that would be like a human suffering extreme vertigo, not knowing where you are or how you are oriented, which is an extremely dangerous situation. 

(Xiaoling) On the test track we have been able to conduct many MRC fault injection test cases. In an extreme scenario we tested the system at 65 mph during autonomous driving with an active lane change. First, we deliberately shut down Server 1, triggering the system to hand over control to Server 2, which continued the lane change and initiated a safe pull-over maneuver. Before that maneuver could be completed, we then completely shut down Server 2 as well. At that point, the Vehicle Control Unit (VCU) automatically took over and brought the vehicle to a safe stop.

Becoming the Backup: Always Thinking and Learning

Q. What are the biggest challenges in validating a hub-to-hub demonstration with a safety driver to prepare for actual humanless operation?   

(Lei) There is a reason we call it continuous validation. There are always new things to learn which expand the system's experience. That requires the expansion of our Operational Design Domain which is one of the biggest challenges in creating a non-human backup driver. 

There is also the consideration of increased system stability as the length of the trip becomes longer. Our continuous validation has led us to physical changes such as optimizing the cooling system to prevent servers from overheating during long operations, adding sensor cleaning systems to adapt for the bad weather and reinforcing hardware components to withstand prolonged vibration without degrading reliability. Without a human being there to handle a hardware failure, the Miles Per Failure requirement is over 100 times stricter to be considered safe. 

(Xiaoling) From the redundancy perspective it is not a simple copy-and-paste exercise. Adding redundant systems actually increases the workload significantly because a large amount of additional system design and software development must go into diagnostics, fault detection, and seamless switching.

Remember the "arms" of the system, the steering system? We made the decision to add a third piece of hardware. Even with a failure it would still be "ten and two", all of the time it is actually "ten and two…and six" or driving with three arms!

Q: What is the key element?    

(Lei) Data driven is the key. We have plenty of daily operation validation data and we learn from all of it. We analyze any issues of the system to find out where we need to expand in order to carry out humanless operation; we pinpoint areas of improvement we can do to enhance stability from actual experience, employing automation to identify efficiency bottlenecks and improve upon them.

The Future Is Now!

Thanks to Lei Wang and Xiaoling Han for helping to understand just how difficult it is to completely remove the human driver from a big rig and the growth of our autonomous driving system. 

Designing this level of redundancy requires not only duplicate hardware but also sophisticated coordination logic, fault management, and real-time decisionmaking to guarantee that any transitions are invisible if watching the vehicle's motion.

Everything about doing something for the first time is a challenge. Especially something as ever changing and unpredictable as the nature of driving. Any second another driver could cut you off, a breakdown could be a little too close, lanes close due to construction, or even the weather changes in a heartbeat. That requires even more knowledge to be able to understand the best path in any given situation. Through continuous validation in all of its forms we not only are ready for "The Big Empty", removing the human backup from the vehicle, but ready to keep the continuous validation coming with each and every run on the road to full commercialization and beyond.