Self-driving cars are revolutionizing transportation, and at their core is a sophisticated system allowing them to “see” the world using sensors. But the crucial question is: how do these vehicles interpret this vast amount of sensory data to navigate and make decisions? The answer lies in the concept of perception, the automotive industry’s term for a self-driving car’s ability to process and identify crucial road information – from traffic signs and pedestrians to other vehicles and lane markings – while in motion. Powered by artificial intelligence, particularly deep learning, autonomous vehicles achieve real-time environmental recognition and reaction, which is fundamental for safe navigation.
This remarkable capability is achieved through an intricate network of algorithms known as deep neural networks (DNNs). Instead of relying on pre-programmed, rule-based instructions like “stop at a red light,” DNNs enable autonomous vehicles to learn and adapt to the complexities of the real world directly from sensor data. These DNNs are essentially complex mathematical models inspired by the human brain, learning through experience. For instance, by exposing a DNN to numerous images of stop signs under various conditions (different weather, angles, and lighting), it learns to autonomously identify stop signs in new, unseen scenarios.
Diversity and Redundancy: Cornerstones of Self-Driving Car Safety
However, a single algorithm is insufficient for the multifaceted task of autonomous driving. A comprehensive suite of DNNs, each designed for a specific task, is essential to ensure safety. This system emphasizes both diversity and redundancy. Diversity means employing networks that handle a wide range of tasks, from interpreting traffic signs and identifying intersections to mapping drivable paths. Redundancy involves overlapping capabilities across different DNNs, minimizing the risk of failure in any single component.
The exact number of DNNs in a self-driving system isn’t fixed and is constantly evolving as new functionalities are developed. To translate the outputs from these individual DNNs into driving actions, real-time processing is paramount. This demands a powerful, centralized computing platform, such as NVIDIA DRIVE AGX, capable of handling the immense computational load.
Below are examples of core DNNs utilized by NVIDIA for autonomous vehicle perception, illustrating the complexity of writing neural programs for these systems.
Pathfinders: Guiding the Autonomous Vehicle
These DNNs are critical for enabling the car to determine drivable areas and plan safe routes:
- OpenRoadNet: Identifies all available drivable space surrounding the vehicle, encompassing the current lane and adjacent lanes.
- PathNet: Pinpoints the most viable driving path ahead, even in the absence of clear lane markings.
- LaneNet: Detects lane lines and road markers to define the vehicle’s trajectory within lanes.
- MapNet: Identifies lanes and landmarks, contributing to the creation and maintenance of high-definition maps for navigation.
Path-finding DNNs working together to identify a safe driving route for an autonomous vehicle
Object Detection and Classification: Perceiving the Environment
DNNs in this category focus on detecting potential obstacles and interpreting traffic signals:
- DriveNet: Perceives other vehicles, pedestrians, traffic lights, and traffic signs, although it doesn’t interpret the specific meaning (like traffic light color or sign type).
- LightNet: Classifies the state of traffic lights – red, yellow, or green.
- SignNet: Identifies the type of traffic sign, such as stop, yield, or one-way signs.
- WaitNet: Detects situations requiring the vehicle to stop and wait, like intersections.
Expanding Capabilities
The array of DNNs extends further, encompassing functions like monitoring vehicle and cockpit status, and facilitating maneuvers like parking:
- ClearSightNet: Monitors camera visibility, detecting conditions that impair vision, such as rain, fog, or glare from sunlight.
- ParkNet: Identifies available parking spaces.
These examples represent a fraction of the diverse and redundant DNNs within the DRIVE Software perception layer. Developing a neural program for self-driving cars is not about writing a single monolithic program, but rather orchestrating a complex ecosystem of specialized neural networks that work in concert. It involves extensive data collection, rigorous training methodologies, and powerful computing infrastructure to bring these sophisticated systems to life.
To delve deeper into NVIDIA’s approach to autonomous driving software, explore the DRIVE Labs video series.
