One year with Udacity's self-driving car engineering program

           
    Pantelis Monogioudis
           
    Head of Applied ML Research Group, Augmented Intelligence & Devices Research Lab
 
    Bell-Labs, Murray Hill, NJ, USA
iPhone

About me

20 years of experience in leading Research & Development projects in multiple countries / cultures. I started in wireless communications (PHY/MAC) in the 90s and in 2000’s started diversifying into Machine Learning initially for wireless networks and later for all sorts of applications. Currently leading ~15 researchers in Murray Hill NJ and Budapest in Applied Machine Learning working on real-time multisensory data fusion applications. 1 (failed) startup, 15 journal publications and 50 patents issued/pending. As of spring 2019 I am also with the Computer Science department of NJIT. For the last 20 years I live in Randolph, NJ with my wife Dora, my two boys Alex & Nicholas (and as of recently my dog Bleu).

We are hiring ML researchers / engineers in Paris-Saclay region - please get in touch with me via linkedin / twitter if you are interested.

DISCLAIMER

Luckily, no human or animal was injured or died as a result of our code deployment into Carla the car.

Technical Areas

  • 01. Computer Vision

    How to use the cameras of the car to find traffic lane lines, traffic lights, semantically segment the environment etc.

  • 02. Sensor Fusion

    How the vehicle's LIDAR and RADAR sensors readings are combined.

  • 03. Localization

    How to localize and track the vehicle

  • 04. Planning

    How to plan the path of the vehicle

  • 00. Architecture

    What is the high level architecture of an autonomous vehicle?

Architecture

Self-driving car subsystems

system
(The camera should have been connected also to the free-space detector.)

Carla is based on the Robotic Operating System (ROS)

ros-graph

ROS Connected Simulator for System Integration

The car is not enough - you need also a real world simulator with a good physics engine and develop the ability to communicate with ROS.

The final test - somewhere in California (March 2018)

Perception Subsystem - Vision

Sample outputs of the vision perception systems

Lane line detection

Traffic sign classification - images in the dataset (32,32,3)

Light Traffic Detection
https://github.com/pantelis/traffic-sign-classifier

Traffic sign classification

  • For German traffic signs classification, the Keras API was used to define a deep neural network and train its hyperparameters.
  • Validation set accuracy with the CNN bellow was 94.830%

Scene Understanding

Semantic Segmentation

Localization & Sensor Fusion

LIDAR

  • Velodyne VLP-16 - indoor scanning .

Localization Problem Statement

  • Localization accuracy must be 3cm-10cm.
  • To localize the car, you need to match the (distance) measurements using the onboard sensors (lidar/radar) to those of the map.
  • This is one of the reasons why good map info is important.

LIDAR-RADAR Fusion

Planning

Planning Subsystem Overview

Behavioral Planning

Trajectory Generation

Highway planning - rule based algorithms

Behavioral cloning

Closing ...

  • We didn't see anywhere in the technology parts of the program something that was considered cutting edge.
  • I am not sure if this was a purposeful made choice - to reduce either the risk of the the product or the technical debt.
  • The self driving car space has cooled down a bit after the initial euphoria - its now apparent that those that will win the race are those that have real cars equipped with cameras and other sensors that can effectively use the data to their advantage. Its difficult to beat Tesla (1B miles) or big German players in this respect.
  • Acquisitions are continuing though especially around companies that can reduce sensor costs (e.g. solid state LIDARs).
  • The next step in the evolution of self-driving is co-driving (collaborative driving between vendors) that presumes a very low cost and very low latency communication technology (probably 5G) that is 2-5 years out.

Thank you!

@monogioudis