NVIDIA CEO Jensen Huang's presentation at this year's Consumer Electronics Show started out focused on the company's traditional strength in gaming computers and display technology. But the parade of new laptops and vivid monitors took an unusual turn when Huang introduced a video of a most extraordinary car.
For eight miles the black Lincoln MKZ sedan zips along the rural lanes of Holmdel, New Jersey, without any human guidance. As the driver sits with his hands in his lap and feet off the pedals, the steering wheel spins on its own, as if controlled by a ghost.
The person behind the wheel is Chenyi Chen, a research and development engineer for NVIDIA, a leading manufacturer of hardware and software platforms for visual computing and artificial intelligence. Chen, a scientist in NVIDIA’s team of thousands working to give cars the ability to “see” the road, earned his Ph.D. at Princeton in 2016.
The information about a car’s surroundings that helps it to navigate comes from several different systems, including radar and laser scanners, but cameras provide essential information that these other systems cannot.
“A laser scanner can only tell you the position of a speed limit sign, it cannot tell you its content,” said Chen, whose doctorate is in operations research and financial engineering.
Chen, who studied under Professor Alain Kornhauser at Princeton, first connected with NVIDIA through Kornhauser’s work on smart driving cars. Urs Muller, the director of developer technology for NVIDIA Automotive, got to know Kornhauser through the professor’s efforts to support smart-driving research in New Jersey.
“Urs learned about the research that Chenyi and I were doing and immediately realized that Chenyi could fit right into his company,” said Kornhauser, a professor of operations research and financial engineering.
Danny Shapiro, also a Princeton alumnus and NVIDIA’s senior director for automotive, said research plays a key role in the company’s mission. NVIDIA started making graphics processors for personal computers, but in recent years the processors’ applications have skyrocketed and are found everywhere from the datacenter to the car. Graphics processing units, or GPUs, now form the backbone of computing technologies used for scientific discovery, as in particle physics, genome sequencing and energy exploration. Transportation, which is increasingly relying on big data and artificial intelligence, is an important new sector for the company.
Shapiro, who earned a BSE from Princeton in electrical engineering and computer science in 1987, said scientists are working to develop software that can accurately perceive the environment and make complex decisions quickly and reliably.
“We have a very large research team at NVIDIA. One group is looking very long-term at building new models of AI technology,” he said. “We also have a group that does applied research, focusing on how to take recent discoveries and apply them to problems that have not yet been solved.”
Because so much of the science is new, the company often works with students and new graduates. “There are a great deal of opportunities for undergraduates through our internship program, as well as for new grads, and postdocs,” he said.
Some of the NVIDIA research in New Jersey involves speeding up the car’s ability to process visual information from the road. Latency, the time between the camera seeing an image and the car’s reaction, poses a challenge. The time lag results from two sources: software, which needs time to process camera images and use them to understand the environment; and the automobile itself, which is a mechanical system.
“Even if at this moment you tell the car to steer to the left‚Ķ the actual moment the car starts to steer is not exactly the same moment when you tell the car to do so,” Chen said. Computers can be made faster, and processing algorithms more efficient, but mechanical delay-something humans deal with routinely-is harder to manage.
This mechanical latency can be mitigated by using the software to predict the future-two images in sequence can be used to extrapolate the trajectories of nearby objects, for example. The autonomous vehicle must also carry a memory of past events to execute actions that require milliseconds (keeping a car in a lane), or seconds (making a lane change), or minutes (driving from one location to another). And some of the algorithms that power NVIDIA’s test vehicle, learn from other, human drivers.
“In addition to our primary focus on safety, we try to teach the car good manners,” Chen said.
Engineering self-driving vehicles is one problem; mass acceptance of such vehicles is another. To the person who has never experienced a self-driving vehicle, giving over control over can be strange and unsettling.
“But after a short while the driver realizes that the car really behaves very, very well. The person behind the wheel become very relaxed,” Chen said, having long overcome such concerns himself.
But these autonomous vehicles will have to exist on the same roads with vehicles piloted by human drivers, who exceed speed limits, or who make mistakes through carelessness or inattention.
“This is actually the tricky part,” Chen says. “There is no magic that can replace all human-driven cars with autonomous cars instantly. These vehicles will have to deal with the bad drivers on the roads and all sorts of other complicated things.”
What does a future look like in which such vehicles are mass produced and widely available? All vehicles will not be replaced by fully autonomous ones immediately, but the technology will be able to augment human driven cars, giving consumers another option, Chen and Shapiro said.
“We are working with a broad range of car makers, truck makers and shuttle companies on a new wave of autonomous vehicles. Some will be fully autonomous robotaxis, and others will be similar to cars we have today, just much safer,” said Shapiro. “In some future vehicles you will have a choice–when you want to drive, for example along California’s beautiful Coast Highway 1, you control the vehicle. However, for long-distance driving, for very tedious, very boring driving, the car can drive itself. And even if a car is not fully 100% autonomous, our technology will make the driving much safer and more relaxing.”