How Artificial Intelligence is changing the game in optronic sensors

Sensors will understand, not only see

In the future, HENSOLDT sensors will be enhanced further with the introduction of more advanced embedded systems, which will include advanced levels of artificial intelligence (AI). Sensor systems traditionally rely on a computer processor close to the sensor, or several servers in an HQ, for data processing and exploitation, but the idea for next-generation systems is that the sensor itself will do much of this processing work.

This will build on existing “smart” features in HENSOLDT’s sensor systems, including automatic object detection, target tracking and moving target indication.

For cameras, this is known as embedded computer vision, which brings “intelligence directly into the camera” and reduces issues such as latency, explains Dr Michael Teutsch, Staff Scientist at HENSOLDT Optronics. “It is better when you do the data processing closer to the camera, because greater distances can increase the risk of signal loss or raise the demand for data compression. In both cases, the quality of data is affected.”

The company’s engineers are now exploring how computer vision – by utilising deep learning algorithms – can be used at the sensor level to improve areas such as threat detection and classification. Deep learning algorithms embedded in the sensor itself will not only allow devices to see the environment, but also understand it.

Learning from the consumer industry

This new era of computer vision is being enabled by the miniaturisation of electronics and a new generation of technologies including Field Programmable Gate Arrays (FPGA) and the latest Digital Signal Processors (DSP). Several industries – including medical, energy, automotive, commerce – are already utilising smart and intelligent embedded device processing (rather than sending the data to a computer or server) to speed up processes and improve areas such as fault detection.

“The consumer industry nowadays is so powerful and puts so much effort into such technology that we in the military industry can benefit here,” explains Teutsch.

In particular, HENSOLDT is leveraging the huge amounts of innovation taking place in the automotive sector and the development of self-driving cars. To safely navigate the complex environments in which they drive, self-driving cars require sensors that provide a deep understanding of the things around them. This sensing needs to be low latency for split-second decision making.

One popular technique for self-driving cars is known as semantic segmentation, which links each pixel of an image with a class label (a person, object, car, etc.). This has been improved significantly with the development of deep learning algorithms.

“This tells us for each pixel in the image, what ‘class’ it may be. It could be vegetation, trees, vehicles, humans or buildings, for example, and this already gives you the chance to distinguish between relevant and irrelevant scene content, object presence, or object motion,” says Teutsch.

“AI will help us to derive information for decision making. Imagine you are sitting in a main battle tank and there is a system that surveys the environment, it will detect a lot of motion and a lot of objects in the world, and now there is the chance to filter this information so that the operator receives an alarm only in presence of a real threat.”

Dr Michael Teutsch
Staff Scientist

This will be improved further as cameras integrate higher megapixel sensors and resolution sky rockets, which ultimately translates into more accurate object detection, recognition, and identification, as well as an increased coverage of an environment. Higher resolution systems bring their own challenges, however, including greater levels of data and how to process that in real time without delay for prompt decision making.

But as those challenges are overcome, the resulting sensor systems will not only outperform the human eye, but also the human brain’s comprehension and understanding of the surrounding environment.

Smart analysis & human decision-making

HENSOLST sensors will be able to provide wide area surveillance and rapidly identify or even predict anomalies in the environment, including how objects behave, even in crowded and congested areas such as urban settings. Sensors will be able to distinguish between a person going about their daily routine and someone acting suspiciously, as well as between a person who is holding a weapon or someone holding a bag.

These types of sensors could eventually find their way onto Europe’s developmental Main Ground Combat System (MGCS) or Future Combat Air System (FCAS) platforms, giving them a significant advantage over potential adversaries.

Human beings are still a key part of the decision-making process and will remain so for many years to come, particularly in cases where weapon systems need to be deployed. “We do not want to get rid of the human altogether, because they are important in making decisions,” says Teutsch. “What we want to do is to ease the process of decision making as much as possible, so not only with algorithms that derive information from the environment, but also with human-machine interfaces that display only the relevant information.”

More HENSOLDT Stories