Multi-mode image fusion technology improves visual recognition when shooting a single image [Full text]

A report released by Grand View Research, a dynamic technology market research & consulting firm, recently revealed: “In 2016, image recognition dominated the application field and gained more than 40% market share. Facebook’s facial recognition function is one of the technologies. The most widely used. However, the promotion of image recognition in the two vertical areas of healthcare and defense is expected to become the industry's growth point in the next eight years. Automotive and financial services are preparing to use this technology to improve operations and improve products."

NEC and Japan National University Corporation Tokyo University of Technology Professor Oto Masaru, Professor Tanaka Masaaki's research team (hereinafter referred to as Tokyo Institute of Technology) jointly developed the "multi-mode (Note 1) image fusion technology", the general camera through AI technology The captured visible light image and the non-visible light image captured by the thermal imaging camera are automatically and efficiently synthesized, and the visual recognition degree of the single image is improved.

This technology is used in various fields where instantaneous visual recognition is required, enabling correct judgment even under harsh conditions. For example, it can also perform normal facility monitoring under adverse weather conditions such as nighttime or heavy fog; even the glare of headlights and darkness of oncoming cars can also be used for automatic driving support; not only can the cracks on the surface of the building be monitored, but also Infrastructure abnormalities such as internal abnormalities can be monitored.

In the past, it was necessary to synthesize images shot by different types of cameras requiring experts to perform complex manual synthesis operations. This technology automatically and efficiently synthesizes images captured by various cameras through AI technology without manual operation. In addition, the advantages of the visible light image and the non-visible light image are fully utilized, and even a scene that is difficult for human eyes to see clearly can obtain a high visual recognition degree.

In the future, NEC and Tokyo Institute of Technology will continue to jointly research and develop image processing and AI-related technologies that support the safe and secure operation of various social infrastructures through cooperation between industry and universities.

background

In recent years, the use of non-visible light cameras such as X-ray, terahertz wave, and millimeter wave cameras with thermal imaging cameras and trapped objects has been applied to nighttime, dense fog, etc. against the background of ever-evolving image sensors and continuous cost reductions. The situation of monitoring and diagnosis under adverse conditions such as weather, backlighting, and shadowing is becoming more and more popular. However, these non-visible light cameras have lower resolution and image quality than visible light cameras, and it is difficult to visually recognize them. Therefore, a visible light camera needs to be used in conjunction with it to monitor and diagnose images by comparing the two images. Therefore, it is difficult to quickly and accurately determine the status of a target object. To solve this problem, the best way is to combine two images into one image, but the past synthesis methods need to be manually adjusted by experts who are well-versed in the camera and photographic environment to extract the parts suitable for synthesis from the separate images. Take care to prevent problems such as overexposure and underexposure, and noise enhancement and image destruction, and perform complex image composition operations. In addition, the features included in the non-visible light for judging abnormal and dangerous substances are likely to be lost due to synthesis, which is also an important issue.

NEC and Tokyo Institute of Technology learned the AI ​​technology that transforms experts' skills. It automatically and efficiently integrates images captured by visible light cameras and non-visible light cameras to improve the visual recognition of objects and conditions, and jointly develops even harsh environments. Rapidly determine whether there is an abnormal or dangerous substance "multi-mode image fusion technology."

Fig. 1: Example of application of this technology (Note 2)

The advantages of new technologies

The AI ​​technology automatically selects images with high visibility from a plurality of images, and performs image synthesis while emphasizing fine features included in non-visible light, thereby realizing highly visual recognition that breaks the limitations of the past.

The AI ​​technology evaluates the degree of visual recognition of various parts of the image based on different conditions such as the thermal imaging camera, camera types such as terahertz cameras, and environmental characteristics (brightness, light direction, and obstacle-free objects). In the automatic extraction of the most appropriate part.

In addition, when the AI ​​technology processes images captured by non-visible light cameras, it analyzes the fine features of abnormal and dangerous materials, and performs appropriate strength and weakness adjustments to prevent the images from being destroyed due to overexposure and underexposure, which is automatically generated in the past and cannot be realized. High-visuality multi-mode (visible-non-visible light) fused image.

(Note 1) Multimodal

Multiple modes and forms. The multimode mentioned in this article refers to images captured by visible light cameras and images captured by non-visible light cameras such as infrared cameras.

(Note 2) Acquire and use the Tough Robotics Challenge of the Innovative R&D Promotion Project (ImPACT) for harsh environment simulation equipment.

The original title NEC "Multi-mode image fusion technology" uses AI technology to improve visual recognition under severe conditions