Self-driving cars, safety for the elderly and even search and research missions can benefit from new techniques for image reconstruction.

MIT-IBM Watson AI Lab: robots take over parts of your task, not everything
Researchers predict that there is still a lot of demand for designers at all wage levels and employees with in-depth knowledge of the industry.

The combination of cameras and computers can do everything from fighting forest fires in California to finding survivors in natural disasters. And MIT uses them to help people look around the corner.

Seven years ago, MIT researchers created a new imaging system that used floors, doors, and walls as mirrors to provide information about scenes beyond normal line of sight. Special lasers were used that produced 3D images.

Based on that original work, MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) has developed a simple computational imaging method that can reconstruct hidden video, according to a press release. The findings will be published next week in a paper at the conference on neural information processing systems in Vancouver.

Using shadows and light reflections observed from a pile of junk, MIT researchers can reconstruct video from an unseen area of ​​the same room, even if it is out of sight of the camera, according to research team member, CSAIL PhD student Prafull Sharma, who is one of the authors of the article is, along with Lukas Murmann and researcher Adam Yedidia, and MIT professors Fredo Durand, Bill Freeman and Gregory Wornell.

SEE: Special function: autonomous vehicles and the company (free PDF) (TechRepublic)

“If you just look at the pile of junk, you wouldn’t imagine we could reconstruct something meaningful from it,” Sharma said. “It is very fascinating to do this, but when you talk about applications, there is this whole scope of application: search and rescue missions, elderly care, autonomous vehicles, etc.”

How they revealed hidden info

To set the tone, Sharma set an example. Imagine standing in a room and looking at an object or a stack of objects. You cannot see what is behind you, or maybe what is happening outside of your vision in another part of the room. However, you can see some vague shadows caused by the stack of objects, which could indicate that something is happening out of your sight.

“The goal was, is this enough information to deduce something about the hidden (situation), and can we actually reconstruct it? It’s a linear problem, which means that what you perceive is actually the multiplication of two matrices,” said Sharma.

The two matrices are light transport and hidden video. Light transport is about the way light travels in a scene, which is used to estimate hidden content from images, he said.

The process begins with turning on a video camera in a room that acts as a field of vision set on a pile of junk. The pile of junk almost acts as a pinhole camera that you would build in a primary school science class, according to the press release.

“(The clutter) blocks some rays of light, but lets others go through it, and these give a picture of the environment wherever they go,” the press release said. “But where a pinhole camera is designed to allow just the right amount of rays to form a readable image, a general pile of rubbish produces an image that is unrecognizable (due to light transport) transformed into a complex play of shadows and shadows. ”

The clutter almost acts as a mirror and offers a distorted idea of ​​the environment. By multiplying the light transport and the hidden video, the team was able to give a rough idea of ​​what was happening in the hidden scene, Sharma said.

To estimate both matrices, the team used convolutional neural networks. “These are a type of neural network that produces image-like structures,” Sharma said. “If you think about it, these two matrices have image-like structures along different dimensions. For example, the hidden video, each frame is an image. On the side of the light transport matrix, for each pixel when it was illuminated, you were getting an image.

“We have a convolution operation on these dimensions to create image-like structures. Together we optimize it so that the output of these two neural networks should multiply and equate to what was observed,” he said.

One neural network produces the distortion pattern of light and shadows, the other estimates the hidden video and the combination is capable of reproducing an idea of ​​hidden information, according to Sharma.

After the camera has recorded a video of the clutter, the team transfers the content to a graphics processing unit (GPU). After the video was produced, the team used the networks implemented in PyTorch, an open source machine learning library from Facebook, Sharma said.

Use cases

Currently, the team’s process takes two hours, but if the hidden information can be discovered in real time, the applications can be very impactful, Sharma said.

“You could imagine that if you wanted to know what was in the room during a search and rescue operation without even entering, you could use such a technique,” Sharma said. “Or for fall detection for the elderly, because you cannot place cameras anywhere, but cameras are likely to be able to observe parts of the room or areas where these features may be present. This allows you to see the hidden part of the room and you can basically estimate whether someone has fallen and so on. ”

An extremely powerful application could, according to Sharma, be autonomous vehicles.

“Imagine a parking space and you can’t always see what’s around the corner, but there may be a pile of clutter that can be detected,” he continued. “If our technique can be optimized to be executed in real time, we can reconstruct the hidden part or situations around the corner. This could prevent collisions, or if people walk by, it can actually give a warning that there might be people ”

As a next step, the team hopes to improve the overall resolution of the system and ultimately test it in an uncontrolled environment, according to the press release.

(Embed) https://www.youtube.com/watch?v=hhEJMpouMS8 (/ embed)

For more information, see MIT’s self-assembling robots, literally on TechRepublic.

Next Big Thing newsletter

Be up to date with smart cities, AI, Internet of Things, VR, autonomous driving, drones, robotics and more of the coolest technical innovations.
Delivered on Wednesday and Friday

Register today

Also see

Image: MIT

Similar Posts

Leave a Reply