AI 2019: The Year Of Virtual Reality Environments

Posted: 6/17/2019

Perhaps, 2017 was the year when Generative Adversarial Networks (GANs) took off and became a serious and fast expanding research subject in AI.

Now in the middle of 2019, it appears that new and drastically improved, very realistic simulators or AV/VR environments might well be the highlight of AI for this year. This is a new class of interactive environments for AI research.

It appears that these new software platforms/environments/simulators are coming ever closer to being very realistic environments for AI. These new environments promise a great speed up in training of robots and other advances in AI.

Here are some relevant quotes (emphasis added):

1. “But researchers believe these systems will learn best if the simulated environments capture the subtle details — like mirror reflections and rug textures — necessary to make them virtually identical to the real thing.” (S1)
2. ““The Replica data set sets a new standard in the realism and quality of 3-D reconstructions of real spaces,”” (S1)
3. “Replica can be loaded up in AI Habitat, a new open platform for embodied AI research. Facebook AI created AI Habitat to be the most powerful and flexible way for researchers to train and test AI bots in simulated living and working spaces.” (S1)
4. “But it is also an important research tool for creating next-gen AR experiences that begin to merge the physical and digital worlds” (S1)
5. “While other simulation engines commonly run at 50 to 100 frames per second, AI Habitat runs at over 10,000 frames per second (multi-process on a single GPU). This enables researchers to test their bots much more quickly and effectively — an experiment that would take months on another simulator would take a few hours on Habitat.”
6. “For example, Facebook AI researchers will explore ways to build realistic physics modeling into AI Habitat so an AI bot can learn what happens when it knocks a virtual glass off a virtual table” (S1) [Too bad that physics modeling is not yet integrated]
7. “We present Habitat, a new platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation, before transferring the learned skills to reality.” (S2)
8. “Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale. Each scene consists of a dense mesh, high-resolution high-dynamic-range (HDR) textures, per-primitive semantic class and instance information, and planar mirror and glass reflectors.” (S3)
9. “On the other hand, this variety of [previous] simulation environments can cause fragmentation, replication of effort, and difficulty in reproduction and community-wide progress. Moreover, existing simulators exhibit several shortcomings … Most critically, work built on top of any of the [previous] existing platforms is hard to reproduce independently from the platform, and thus hard to evaluate against work based on a different platform, even in cases where the target tasks and datasets are the same.”
10. “By operating at 10,000 frames per second we shift the bottleneck from simulation to optimization for network training” (S3) [This sounds very desirable!]

Sources (S):

1. How Facebook researchers’ realistic simulations help advance AI and AR Replica and AI Habitat are key to developing experiences like social presence and intelligent assistants (published 6/14/19)
2. Habitat: A Platform for Embodied AI Research (published 4/2/19)
3. The Replica Dataset: A Digital Replica of Indoor Spaces (published 6/13/19)
4. EvalAI: Towards Better Evaluation Systems for AI Agents (published 2/10/19)
5. Evaluating state-of-the-art in AI