Reverse engineering the sensory-motor control for schooling in zebrafish using virtual reality

Amazing stuff! This could be a breakthrough!

"... Now, using an immersive virtual reality (VR) setup, scientists have figured out the sensory-motor mechanism underlying this harmonious behavior.

While inside their 3D arena, which researchers dubbed “the Matrix,” individual zebrafish could interact with “holographic” projections of other fish. By analyzing the animals’ reactions and tweaking the movements of virtual “leader” fish, the team determined that “follower” fish used the perceived position—rather than the speed—of their neighbors to guide their own movements. “We were surprised by how little information the fish need to effectively coordinate movements within a school ,” ... Using this natural control law, the researchers programmed a variety of wheeled robots, drones, and robotic watercraft; they were able to follow targets just as well as robots programmed with an existing, state-of-the-art controller—at a fraction of the complexity. ..."

"To the point

• Innovative method: A team of biologists and robotic engineers have developed a virtual reality system for fish to decipher how they school
• Discovering nature’s algorithm: They uncovered the natural ‘control law’ that is used by zebrafish to coordinate behavior with others, a behavioral algorithm that has been tuned over millennia to facilitate effective collective motion.
• Implications for robotics: They tested the natural control law in groups of robotic cars, drones and watercraft, demonstrating its potential for the control autonomous vehicles in the future.

... Using a virtual reality (VR) setup that mimics natural schooling, researchers placed individual juvenile zebrafish into networked arenas where each fish could freely interact with ‘holographic’ virtual conspecifics. Each virtual fish was a projection of a real fish, meaning that fish could swim and interact together in the same virtual world.

The fully immersive 3D environment lets researchers precisely manipulate visual stimuli and record how the fish respond. This high level of control allowed the scientists to isolate exactly which cues the fish were using to guide their interactions with other fish. In other words, they could reverse engineer the behavior of schooling in zebrafish to understand how fish solve the complex problem of coordinating their motion.

The solution, they discovered, was a simple and robust law based only on the perceived position, not the speed, of their neighbors to regulate their following behavior. ..."

From the abstract:

"Revealing the evolved mechanisms that give rise to collective behavior is a central objective in the study of cellular and organismal systems. In addition, understanding the algorithmic basis of social interactions in a causal and quantitative way offers an important foundation for subsequently quantifying social deficits.

Here, with virtual reality technology, we used virtual robot fish to reverse engineer the sensory-motor control of social response during schooling in a vertebrate model: juvenile zebrafish (Danio rerio).

In addition to providing a highly controlled means to understand how zebrafish translate visual input into movement decisions, networking our systems allowed real fish to swim and interact together in the same virtual world. Thus, we were able to directly test models of social interactions in situ.

A key feature of social response is shown to be single- and multitarget-oriented pursuit. This is based on an egocentric representation of the positional information of conspecifics and is highly robust to incomplete sensory input.

We demonstrated, including with a Turing test and a scalability test for pursuit behavior, that all key features of this behavior are accounted for by individuals following a simple experimentally derived proportional derivative control law, which we termed “BioPD.”

Because target pursuit is key to effective control of autonomous vehicles, we evaluated—as a proof of principle—the potential use of this simple evolved control law for human-engineered systems. In doing so, we found close-to-optimal pursuit performance in autonomous vehicle (terrestrial, airborne, and watercraft) pursuit while requiring limited system-specific tuning or optimization."

ScienceAdviser

In VR school, fish teach robots (original news release) "Scientists use virtual reality for fish to teach robots how to swarm"

Reverse engineering the control law for schooling in zebrafish using virtual reality (open access)

Fig. 1. Schematic of the study of SMC of schooling behavior.

Fig. 2. Reverse engineering SMC of fish to a bioinspired proportional-derivative controller, BioPD.