Brain surgery training from an avatar

Benjamin Warf, a renowned neurosurgeon at Boston Children’s Hospital, stands within the MIT.nano Immersion Lab. More than 3,000 miles away, his virtual avatar stands next to Matheus Vasconcelos in Brazil because the junior doctor performs a fragile operation on a doll-like model of a baby’s brain.

Wearing virtual reality glasses, Vasconcelos can watch as Warf’s avatar demonstrates brain surgery before replicating the technique himself and asking questions of Warf’s digital twin.

“It’s an almost out-of-body experience,” Warf said as he watched his avatar interact with the residents. “Maybe that is what it seems like to have a similar twin?”

And that is the goal: Warf’s digital twin bridged the gap, allowing him to functionally be in two places without delay. “It was my first training with this model and it performed excellently,” says Vasconcelos, a neurosurgery resident on the Santa Casa de São Paulo School of Medical Sciences in São Paulo, Brazil. “As a resident, I now feel more confident and comfy using the technique on an actual patient under the guidance of a professor.”

Warf’s avatar was created as a part of a brand new project launched by a medical simulator and augmented reality (AR) company EDUCSIM. The company is an element of the 2023 cohort START.nanoMIT.nano’s deep tech accelerator, offering early-stage startups discounted access to MIT.nano’s labs.

In March 2023, Giselle Coelho, scientific director of EDUCSIM and pediatric neurosurgeon at Santa Casa de São Paulo and Sabará Children’s Hospital, began working with technical staff on the MIT.nano Immersion Lab to create Warf’s avatar. In November, the Avatar trained future surgeons like Vasconcelos.

“I had the thought to create the avatar of Dr. “Thought to create a proof of concept and asked, ‘Where on this planet would they work on technologies like this?’” says Coelho. “Then I discovered MIT.nano.”

Capture of a surgeon

As a neurosurgery resident, Coelho was so frustrated by the dearth of hands-on training options for complex surgeries that she built her own model of a baby brain. The physical model incorporates all of the structures of the brain and might even bleed, “it simulates all steps of an operation, from incision to skin closure,” she says.

She soon discovered that simulators and virtual reality (VR) demonstrations shortened the educational curve for her own residents. Coelho launched EDUCSIM in 2017 to expand the variability and reach of coaching for residents and experts who need to learn latest techniques.

These techniques include a procedure for treating hydrocephalus in infants developed by Warf, the director of neonatal and congenital neurosurgery at Boston Children’s Hospital. Coelho had learned the technique directly from Warf and thought his avatar could offer surgeons who couldn’t travel to Boston the chance to learn from his expertise.

To create the avatar, Coelho worked with Talis Reks, the AR/VR/Gaming/Big Data IT technologist at Immersion Lab.

“Access to a whole lot of technology and hardware will be very expensive for startups at the start of their business trip,” explains Reks. “START.nano is a approach to enable them to make use of and pay for the tools and technologies now we have within the MIT.nano Immersion Lab.”

Coelho and her colleagues needed high-resolution, high-fidelity motion capture technology, volumetric video recording, and a spread of other VR/AR technologies to capture Warf’s deft finger movements and facial expressions. Warf visited MIT.nano several times to be digitally “captured,” including by performing surgery on the physical baby model while wearing special gloves and clothing with built-in sensors.

“These technologies were primarily used for entertainment or VFX (visual effects) or CGI (computer-generated imagery),” says Reks. “But this can be a unique project because we are actually applying it to real medical practice and real learning.”

One of the most important challenges, Reks says, was helping develop what Coelho calls “holoportation” – transmitting Warf’s 3D volumetric video capture in real time over the Internet in order that his avatar could appear in transcontinental medical training.

The Warf avatar has synchronous and asynchronous modes. The training Vasconcelos received was in asynchronous mode, where residents can watch the avatar’s demonstrations and ask him questions. The answers, provided in various languages, come from AI algorithms based on Warf’s previous research and extensive database of questions and answers.

In synchronous mode, Warf controls his avatar remotely in real time, says Coelho. “He could walk across the room, he could refer to me, he could orient me. It is wonderful.”

Coelho, Warf, Reks and other team members demonstrated a mix of the modes in a second session in late December. This demo consisted of a live volumetric video capture between the Immersion Lab and Brazil, spatialized and viewable in real time via AR headsets. It’s a major expansion from the previous demo, which streamed volumetric data in just one direction across a two-dimensional display.

Powerful effect

Warf has a protracted history of coaching desperately needed pediatric neurosurgeons all over the world, most recently through his nonprofit organization Neurokids. Remote and simulated training has been an increasingly necessary a part of education for the reason that pandemic, he says, although he doesn’t consider it can ever fully replace in-person, hands-on instruction and collaboration.

“But if someday we could even have avatars like this of Giselle in distant places, showing people the way to do things and answering questions for them, with no travel costs, no time commitment, etc., then I believe it could possibly be really powerful.” says Warf.

The Avatar Project is especially necessary for surgeons who serve distant and underserved areas equivalent to the Amazon in Brazil, says Coelho. “This is a approach to give them the identical level of education that they might receive elsewhere and the identical opportunity to work with Dr. Threw to remain in contact.”

According to Coelho, a baby recently treated for hydrocephalus at an Amazon clinic traveled 30 hours by boat for the operation.

She says training surgeons with the avatar “can change that baby’s reality and the long run.”