Teaching Digital Neurosurgery as Fieldwork: Mixed Reality and 3D sketching at Speculative Realities Lab

At the Digital Neurosurgery seminar led by Prof. Thomas Picht this week and the next, we had the pleasure of hosting a demo by the software editor BrainLab. One of their researchers visited us at the Speculative Realities Lab, bringing along crates of high-tech gear. He presented to the students, and later to the staff, how the mixed reality could play in their Suite. Using Magic Leap AR glasses, a group of seven could explore a brain data set through the Elements neurosurgical planning software suite: a head of a patient with colorful tracks running through the white matter was suddenly floating in space. They multiplied, as students were invited to play around in the room with them. They could manipulate them by poking at them with a virtual laser.

This tool is introduced as a way to plan surgery, yet also as a powerful pedagogic tool for medical students. I could have a look at the questionnaires filled up the next day by the students and they highly appreciated the experience overall, most of them find it a useful addition to the program —an additional modality into the “anatomical intermediality” of medical training (Hallam 2020). The appeal of 3D images is used as a prestigious mark of futurity.

“Students ‘engage in reshaping their experiential world’, learning specialised vocabularies along with ways of seeing and acting that, it is argued, reconstitute persons – whether living patients or dead bodies – as ‘object[s] of medical attention’ ”

— Hallam, 2020, p. 102 quoting Good, 1994, 72-73

In the next room, on the other side of the wall, I was organizing a very rudimentary 3D sketching workshop using the commercially available app Gravity Sketch. The students ended up creating collaboratively a fuzzy scene, with a parrot and a palm tree, and lots of other scribbles.

This was a lot of fun, and curiosity spread as they were exchanging the headset to share what they had created with one another. The students reminisced about the white matter tracts that they had just seen on BrainLab. This same type of eye candy makes the neuroanatomy mixed reality application look so futuristic and appealing. 3D lines become a kind of play dough and the engagement was obviously higher than with the neurosurgical software. We concluded that combining both of these setups, a joyful interaction together with serious neuroanatomical content, could lead us to an even more potent pedagogical experience. During the next session, we explored the visual power in action in the digital image and how they shape medical practices (Burri 2013). We explored the multifarious ambiguities lurking underneath the apparent opposition between the “beauty” of an image and its “objectivity.” The conversation elevated as we went through a strange case: how and why is it that a medical practitioner can call a tumor “beautiful”? It feels like eye candy comes in many brands and many flavors.

Fieldwork with: Hyper Haptics, seminar @Weissensee Kunst Hochschule Berlin, 02.11.20

Originally invited to present my ongoing design anthropology project on haptic technologies, I’m currently taking part into and observing with great interest the product design seminar of Carola Zwick, taught by Judith Glaser and Felix Rasehorn at the Art School of Berlin in Weissensee. The presentations of week 2 were yesterday.

Prompted to explore the design of sensations, students have been given a cue to work from: an “onomatopoeia” inspired from a poetic sentence that they have picked at random. They have also been given a specific technique to explore: a memory alloy wire that moves when charged up with energy. During a series of “sprints” workshops and presentations, they are making up haptics sensations through forms and motions animated by these “muscle wires.”

This student has built diverse devices to work on the idea of the “rustling” – a sensation that can be translated in sound as well as on tactile sensation. These moving platforms are tilting under the hand of the user, creating an effect of rustling that is remotely reminding me of frolicking in the high weeds in the high summer. Colleagues students suggest to amplify the sound to make it more “rustly.”

Presentations are projected on the screen: an emphasis on the process and on the documentation of the many steps, as well as its narratives, is strongly encouraged by Carola Zwick and by the teaching team. “This is also design research,” says Judith, “things that should now appear on the CV of designers. Also, we train them to maybe continue onto the academic path.” The exercise of making quick design “sprints”, a term borrowed from computer programming, brings the students to explore ideas in their materiality more dynamically: here, a “slithering of the snake” is transduced into paper and muscle wires in many different iterations. On this phase, none of them is satisfying to the student but nonetheless, the “journey” is captivating.

This student is working on the fluidity and qualities of water. How can we translate them into these semirigid materials? The week before, he started from the fascinating folding of a single paper leaf with the muscle wire. Today, he first shows us the motion of a wave through a cut-out paper shape. Using motion sensor, he tries to convey the impression that the gesture of his hand has triggered a wave reaction in the paper material.

“The final product, which is the key feature and the main goal of most other seminars, generally outshines the whole research process. Usually, in the second week, students are working on the mood boards, doing some field research. With us, they are already in the second rounds of presentation.”

The student expressed some humor, gluing a smiley on his hand to simulate the swimmer’s movement with his fingers. “It’s always nice to have to trigger a smile in a presentation”, says Judith. She also advises to speed up the videos, to magnify the phenomenons, and get them to gain some spectacular features. “Trick it!” – however, the subtlest motions are much appreciated, an insider joy exclusive to the specific aesthetics of design researchers and other technical poets (Simondon would probably love it).

Reading an earlier version of this post, Judith thus commented on this point: “Just to make sure: This is about early prototypes where they maybe did not manage to fully implement what they were looking for yet want to communicate a certain quality, aim or finding. Post producing physical prototypes is an appropriate way to argue concepts and make a point. Because a prototype is about communicating ideas it is an »Illustration of concepts –meant to allow the user to imagine and enact the use case to open discussions.« (Massimo Banzi).”

It’s also a way to get better at writing these stories and cultivating the art of presenting the idea. As Carola Zwick pointed out in the previous session: “your clients are going to pay a lot of money for your ideas. So you own them to be able to tell them in a convincing way!”

Fieldwork: Visit at the Studio 7.5 (with Carola Zwick and Burkhard Schmitz)

Today we went to visit the Atelier 7.5 in Charlottenburg (Berlin) with the project team of Cutting. Carola and Burkhart told us all about the birth of their studio, the development of their signature chairs, the relationship that they have built with the leading furniture company Herman Miller, and the role of materiality and prototype building in their creative process.

View fullsize

In the kitchen of the studio, a central space in the social life of the studio and the theater of operation of a cook that is permanently employed by the studio, Carola and Burkhard deliver a brief history of their company. How small the first studio was, how they got the interest of that major furniture label, how they took some radical steps that made the business take off more slowly and how this cutting-edge attitude paid in the end. The studio now employs around ten people including the founders and it is now recognized as a major asset in the catalogue of Herman Miller and their products are famous world-wide.

View fullsize

Kathrin and Habakuk are testing out a new concept of chairs designed to be ergonomic even when working with a laptop: “start-up cool” and yet healthy through design! The room has tall windows that give a lot of light and a view on the water surface of the canal outside. It is filled with samples materials, on tables and on a stylish catwalk that crosses the room, on which forms are being displayed.

View fullsize

Carola explains us the story of a new concept that they are struggling to get into production, a mobile battery for conferences and offices. The many iterations of shapes and material silently testify for the hard work that is concentrated in this one latest prototype.

View fullsize

The latest product of their pioneer design line is called “Cosm”. Lucius seems convinced and is having a very brief moment of quiet. He is here caught day-dreaming of a budget extension that would allow him to sit on this chair everyday at work. Kathrin is having a look at the morphological evolution of the chairs’ iterations, which Carola brings to our attention.

View fullsize

Lucius and Thomas are listening intensely to the stories that Carola unfolds with gusto. At Studio 7.5, the iterative process and its material contingencies are core to design craft. A series of prototypes stages the development and the overcoming of technical difficulties, slowly giving shape to the latest model, which is never understood as a final development. Very comfortable chairs indeed, and especially with that recliner twist!

View fullsize

A reliable participant in the design process: the artificial human bottom. We also discuss about the possibility of making chairs to measure, or in various size. It doesn’t work, says Carola, there is some politics at work in design, esp. in the USA, and everything has to be design for everyone: “Figure yourself a situation into which they wouldn’t give you the job because you objectively don’t fit on the chair!“

View fullsize

Studio 7.5 has been experimenting for many years on additive manufacturing, from polymers to concrete. Here, Kathrin is testing the solidity of outdoor furniture that looks like a mesh of lines spread by the machine. Behind the grey cardboard room separator, a co-worker is silently making progress on a 3D-model. The studio has come a long way in controlling each aspect of the production process through hands-on experiments and prototyping. They have become experts in bridging digital models and materials products. Carola explains that they are being paid on a royalty model, as it constitutes a fair share of risk and reward for both. They are considering ourselves not being a service provider but a development partner.

View fullsize

“Demo or die!” claims Carola, the motto of the MIT of the early 80’s converted from programming to product design. In the workshop, Krzysztof is working on a piece. All has to be tested at very early stages with material models. They refrain buying new machines, because devices have great influence on the creation process. The visit is over soon after. The performance is very convincing and is a demonstration of how staging is an intrinsic part of the job of top-level designers. Materials, devices, prototypes: the true actors of this theater are given full credit for their part of the job.

Fieldwork at the Image Guidance Lab (Charité): how to "preplan" a neurosurgical operation

Today, Thomas Picht and Lucius Fekonja were teaching a group of seven students how to prepare a brain surgery. All of it happens on Brainlab, a software that is considered the “Apple” among the medical software, says Thomas.

The software is available from any computer in the hospital. The data of the MRI appears directly in the program. First part is to find the patient and access to the relevant images.

The next step is a fusion of the images that are available.

The scan appears in three dimensions, according to the series of images generated by the MRI. From those 3 series, the software generates a 3D visualisation, on the up-left corner.

The next step is to “draw” the tumor of the patient. Several tools exists in the program. On one of the three 2D images, the student “draws” the tumor by manually highlighting the borders of that slightly zone. An algorithm in the software guides this drawing by coloring automatically the tumor. The “drawing” appears as one “slice” of the tumor in the 3D representation.

After having done the same work on another 2D, the tumor appears now in 3D. the student drawing on a second of the 2D images, the tumor now appears as a 3D representation in the viewer. This has to be refined now, and depending on the clarity of the definition of the tumor, the manual result can show a sensible difference with the result of the algorithm. Thomas explains that even very experienced users can obtain slightly different drawings: the challenge is that the border is never completely sure. They discuss now about how to access that tumor, there are several possibilities with various difficulties.

What can we do after this, asks Thomas? “Tractography!” reply the students. They have been learning the day before from Lucius about that relatively new technique of visualization of the most important networks of the white matter: those related to motric and speech functions. Those “circuits” are statistically computed from MRI data (that’s for another post). Most importantly, drawing those tracks enable the surgeon to avoid damaging them during the operation.

This track, in blue, was highlighted successfully by the student. “When the track is so neatly appearing on the visualization software, you can be sure that it’s an important one”, says Thomas.

That was it for the day. Last discussion, Thomas explains about their current research interest in more advanced modes of data visualisation. Was the software clear enough? Do we need an hologram to visualise the objects in 3D? The opinions are divergent. But all seem very satisfied by this thourough hands-on introduction. Thomas conclude: “The surgeons do exactly the same that you did today. They only do it a bit faster!”