Infrastructuring the multimodal: Workshop at the University of Cambridge


Caroline is singing us a song. And then in the end she asked us to sing together for ourselves. Not to save anyone’s soul, just to feel how it is to be in the song mode.

For two days, Iza Kavedžija, Liana Chua, and Natalia Buitron brought a group of inventive ethnographers from across Europe to the Center for Research on the Arts, Social Sciences and Humanities. They invited us to three panels — visual, sonic, and literary practices — for a conversation around the future of multimodal ethnography. How can we value and evaluate academic works that operate at the border with artful practices? How can we accommodate unusual academic outputs in a journal such as the Cambridge Anthropology Journal?

The workshop was strongly reminiscent of a project conducted by three of my colleagues at the Humboldt University. Iza opened her introduction with an acknowledgement of this earlier project, which bears the same subtitle: “Evaluating multimodal forms.” I know these colleagues well. I co-taught multimodal ethnography with two of them in recent years. Back in Berlin that same week, I met with Andrew Gilbert and Ignacio Farias. They were a bit perplexed by the workshop in Cambridge. They were surprised they weren’t invited to that Cambridge gathering —but it’s just that the word hadn’t gone around yet. These things happen. Tomas, Ignacio and Andrew have just completed a manuscript of their toolkit for multimodal appreciation, which they are submitting to the multimodal section of the American Anthropologist journal.

It is significant that we met to evaluate two master's theses, getting the students through their defences. Both students were very inspired by the multimodal ethnography teaching. Max did the ethnography of a VR project related to ageing. Alexander got started with fermentation workshops and went on to develop a thesis on fungal biohacking communities in Berlin. Both of them could have used multimodal material in their thesis. Yet they made only use of text, though, in a rather straightforward academic way. Not that the guidelines are especially rigid. It just doesn’t pay extra to go that extra mile, and the students seem to be acutely aware of it.

It doesn’t seem to pay more to go the extra mile in professional academic ethnography either. And this was at the core of the conversations we had during these two days in Cambridge. The extra work is not accounted for in the point systems of the neoliberal university. So why do anthropologists, like the group that was gathered in Cambridge that day, keep pushing that border? Two arguments came up a lot. 1) Other media and mediations make us see and feel things differently, and the ethnographic media can be integral to the ethnographic argument. 2) Making ethnography different is fun and keeps us alive. These two things, of course, go together. But aren’t these two arguments applicable to any good ethnography today? That’s the question I asked Julia Offen, the editor of anthropology and humanism, and the main invited provocation on the literary panel. So there is the boring stuff on one hand and the good, creative stuff on the other hand? Wasn’t ethnography always multimodal? In short, yes, replied Julia. Until the discipline weighted itself down with value criteria derived from big science to justify its existence (and its funding).

American Anthropologist was the first journal to get a multimodal section going. Andrew, as well as his colleagues of the “Multimodal Appreciation” project in Berlin, aren’t convinced that having a separate section for multimodal work is the way to go. The term will die, Andrew tells me over coffee, because it will be absorbed by the field of anthropology again. He quotes (loosely but precisely) a sentence from the last editorial of the Entanglement journal, a multimodal venue that closed shop in 2022: “This rush to tell the story of multimodal ethnography, to stake a claim and to define, feels really premature; it forecloses, it feels to us, experimentation which is at the heart of multimodality.” (1)

So maybe it’s not about staking a claim at all. Not about carving out new territories or formalising yet another sub-discipline. Maybe it’s about resisting that very impulse — the impulse to fix, to define, to institutionalise too early. Maybe it’s about staying with the looseness, the makeshift, the experimental — not because it’s easier (it’s not), but because that’s where ethnography comes alive. In the tentative, the collaborative, the not-yet-named. What if the point is not to assert what multimodal ethnography is, but to keep asking what it can be? Maybe it’s about holding open space for that singing-together moment — not to save anyone’s soul, but just to be, for a while, in the mode of the song.

(1) Nolas, M., Varvantakis, C., Long, R., Walton, E., and Logan, B. (2022). Last but not least, entanglements, 5(1): 1-7

Fieldwork in the art-science VR scene: Sophie Erlund & the Experimenting, Experiencing, Reflecting group @PSM Gallery

Yesterday at the PSM Gallery in Schöneberg (Berlin), Sophie Erlund was presenting half a day of workshop and presentations around her VR piece Nature is an event that never stops. She has been working on that piece for the last six months, and it will be on display until 25.02.2023. It’s fantastic, definitely worth a trip to this part of the city where a cluster of galleries is established, on the opposite side of the canal from the Neue Nationalgalerie.

The artwork is a collaboration with the EER group, a mixed bunch of scientists and artists interested in playful approaches to inquire into perception. They offer pathways between cognitive science and installation art, using a “French” method to conceive and evaluate these hybrid setups: microphenomenology. The cognitive scientist Katrin Heimann interviewed visitors and presented her first conclusions in the first part of the workshop in text-heavy slides. She invited us to close our eyes as she channeled the experiences “in first-person” that she had collected. In the second part, Karsten Olsen presented his work on “cultural transmission,” the cognitive mechanism that is explored in the piece by Sophie Erlund. How are sketches or colors transformed as they are memorized and transmitted?

Microphenomenology is the child of the philosopher Claire Petitmengin. The method is elaborating on a technique called the “elicitation interview” which was invented by Pierre Vermersch (1994) and further developed in collaboration with Nathalie Depraz and Francisco Varela as a “phenomenological practice” in the book On Becoming Aware: A pragmatics of experiencing (2003). It is a delicate practice, a deep dive of the interviewer and the interviewee into the sensations and thoughts that occurred during a targeted experience. Claire Petitmengin has conducted research in contemplative science for the last 30 years. The method she teaches indeed shares many principles with meditation, evolving self-inquiry into interpersonal approaches. The method came to me in a new light influenced by the work of EER member Joe Dumit: what if we took it as a contact improvisation exploration inside and around a phenomenon? That day, we centered the interpersonal perceptive field on the lived experience of unprepared visitors going through the VR “film” of Sophie Erlund.

The final panel featured the artists Sophie Erlund and Helene Nymann, the neuroscientist, poet, and brain activist Pireeni Sundaralingam, and the anthropologist and cognitive scientist Andreas Roepstorff. It was introduced by Helena Herzberg, the art manager of PSM Gallery. The session was touching on many topics ranging from cognitive flexibility in school kids to funding schemes for art-science endeavors. Excellent both in content and shape, this two-hour conversation strokes a powerful chord. The participants took the context of EER and the production of the VR piece as a start. The focus went quickly to the serious political implication of bringing together in playful ways the knowledge of cognitive neuroscience and that of artists. Challenged by the audience, speakers stated the urgency of encouraging resistance against the pervading rigidity of the educating system, as well as the uncertainty-adverse societal context. Staging and sharing more-than-human perceptions, as Erlund does so masterfully in her VR film, is one of the many ways to champion tender attention to other people's experiences. As the EER member and play scholar Amos Blanton puts it in the end: bringing children and adults into creative exercises brings up the fears of failing and coming short. Yet the moment when they become engaged in the aesthetic activity, these fears dissolve and the joy of tinkering takes over. Hearing these words, I lifted up my head from my iPad for a short moment, reflecting on graphic ethnography and my experience of teaching ethnography through drawing. So much food for thought today, and great insights on how to frame an art-science VR experience. The panel, as well as the other interventions of the day, were documented and will be available soon on https://www.eer.info/

All the info is on the website of the gallery.

Sharing my sketchy fieldnotes on the spot with anthropologist colleague Andreas Roepstorff w/ ©Yoonha Kim, exchanging first-person impressions in embodied way at the end of the day.

Teaching Digital Neurosurgery as Fieldwork: Mixed Reality and 3D sketching at Speculative Realities Lab

At the Digital Neurosurgery seminar led by Prof. Thomas Picht this week and the next, we had the pleasure of hosting a demo by the software editor BrainLab. One of their researchers visited us at the Speculative Realities Lab, bringing along crates of high-tech gear. He presented to the students, and later to the staff, how the mixed reality could play in their Suite. Using Magic Leap AR glasses, a group of seven could explore a brain data set through the Elements neurosurgical planning software suite: a head of a patient with colorful tracks running through the white matter was suddenly floating in space. They multiplied, as students were invited to play around in the room with them. They could manipulate them by poking at them with a virtual laser.

This tool is introduced as a way to plan surgery, yet also as a powerful pedagogic tool for medical students. I could have a look at the questionnaires filled up the next day by the students and they highly appreciated the experience overall, most of them find it a useful addition to the program —an additional modality into the “anatomical intermediality” of medical training (Hallam 2020). The appeal of 3D images is used as a prestigious mark of futurity.

“Students ‘engage in reshaping their experiential world’, learning specialised vocabularies along with ways of seeing and acting that, it is argued, reconstitute persons – whether living patients or dead bodies – as ‘object[s] of medical attention’ ”
— Hallam, 2020, p. 102 quoting Good, 1994, 72-73

In the next room, on the other side of the wall, I was organizing a very rudimentary 3D sketching workshop using the commercially available app Gravity Sketch. The students ended up creating collaboratively a fuzzy scene, with a parrot and a palm tree, and lots of other scribbles.

This was a lot of fun, and curiosity spread as they were exchanging the headset to share what they had created with one another. The students reminisced about the white matter tracts that they had just seen on BrainLab. This same type of eye candy makes the neuroanatomy mixed reality application look so futuristic and appealing. 3D lines become a kind of play dough and the engagement was obviously higher than with the neurosurgical software. We concluded that combining both of these setups, a joyful interaction together with serious neuroanatomical content, could lead us to an even more potent pedagogical experience. During the next session, we explored the visual power in action in the digital image and how they shape medical practices (Burri 2013). We explored the multifarious ambiguities lurking underneath the apparent opposition between the “beauty” of an image and its “objectivity.” The conversation elevated as we went through a strange case: how and why is it that a medical practitioner can call a tumor “beautiful”? It feels like eye candy comes in many brands and many flavors.

Awake Surgery as Fieldwork: snippet from the "Sketching Brains" participant exhibition project at Charité Berlin

Awake surgery has been a hot spot of neurosurgical research and a growing site of interest for journalists and scholars. The idea is simple yet radical. The patient has a tumor that lies close to the “eloquent areas.” Removing this tumor puts at risk their cognitive abilities, in particular those related to speech. Instead of waiting for them to wake up in order to assess potential damage, which may then be irreversible, why not keep them wide awake during the process, and simulate with low-current electricity the effect of potential lesions on the brain? The result is a very atypical surgical situation with a speaking patient, a patient which is aware of what is happening in the room. This calls for a mastery of the atmosphere around the patient, an atmosphere that requires perfectly rounded-up teamwork to even out possible stressful situations. Here is a draft of one of the graphic ethnographic “series,” that I am presenting in the “Sketching Brains” participant exhibition at Charité in Berlin.

Credit MLC

That morning as I follow Katharina Faust to the OR, she tells me about the “very big tumour” she will have to expurgate from the patient’s brain. The patient is an artist. The OR is quiet, that particular patient seems quite sedated. Katharina bends over the patient, and with a very soft voice, greets her and asks her how she’s feeling today. Mehmet Tuncer, a junior resident who will be stay in contact with the patient during the whole operation, is already at her side. He has already built a relation of trust with the lady? The sounds are ushed and the movements are quiet.

Credit MLC

Three students are in the room this morning: the OR keeps it character of theater and it is very usual to have a small audience for the neurosurgeons at the University Hospital of Charité, which is the leading training center in Germany and in Europe. A few more signatures for Katharina, who seems to be signing papers all day, even a few away from a feat of surgical mastery.

Credit MLC

The skull of the patient is held into a metal clasp, between two dented spikes, as in any other neurosurgical intervention. What is usually a routine affair of a few seconds take longer when the patient is awake, as this is one of the least comfortable feature of the intervention: feeling one’s head stuck into a jaw of steel. A decent amount of local anesthetics makes it a bit easier. The instrument is shown to the patient to reduce her anxiety, with a few sympathetic words.

Credit MLC

A little electric trimmer buzzes in the hands of Katharina as she clears the space where she will incise the scalp. At the site where the outside world is about to smuggle in, a generous spraying of antiseptic solution is keeping the danger of a bacterial incursion at bay. 

The head of the patient is also “registered” for the virtual representations of the screen to match with the actual body. A stereoscopic set of cameras is helping the system keep track of the scene, orienting itself of the bearing of the four balls assembled into a cross. On the three big screens of the OR, the scans show the bulky tumour in dark, dangerously close to the colorful lines hinting at the presence of language tracks pathways in the connective tissues of the white matter. 

Credit MLC

When the trimming is done, Dr Faust covers the patient’s head with adhesive blue paper protections. Only the site of the craniotomy will be apparent from this side. On the opposite side of the bed, an assemblage of machines has taken over part of the vital functions of the body for the length of the operation.

Credit MLC

My tablet is running out of battery. The last image I can sketch is in the middle of the langage mapping: as Katharina is touching the brains with a pair of electrodes, triggering a temporary inactivation at a determined point, Mehmet is showing to the lady a set of images. Every time Mehmet says “anomie,” it means that the word, like “hand” or “cat,” came out wrong from the patient’s mouth. Katharina then duly puts a numbered piece of paper on the area she just probed, and registers it into the machine with a long pointer. Another round of images focuses on actions. This series of linguistic tasks, coming straight from neuroscientific experimental processes, makes it possible to assess if that specific part of the cortex is critical to the speech functions. The lady is getting tired, but the key information is in. The resection can start! It’s a impressive moment which I won’t be able to render today in the graphic form, as my iPad calls it a day. Next operation is two days, later. That’s exciting!

Conference-as-fieldwork: EASA 2022 in Belfast

So happy to be in Belfast for a dip into our fleshy community of scholars! Ethnographers can’t help it, they attend conferences ethnographically. Here is a graphic report.

The ethnographer at work.

Sitting in the corner that will offer the clearest picture is more often than not a deliberate choice.

Teaching-as-Fieldwork: Open Design studio course "Growing Senses"

The Open Design Master students are moving forward on their Laboratory Project, with the common topic of »Growing Senses«. Laboratory Project is the main studio teaching of the third semester of this program between the Faculty of Architecture of the University of Buenos Aires and the Institute for cultural and media studies of Humboldt University. As the mid-term presentations are approaching, I sat with my colleagues designer Frank Bauer and architect Bastian Beyer for a feedback round. Many projects are emerging: we are joining them to design shape-shifting shading devices, protocols for sensing emotional territories, outer heart fountain flows, plant ritual accessories for extra-planetary life, mycelium spatial collaborations, and garden-based divinatory systems.

“In the classroom as in the field, anthropology demands an openness to encounters with the unexpected, a disposition to the moment and its surprises, their unpredictable knocks and occasional gifts.“ (Pandian 2019, 67)

This is a quick graphic report, straight from my fieldnotes.