Thank you. Teksting av Nicolai Winther Thank you. Wow, what an intro. I like it. Hello, everybody. Welcome to the next session. This is going to be about embodiment. My name is Katya Zybrek. I'm going to be the session chair for this session. I was cut off. Yay! So, yeah, this is the embodiment session. Welcome. We're going to have a couple of really exciting talks today. I'm really excited about it. I'm a researcher in psychology and technology, so we work a lot in VR and with embodiment. So this is something that I am very interested in. But it's true that I think the way we're going to see embodiment in this session is going to be a little bit more different, expanded, should I say. And so, without making too much of a story about it, I'm going to welcome our first presenter. This is Dan Xiyiao, who will present their work, so Wolf and Tar, Designing Mixed Reality Dance Performances for Expanded Theater. And she's also here with Anita Lam and Andy Stokes. Thank you. Hello, everyone. Hello. Hello. Hello. Hello. Okay. Hello everyone. My name is Dan Chiao I'm a creative technologist from the collective a digital art studio based in Hong Kong and I'm with Anita Lam and Andy Stokes Who are the co-founders and the curator and directors of the collective and Alan? We are here to present our art paper Wolf Wolf and Tar, Designing Mixed Reality Dance Performances for Expanded Theatre. So before I start to talk about this project, I want to share a bit about what we do and what kind of mixed reality projects we have done before, and then how those projects bring us to this one. So we are ELLEN, an NGO from Hong Kong. ELLEN stands for Artists Who Love Animals and Nature. We curate exhibitions and create artworks about animals and nature. We did exhibitions about marine life, Sumatra forest, snow mountains, and urban city with wildlife. We mainly do interactive installations and mixed reality experiences. So we started to explore mixed reality since 2023. So on the left hand side is our first project, which is an MR underwater experience. So the audience wear the VR headsets, they can still see the crocheted coral artworks in the reality. And at the same time, they see the virtual animals and the stories about the fishes around them. So that's how they feel that they dive into the ocean. And this is what's called mixed reality experience. They see the reality and the virtual elements at the same time. And on the right hand side, it's our second project. It's an MR rainforest experience. So we extend the recycled paper rainforest artwork into a virtual one where the audience can interact with the world. We also did MR performances. We turned outdoor space into a virtual garden with dancers in which the audience can interact with the virtual elements by using their hands. We also brought MR performances into a theatre. All these experiences brought us to this project, Wolf and Tar, a narrative MR dance performance. This is the video of it. This is the video of the dance performance. This is the video of the dance performance. and Ta, a narrative MR dance performance. This is the video of it. It is a practice-based research and public performance we did with the Hong Kong Academy for Performing Arts in April this year in Hong Kong. In this show, the audiences are allowed to room freely and choose their own view. And in the show, there are two dancers. One plays the role as the wolf and the other as the tar. The story is about predator and prey. So they fight with each other in the blizzard on the snow mountain. And then they got trapped into a cave because of an avalanche. And then they have to work together to get out of that cave. So that's basically the story. And the story reflects the relationship between the species and also the relationship between human and nature. In this project, we experiment body tracking as well. So as you see in the video, one of the dancers is wearing the headsets and the motion track is on her. So that's how we get the location data of and the motion track is on her. So that's how we get the location data of all the body joints of this dancer and then the audience can see the virtual elements following the dance movement in the real time. And the whole show is a 10 minutes long performance. So within the first 8 minutes the audience watch the show with the MR headset on and then the performers will help the audience to take off the headsets and then they watch the live dance without the headsets with the projection on and with the lighting. So that's basically the overview of this project and in our art paper we raise up four main challenges when designing narrative in my performance. I'm going to go through one by one and with our approach to this challenge and with examples. So the first challenge is the immersion versus presence. So most of time, the virtual content is rendered over the camera view. So how can the virtual layer support the narrative and immersion without obscuring the dancers? So our approach is to balance the virtual environments with live performance through selective rendering, masking, and control the visual behavior. So here is the example. So as you see in this photo, this is the physical stage design. It's very minimal with the plain steps where the dancers dance on and the playing walls. And what we do is in the first chapter, we rendered a virtual snow mountain from the floor level with the gradient at the bottom. That's how we make sure that the floor is visible, the dancers is visible, and the virtual snow mountain hide the dancers who perform on the steps at the back. And as the performance begins, the mask moves upwards. So imagine the virtual snow mountain hide the dancers who perform on the steps at the back. And as the performance begins, the mask moves upwards. So imagine the virtual snow mountain like a curtain on the theater stage. So it moves up. Once its edge passes the dancer's height, the wolf is revealed at the peak. So that's how the audience understands the steps as the mountain and the dancer on the top as the wolf on the peak. The virtual layer gives the meaning to the physical space. The two layers stay aligned without obscuring the dancers. We also do building the atmosphere without occluding the dancers. On the right-hand side is a cave we rendered in chapter 2. It's a cave that's bigger than the whole space. A full 360 enclosure of the cave with the chapter two. It's a cave that's bigger than the whole space. So a full 360 enclosure of the cave would hide the performers in the reality. So only the upper parts of the cave is rendered, but Blizzard and the sky remain visible through the opening hole on the top. So there's no drift scene, so the audience still reads that we are now inside the cave, even though the bottom part of the cave is not rendered, the dancers remain visible. So that's how we balance the virtual environments with the live performance, through selective rendering, masking and controlled visual behaviour. And the second challenge is the attention in 360. So as we know that one of the advantages of MR experience is the audience are allowed to room freely and look anywhere. That means that they may look at the back, they may look at somewhere else. Then how can the attention of the audience be directed to the key story moments without restricting their movement? So our approach is to guide attention with location-based cues. So for example, at the very beginning of the performance the audience are given a short period of time. They can walk around, they can look around to get familiar with the snowy environment. And then as the opening of the performance a glowing ball appears in front of each audience member and drifts towards the wolf at the mountain top. So it travels while the mask reveals the dancer. When it reaches her, the wolf's performance begins so that every viewer's gaze arrives at the same moment. And on the right-hand side is another example. At the end of chapter one, there is an avalanche happens. So it's pulled down from the peak to the foot, then traveled towards each viewer's own position. So wherever they stand, the avalanche feels their view of view and then transits into the next scene. So that's how we make sure that audience don't miss key story moments, by using location-based cues, no matter where they are standing or looking at. And then the third challenge is the two layers, one choreography. So as we know in the MR experience there are two layers of information. One is the live dancers in reality and the other is the virtual elements. And sometimes these two compete for the same attention, which means that when you focus on the virtual elements you may ignore the live dancers in the reality. And when you focus on the live dancers in the reality, you may think the virtual elements as a distraction. So how can we choreograph them as one performance, not two? Our approach is to bind virtual elements to the dancer's movement so that the virtual and the real moves as one instead of competing for attention. So for example, in the first chapter, we have a large wolf head model follows the head dancers head movement and also this wolf head names the character and gives the audience a strong visual anchor and at the end of the story and we have some paddles following the hands movement of the dancer. What we do is we ask one of the dancers to wear the headsets and the motion trackers and then that's how we get all the headsets and the motion trackers. Then that's how we get all the position data and rotation data of each body joint. And then we send necessary information from the dancer to the server and then broadcast them all to every audience headset. That's how they see the virtual elements following the dance movement in the real time. And yeah, that's how the virtual and the real move as one instead of competing for attention. And the last challenge we have here is MR as a partial mode. So we know that the headset is heavy to wear, so it's not comfortable for the audience to wear for a very long time for the whole performance. So we explore MR as a partial mode, which means they watch the performance with the MR headset and then watch the rest of it without it. Then here comes the challenge. How can the audience move from headset to live performance without a break in the experience? So what we do is in a performance, the headset removal is done by the performers turning a potential break into part of the show. So as you see in the picture the dancers remove each other's headsets first so it's a performative gesture so that's like the wolf and the tar set aside their animal identities and returning to a human state. And then the dancers help the audience to remove theirs. So the audience already gets a hint that it's the timing for removing the headsets. And then the shift in viewing mode is carried by the performers. So it stays continuous with the dance. And then finally they watch a live dance without the headsets, with the project is done and the project continues with the dance. And then finally they watch a live dance without the headset with the projection and the light dimmed. That's how we deal with MR as a partial mode. This is the headset piece design of our show. Due to the limitation of time I'm going to skip this part. the. We're going to have a new project coming in next year which is also a more performance about the relationship between owners and the pets. We're going to further develop about the interaction of the audience and also how MR can work as a partial mode of viewing in the performance. This is basically all of the sharing, and if you are interested in what we are doing, you can scan the QR code. And we also bring a headset with us. So if you are interested in experiencing on your own, you can come to us any time. And I promise you, it will feel very different from just viewing a video on the screen. Thank you. Thank you. Do we have any questions? There's one here. Thank you for... Hello, hello. Thanks for the wonderful presentation. I have a question about the experience of the audience or the viewer when you are removing it. 안녕하세요. 시청자의 경험에 대해서 질문을 드리고 싶습니다. 여성의 관점에서, 메이크업이나 다른 것들에 대해서, 다른 사람들의 경험에 대해서, 헤드셋을 제거하는 것에 대해서, 편안한 경험이 있다면 어떻게 생각하세요? the headset and whether that is a pleasant experience? We have a short preparation time before it starts. So we have helpers on site to help wearing the headset for the audience. And then I think that's where they already get familiar with people helping them with the headset. And at the end of the show, we have two performers. One is a boy, one is a girl, so they will help remove the headset from the audience. And that's what we do for this performance. Do you consider it kind of a significant gesture that someone removes the headset for them instead of them removing it themselves? Yes, and our design is that the performer removed the headset for them with some body movements, so it's still part of the performance. And then that's how the audience know that that's not a signal of the show ending, but it's still part of the show. So they won't leave after the removal of the headset, but still watching the rest of the show. So they won't leave after the removal of the headset but still watching the rest of the show there. So yeah, that's a part of the performance. And if they remove the headset their own, I think they will consider that's the end of the show. So that's why we choreographed them in the performance as well. Thank you. Thank you. Thank you. Hello. Thank you. Thank you so much for this very interesting presentation. And I'm also a media artist. I also work with theater. So I find really like you have really precious solutions to a lot of problems nowadays that happen in XR theater. So I have a question regarding the dramaturgy and the choreography of this. So according to what we have seen in your presentation, it's kind of like an animal fairy tale thing, and introducing people to the natural environment and get them through that. But what if it's a very complicated story, like something that's already have its own storyline, so how do you consider integrating these technologies and all your solutions, like taking off the headset or using something to get them into some dramaturgy that's more complex? So I'm just wondering if you have thought about this and I'm really glad to hear that. Thank you.. I will pass the mic to our curator of this show, Anita Lam. Thank you for your question. That is exactly the same challenge we are facing as we are preparing our next show, which is a full performance around one hour. So that will include putting it on and taking it down. the story is going to be a little bit more complex. We want to have a story that is not just about one hour, so that will include putting it on and taking it down. So we don't want to have a topic around technology because it makes it so easy that, okay, put it on and take it down because it's about ML, then it makes sense. But we want to have a more complex storyline. So we also need to include how to embed this action into the story, right? You can't just having a headset just put it on. So we're considering first, like, can we make a headset like a hat piece that is included in the story? And also, why is there a reason for mixed reality to exist in theatre? You know, like, because it's about imagination, it's about expanding the theatre, expanding the stage. And we need to have a reason for that. So these are all the challenges we need to put in. And for creating a show, because not every performer or the show director understand how mixed reality works and also the visual director how things would work. So to prepare that to happen, we need to all sit down together to discuss how to utilise the ML without making an ML as just a technology demonstration.... demonstration. I think that is super can expand ML experience more than just technology showcase. reason and to have that reason, everyone in the design team, in the creative team, have to understand the possibilities and the potential of it rather than just a checkbox of using ML. So yeah, it's a challenge, but let's see how it goes in the next show. But this 10-minute show, we did a survey with the audience to try ticking it off. Does it break the energy? And it did not. So it seems that we able to embed the storyline into removing so that they don't feel that taking it out is a break of the story. Thank you. Okay, in order, sorry, in order to keep in time, we, you can leave the questions for after this session. I'm sure the authors will be around so you can ask them more questions. For now, let's thank the speakers. Thank you. And we welcome our next presentation, which is by Kenji Maverick Lee and Philippe Posquier. They will present Revival, Meta-Creation, Co-Creation, and Digital Afterlife with Musical Agents. Hello, everyone. We don't have a jingle? I'm a little bit disappointed. I'm just going to bookend Kian's presentation here. Fantastic. I'm going to just bookend this presentation. Looking at the pre-Ars Electronica catalog, they compiled statistics. And this year they announced that over 50% of the artists use AI in the making of their artwork, which I think is a really shocking number that we need to think about and question. And they further reveal that the artists are using, for the most part, so there's a list of tools that are most used, and that's ChatGPT, Gemini, Cloud, Midjourney, no surprise, like big corporation type of tools. And so at the Meta Creation Lab, we're in the business of using creative AI to automate partially or completely creative tasks, but we don't use anyone else's tool. What we mean by meta creation is really in the tradition of generative art, making a machine that make art partially or completely and I think that's an important distinction. And in order to do so, we have a research creation methodology in which we actually do develop algorithm, we work with the industry and we release open source software that we develop for everyone to use and artist to use and therefore we work with artists and both second person methodology with artists with artistic residency and first person ourselves as artists with our tool and we really find that it's important to have what we call participatory design. This idea that if artists are involved in the making even of the algorithm, the interfaces and the development of those tools then we end up somewhere else. And so that's the distinction I wanted to start with, and with that I give you Kian. Thank you. So yeah, let me start with the concept for this talk. We'll begin by defining four terms that guide the talk, and by musical metacreation I mean the design of computational systems that can participate in creative music making and by musical agents, I mean computational performers that can listen, respond, transform material and act within a musical embodiment. Co-creation refers to a process where human and machine actions shape the artwork together rather than one simply controlling the other. And then lastly, the digital afterlife refers to the reactivation of historical musical traces through computation, not as resurrection of the exact replica, but as transformed presence within a new performance context. And so in revival artwork, the more interesting question is how agency is organized in performance. So here, the central argument is that revival artwork is a live metacreative system rather than a fixed AI assisted composition and musical form emerges through real time improvisation and feedback among human performers, curated sonic traces and musical agents, generative visuals and audio visual systems. Each action changes what can happen next, so the work is continuously shaping performance. So here the key idea is liveness within an ecology of relationships and the system is neither fully performer controlled nor autonomous. So responses are shaped by the corpus, model design, live electronics, mappings and improvisational decisions. So authorship is therefore redistributed rather than removed across curation, system design, rehearsal and live interaction. So in revival artwork, like composition becomes the design of conditions from which form can emerge in real time. So here, our paper is grounded in three conceptual anchors. First, revival follows a research-creation approach where artistic practice, technological development, crafting, trading generative system as artistic instrument shapes through data set selection, parameter and interface design. And also I mean including rehearsal of course as an iterative design process. And third, it adopts a small data ethics, transforming and re-conceptualizing specific historical and collective materials rather than simply reproducing them and trained on large-scale data set. So for us, model crafting also means the learning, tuning and rehearsing the technical system like a musical instrument and these technical choices are therefore also artistic decisions. And small data ethics similarly shifts the focus away from large-scale extraction toward the responsible transformation of specific materials within a live artwork. And here, Makat and Makatart are the two main audio-based agents in the revival artwork, and both begin with the curated recording selected by the artist and the system segment and analyzes these sounds, turning the corpus into a landscape of related timbres and remembered sequences. And Makata-eigen explores these landscapes through temporal modeling, allowing it to propose where the music might move next, and Makatarte-eigen listens more directly to the live performance and finds related sound fragments and recombines them as an evolving music mosaic. And it can operate both reactively and proactively. And artistically, the agents do not simply replay the corpus, they reshape remembered material in the moment and creating an ongoing dialogue in which the performers also listen, respond, and redirect the system. And here, the visual system extends the co-creative approach into image and lighting. Revival combines Autorium, a small data GAN-based visual synthesizer with Touch Designer and performer-facing Touch OSC interface. And the VJ can shape scenes and effects in real time while also audio features such as density, amplitude and rhythm influence visual motion and lighting. Rather than serving as a background decoration, the visuals become a performable layer of the work and allowing image and light to participate in the same feedback loop as the musical agents and the human performers. So here in performance, improvisation connects all parts of the revival artwork and the percussion gestures can be captured, transformed or extended by the agent and while live electronics, visuals and lighting continuously reshape the texture and perception of the performance. So rather than following a simple cause and effect chain, the work operates through recursive feedback between human actions and system responses. So the MUTEC Montreal performance was particularly important because it tested these relationships under real performance conditions. And the rehearsal helped us explore the balance between control and uncertainty while on stage those relationships became more dynamic. And the performer changes the system and the system influences the performer's next decision and the visual and lighting layers reshape how those interactions are experienced. One result of this ecology is distributed agency. During performance, it can be difficult to tell whether an event comes from a human performer, an agent, or several layers of transformation, and our paper treats this ambiguity as an aesthetic condition. And this also connects to digital afterlife, and revival artwork does not attempt to resurrect or imitate DC's composer composer as exact replica, instead historical musical fragments become transformed traces reactivated through machine listening, recombination and improvisation including live performance. And in this sense digital afterlife refers to how musical materials and gestures continue to circulate through new technological and performative contexts where memory becomes unstable, transformed, and shared between human computational and historical elements. So to conclude, Revival presents a co-creative audiovisual system where musical agents, generative visuals, and human performers shape the work together, and it also explores a small data approach in which historical materials are transformed rather than simply reproduced, and while research, artistic practice, and system development evolve together. So more broadly, Revival Artwork treats AI not as a replacement of artistic agency, but as part of shared and negotiated performance ecology. And so the value lies not only in the final audio-visual result, but also in the process of building, rehearsing, and reflecting on these relationships. So thank you very much for listening. So we'll now briefly watch a short excerpt from the revival performance. short after from the revival revival performance sorry the sound is not coming true Increase the volume or something or hear the sounds All right. And so to conclude here, this is a good example of a project that brings together a number of tools that are not just used in this project, but are also released. And it's important to say that because we live in a world where there's increasingly in our field and in creative fields a polarization around AI with a lot of haters and a lack of nuance in the discourse. And that's totally understandable because if you're being told, and you are, that your work is being stolen and we're going to try to replace you, you're going to be pissed. So people are pissed. But it turns out there is a light at the end of this possible tunnel because there's diversity in AI like in everything else. It's hard to compete with the corporate discourse relayed by the media. It's hard to surface above that noise, but there is a truly option there. And so some of the tools that are used in Revival are available for everyone to use, and in fact there's training material online about those tools, whether it's MIDI GPT, a computer-assisted composition system that is used here to extend Wagner's opening of the Rheingold as part of this project of reviving dead composers on stage, Autolium, the small data and model crafting visual synthesizer, and Fragmenta, which allows to do audio generation and crafting small personalized model, as well as a collection of model bending object in a Comfy UI. All of that is available and doesn't require a line of code because artists don't want necessarily to become coders. And with that, I'll conclude by thanking the funding bodies that are supporting this work in Canada, the CHIRC for social science, the National Science and Engineering Research Council for the technical work of developing those tools, and the Canada Council for the Arts for supporting the making of artworks. And I thank you all for your attention. Thank you. Any questions? We have time for maybe two short questions. Okay, maybe I'll start with one. I'm actually, it's really interesting to think about co-creation with AI, but what is the subjective experience of the artist when they're co-creating with AI? Is there a time when you go like, oh, I didn't like what they're doing, and oh, I want to control it more, but I can't. Is there some sort of this kind of interaction that can be quite negative or very positive? Yeah, well, we rehearse with the musical agents, and in fact, in some of the latest models of the musical agents, we can give them feedback of what happened on the rehearsal because like any co-creation between humans we talk to each other about what's happening typically and then we try to do that with the agents as well by saying hey I like that, do more of that I don't like that, do less of that so that eventually those rehearsals are not just about fine tuning the data sets and retraining things so that we are up with it, but that we can eventually really iterate and craft those models so that they act as we want. But there is an inherent risk in putting a machine playing live with you on stage and when things don't go well, which with technology we all know is more fun than we would like to, then that risk can manifest itself. And I think Ryan over there had a question, and I see one at the front. Hi. So generally, when I worked on music visualization, there's a communication layer between the music and the graphics, and usually it's done with FFT or MIDI or OSC. What I find interesting here is because both are generated by the same agent, you don't need that communication. Is that correct? Do I understand that correctly? Yeah, there is part of that, but there is also Vijay Amagi, who is live, intervening on some of the visual parameters. And so there's, here again, a bit of co-creation between Vijay Amagi, so human Vijay, and what the agents are doing. Cool, thank you. Yeah, there's another, oh, there's another, yeah, sorry. Take this one, Kian. Hi, I'm also a music producer, co-create with AI, but I'm very interested in this software, but what the main key differences between this and like mainstream platforms like Suno, etc? Yeah. Okay. Yeah, so our system is nothing related to Suno and Udio. It's more of a curated by us and we also trade our material as a small data approach as opposed to big data approach. So it's like a, that's the main difference. And we tend to have more control and the system that we are using is, most of the people we are using is like a techno-fluent musicians. They know how to code and like a Maxim SP and all that, and they craft their tools based on specific needs. So it's like a fairly different from the, like a big data model, like a Suno and Udio. Personally, I'm opposed to Suno and Udio. Yeah, I would, you know, those are very radical different sort of approaches. And here we're talking about very small asset, a few hours of recordings. And so it's my data, it's my model, it's my artwork. When you prompt Suno, you don't have copyright over what comes out. So that's, I think. And the question, the authorship when you use Suno seems to be a bit diluted and so I think the differences for us are important. I think we need to move on. Thank you so much, thanks again to speakers. And our next speaker is Albert Borek-Duran, who worked with Ada Loura-Docrespo on Synapticon, creative alignment, and the brain AI model interface. Good morning everyone, thank you so much for being here. Imagine a building in which every room can be seen from a central tower. The people inside the rooms cannot know when they are being watched. They can only know that they might be watched. This is the essential mechanism of Panopticon. Its power does not depend on a guard looking continuously. The architecture itself performs the act of observation. The possibility of being seen becomes something each person carries within them. And Jeremy Bentham imagined this structure as a practical machine and then later Foucault made it a philosophical one. A model of understanding how visibility can organize bodies, attention and behavior. But what happens if we turn this architecture inward? What if the rooms become synapses? What if the central tower becomes a computational model? What if the object of observation is no longer the body, but a trace of our cognition? And at that point, who is watching whom? This speculative reversal gives the project its name, Synapticon. Synapticon is an artistic research project situated at the interface of the brain and foundational AI models. It brings together brain-computer interfaces, VCIs, LLMs, large language models, real-time audiovisual systems, and live performance. And at its center, there's a simple sequence. And at its center, there's a simple sequence. Brainwaves become natural language. Natural language becomes aesthetics. But the simplicity of this sentence is a bit deceptive, I would say. Nothing passes through the system untouched. A neural signal is captured, filtered, divided into time windows, encoded, interpreted through the prior structure of a language model, and then routed into an audiovisual environment that has been carefully authored in advance. So Synapticon is therefore not a transparent window into a mind. It's an apparatus for staging everything that happens between an oral event and its public meaning. The system begins with a 16-channel OpenBCI headset and across 30 EEG sessions we collected a few thousand paintings between neural recordings and text. The training material was intentionally bounded, 108 curated sentences comprising 623 words which this makes a total of 3,210 trials. During data collection sentences were presented and internally vocalized. 10 second EEG windows were aligned with those sentence and in performance the system works with a streaming windows of approx five seconds and produces partial textile outputs. These numbers matter because they define what the work can do and partly because they define what it can honestly claim. The project makes something very fragile public which is a probabilistic relation between neural dynamics, a learned linguistic space and an out-of-order performance system. The text that appears is neither a direct transcription nor arbitrary noise. It lies in between. The language model imposes a linguistic prior on complex and noisy neural features. So in other words, it gives the signal a field of grammatical, lexical, and associative possibilities. This means that the machine does not simply find a sentence waiting inside the brain. It participates in producing the conditions under which something sentence-like can appear. For us, decoded text is best understood as creative index. An index points. It does not contain the thing itself. Like smoke can index fire, a footprint can index a body that has already left. In synapticon, a textual fragment points towards a relation among intention, signal, model, and context. It becomes a cue that can organize transition, images, sounds and attention. And this distinction I think it's very crucial. If we call this mind reading, the output acquires an authority it has not really earned. If we call this meaningless noise, We erase the situation relation that is producing it. So the artwork occupies this unstable distance between these two claims. It asks the audience to feel that instability rather than resolving it for them. After the coding, the text enters a pre-created library of audiovisual scenes built in TouchDesigner. The prototype does not depend on unconstrained real-time image generation, although this is our goal. This is an important artistic decision right now. Images, transformations, filters, compositing rules, scene changes, and temporal behaviors are prepared as vocabulary. The decoded text enters that vocabulary as a semantic and performative layer. The Navleton Live operates as the sound engine. It also sends control message to Touch Designer, coordinating part of the movement between the visual chapters. And the sonic architecture is also site-specific. Pure tone tones are calculated in relation to the physical dimensions of the venue, of the room, exciting low frequency. And at certain acoustic notes, the audience can feel vibrations moving through their bodies from the floor upward. So the brain produces data, the model produces language, and the room answers with pressure, with light, with rhythm, and with scale. So what began as an internal neural variation returns as an external collective sensation. To describe this relation, we use the term creative alignment. In conventional AI discourse, alignment often means bringing a system's behavior closer to our human goals or values. Here the question is a bit more specific. How can a live system sustain an intelligible relation among performer intention, neural capture, model priors, audiovisual design and audience perception? So creative alignment does not eliminate ambiguity, it makes ambiguity usable in some sense. And we describe it through four connected layers. Signal alignment asks how noisy low signal neural activity becomes technically processable through windowing, filtering, and temporal encoding. Semantic alignment asks how those features enter the learned structure of a language model and emerge as fragments of natural language. Narrative alignment asks what happens when those fragments are mapped onto images, sound, pacing, transitions and sensory intensity. And finally, social alignment asks how an audience interprets what it encounters. Does it attribute an output to the performer, to the model, to the artist slash creative technologist, or to the situation created among all of them? Each layer aligns something, but each layer also introduces a new displacement. There is no single moment at which meaning arrives complete. Meaning is negotiated across the whole chain. This changes, obviously, how we speak about authorship. Synapticon is not authored by the performer alone. It's not authored by the model alone. And it's not authored only by the people who build the apparatus. The performer generates neural events, the model turns variation into textual possibilities, the artists and developers decide how that text can act inside the audiovisual system, and the live environment shapes timing and scale. And the audience finally resolves all that ambiguity into experience meaning. Authorship is distributed, but it's not equal. Agency is distributed, but it's not equal. The point is to make these differences or these different agencies and their different responsibilities perceptible. On stage, the boundary between intention and inference remains visible. That boundary is not a technical defect. It's actually the material of the work. And when brain-derived outputs become public, ethics is not an appendix. It's part of its form. Neuro-rights scholarship gives us the necessary vocabulary. Cognitive liberty, mental privacy, mental integrity, psychological continuity. These ideas are often presented as protections for a rapidly approaching future and in this work they are already practical questions. What is exposed? Who controls the exposure? Who can interrupt it? What authority does the audience grant to the machine? And what remains private even after a fragment appears on screen? This is not a contradiction of the project's openness. It's a boundary within it. Perhaps cognitive liberty also includes the right to opacity. The right to remain opaque, even to the systems who have trained on ourselves. This is why Synapticon is both an artwork and an apparatus. As an apparatus it includes the headset, the data set, pre-processing choices, model priors, decoding parameters, interfaces, timing systems, and a stage logic. And as an artwork, it composes all these elements into a public encounter with partial cognition. So the research does not exist only before the performance in the lab or after it in the analysis. The performance is itself a way of producing knowledge. The performance does not demonstrate that the interface has disappeared, it actually gives the interface a sensible form. We can see it, we can hear it, and even at certain frequencies we can feel it vibrating in our bodies. So let us finally return back to the panopticon. In the classical model, power belongs to an observer who may remain invisible. The watched subject is made visible while the machinery of watching disappears. Synopticon tries to reverse that condition. The performer is visible, but so are the mediators, the electrodes, the filters, the windows, the training sentences, the model priors, the mappings, the scene libraries, the sound cues, and acts of interpretation. It can show that every output has traveled through an architecture and perhaps the most important output is not that sentence that appears on the screen. If the panopticon asks how architecture governs what can be seen, synopticon asks who authors the visible when the material is cognition. The mind is not opened yet. The relation is performed and cognition keeps expanding. Thank you. Thank you. We have maybe one quick question time. Okay. I'll just have one quick question. So do you think that in time, if you do a lot of these kind of interactions, do you think that it might actually converge where the language model will be able to respond to the user and to the audience in a way that's actually meaningful to us? I guess, yes. I was thinking like in the way we trained the model, In this case, it could be even interpreted as a toy version, because we're talking about a very unipersonal data set, just with myself, and with very strict boundaries and conditions. But again, if we start playing around introducing new variables, new ways of this language model to express itself, let's put it that way, I'm sure we could find ways of incorporating, of adding, in this case, you were suggesting the audience, even to enhance somehow this feedback loop between performer and audience, for sure. I mean, being performed this for some time now, I have to say that the audience changes a lot how the performance goes. I mean, because I'm in that specific environment, I'm feeling the vibes or how the audience reacts to what they are seeing, and that thing influences me, which is in real-time influences the whole architecture that I was sharing. Thank you. Let's thank the speaker. I believe the next presentation is online. This is by Shachma Abdullah. She's going to present Ouroboros, the anti-game, diagnosing the achievement subject through real-time EEG. Hello, everyone. My name is Shachma. I'm a student atciplinary artist, designer and researcher joining you virtually from Singapore. I'm really gutted I can't be there in lens with you today due to some administrative issues, but I'm very excited to share my paper with you. The project I'm presenting is called Ouroboros, we init the anti-game diagnosing the achievements subject to real-time EEG. This project is a BCI interactive installation driven by real-time brainwave data but at its core it's really an exploration of why we literally cannot stop trying so hard. It is a diagnostic tool for what we might call the achievement subject. This is the setup of the installation. A participant walks up, sits down at a small console. First, an iPad prompts them with a single, slightly loaded question. What are you currently working towards in your life? After entering a personal goal, they put on a Muse 2 EG headband and pick up a controller they have been handed. However, the controller is merely a decoy. What is actually driving the visuals is not their hands, but their brainwaves. The screen in front of them begins rendering a live environment of pixelated forms that gather and fall apart. For this experiment, I observed 14 participants across different ages and briefing conditions, yet every single one responded the exact same way. Upon discovering that their effort caused the visual to collapse, they applied even more effort. caused the visual to collapse, they applied even more effort. Seven of them were told in advance in plain language that trying would destabilize the image. They could recite the rule back to me, but it wouldn't matter because the reflex of theirs simply ran faster than the knowing. That gap between what the cortex understands and what the rest of the nervous system does anyway is the conceptual center of this entire project. The harder a person tries to stabilize the visual on the screen, the more violently it fragments. Touching briefly on the project's lineage, PCIe Art has a strong tradition of using biological signals as raw material, seen in the works of artists like Anadol, Aksin, Terrell, and Ikeda. However, almost all of these works rewards calm and pernicious agitation. While that is a comforting orientation, it quietly reproduces the exact same logic as the productivity software it is sometimes pitched against. And I know there are a lot of works that are BCI related today, and I'm excited to experience them, but I could not find a sustained version of a reward gradient in the existing literature and I treat that absence as diagnostic in itself. So my project aims to build an anti-game, an installation that sustains this inverted gradient. So to understand why we double down on effort even when it actively hurts us, we have to look at the biology of striving. And I very briefly touch on four key sources that built this biological case. Perridge and Robinson tells us that wanting is dopaminergic, it is structurally insatiable. Shules demonstrates that dopamine flies on prediction error rather than upon arrival at a goal. Furthermore, Sapolsky and Lemke point out chronic stress actually erodes the prefrontal capacity that would otherwise allow to step back and reflect. So simply knowing that you are in a toxic loop does not interrupt it because your subcortical system acts much faster than your conscious reasoning. However, biological vulnerability only evolves into a civilizational structure with a little help. So I have layered the Calvinistic work ethic, the Benetian manufacture of desire, and finally what Byung-Chul Han terms the achievement subject in his book, what Byung-Chul Han terms the achievement subject in his book, The Burnout Society. We now command ourselves so efficiently that we no longer need an external boss to crack the whip. And as Han argues, burnout is not a malfunction of the society, it is actually treated as proof of your capacity. Now, how do we measure this in the gallery? The anti-game maps beta-bad activity, which we generally associate with focused goal-oriented cognition to visual fragmentation. When your beta waves go up, the world falls down. Ironically, the exact state that every school, workplace, and productivity app has trained you to maintain is the state that accelerates collapse in this installation i want to be precise though because the eeg literature on beta waves is contested so i'm using it strictly as an operational marker within this specific paradigm not making a universal claim about what beta means everywhere and technically speaking we pull a 256 hertz signal from the Muse 2 via OSC, run a rolling average, calibrate the 30 second baseline and push it through a residual low pass filter to drive a live GLSL shader and touch designer. Now I know that the Muse 2 is a consumer grade device, it cannot cleanly separate deep cognitive effort from, say, jaw tension. But this imprecision is not a bug, it's conceptually load-bearing. Every wellness dashboard and engagement metric quietly presents this flawed, imprecise biological surveillance as authoritative, and I'm just being honest about it in my work. So two more design decisions, the pedestal is graphic on purpose, the loop it's reading runs from Calvinistic soteriology to Venetian advertising to the platform engineering of the present, and it shouldn't look contemporary. And at the end of the session, the participant receives a printed diagnostic that closes with a single recommendation which is to continue the recommendation of every system that has ever needed the participant to keep going. Now returning back to our participants, 14 out of 14 intensified their effort. They leaned forward, narrowed their eyes and tried to verbally outsmart the rule. One participant put it beautifully, the part of her brain that knew was watching the part that was doing and the watching part couldn't reach the doing part that's very confusing while i acknowledge that a sample size of 14 is not statistic statistical proof of a universal human condition it is an empirical homogeneity that certainly warrants a structural reading. And now, when stabilization did finally arrive, it came through one of these three routes, which is depletion, frustration, or accidental destruction. None of these were deliberate decisions. Nobody chose to let go and successfully held that choice. that absence is crucial for the reading that I want to advance because intentional release is simply not a strategy that any of the participants had access to. I want to be direct about the limits of this work. Three alternative readings deserve serious weight. This could merely be a general problem solving. The game-like framing itself could be priming people to solve, or it could be a universal physiological response to any unpredictable stimulus devoid of cultural specificity. I cannot rule any of these out at this sample size. What the theoretical framework provides is a prediction that matches the observed consistency, not absolute proof. I also want to take the deepest vision of this concern seriously. The insulation may partly produce the very behaviour it attempts to diagnose. That is exactly why the next-scale version of this insulation has removed the puzzle mechanic altogether, replacing it with a categorical reading system. And those findings would be for my next paper. The most ambitious claim in this paper, I would say, is this proposal that is a category of design environments that I called diagnostic rather than optimizing. every environment touching cognitive label right now is to build the extract out of it or to extract more out of it and none of them measure the cost a diagnostic environment conversely points the instrument inward so there are two questions this preliminary study cannot resolve on its own which which is, does this hold at scale and is it strictly bound to the gallery setting? So the next phase, which is already underway, is a multi-part spin configuration, as I mentioned. And ultimately, this project began as about ambition and it has ended as a project about conditioning that produces ambition as the next involuntary reflex. The diagnostic print's final word is to continue. It doesn't tell the participant to stop. All it does, and it cannot tell the participant to stop, all it does in three short minutes is using their own brainwaves, is show them that on this occasion they didn't. What they do with this knowledge is not mine to make or decide. Thank you so much to everyone here for your time. I would love to hear questions and thoughts. So please do reach out to me via Gmail, LinkedIn or Instagram at shalmaxart. You can scan the QR code here. Special thanks to the expanded team, Arts Electronica, for this opportunity and letting me present virtually. Thank you as well to Associate Professor Ina Konradi for her supervision and to all 14 participants. Thank you. Yes, please, if you have any questions, please reach out to Shahma online or through social media. But we're moving on to our next, to basically our final speaker. So it's going to be Oliver Gingrich, Alain Renaud and Erika Mundria. And they will present Kima, Resonance, Humming in Harmony and Resonance for Creative Health Outcomes. Thank you. Thank you so much. Yeah, my name is Dr. Oliver Kingrich. So, thank you so much for the introduction. My name is Dr. Oliver Kingrich. I'm from the University of Greenwich. I'm joined online by Dr. Alain Renaud and Erika Mondria from the Art University here in Linz. We're presenting ChemoResonance, which is a project that rests on a decade of research by the Analema Group and supported by Erika Mondria through her own research also as a neuro expert. The Analima group has been looking at visual properties of sound in mostly massive and participatory contexts. And we've initially looked at physical properties of sound, studying some of the work by Ernst Chladni into semantic patterns, really quite fundamental relationship between matter, physical matter, granular substances and resonance and how these visual properties look in the real world. We studied this and then also developed our own mostly digital responses to this. And here you can see a commission by the Roundhouse in London for the 50th anniversary, which was one of the first immersive House in London for the 50th anniversary, which was one of the first immersive participatory installations that we did. I hope this works as a video, and if it doesn't, oh, it takes me out. I'll do this very briefly, and hopefully I can jump back afterwards. So here you can see a 360 presentation using early forms of machine learning and mapping this into a 360 immersive environment. This was really co-designed with the audience who basically becomes an actor within the installation. I hope I can jump back to the presentation. Okay, so the project is also part of a larger research project into participatory arts practices in digital and online context and their effect on health and well-being. And we've been working with a lot of different organizations supported by the Arts and Humanities Research Council, the NHS, Brunel University, RNIB and others. And I just want to briefly talk about this wider context. Specifically, we're looking at social connectedness, which is a concept that has been defined really as a feeling of including how emotionally distant or connected one feels to somebody else. We know also from the HEART study and other studies that there's a strong link between artistic engagement and social connectedness and that this has a possible effect on perceived loneliness. We also know that social connectedness has been correlated with higher self-esteem, with higher life expectancy and to health overall. What we don't know is whether this applies to digital context too. And so part of the PARTICIP participate research project was to look at digital interfaces and to see whether we can translate these experiences, this effect into these digital context and what the challenges are specifically in the context of accessibility and also some of the data ethics around this. We've been working with the Annalima group and also with other partners like Neurocreator here in the room on understanding some of these implications better. So one of these interfaces, Kima Voice, was the predecessor to Kima Resonance and here what we were doing was to map vocal harmonies onto this Fibonacci spiral that you can see here. This was co-designed with a charity called Show Off Sound. They're working with people with disabilities, physical disabilities but also cognitive disabilities, and together we tried to understand what are the limitations, what are the challenges when facilitating these artworks but also when designing them. And we combined this with a small study with 21 people, and we found that there was indeed an effect between these artistic engagements and social connectedness and health and well-being. And so yeah, this builds kind of the research context for chemo resonance, which we presented here at Ars Electronica Center last year in 2025 in the citizen science lab downstairs. And here we have an interface for two people. It's a physical situated site-specific installation with a holographic screen. Two people are singing. Their vocal harmonies are being visually represented in the center. And you see an outside halo. This is a third person who's sitting here behind the screen and who is listening to these vocal harmonies, to this idea of humming. And what we tried to understand is, can we actually create this meaningful visual representation of the effect of humming on the self? And I'm going to hand over to Erika Mondria now to talk a bit more about the facilitation. Thank you. Yes, so I will talk about to you about the facilitation and the process. I will talk about humming, listening, and the interaction. So we're talking here, the project is about embodied interaction. Therefore I ask you a question. Do you carry your body, or is your body carrying you. When you were a baby, maybe some of you had the pleasure to experience that your mom or your dad were humming you while they laid you down in bed. And you might be thinking, oh I don't remember this. But humming, as everything else, but humming has, does create a network from the brain to the body. And even if you don't know, maybe you were humming when you were an older child, or even, I don't know, anywhere. Humming creates a vibration in your body. And this vibration, if you put your sensory perception inwards, you can feel it maybe in your whole body till down to the toes. So this is one part, the humming, and the network about it in the body. And now I would like to talk with you about listening. So everyone has maybe had the experience to listen to someone and actually there are a lot of thoughts in your mind like your opinion or your impression or the shopping list. Some thoughts and you actually should listen. And listening also is something, there is a pattern in your body. When you were a baby and you were listening to something, there was no opinion in your head. There was no shopping list. There's also a network for this and this is what we call active listening. That's what the third person's task was. The one wearing the BCI. So in the setting we had three participants. The ones, they needed to be encouraged of course to hum in front of the public, because humming is not something we like to do in public. And the other one, we mounted the EEG. We mounted eight gel electrodes on a temporal lobe. And all this person had to do was listen. lobe and all this person had to do is listening. Stand behind the transparent screen and listen to what he or she feels, hears and senses in the body. So this was then visualized. The voices you can see in the picture, the square, and the listener was visualized when this person was touched and moved very calmly, then the color would turn darker and darker. And when it was exciting for the person, then it was lighter and lighter. So the findings, maybe is connected to the own body, the embodiment, the more the social connection works. Yeah and this really ties up a little bit with the findings from the PARTICIPATE study. So the beauty was that it was truly a participatory interface in that the participants created the visual representations and we had this part of active listening where you were just literally observing, listening inwards, looking inwards almost, and then observing your own brain waves coming to life as these outer rings that you can see on the image down here. So very briefly, we're going to keep the technical presentation quite quick because you can read all of that up in the paper, but we used TouchDesigner for the visual representation, we used MaxMSP for the sound analysis. There was an immersive sound design that went along with the installation as well, so that supported the interaction. And as Erika was saying, we used a holographic screen, a semi-transparent screen, to project the work and to create this really interesting triage between the two participants and the active listener. Yeah, and the findings, just very quickly if this works, I'm going to show the video. I'm leaving the presentation again. social events music Is it possible to turn the sound back down? Yes, briefly about our findings. So there was really a positive response by the participants, which did show that they were genuinely engaged, immersed and absorbed in the installation. Several participants noted they found it relaxing and meditative. And one participant mentioned that they really saw a link between the art and health and listening to the sounds of humming and the relaxing effect on the human mind. Yeah, and we really enjoyed that there were so many different generations, different backgrounds that all created these sounds together and literally just tuning into the humming of their own voices. So just very briefly about the context, the political context. We tried to write up some of these findings. There's guidelines on digital participatory arts that we've published. We were invited by the National Centre for Creative Health to present some of the research findings at the House of Lords in the UK. And there's a summary report on digital creative health and participatory arts that was published with our research partners and that you can see online. Also, the guidelines are available online. And I will share with you a QR code where you can access them. And this is us together with a colleague, Shama Rahman, at the House of Lords presenting the research and then holding our own parliamentary roundtable as part of the old party health group and what is so interesting is that there is really a bit of a political deficit in acknowledging the role of digital and to think about regulations about pathways about accessibility about co-design is something that is built into the funding landscape. And this is part of what the summer report is trying to raise. And I think with that, we're probably at the end of time. And we're going to leave it for any questions. Thank you so much. Do we have any questions? There's one question. Hello. Hi. Thank you for the wonderful presentation. I was curious, do you collaborate with experts in somatics or well-being? How big is that role in your practice? Because I imagine it's included in the picture but I don't really understand in which moment of the research or perhaps after the results if there are some social impact actions where you have people that are more close to this kind of events? Yeah, thank you for that question. So part of the participate project, one of the co-I's, the co-investigators was Central Northwest London NHS Foundation Trust and the co-I there, Claire Grant, she's a music therapist so she has this background from a creative health context and she was part of the initial development of Kimo Voice and so really supported this whole process also working on these co-design sessions with the charity Joy of Sound so yeah that actually was the foundation for for this project that we then brought here to Ars Electronica. Thank you. Hello. What was your decision about the visual, the cube and the rings and how do you find these kind of forms and colors? Yeah, thank you. So this mapping is meant to be super intuitive, right? So it needs to work for everybody and anyone. Yeah, no matter how old they are or which cultural background they have. And so we were with these forms, we were visualizing vocal harmonies. So the cube, for instance, alludes to a fourth. We had a triangular shape, which was alluding to a perfect third. So these are kind of musical harmonies that you can... If you're a musician, you would know, but if you're not a musician, you're a layman, they still are intuitive enough. And what happens is that they change quite instantly. So we detect these vocal harmonies. When me and you are making a sound, we hum together and there is this meaningful interval between our voices, then these shapes, they merge. So the mapping is quite literally connected to the type of vocal harmony that we detected sonically. Any other questions? I actually have one comment. It was very interesting when Erika started speaking I felt a very huge shift in because I think you were trying to be very present and speak from a very present place maybe this is just my perception I was sitting right here and I was I could kind of feel it and I actually it did what you mentioned later it kind of focused me out of my head and more into what was happening around me so I'm really interested in this connection between feeling embodied and social connection. Do you know why there is a connection or have you found this maybe in some theories before? Just some comment if you want. Yeah, it's right connected to my research because I haven't found it in the series. I found it in practice when I do my projects and I believe, well that's my belief, if you're not connected to yourself really deeply into your body because you're, in my opinion, you are your body. And if you're not connected, you cannot connect to anything else. You cannot connect to Earth, you cannot connect to the climate change, you cannot connect to another person. And once you experience being really connected to yourself, that's when you have the pleasure to experience, to be connected to someone else or to a group. And this is not something you can read in theory. You can, of course, but it doesn't change how you feel. That's something, it happens. And with the project, humming, it does. Because you're brave, you stand there, you just do humming. And I remember maybe 15 years ago, I was leading the brain lab and many, many visitors came and asked me a lot of questions because I'm in the brain lab. And she said, oh, my daughter, she's always so nervous when she has an exam. What can she do? And I just said to her, just maybe she can try. Go to the toilet before you have the exam. Go to the toilet and just hum. Hum for yourself. And feel. When you're humming, you cannot think. If you not think, then the body relaxes. And it's, you know, you don't need to know this theoretically, you need to do it. That's beautiful, thank you. Thank you very much. This we conclude the embodiment session. I'm not really sure what's next on the program, but maybe it's time to eat perhaps? Thank you.