Thank you for watching. I'm sorry. Hello everybody and a very warm welcome to our second panel of the day which we've called hearing colour seeing sound and in this panel we've brought together three papers which are really about collaboration there's a collaboration between peers and in the first paper and then in the second two there's a collaboration between different disciplines so our first paper I'll introduce to you the the papers first and then we'll go into each speaker and if you have any questions that come up to you as we are going through the different presentations please type them into youtube and we'll ask them at the end so speakers please don't answer them when they come up but wait till the end so please enter any questions into the chat okay so very pleased to um introduce our first speakers Nadea and Alexandra Antonopoulou and Elena is an academic and critical technologist, Dr. Elena Nadea and her PhD comes from Goldsmiths from the Department of Computing. She was formerly a reader in digital media and head of program at Royal College of Art, but she now works with Central Saint Martins and Cambridge University. And she lectures and supervises PhD students at a number of different universities. And Alexandra is course leader at University of the Arts London and a designer. She's taught design, story making and immersive environments at Goldsmiths and the Royal College of Art. Her practice has been showcased in many different galleries, including Victoria and Albert Museum, Tate Modern and London Design Museum and she also holds a PhD in design from Goldsmiths. Following their presentation, we have a collaboration between Kate Steenhauer, Andrew Starkey and Jack Craven on painting music using artificial intelligence to create music from live painted drawings. And the collaborators are Kate Steenhauer, who is a visual artist whose practice looks at the dynamic and interactive capacity of drawing dialogues with other art forms, technology and their relationships with the audience. And she's collaborated with lots of different disciplines like dance, opera, music, sound, text and artificial intelligence. Andrew Starkey, who she's working with on this project, is senior lecturer at the School of Engineering at the University of Aberdeen, with over 20 years of experience in artificial intelligence. And they're also working with Jack Craven, a software engineering and aspiring entrepreneur who's a recent graduate from the University of Aberdeen. And then finally, our final panel, looking at collaborations between music and animation. Music as a Path Guide to Animate is by Elaine Gordieff, who also has a PhD in Fine Arts from FBAUL in Portugal. She's an independent animator, professor and researcher, a member of ACIFA and the Society for Animation Studies. And she's also a producer and animator and director of more than 16 animated shorts, which have been shown at festivals all around the world. She's written a number of texts on animation and is the author of the book Aesthetic Interference, the stop motion technique in the animation narrative. the stop motion technique in the animation narrative. So very pleased to have such an amazing panel of different speakers. And let's start with our first talk by Eleanor Dare and Alexandra Antonopoulou. Thank you. Hello everyone, I'm Alexandra Antonopoulou from the University of the Arts. And I'm Ellen Ladare from Cambridge University. So this talk will examine how virtual spaces become the focus for a collaborative animation project conducted online under conditions of pandemic lockdown. In the absence of intimate proximity we mediated our physical research methods, deploying VR spaces and ping pong as a conversation method in the indie game engine, and also an artificially intelligent third collaborator, or what we call the co-agilent, to evoke our practice of walking, writing and researching, while in the world of parks, food play and sensory engagement, all of which we have lacked under lockdown. This also involved inventing and animating fictional collaborative islands where we could meet weekly to research and write and to share drawings and animated works. So in the first slide we have one of those videos of an island, some of the work that we've created. Okay, these methods are not nostalgic. They're not a nostalgic practice, but a means to examine how new forms of visual, sonic, animated and embodied storytelling might become possible across distances and circumstances. In a way, we reassembled our pre-Covid sounds and senses, memories and collaborative methods within virtual space, while being acutely aware of the surveillance nature of those spaces and fatigued from being online mediated for most of the day. Here, the Riverine Archive, an unstable repository for our collaborative work. And here is one of the islands of the animations that we've created while thinking that under lockdown we're meeting in different islands. So within that, to avoid a neoliberal framing of technology as idealised or utopian, we've deployed Brecht's 1964 alienation effect, or A-effect, with our animation, which we call an immersion. These are cures when people are encouraged to question their preconceptions and look at the familiar in a new way, that is to make it strange, but also to surface where power is positioned, where corporate or explicitly political. In our case, one could argue that we've been actors rather than the audience, but at the same time, while we've been visiting each other's imaginary islands, that is one of those, we've been spectators, witnessing how each other's imaginary worlds were unfolding. So this was by watching each other's animations or images that we've been sent to each other, reading each other's texts, or even improvising music while meeting in these imaginary islands. This way we maintain the practice of what we call immortality, which for us is a critical immersion, to remain mindful of the platitudes and hype so often associated with virtual and digital technologies. So imerticality is a portmanteau term created by us in 2020. It combines the word criticality with immersion, evoking a form of involvement which avoids the hypnosis of being unaware of mediation, influenced by breast alienation effect, as Alexandra said, a distancing process which seeks to move audiences between their immersive involvement in drama and a sense of the conditions under which representation takes place. of the conditions under which representation takes place. Immortality is political and effective, asking of those who engage with virtual extended reality to also consider the conditions of its making, its rhetoric, for example, the so-called empathy machine, Chris Milk, this hyperbolic term for the supposed power of VR to evoke empathy in audiences. Immortality disputes such-determination and hype. It questions the construct of empathy which negates analysis of technology's entanglement with militarism." So this video is interesting. It comes from the glitters of our own collaboration. Our work moves between immersion and sober separation between poetic spells and unbeguiled analysis so these kind of conversations are important for us at the same time the nausea and the river in archive acts as an immertical mechanisms to break the metaverse fantasy of unsituated views from and the reminds us of our bodies, it makes and breaks a spell of mediation pulling us like a spring tide in it and out of the waters of immersion. The river is a key theme within our work, an agent and collaborator. We've used the tides of the Thames to define the patterns of a collaboration to carry away biodegradable work we no longer wanted to own. We've used the river as a metaphor and walked within the Thames and the river ravens born while recording narratives. The colour of the Thames captured on CCTV has also been used in our live performances and workshops, hence our framing of the river as a kind of agent, one which always has the potential to capsize our truths and destabilise what counts as knowledge. What follows here is an attempt to explain and analyse our work and riverine relations, which may at times only confuse further. While locked down, we also deployed an Olympian animation approach, returning to the algorithmic process of our earlier work, the five books, to re-energise our collaboration under conditions of anxiety and isolation. We developed and used a chatbot to represent a place. So in the next slide, this is the chatbot, this brought to our project a new set of agencies and imperatives somewhere between chaos and order, defamiliarising familiar patterns of writing and deploying it in an AR animated works. We mediated ways with methods of material thinking in the same ways that Carter in 2005 uses the embodied nature of material to produce new understanding about ourselves, our histories and the cultural we inhabit. For example, thinking through drawing, animation, model making and sound which all of them we've produced during our collaborations. Our work may be seen as a manifesto for a representational shift within contemporary technological practices in particular VR and virtuality. We urge artists, developers, directors, viewers and audiences to turn away from mirrors of realist correspondence and immersion and instead to engage with virtual reality of dynamic practices and actions. This is an assault on the Cartesian split between subjects and objects. It is intended as a provocation and the spur for new forms of virtuality, animation, pedagogy and practice. For me, the issue is this. How do we stop everything being subsumed into a near liberal model of everything as business it is connected to the construct of platitudes about a systemic empathy heavily associated with virtuality which most of my work has addressed in recent years for us as workers in the uk he system is also connected to the pervasive assault upon the humanities under the guise of a disingenuous commitment to STEM, creating an artificial binary between disciplines. We challenged the idea of an archive as a fixed account. We've been interested in the data lost during that time and since the process of this archive and its intentions would contradict our methodological approach. This is based on all our projects are based on physical body-to-body performance and relationship. This paper instead introduced the idea of an archive that is in constant flux like it's a mutable structure one that celebrates the idea of occlusion and the value of intersubjective emergence. We're interested in an idea of a mutable archive that resists to fix documentation and leave space for the non-recordable energy of togetherness. Togetherness, collaboration and performance are all methods of our work. And this archive allows for the data to dip beneath the surface and re-emerge when variables come together. Our work as designers, critical technologists, writers, as well as academic researchers, seek to identify the chances and challenges of virtual collaborative environments, adapting an interdisciplinary and intermedia approach at the intersection of gallery, games, film, theatre and fine art. Here is an augmented reality version of our models from five books. So we have chosen to work with a flow of data that's essentially position between chaos and or. In this way, we define the riverine archive as a system having what in Deleuzian terms might be described as a diagrammatic quality, such qualities are contingent, nascent or diagrammatic. These are conditions that Gilderloo's described as being necessary for generating the new. His description is highly resonant of a river. The diagram is indeed a chaos, a catastrophe, but it's also a germ of rhythm. It is a violent chaos in relation to the figurative givens. So we use image-making, narrative animation and technologies as a way of becoming disorientated and defamiliarised with our everyday working lives in our rooms last year during COVID and last you know year and half during COVID and to escape our routines and reinvent aspects of our work. We both longed for the times we used to meet at the Healey Fields which is a leafy park in southeast London with a cafe, a bowling green, a children's playground, a ping pong table, all of which featured in our virtual version of the park. During the second lockdown in 2021, we decided to recreate those meetings, those physical meetings at the Healy Fields, while also recreating the chatbot and training it to become the place, the Healyfields virtual space. So we wrote down some of our memories from the Healyfields to feed our chatbot and make it learn about this location, this park in South East London. So at Healyfields, ping pong was an important ideation method for us. We've been playing ping pong there and while playing ping pong we've been hitting ideas back and forth. We've been creating ideas with the body through the movement of the hand, through the anticipation of the ball and during the pauses around the table. So ping pong became a very important ideation method and this would resonate when we created a Healyfield chatbot during that lockdown, during the second lockdown. So we've used co-agents, including ourselves, other people, algorithms and places. The chatbot was created in Python by me deploying the Chatterbot library. This enabled us to train it with a corpus of common terms, but also through the unique conversations we had about hilly fields. We also trained the chatbot on our creative and academic writing as well as general knowledge about southeast London. After a few iterations of training it could quite effectively evaluate probable responses. Sometimes the responses were nonsensical but other times they were uncanny portraits and uncanny mirror of ourselves. So, virtual helifields creating unity with a soundtrack from Zoom conversations and throughout our work with Imeriticality we've explored what happens in the moment of breaking, of creating meta-narratives and analysis, how can it be framed as a critical immertical space, a form of meta-immersion, and what's the difference between creative flow and immersion. The work was extended in questioning ways of framing the technological interactions that happened during technologically mediated collaboration and the implication of scenography, world building, dramaturgic grammar, recording and post-production when we collaborated online. The archive of our work is in constant flux, like the Rivering archive, but also all our work. It's a mutable structure that celebrates occlusion. And in a way that what is interesting for us is that it allows for data to dip beneath the surface and reemerge when variables come together. So in this context, we're always at risk of being observed by our employers, observed by platforms, observed by Amazon, observed by Google, at risk of being commodified. We've explored how we can move away from these algorithmic constraints to establish more fluid ways of collaboration. We feel that when our collaboration was established we were able to improvise in the way that musicians do. This approach has direct applications in performance based pedagogic methods we've used during our online and blended learning and teaching we realized that throughout our work we've invented these third parties these agents and collaborators like the chapter these help us to de-center our authority and remove ourselves as the main characters we frame them as coagulants and agents between the fluid and this and the fix and this helps us dip in and out of our agencies and authorities as authors creators and educators I believe visual art does not need to be a static experience. It can be an evolving and interactive art form, have a transformational, transient and temporal nature. The combination of drawing with other art forms, music, dance, verbatim, is compelling due to its fluidity in the act of making. Creating an evolving dialogue, a dynamic sequence of movements unfolding in time and pulsating in rhythm while capturing the essence of a figurative pose, gesture or any other subject matter. The fluidity of drawing lends itself therefore very much for a performative application, outputs, in film, theatre or live installation. The technology transposes, integrates and presents drawing into a performative setting on a multi-platform level, physical and digital. But technology can also investigate cross-disciplinary relations, for example between drawing and music. Drawing in a performative setting tells a story in space and time, perpetually considering audience engagement as actively and continuously as for instance theatre and film engages them. It requires considerations towards varying the style, perspective, aspect ratio of the drawings and the materials you use. With individual gestures and mark making considering elements of creative movement commonly found in dance. How do you adapt inherent levels of reality due to the presence of the artist's hand, body and the materials? How do you renew your canvases and maintain the flow of the production between drawing scenes. The player's self yet too can be a character alongside the drawing component. An evolving drawing has the capacity of going through a phase of abstract form and shape before entering a period where a drawing reveals what it is representing. How innovative can you be in your mark making? How different can you describe subjects from reality? How to create a sort of liminal space while the audience tries to make sense, more actively engaged, involved or appealing to their own imagination, rather than just waiting till the drawing becomes officially known. I'm sorry. And then there are considerations towards cinematic techniques. How do you frame a painting from a traditionally static presentation to one that is perpetually evolving and transforming? Camera shots, camera angles, camera movements. And how do you collaborate and create a meaningful dialogue? Many of my productions have put this into practice collaborations with writers, performers, dancers and composers. Feedback from these shows often focuses on the trance-like state many people enter observing the live drawing, on the experience of figuring out what I'm drawing, on how responsive and emotional the art forms are to each other, and in particular on the compelling combination of drawing and music. Why is that, I wondered. Many artists have examined multi-sensory ways of perceiving, analogy between art and sound have been played with, but now, with cutting edge artificial intelligence techniques available, can we approach the relation between music and art more scientifically? And this is how Project Painting Music was born. Born from a more colded perspective than in my other productions. Born to explore more fundamentally the correlations between visual art and music. Can artificial intelligence replace a human? Can artificial intelligence replace a human? Painting music uses AI to create music from live painted drawings, in real time and unique for each performance. The AI is based on the type of learning used by the human brain and translate tangible painted marks into audible sounds. You are about to witness the results of artificial intelligence being used to create music from art. intelligence being used to create music from art. There are similarities between art and music. White space on a canvas is similar to silence in music. Highly detailed artwork is like an intricate melody. We can measure the white space on the canvas and input it into a computer brain. These inputs are like the sensory inputs we take for granted. Sights, sounds, taste, touch. Based on these sensory inputs, the computer brain can then make decisions on what type of music to produce. Our brains have 100 billion neurons. The computer brain has 25 neurons. What can 25 neurons accomplish? It can still make decisions and it has learned to understand these artworks by analysing over 1000 images being painted. It then creates notes based on what it has previously learned and what it sees in the painted object. Is the computer brain showing intelligence? This table is relatively subjective but shows what physical parameters we measured in the artwork and what we used as its counterpart within the music domain. The analogy is just our interpretation and also just to get started with our very initial prototype. The computer brain has 25 neurons representing the amount of white space on the canvas, how grey the picture is or how complex the image that is painted. Each neuron in a computer brain becomes an expert in identifying one aspect of the painting. We can use this knowledge to convert what the neuron tells us into music. The AI brain has also learned musical structure like chords and harmonies. It does this based on how the painting develops, so you will hear the music grow and change as the canvas becomes more full. We are currently working with composers and sound technology experts to improve the fundamental music theory within the AI system and a correlation between the elements and principles of art and music. So in real time during the performance the computer takes a snapshot of the evolving painting, the image gets analyzed, the AI extracts the attributes and produces a musical motif and integrates within previous music. There are different ways to create an AI brain in a computer. We are using the AI approach closest to our brain as this allows the learning to be explainable. This is important since we need to know what decisions the AI makes as this information is used to create the music. These are extracts from a 2019 performance where we just developed the first prototype. © BF-WATCH TV 2021 Sous-titrage FR? Субтитры создавал DimaTorzok How do we learn? As children we learn through play. We play with our environment. We improve every time we play. We learn, and we like winning. And we like winning. And we hate losing. If we gain knowledge through play, is playing a sign of intelligence? Have we as humans evolved to be able to play with the world around us? How can a computer learn? How can a computer play? How can a computer play? If it plays, does it like to win? Does it not want to lose? How can a computer understand our world? How can it interact with our world? Mark making has a direct implication not only on the composition of the drawing, but also on the development of the music, and then the audience themselves. So although drawing is often considered a static art form, the music output transforms this into a temporal and transient product that is dynamic and depends on the order and choice of mark making. And the nature of the drawing process means that the mark making will always be different, even if the same drawing is undertaken. This is also the case for the music, due to the built-in random process for the development of the notes and chords. This lends the entire system to a performance, with no two performances alike. The whole idea of the AI working in real time, so this is not something that is recorded, we do not know what the AI will produce for music and every time we do it it is different because the AI is choosing what to play based on what I paint and I'm also not painting exactly the same painting each time. So there is an element of risk and an element of danger in there. And this edginess to the performance emphasizes and enhances the story of painting music, centered around the question, is AI good or bad? Exploring thoughts and fears over the application and impact of AI and its prevalence both today and in the future. The project aim is not to create a process that always results in harmonious music and melodies but that the AI can make mistakes. Harmonious notes may be considered uninteresting by some people but beautiful to others, or may result in an interesting and novel musical form for some listeners but jarring for others. Likewise, conflict transpires in the visual component through the subject of each drawing and the application of dramaturgical drawing techniques. Dramaturgical drawing transforms a relatively representational drawing to an abstract composition, forcing the audience to consider and question the meaning of the original drawing, as well as the reaction evoked by the development of the drawing. It can induce quite an emotional response by the audience, occasionally expressed as a gasp or shock running through a show. The investment of an audience into a drawing brings on a certain degree of conflict when that same product is transformed into a new artwork. Dramaturgical drawing is also sometimes quite sudden, with a relatively big consequence, both movement and impact being further amplified on a huge screen. How much further can the AI go? Could it learn from an audience reaction to improve the music it creates? Can it learn from itself? Could it start to produce a new form of music? And what of other developments in AI? Companies are using AI to monitor and respond to how we interact with our world. This AI has been exploited to shape our thoughts and our discussions, to provoke discord and disagreement where perhaps there was none. Humans are physical. We have hands, we have eyes and ears. I can touch, I can feel, I can see, I can hear. My world is infinite. I have emotions, I can laugh, I can cry, I can be nervous or excited. computers are digital they have no hands no, no ears, unless we give it to them.. Computers have no emotions. They cannot laugh, they cannot cry, nor understand what it is to be sad. How then does the AI judge the music that it creates? Its world is binary. Zero or one. Yes or no. On or off. How can humans and artificial intelligence speak together? Will we ever have a common language? A common culture? Is artificial intelligence really intelligent? Or just a poor imitation of ours? Can we join together in a community? A change is coming, a revolution, an evolution, a technological arms race. Should we welcome what is coming with open arms? Or should it worry us, make us anxious? Is this the dawn of a new age of man? A combination of humans and the digital world with artificial intelligence? Can we keep control of our world that was shaped for so long? Or will it now start to shape us? Technology is a tool within a story. It allows an additional dimension to the artistic capabilities of a piece and moves visual art as a traditionally static experience into a dynamic interactive art form that is temporal and transient in nature. Painting Music won the Connected Innovators Award run by Creative Edinburgh and Creative Informatics. As part of this program we are designing a portable standalone system that has the ability of live broadcasting to audiences and creators through any physical and virtual platform. This bespoke product consisting of multiple cameras, lighting, painting board and AI brain allows me to interact and perform with creators and audiences from anywhere in the world reacting to current affairs related to AI that is already firmly rooted at the heart of our society and perpetually underlying and impacting global decision making and experiences. First of all, I would like to thank the organizations of this panel and Ars Electronica for the opportunity to talk about animation, an so important medium to a wide and high-level audience. My speech beholds the field of artistic animation when it works strongly with music. The objective is to analyze how to create images in movement from a piece of music, considering their nature and the personal experience as an animator, and trying to clarify barriers, which are the main aspects that wake up the author's imagination to create figures and animate them, as well as how can music influence her or his aesthetic choice. I will not present any deep analysis on visual or auditory perception, however, just the necessity to understand the core of this research. I also declare that I do not have any music in the terrace. I would like to stand out that music has always been a significant element of my animation works to achieve the emotional goals that I want to express. To target the goals, the methodology is the case of study of two animated shorts in different contests, a supported and no supported production. The first short is Mildinho by Claudio Roberto, is a 2D and paint animation videoclip made over the harmonious music by Heitor Villanobos, a famous Brazilian classic musician, and performed by Duoges Branco, a duo of women pianists. The animation was a result of a video clips contest from Brazil, so it's a case of sponsored animation. The second shot is modern, a film d'auteur and mixed technique. It is the product of a will to create an animation from live action shot as animated sequence. From this premise, I developed the hand time-lapse to this initial sequence and just after found a piece of music on a commercial site to create the whole animation. I constructed the analysis based on the studies of distinguished scholars and also considering some indispensable testimonies. The testimonial on the animated productions. Just the beginning, Miu-Jin has music as the ground to the image arrives. We knew about Villalobos' life. He had made many tours into the country territory to listen and register the sound from the forest, birds and nature, and identify this spirit in Mildim. At this point, it is important to highlight that as Brazilians we appreciate Villalobos work and the diversity of our music. Probably if we were foreigners, we could not identify some rhythms within the new theme. Therefore, the personal creative context of the artist was fundamental to perceive the information from music. In this case, we identified three particular moments. First, our cadence, deep, pound and irregular rhythms. Second, our fun, free and variable with five subdivisions. And the last one, a repetition of the first melody. Within the second music section, we couldn't identify rhythm strays from each one of the five Brazilian regions. The baião rhythms from the north-east region, the aibão rhythms from the northeast region, the ibexiana flute from the north region, the right-pitched and fast rhythms that sings the viola music of central Brazil, the swing melody that was the region of samba from the southeast region, and the last one with the regional beats from the sound of Brazil. So after the identification process, we start to create the image to represent what the music graph was, according to these efforts. The image should present different musical moments to represent the middle part of the music. We would use the flora and fauna from each region as reference, the last music segment should have a similar visual to the first one. However, both are very diverse from the middle part, so if the middle would be in color with organic figure, the other two parts should be in black and white and abstract. We also watched films by Norman McLaren in Landline in Oscar Fischinger. He gave us the idea to work with geometric elements, which open up paths to represent piano keys in some moments. At the time, the animation had the structure defining, a small clash between light and darkness that resulted in life with colors and a rich rhythms. It defined how the logical and the structural would bring emotional colors, the sinuosity of the movements. The ending is a result to the beginning. After this work concept we went to the hand test to design the storyboard, count the music times, look for image and movement reference to the animals that we would use in the animation, and so on. The second shot is modern. There was no idea, script or music being an undefined, unpredictable procedure. I consider using a design process while models are an artisical one. I stopped my thesis because I needed to handle the subject I was studying, the difference between animated and live-action images. From this context, the live-action sequences were formed into image series, but choosing each picture to create a new sequence. That gives me the idea to apply the image filters individually, what I called hand time lapse. Most of the sequences were shot from nature. I liked some of the new sequences, but I needed to join them. I did not have financial support for time, so I went to look for a piece of music on the internet. The sound would help me join the time-lapse sequence and stimulate my creativity to create another animated image. After some week, I found three spots that moved me. But just the music in my time-lapse seemed to work well with the sequences I already had. I do not consciously know why this music moved me. I know that it woke up more images in my mind, being the reason for my choice. It worked with many time variations, dramatic and emotional moments, and piano and violin sounds reaching in the spotlight over the music. Some moments remind me of a drop of water. Consequently, all in the stage process I realized that water was the theme, since also the better sequence I had were from water, river, sea and clouds. After that, the image creation process became less blind. I could think about water and join with my emotions listen to the music. Therefore, music gave me all information to create. It differs from Mildinho, which came along with Villalobos' story and characteristic as an artist. The source of the ideas for MOTHER was me and the music itself. I tried to listen with my emotion and not with my ear to bring to my mind anything. So Mauder was the result of a search for a way to express emotions inspired by music. Some concepts for important study. The first approach is based on Laurent Julliet, the scholar's research on the operation of human reception in memory and some points turned out. Mainly the perception, cognition, emotion, the sound images and auditory attention. In the process between visual and auditory attention and memory, we have three possible situations the pre attentive process working memory and long-term memory the first is related to the vision and ambition which have simultaneous process in our brain but without pay attention our brain makes a comparative action through wild time with our long-term memory, to know if there is something dangerous or important for us to be attentive. The second situation happens when we pay attention to something. Paying attention to what we hear determines our understanding in its registration by the brain. And the third case, as inside doubt has shown to us, is long-term memory. That is created when we pay attention with an emotional charge included, as a day of our thesis presentation. Some studies identify it as a combination of a semantic memory, addition to the episodic memory from the life moments, or a declarative memory plus procedural memory. These two contexts are used by the audiovisual reception process to understand what we see and hear. It is good to stand out that this process is the same to the viewer and the creator, since both work and react under these memory connections unconsciously. And that is a great point, as Julia also affirms that this perception, cognition, emotion is unbreakable. When the sound enters the eardrum, it is activated and sends information to the brain that processes it, identifying or searching memory for some registered reference. This determines the interpretation of what is heard. So, a sound that reminds me of a good or bad memory, that emotional data will contaminate the interpretation process. Our emotional impressions color everything that we perceive, and this is another great point. At this moment, we can understand the sound image. Julliet uses this macadam's term to explain that anyone can create a scene just hearing some noise, as a field falling from the tree. Anyone can understand and feel, due to the old information from the episodic memory. So, when we receive both hearing and visual information, that is very strong to our understanding. That synchronism generates a single apprehension and even seen as something unique and not as a combination of two different assimilated sounds and lines. This union is what Philip Murphy calls symphony, symphonic experience, observing Walt Disney's work. The scholar affirms that fusion is the best term to describe such a symbiotic relationship between animation, an artificial life force, and the sound, an organic life force. Frank Thomas and Ollie Johnston, who worked with Disney, called that synchronicity Mickey Mousin due to the force of steamboat wheels impact, but they state that music can add to image, emphasis, meaning meaning and any other single ingredient. And that was a substantial entertainment aspect of animation cinema, as observed by Gunnar Storm. And in Disney's productions, the beat and instrumental sound effects were created to the laugh and funny of the audience. However, in Fantasia's case, according to Thomas and Johnston, it was the music that gave the ideas and time to animators' work. Norman McLaren states that many times he also used music as a script, and that the organization of the music sound is similar to the organization of animation. McLaren created his own music to his short neighbors directly on the sound band of the film, but also animated basing on music like Boudioudi to the short Boudidodo. He knew the importance of the sound or the lack of it to complete that scene. Considering all image and sound as a couple working together. Other two scholars and professionals on the audiovisual domain, Michel Gion and Michel Farrell, agree with McLaren. Although they present different conceptions, Michel Gion developed the no discontinuous acoustic conception about the sound to cinema. Thus, there are cuts and different kinds of sound in a film, so different attention from the public. While Michel Varone developed the continuous sound concept, since for him everything is music, is all film sound, noises, effects, dialogues and the music itself. The scholar also identified through Veronique Camp's work that the sound does not exist only according to its source, but there is a contest to its existence which implies iconic information. On observes that the sound continuous and iconic sound concepts by Faro are important to this study. So, considering that, how did happen the creative process for new Jim and mother? Music results from a set of sounds created by musical instruments present a certain melody with the tone, time, duration, loudness and attack decay. It can also be a combination of repetitions. This is the case for both shorts and plush. And when creating animation for a piece of red-laid music, that happens in some circumstances. Music has a story, like Neil Gini, or it may be completely unknown to the creator, like in a magic time-lapse. In the first case, there are references for the creator to start working. They are consciously investigated or unconsciously, for some information rather absorbed by the creator's life, but both interfere with the creative act. Mildino has historical, artistic and cultural references. In addition, we are Brazilian, so there is a use of our episodic memory. In the case of an unknown music, there are two other situations. The choice of music is a question of liking, unconsciousness, and a piece of music with no reference offers more freedom to the imagination. That divers of support projects. During a process of the creation of the image, there was a division of the music, observing the music sound wave. In other words, there was a selective listen, based on the select perception of each part of the music. From each stretch, the beat of the main instrument served to imagine what movement it inspired. Once Mother's theme was defined, the image of drops of water should be different. Isolate drops, rain, firm thoughts, tears, snowflakes. Somehow the music stacks words as icons to my episodic memory and my imagination. The same happens to new genes our episodic memories or our status. Since the sound has a meaning beyond what is heard and for who heard it, music has the capacity to join anything imagined. These images are the result of the records of the objective and subjective information that the creator has observed consciously and unconsciously with music, that enforces the alignment to the final studies. Conclusions The creation of the image from an artist will be different from another one, even when inspired by the same music. If ten animators are asked to create a moving image about a piece of music, we will have ten different animations. Each animator will have his or her personal interpretation. Music is a different index or information icon, we can say that, for diverse people. In other words, when creating an image for it, this is a potential open world that becomes connected during the process of creating the image. And then, once you join this image and the music, we have another result. As Eon and Fanon affirms, it would be heard and seen as one of the visual pieces of information, since its perception as a separate element does not exist. And it is noteworthy that the audience also interprets any visual and auditory information according to their episodic memories, and animations, as with the gene or model, start another process, the search for a narrative. That is unconscious action. If it is not possible, the audience can let the mind open and free to perceive what is on the screen and not a conclusion about what they perceive. Only when the projections end, the spectator is able to interpret the short as a whole and answer the questions. There is a process involved in this creative act. The animator's willingness to create images due to a paved work or artist's impulse. The relationship between creator, creating for music and emotional creative act, as Damasio says, if there isn't some kind of empathy, it's harder to create if it works. And it's necessary some life experience for music to wake from memories and emotional or physical sensations that then arise in the artist, experience in the imagination, the story of music, the references that, by chance, it brings to the process. So, the common element to our point is emotion. That is the primary element to every human being or animal. Joy, sadness, ills, disgust and anger are secondary emotions to human beings. Therefore, it is the emotion that musical wakers favor in creativity. The artist must be open to feeling. And how can music influences her his aesthetic choice? The aesthetic issues is always linked to the style and personal taste of each creator. However, music is a kind of territory to artist's work and each artist will interpret it according to her his experience, which works as a kind of filter. In other words, music will never be the determinant element in the creation process. Obviously, whether the music to be animated is a rock'n'roll, that already frames the creative space to work in. However, how it would be explored depends on the emotional charge of the art absorbed in memory. The intersection of what arouses as image and emotion from her or his memories and other objective work elements is what will determine the artist's choice, taking colors black and white, brown and brown, etc. depends on her or his technical capacities, of course. To conclude, according to this research, the key to all this process is the emotional charge of the artist's life. Our emotions are source and working elements to our creations. Thank you. You can start now. So hello everybody and let's start with some questions and encouraging all our viewers to write questions in the chat on YouTube and we can ask them as they come up. I thought I might start out by asking each paper group some individual questions and then I'll go into a more general discussion. So if I start with Eleanor and Alexander could you start by explaining Oolipian because it's in the title of your paper and then you didn't develop that further. Yeah great great question thank you. So the Oolipo movement was this movement for potential literature from the 1960s onwards with particularly people like Georges Perrec and Raymond Queneau and they used algorithms or rule sets, small of rules finite with finite endings and beginnings which are very reminiscent of you know coding practices that we use in our work and we've used them extensively to in the way we try to explain where there's an element of chance which I think has a lot of resonance with Kate and Andrew's work there's a sort there's a potential for chaos there's a potential for order there's a potential for innovation There's a potential for chaos, there's a potential for order, there's a potential for innovation, there's a potential for dreariness and we kind of really like that. And also that just to add that the Olympian approach was used especially initially when we used algorithms to set up collaboration and have it as a as a theorem for collaboration and later on they were kind of dismissed and went more into the the trance and kind of poetic literature side of it that is used in our latest pieces of work like the river in archive or the hilly fields work or even the the ping pong exchange or the ping pong as a ideation method and an accessible example of that that I'm aware of is a book called 99 ways to tell a story so there's it's a graphic novel and it's the same story told 99 different times so it's setting up a rule isn't it and following in different ways and i was i was really interested in the islands you designed so did you design them together or did you or i imagine did you design them as a surprise and a present for each other yeah yeah it was a surprise and a present for each other each week we would invite each other to each other's island and we've been waiting for an animation a a letter or something, and we would meet in that island and in that space, in that imaginary space, we would improvise, play music, but also kind of respond to each other. Yeah, yeah. So I suppose in the Mozilla Hubs, Alexandra, we were designing those together as well. So there's an element of that, as well as a present giving, which I think is a really interesting part of the collaborative process with machines as well. So some of these ideas, Mouse's ideas about the gift, I think are resonant with everything I've heard today, actually. I thought that was a really nice idea during lockdown that you'd have this gift for each other. for each other and um and i'm also really interested in to pick up on something you were talking about which is um critiques of neoliberal and uncritical uses of technology you're you're particularly um critical of kind of immersive shamanistic trance-like approaches to um virtual reality where it becomes almost like a drug type experience oh no no no no i love james marx's i love some of the words i know you love that psychotic uh psychosis with i think they're really interesting questions but i what i'm what i personally have worked a lot against is um a sort of uh neoliberal rhetoric about generate its deter predetermining rhetoric about its ability to generate a kind of a construct of empathy, which I think is extremely neoliberal in presupposing the outcomes of a human encounter with the digital. There is no reason to believe in this predetermination and very, very unconvincing evidence. But from gifting to charities that they generate this thing called empathy or have any agency in making social change that's all i disagree with i love i love the stuff with shamanism that's really interesting okay did you okay so and then finally since today we were looking a lot sound i was interested in your use of sound you said you use sound from your zoom conversations how did that work yeah yeah you we've used sound throughout the projects from uh just using it initially to punctuate um the the five books rooms and and kind of create sounds that that give an essence of the rooms that we've been imagining uh you've used we've used the ping pong the ping pong sound the part the sound of the ball as a way to think about exchanging ideas um we've used um with the healy fields um uh work we've done lately we've used sound that was a reminder or a memory of what we've been doing during our walks in healy fields and i found that quite fascinating because there's a word uh remember do you remember do you remember that is constantly kind of replayed and it's interesting how that that space that we used to walk becomes like a memory space and how it's changing also through the technology but also through our own memories through through the retelling of the space uh Eleanor would you like to talk yeah we re-edit we re-edited a lot of our zoom conversations and then put them into the unity game engine with our work with with unity and actually for me that was some of the most interesting work it made me think this bland platform of zoom which we all the most many of us working on all the time it's so dreary it's so tedious so tiring and so innovating and actually remixing it including the glitch and glitching it deliberately it made it into a sort of material that i found but we both found more more creatively conducive just you know it was a relief from the tedium of our work in five many ways and this was almost like a exquisite corpse game between yourselves and the chatbot yeah yeah nice way of putting it yeah so if i if i move on now to um kate and andrew and jack um i'm interested in the first of all sort of in the live performances what was the role of the dancers um actually the dancers were not included in our painting music project. That was just a build-up to what I was initially doing and how my audience feedback got me into thinking about the relationship between art and music. And from then on painting music was really born because let's investigate it more scientifically if we can. And so I was wondering, you as the performer of painting, so that you're creating music via AI, was it totally unpredictable? Did you start to recognise that certain shapes or movements would generate particular sounds? Because for for me it must have been like playing an instrument you'd start to recognize the rules of the system you were working with you'd start to make particular marks in order to generate particular sounds did you do that and i was thinking particularly about norman mclaren's pinpoint percussion experiments where he was painting on the sound strip of film and generating sounds from image through painting but you know using film. Okay okay well to be perfectly honest we literally finished the AI software like maybe an hour before the performance so the one that I was showing you so there was no chance really to get familiar with like how the AI was going to react and things like that. In the meantime, we will be working more towards how the AI's relationship is with the music. And currently the music is quite simplistic or very simplistic. And we are building yeah with um yeah composers we will be um adding fundamental music theory so in that sense to answer your question you will be um there will be a higher correlation really between elements and principles of art and music so if you make a really big um dark shape you will find a low frequency of that in the music and so yeah you can build towards a more predictive sense towards that as well but in a way yeah the randomness is still like inherently yeah built in. Andy I don't know if you want to comment on that? Yeah I could I mean yeah it was unique each time and it was doing a different tune every time, you know, it was responding to Kate's paintings. But the thing that we recognize the weakness in the project is the understanding of music, let's say. I mean, it's not a trivial task to compose on the fly. and even with the AI. So the AI is giving some indication as to what notes, what range of notes and kind of what tones and what kind of duration that it wants to create. And then there's some heuristics. There's another form of AI that sits on top of that too, which we don't really talk about, but that's also guiding the process. And in all of that, there is a random choice that the AI whole system has, which means that although Kate might have been familiar with kind of roughly what it would sound like, it was generally different each time and it would vary its tempo perhaps more than we anticipated and things like this. And so it was very different. I mean, we had the two performances like Kate says when you saw the film and we had the rehearsal and then we were pleasantly surprised how nice it sounded because we just we didn't really know and then obviously the performance itself we in front of the audience we also were unsure what it would be like and it was it was some of the tunes we thought were better if you like some of them were worse and but that was that's the very nature of the project is that there is this kind of randomness but the other thing to add in is just also we had it learning styles of music so we had it to learn it wasn't mentioned but we had it learn Beatles, we had it learn Philip Glass style music so it would kind of use those as templates for the type of notes and chords that it would then create and we want to go further with that with proper styles like country or jazz or blues and things like this and that's kind of the current direction that we're on. And Eleanor raised the point of Daphne Oram and Oramix with her sort of machines for generating, do you want to talk about that Eleanor? Oramix? No, it just remind me in a really great way, I mean, Daphne Oram is one of the absolute pioneers of synthesizer. I mean, that might be the question, is you see it as a synthesizer in a sense? Yeah, yeah. So actually I've been working, so I work in the School of Engineering at the University of Aberdeen, and I've been working with the music department there as well. There's a guy called Peter Stollery, so he's very interested in the sound aspect and using ai to to do that the synthesization so how can it how can ai could it can it create new sounds a new sound for example or how can it create something that is interesting or not interesting or whatever and how can it do that so that kind of process which is just about the the formation of the notes and it is part of it as well and that's definitely on our forward trajectory too I think. I was wondering about the translation because as you know Elaine's showed there's different relationships between animation and sound you can have a direct translation or you can have counterpoint you can work with harmony or rhythm or pitch or tone. I wondered what principles You talked about white space. I wondered how much you were getting into all these different aspects of music Yeah, I mean I could comment on a first and Kate could come in I guess and that's exactly the sophistication We want to build in one of the things that Kate wants to look at is symmetry For example in the painted object and some of the more principles of art let's say applying those and working out what they're what their what the relationship to music would be and how we get it to do that on a more fundamental level I mean my research is about AI and although we're using in some respects a simplistic form of AI and the reason we're using that is because it has to be explainable it has to be something that can can give us a logic as to how it's doing what it's doing which then means form of AI and the reason we're using that is because it has to be explainable it has to be something that can can give us a logic as to how it's doing what it's doing which then means that it can do the type of things you're saying yeah sorry oh I was going to say we had a speaker last year who was looking at Islamic geometry and translating that into music and he had a very mathematical correlation between specific tones and notes and rhythms. So yeah, I mean you can definitely do that but what we want, ideally what I'd like to do is have the AI learn music and learn the corollary between the two and actually then start to produce something, actually start to compose something using its own internal rules that it's learned from the music that it's seen already. I mean one the things we talk about is is that music is a reflection of of of of human history i mean music is a language which we you know we've grown up on and we know the history of it we know so for example you know rock music came along if you played that 100 years ago it would have been jarring but we get used to the language of it and and that's kind of what we want to try and build on and get ai to understand simple things and then try to get it more sophisticated well you're going to add anything to it uh no i'm ready for the next question okay so um i was thinking about, which one should I go to now? Oh, yeah. So this morning, Lily Husbands, sort of following Norman McLaren, was talking about abstract animation as a form of dance in which the viewer experiences kinesthetic empathy with the abstract shapes. And I wondered how your audiences responded to your live performances. Did they become, you know, did they feel some kind of empathy with the painting? Did they just see it as a novelty or what was the audience response like? Yeah, generally the audience, well I'm told it's very mesmerizing and like quite, yeah, and also, yeah, the emotional reaction, well, the most clearest emotional reaction is when you transform a painting, so when at the end, yeah, I call them dramaturgical drawing gestures, so, you know, you pour water over it, but as long as there's meaning then but yeah often you do hear a gospel shock because people have invested in that drawing and to change the meaning of the drawing because that is what they have to then reconsider and and it's amplified on a big screen so there is quite a shock to it and uh yeah i've had people like crying in performances in different productions. I don't know it's I'm not saying this is necessarily drawing but it's the drawing and the music combined I think that somehow works quite well with our brain. Talking and drawing sometimes can become quite overloading especially if the drawing is very illustrative and the talking is very illustrative it's still overloading so I have to calm down illustrative and and the talking is very illustrative it's it's still overloading so I have to calm down my visual so things like that you have to play with and be aware that you're not overloading the audiences and and get the full attention that you want them to have and and invest in the whole process but yeah there's definitely a lot of emotional reactions to yeah to some parts of my productions and to follow on these points about being critical about the technology we used it's quite interesting you talked about music and sophisticated and understandings of music being needed to underpin the ai and i was thinking about you talked about the use of ai as somehow scientific in its analysis of the image, whereas researchers such as the Algorithmic Justice League show us that there are prejudices, our limitations are hardwired into the algorithms that we write. And I wondered what you thought about that. Yes, it's true. I mean, like AI, it's, you know, the data gaps that are existing in our society, it's really scary. And if you think AI is going to, you know, slowly take over, those data gaps are only going to be amplified, you know. So on a statistical level, it's scary, because, you know, well, especially some of like, you know, feminist issues are not going to be taken into account and only going to be amplified and make worse. it is in a way it's quite a scary thought uh what ai is doing and it's it's not the fault of the ai it's just what do you what you put in you know is what you're going to get out so there is yeah inherent bias so um yeah it's uh and it comes down to ethics in that sense. So, yeah, I forget. So I forget the question now. Sorry. Well, you answered it. Yeah, okay, good. And just find it. Oh, yeah, sorry. Can I just comment quickly on that? Because it's an AI question as well. And I think the issue here is that most AI is black box. And that's where we get the problem with the bias you're talking about and the morality and ethical questions that come with the use of AI because it can't explain itself that's at the heart of the research that I'm doing it has to be explainable and actually it's the heart of painting music because it is explainable so painting music isn't and it doesn't have the same bias or if it does we know what the bias is because it explains to us what it's doing and the 99% of AI applications don't do that and that's the problem. They're not explainable and therefore the bias is hidden and then that's where the danger comes from. Well you're going to make a comment Eleanor, you wrote something. Yeah I just wanted to say is there not also the problem that a regression algorithm you said that it was learning from styles if if those styles reinforces domination of a particular culture what what what do you do to avoid that that's the problem with the racism that emerges through not not your work but through through uh no exactly it comes down but what we're talking about is how it makes its decisions so the racism issue for example comes from the fact that it's making a decision which is wrong well the inherent logic is regression algorithms replicating the status quo aren't you not explainable yeah exactly that's the point it's not explainable if it can explain itself then it would be saying i'm choosing this because of a black person but that's wrong the logic is wrong so but because it doesn't even explain what that logic is then it's hidden and then we see this that the misuse of ai and if ai is explainable then all of a sudden it's transparent and these these these issues would be would be able to see them would be able to say oh it's making this decision in the wrong way we see the same thing with self-driving cars it can't explain itself it can't explain how it makes any decisions and that's where we have the the tragedies of you know, with the woman that was killed by knockdown by the Uber car. That's why, because it can't explain itself. Same issue. Okay, so let's, I have to ask just one more question because there's one in the chat. Leopold Zika says, why does the ai not paint the human painter is redundant i was going to comment on this so in theory we can go both ways again because for this because of the same reason actually we've been discussing because the ai is transparent we can actually go to the other direction so if someone's creating music we could then use that to create images so we could in theory do it but it would require an additional AI to do the painting part which we haven't developed but in theory we could easily do that but obviously just remember the question on YouTube is kind of insinuating that yeah the human paintings were done that it's not we need something to seed the process of the AI to create the music all the other way around so we would need music to create to create painting or we need the painting to create music. Okay so let's go over to Elaine now. So I was interested that in your paper, you were rather than some people who had very abstract relationships between music and image, like this morning there was people was people they were mainly using abstraction um quite um you know lines and color and so forth and i was interested your use of kind of leitmotif so almost returning to classical music we might who might be thinking of trying to replicate birds or nature that you were looking at specifically brazilian cultural memory nature, que você estava olhando para a memória cultural brasileira e evocando isso, para que fosse mais evocativo. Sim, na escala de Nildinho, até mesmo a história do músico, porque ele também era brasileiro, the sound of music, that seemed like the main sound from Brazil. So there is no way to take all this information, take it off and begin a new thing. That is impossible. Even because the creators, me and Paulo Paul Colbert, we are Brazilian so there is no way to make this things a part of us so I think in this case it's better to go in this subject, go deep. Yeah because some people working with visual music might literally say oh there's this particular note i'm going to use red and you know they put very linear correlations between properties of the music and properties of the image but you you raise raise cultural memory, vocation. Yeah, yeah, and in this music specifically, there is a middle, as I said in the presentation, that reminds us some rhythms from Brazil, from each region from Brazil. So to us, as Brazilians, that was very, very strong, strong force and emotional force too, because music always dancing with our emotions, in a way, in a good way, but not that way. So it's not the point, but this movement, I can say that, it's a movement that action, it's like an action inside us. It makes a click in our mind. As I said, it's our exotic memory. It's an unconscious action. It's not conscious. and you can create the english it is it can create in this creation it's not a rational to me it's unrational it's unconscious you don't know why you think about the flower you think about the water you think about the flower, you think about the water, you think about a boat, you think about the sun. The music brings to us some sensations and these sensations that create, make us create image. And this image make us also create movement because music is also movement, it's rising, it's movement. It's this movement that moves us. I really like that about Lily Husband's paper this morning where she compared animation with dance and how we feel a kinesthetic empathy with a kind of more abstract animation and we're identifying with the the visualization of movement yeah yeah felt the paper about ocean yeah said about it yeah yeah so i'm aware that we we did start a bit late so i'll give us five more minutes so would each um panel like to just say a little bit in concluding remarks of any sort of synergies they see with the other papers? So let's start with the first paper. Eleanor and Alex, what kind of synergies do you think your could um your paper spoke to would be others yeah or maybe they're right when they're not synergies at all i mean there's not there's not inevitable synergies there um i don't think that i don't artificial intelligence publications okay i don't artificial intelligence isn't just a tool it enables and disables it discriminates it embeds cultural prejudices as well as interesting things that's far more than a tool and increasingly if it's used as a determining to discriminate in terms of policing in terms of mortgage payments then it will have a material impact on people's lives and their ability to to either be free or be in prison it's part of the school to prison pipeline in american racism for example british racism so i think it's far more than at all and that that's why i'm left with a couple of other great things about all the work but alex yeah and i i agree and also i i also find fascinating the differences between the papers and the different uses of drawing, the different uses of sound and movement and how, as you said, how movement and sound moves us and how that relates to technologies and the way we use technology through that. So I think it's really important to also look at the different uses of technologies within the papers or throughout the papers. And Kate, did you have any concluding remarks that sort of brings together the papers or thoughts provoked by them? Well, I don't know, just thinking about the AI as a tool, I think I would say, the AI, you know, I have a problem is that you use a technology, whatever that may be, as a, say you get the most innovative kind of technology out there, people often use it as a gimmick you know it becomes a gimmick and the story is lost so this is one of the reasons that when i say ai is a tool in that context i think i mean um you know the story it ultimately is about story art is about the story and and often it's about people and technology can you know enhance that experience or can help that experience or disable that experience but ultimately you know it should play a subsidiary role to the story ultimately I mean my painting is a gimmick you know it's the story that counts but and second the second point on that as well, because it is an interesting, because sometimes you have an artist in Canada, I forget her name, and she paints with robots, and she has this big robotic arm, and it looks like they do synchronized dance, and she advertises or PRs herself as like collaborating with robotic arms or robots. And that's obviously, that sounds very nice, and you know, arms or robots and that's obviously that sounds very nice and you know a PR kind of you know set up um but it kind of annoys me a little bit because it's not you know a collaboration because if you look up the definition of collaboration there are people involved you know so people create this technology people provide the bias, people are, you know, the cultural references that we are like taking forward, that is built into AI. Those are the tools that then create and make decisions in our political and, you know, the yeah, decision making. So it has a huge impact, but I still think you know it's a tool and not a collaborator and in that sense. Thank you Andrew, do you want to say any concluding remarks? Well I'd say Eleanor summed up justification for all my research better than I could I think because I mean what she stated is exactly why I'm trying to develop new forms of AI that are explainable, which meet the things that Eleanor talked about. I mean, I found a lot of what everyone talked very interesting because it feeds into the philosophy of what we're trying to do, which is make something, which is marrying these different art forms together in a way that's novel and interesting to the audience. So yeah, there's actually some things to go away and think about from what we listened to this afternoon. Elaine, did you have any concluding remarks? I think it was very interesting because there is two points of view different from the drawing of music and sound more technological less technological but very interaction, more emotional as I said so I think it's a good panel to everyone discuss and think about this couple music and sound music and picture and drawing because you all had a drawing in there somewhere as well yeah you know image technological image or not technological yeah okay well thank you all very much and hope you join us for the next panel a bit later on. Thank you. Bye. Bye. Thank you. Thank you. Thank you. Thank you. Thank you. Diolch yn fawr. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Diolch yn fawr iawn am wylio'r fideo. Thank you. Thank you. Thank you.