Thank you. So, good morning everybody. Hope you had a nice evening yesterday. Looks like a lot of people are missing still, but I'm sure they come around. Yeah, in this first panel of the third day of the Expanded Animation Conference, we would like to present you two very experienced artists that worked for a long time in the field they are talking about. And yeah, we will start with Jan Pinkava and then follow Bonnie Mitchell. Both will talk about and give insights about their work and the latest things they did. So, Dr. Jan Pinkava is working in the field of animation storytelling for more than 30 years now. At Pixar he wrote and directed the Oscar-winning short movie Gary's Game, which I guess many of you know. And he created and also co-directed this Oscar-winning movie Ratatouille that most people have seen. Ratatouille that most people have seen. He was then co-founder of the Google Spotlight Stories where he created new forms of interactive storytelling and was awarded with an NME and got nominated for an Oscar for outstanding innovations in interactive storytelling. Luckily for the Film Academy Baden-Württemberg. He is now the director of the Animation Institute, which is very well known and famous for all his productions and he's working there in the field also for animation, VFX and interactive media. And he will talk about the problem of interactive storytelling, which is called the illusion of choice in Interactive storytelling, so please welcome Mr. Jan Bakkever Thank you very much. Thanks Hello everybody good morning. Thank you for getting up on a Saturday morning after I'm sure a fun night and for coming to this talk Illusion of choice I feel I feel in this conference especially because it's so themed on AI gyda chi. Mae'n ddiddorol iawn. Diolch yn fawr i chi am ystod y bore dydd, ar ôl ystod y nos hwyr, ac am ddod i'r sgwrs hwn. Yliadu o ddewis. Rwy'n teimlo, yn y cyfran, yn enwedig oherwydd mae'n ddewis ar AI, bod yna rhywfaint o ddwyloedd gwirioneddol. Nid oes AI yn y sgwrs hwn ati, hyd yn oed er bod hynny'n rhywbeth rydyn ni'n ei wneud yn y Academi Ffilm. Mae'r pethau hyn yn ddewis yn ddewis yn ddewis. Mae pobl yn dweud, dyna oedd yn ôl ym mis 2022. Felly mae'r holl hyn yn ddweud, mae'r hyn rwy'n mynd i siarad amdano, yn eithaf ddewis, ond rwy'n gobeithio, mae'n dal i fod yn wir a'n ddefnyddiol. Mae'n ymwneud â'r hyn sy'n digwydd pan fyddwch yn rhoi'r dewis i'r cyflwyniad. Byddaf yn ceisio rhoi ychydig o dewis i chi yn y sgwrs hwn, p'un a ydynt yn ddewis neu ddim. Iawn, felly gadewch i ni weld sut mae'n mynd. Felly rwy'n arfer, yn ffodus, yn yddwch yn gweld. Iawn, felly gadewch i ni weld sut mae'n mynd. Felly rwy'n nawr yn ddigon ddigon yn y Academi Film Baden-Wurttemberg yn y Sefydliad Gain, rwy'n fy modd yn ddiddorol iawn o fod yn ysgol hwnnw'n gwych, ac mae hynny'n debyg i'r rhan oherwydd bod hyn yn mynd i fod yn bwysig. Rwy'n hoffi gwneud rhywbeth sy'n ymwneud â chi heddiw i geisio rhoi un cynnydd i'w gael. Ac oherwydd rwy'n Rheolwr Sefydliad Gain, rwy'n hefyd yn Cyngor Cyfrifol FMX, the conference in Stuttgart, which is happening in May. It's Europe's premier animation, visual effects and digital media conference to which you are all invited. So consider that. So yes, I'm known for Jerry's Game, which won the break to develop features and came up with Ratatouille, the story of a rat who wants to become a chef, which is a concept that pitches itself. If you ever get an idea like that that's understandable in a sentence, that is gold. Hold on to it with your dear life. Actually, it was Steve Jobs who greenlit that production. Much later on at Google, at Google Spotlight Stories, we began with another rodent called Pepe the Mouse in a story called Windy Day, which I directed. And the goal was to make immersive stories and the immersive experience to be a story. The concept was to give the camera to the audience. We began with phones and then Zuckerberg bought Oculus for two billion or something like that and everyone was doing VR. So we transitioned to virtual reality but it was essentially putting the audience in the world of the story. And then if you're immersed in the story, what does that mean? y cyflwyniad yn y byd y stori. Ac yna, os ydych chi'n ymrwyso yn y stori, beth mae hynny'n ei olygu? Ac fe wnaethom tua 16 cynhyrchiadau gyda tîm gwych, tîm gorfforol o technolegwyr a artistau gwych. Roedden ni'n gweithio gyda rheolwyr gwerthfawr o tua 2013 i 2019. 16 o'r ffyrdd. Nid oeddwn i'n cael y nomineiddio oscar, roedd hyn, that was Patrick Osborne, but the work won many Emmys and Annies, Peabody's got an Oscar nomination. Only two or three of those shows were to be ashamed of. We were quite proud of the work we did. All of them were experimental. And we had the first VR Google Doodle, found a bigger audience than the rest of Google VR. And the miracle was that we were doing content in a tech company. We were tricking Google into making short films. And it was very successful. And we made all these things in multiple formats, and that was what was exciting, that we simultaneously released each title in 3-degree of freedom VRDVR, the magic window of a phone, stereo, mono, 360 video, all the different formats you could think of. And each one of those was slightly different. And by doing all those different formats, we learned a lot about what was necessary to tell a story. was necessary to tell a story. I used to like to show this a lot, this image of a classical theatre, to signify the ancient history of storytelling and also that this theatre in the round is kind of the inverse of the experience of virtual reality. This is the audience surrounds the action. In VR, it's an audience of one being surrounded by the action. It's a flip of that geometry. And because VR and other formats that are immersive give us the opportunity to do something interactive with our machinery, there's the possibility of doing something subtle with the audience. A couple of years ago my old Google colleague Rashid Al-Gharab and I founded the special interest group on interactive storytelling which was under the aegis of Jean-Michel Blottier's real-time society which is sadly no more but we're still continuing with this and we're an invitation only group of professionalsodiadol, XR, gêm, theatr live, unrhyw fformat sy'n gysylltiedig â'r cyflwyniad, ac rydym yn chwilio am gyfathrebu rhwng y rhain a chyfathrebu am beth mae'n ei olygu i wneud storïau gysylltiedig. Mae rhai ffyrdd o'r diwydiant yn y diwydiant, ac rydyn ni'n siarad am lawer o bethau gwahanol. And we talk about many different things. Real-time, interactive, dynamic narratives, immersive stories, reactive narratives, transmedia storytelling, all these words. And nonlinear stories like Steven Soderbergh's Mosaic, which is up there. Sam Barlow's Her. The Bandersnatch, Charlie Brooker, Netflix. I don't know if you're familiar with that, digital characters and virtual beings, immersive films like Spotlight Stories or Felix and Paul, all those VR companies that do interesting work, and narrative games, Last of Us, Assassin's Creed, Dead Red Redemption, Walking Dead, lots of death. And so any format, that's what we're interested in, and we're trying to define and understand each other. And language is the problem. So what is an interactive storytelling? It's a form of narrative which the story experience is affected by the audience, right? And really, it's the audience may participate in the story's progression somehow, somehow, explicitly or implicitly, and that's an interesting category. And it influences your perception, your perception of the narrative. Not necessarily the narrative, but your perception of it. Sometimes you get a unique story experience. That's a very vague definition that seems to put our arms around everything and then things get more specific. It's hard to do interactivity well, and the harder it gets, it gets more hard the more open-ended it is. That's a linear story. If you're adding complexity of branching narratives, it gets harder. If you're adding a whole web of different situations, it's more difficult. It goes all the way up to in complexity, people are now having to deal with looking for AI to help us with structures that are essentially simulations of the world that have been going on since The Sims and beyond, that are creating world building, but with the intention of narrative coming out of it. The intention that there's somewhere in this world that we've built, there's a story. And that's hellishly complicated. And people like Neil Stevenson, the creator of the word metaverse, are turning to AI in the hopes that it will help us deal with that complexity, and that's what I want to talk about today, the complexity. I want to talk about it down here in the simple part and how complex it already is down in the foothills. So there's more content to author as things get more immersive and there are more possibilities for the audience to explore and learn, and it gets more and more complex. But it's worth remembering that stories have always been interactive. Always. This is one of Rashid's favourite images of a traditional storyteller on the streets of Marrakesh, his hometown, nuancing the story for that particular audience at that particular time. Oops, I like this one, the famous 1947 picture of an African Bushman storyteller. It just encapsulates what storytelling is, right? This is real-time, interactive, immersive storytelling. The narrative is clearly alive in the minds of the audience and that he has their rapt attention everyone's in the story you can see it and he's doing something we cannot yet replicate with technology we cannot do this yet right nuancing the telling of the story to that particular audience at that particular time responding to the flow of their attention to make the story as vivid as possible only live theatre and maybe reading a book, can do this as well. In our interactive storytelling thing, we're always talking about words. What is a story? So I'm going to just, for the sake of this conversation, define it for you, even though I may be wrong. Okay? Because technology challenges our definitions, doesn't it? So as new ways of doing things come up, we say, oh, maybe this is this other thing than that. And the question of what is a story keeps coming up. What really is a story? And is that a stable idea even? Some of our members, for instance, Stephanie Riggs published a book recently, The End of Storytelling, with a provocative title like that. Not a title I like, but it's an interesting provocation. So I'm going to define this for you with a Pixar way, the view from Pixar. Pixar's success was built not on technology, not on the new medium of computer animation at all. Although that was crucial and fundamental to how it worked, the mantra inside and outside the company was story, story, story. That was the most important thing. Half the work on any given production was in story, not in production, not in technology, not in the look and feel and animation and everything else. Story. This was the message. This was because one day John Lasseter went to LA, got religion and came back and we all went down to see Bob McKee who had a famous screenwriting class which he had the chutzpah to call story and it's about story structure and how to write. Three days of scripted ranting, he's a wonderful actor and he stands up and talks for three days. If you've ever tried that, I haven't, it's very, very hard to do. And so he starts with the message like the music man, we got trouble right here, right here in Ravicity, everything's going bad, you know. We're getting decadent. When the storytelling goes bad in society, the result is decadence. This is the message, what are we gonna do? This is actually a quote from this guy. I said I'd go way back in history. This is Aristotle. There are a few more dead white males involved later. And Aristotle, in hisynhyol am drama, Poetics, a gwybod bodwch chi'n ei ddarllen, roedd yn theoriadu yn analytig am sut mae drama yn gweithio. Roedd ganddo list o bryderon. Y peth mwyaf bwysig, plot a chymeriad. Y stori a'r chymeriadau. of priorities. The most important thing, plot and character, the story and the characters, and then the thought, the concepts and the language, the way the language is expressed, music and what he called spectacle, which today we call visual effects. It's at the bottom of his list unfortunately for visual effects artists. It's the thing that cannot save a bad story, but it's something we'd love to see to create a great experience. So, I'm actually, who am I to but it's something we'd love to see to create a great experience. So I'm actually, who am I to take issue with Aristotle? I'm not sure he's right about music. I'd put it higher up on the list myself, because I think it's the most direct emotional impact that we can deal with. But anyway, that's Aristotle's theory. But the main things, the two top things, are plot and character, right? And that is what essentially a story is. Characters doing something. Things happen with characters. And these are two sides of the same coin. The characters are inextricably intertwined with the story somehow. And the story is about the characters. The characters are there to act it out. The characters, as it were, instance the story somehow, and the story is about the characters. The characters are there to act it out. The characters, as it were, instance the story. A story without characters is an abstraction. You can have abstract stories, but without some concept of a thing that is experiencing something. It's a more interesting question. And when you're pitching a story in Hollywood, you don't pitch plot. You don't say what happens. You're pitching the big idea. A rat wants to become a chef. That sums it all up. We're talking about drama. Drama is the Greek word for action. That's all it is. Things happen. When the cameras are rolling on a set, the director shouts, action! That's the magic word. Action. You want to see something happening. There's some change. So drama is action and the main thing is in this theory of story, actions have consequences. Actions have consequences which is meaningful because we have an idea of something called free choice, free will. And the concept of the feeling of free will is very, very important to why we care about stories at all. Characters make choices. And dramatic choices, dramatic choices are those choices that affect the course of the story. They change what happens one way or another. As opposed to what you might call retail choices where you know nothing really changes in the story it's like which which hat did you wear today or what what did you have for breakfast if it didn't isn't important for the story it's not a significant choice not a dramatic choice and and it all adds up in the end to some sort of crisis, some sort of big decision, a big choice, right? And that reveals who you are in a traditional Hollywood story especially. So for example, in Eddie Izzard's famous Dressed to Kill thing in 1998, he's now Lucy Izzard I believe, if I'm not mistaken, the choice was cake or death so your choice cake or death which would you like if you're into comedy you choose cake like Eddie is it if you're into tragedy and real serious drama and memory you go for death like Socrates who chose to to die rather than to give up his right to think freely, which is perhaps why we remember him, or at least according to Plato, that's what he did, right? So tragedy, comedy, cake or death. It's fabulous. So these are dramatic choices. And all of this is relevant to immersive, interactive storytelling. And I actually have a couple of traditional examples which of movies you've never seen so who's seen in the heat of the night with Sidney Poitier one two okay I'm gonna skip over this example because it really won't make any sense to anybody but it's full of really great dramatic choices and beautiful acting and I recommend it to you. I'm just going to actually show you, this is real quick. In the Heat of the Night, 1967, going way back. Recommend it to you one day, do see it. But cutting right to the modern era, let's say, let's go to another movie that some of you may have seen. Mae'n ddod yn dda i'r era modern, gadewch i ni ddod i ffilm arall y gall rhai ohonoch chi ei weld, yw'r un hwn, Avengers Endgame, yn 2019, ymlaen yn 2019. Ac mae'r lle, ar ôl golli'r gwaith o'r holl gwrs gy with Thanos, Benedict Cumberbatch as Doctor Strange signals to Iron Man that this is the one and only, one in a gazillion chance that he has, that he's foreseen in all the branching possibilities of Iron Man's future. This is the one, one time. And the revelation hits Iron Man like anodd iawn. Nid oedd y stegau yn uchel. Rydyn ni'n gwybod hyn, nid oherwydd bod Robert Downey Jr yn actor gwych, ac mae'n actor dda iawn, ond oherwydd popeth sydd wedi digwydd i'r pwynt yng nghyfweliad, rydyn ni'n gwybod bod hwn yn lle oedd y climax, y crisi o'r stori. Ac mae Iron Man yn cael dewis. Dyma'r crisi. Gwerthu'r amgueddfa? Ie neu na? Yn wir, gwerthu hanner y amgueddfa. And Iron Man has a choice. This is the crisis. Save the universe? Yes or no? Actually save half the universe. That makes it a little less obvious. Or does it? I don't know. It's half the universe, come on. It's not a choice we can make for him. It would be really weird if this was a choose your own adventure interactive show. Wouldn't it? We know what Iron Man is going to choose, and we know he's gonna make the ultimate sacrifice in this story, because anything else would be unacceptable. Everything else has run up to this point, and if he didn't, actually no, I'm not gonna say, go what, I wasted my time, I want my money back. And it's not a choice we want to give to the audience dramatically. And even though there's no doubt what he's going to choose there is we need to make the scene satisfying and it's up to the screenwriters to figure out a way of how it's going to happen in a surprising way so we know he's going to do it we don't know quite how and the ending feels both inevitable and surprising or at least entertaining and because this is the dyn hon, Iron Man, rydym yn gwybod ei fod yn y prifysgol o'r stori. Mae ganddo'r dewis ar y crisi o'r stori. Dyma ei dewis. Mae hynny'n deillio'r prifysgol. Felly dyma'r ddweud ymlaen. Ystodion yw stwctriau sy'n cynnig ymwneud â phethau. Ac efallai hyd yn fwy, impose meaning on events and maybe even bolder stories are structures that give meaning to our lives I put it to you this is this is possibly true that we are in fact that we are the stories that we remember his his beginning of one of my favorite books from Charles Dickens, David Copperfield. At the beginning he writes, whether I shall turn out to be the hero of my own life or whether that station will be held by anybody else, these pages must show. It's a beautiful beginning to a story, isn't it? It's like, this is about me, the hero of this, or am I? You read on and you decide whether I am the protagonist of this story. So we see our lives as stories. We are constantly repeating to ourselves what just happened on different time scales in our memory as we try to make sense of what's going on. Don't take my word for it. Ask a scientist. This is Daniel Kahneman, who actually just died this year. He's a Nobel winning behavioural economist, one of the founders of behavioural economics. This is him speaking at Google. In 2011 he wrote a famous book called Thinking Fast and Slow about the irrationalities of the rational choices that we make and why...am y ddiffygion irasgolol y gwirioneddol rydyn ni'n eu gwneud.....a pam. Ac yn y llyfr hwn, mae yna ddau pethau ddiddorol.....rwy'n hoffi eu nodi, ac rwy'n ei argymell i chi. Ychydig o'r bennod mwyaf ddiddorol, mae'n siarad am ddwy o ni. Ddwy gynnyrch. Ac mae'n disgrifio brofiad â phatient.....gymryd ymddygiad meddwl. There are two selves. And he describes an experiment with patients undergoing a medical procedure. Actually, I think it was a colonoscopy. It was a painful procedure without anaesthetic. And they had a way of reporting in real time how painful it was, the pain and discomfort, some gizmo. And then after it was over, they were asked what it was like in their memory. So real-time reporting of experience and then the memory of the experience. And surprisingly, the memory of the procedure had absolutely nothing, no correlation to do with the duration or the overall pain level of the actual experience as they reported it in real time. neu'r lefel o ofn cyffredinol o'r profiad yma, fel y dywedodd nhw'n ymwneud â'r hyn sydd wedi'i ddweud. Yr unig beth a ddywedodd y cofnod, y cofnod ymwneud â'r profiad, oedd y gofnod yn ei gwael, y pwyll, ac yn y diwedd. Ac mae'r cyfrif o'r ddwy bethau yn cyflawni â'u profiad o gofnod yn ymwneud â'r gweithdariad. Mae hynny'n syniad ddiddorol. Roedd hyn yn cael ei enw'r Rol Gwaith Ddiweddol. Yr emosiwn mwyaf a sut mae'r profiad yn diwedd, yw'n gwirioneddol ein algorydd o gyfresu am bywyd. Dyma sut rydyn ni'n casglu ein profiadau a'u troi i mewn i'r cofnod. Yn un fath o hyn. and turn them into memory in one aspect of it. And it suggests that, in fact, we are story creatures. We compress our experiences into these mini stories. I always wonder why we read obituaries. I do. You know the old joke, I get up in the morning, I read the obituaries. If I'm not there, I have breakfast. But why are we interested in how people die? Why, really? It's the end of their story, and it somehow has meaning. Because we think of people's lives as stories. So, two selves. So there's the experiencing self and the remembering self. And Kahneman says, "'Memories are all we get to keep "'from our experiences of living and the only perspective we can adopt as we think about our lives is that of the remembering self and I put it to you that hardwired as we are like this stories well constructed stories are pre digested memories that work that are very good for us easy for us to metabolize as as memories that work, that are very good for us, easy for us to metabolize as memories that we're well made to remember. The other thing that's very interesting in the book is in chapter one he introduces the characters of the story he calls them, terms used in psychology for several decades, two systems, system one and system two as the way we do our thinking. System two has beliefs and makes choices, rational, deliberate, and believes it understands system 2 fel y ffordd rydyn ni'n gwneud ein meddylch. System 2 mae credu a gwneud dewisau, rhyfeddol, ddelybredig, ac mae'n credu ei bod yn deall beth mae'n ei wneud. Nid oes stwctriau fel hyn yn y bren. Nid yw hyn yn beth gwirioneddol. Mae'n ffordd o feddwl am sut rydym yn meddwl. Ac mae'n cymryd llawer o effeithio i wneud y math hwn o feddwl. System 1 yw ddiddordeb, emosiynol. Mae'n delio â pethau fel detegu a yw un adborth yn barod neu'n agos i ni, neu pan fyddwn yn clywed swn, rydym yn ei oriannu ati. Y pethau cychwyn, y pethau gyflym yr ydym yn eu gwneud. A chyfathrebu llyfrau yn y sôn neu pan gallwn ni ddrys car ar ddwy ffordd ac rydym yn ffaze allan, rydym yn defnyddio ein system un. Sut y gallwn ni ddeall gweithgareddau syml, y math hwnnw o beth. and we sort of phase out. We're using our system one, how we understand simple sentences, that kind of thing. And Kahneman apologizes for using these as characters because, as I said, there's no such, he doesn't want to commit the homunculus fallacy, there is no such structure in the brain, but they're functional ways of thinking. And he says people are good at thinking about agents, that is characters, we're good at that, thinking about characters. It's a good way to remember these ideas. So here, thinking fast and slow. And the interaction of these two systems is how we deal with thinking and feeling about the world, and the only important thing to remember from this talk is that it's system one that generates associatively coherent interpretations of experience. Basically, everything that's happening to you, you're trying to make sense of it all the time and make it coherent, unambiguous, understandable, and that's where you make most of the mistakes in life, according to Kahneman. Okay. So, story is structure. Every screenwriter is told about structure. Traditional three-act structure. This is Rebecca Stockley's improv class. Once upon a time there was something and every day something until one day something, inciting incident happens at the end of the first act. Something happens and then the story begins, having set up everything else, the characters. And because of that something happens and because characters. And because of that, something happens, and because of that, and because of that, until finally, this is a classic structure. Maybe it's like this, but it's one, two, three, three-act structure, sequential linear storytelling of the old tradition kind. We can represent it like this, one, two, three. If we open up the second act of the story, we have scenes, scenes or sequences, which we all understand like old-fashioned going back to DVDs, the chapters in a DVD, that kind of thing, that's those. And if we open one of those up, these are the shots, the events that are happening on the screen that constitute a scene or a sequence. And it's at that level that we experience the flow of a movie, a traditional movie, because life is one damn thing after another, as someone famously said. We experience life sequentially. That's how we experience. But that's not really interesting for our purposes. It's not the way the story is constructed. That's not the thing that's going on in our heads. And as you only have to ask Wim Wenders, the famous practitioner of anti-structure cinema, or the old communist Jean-Luc Godard, that every story should have a beginning, a middle, and an end, but not necessarily in that order. Or Chris Nolan, Memento 2000, famous example of non-linear cinematic storytelling, which I actually haven't seen, by the way, so don't ask me about that. You can mess with a straightforward structure quite a bit, as long as the audience can understand the sequence of events and the story eventually becomes coherent to the coherence seeking system one in your brain. So imagine this is the sequence of scenes that's showing on the screen and the linear time flow. Let's imagine we have what kind of connections need to be made in our system one brain just in abstract. I'm trying to draw you an abstract picture. Tell me if you have no idea what I'm talking about. So we're trying to make a coherent sense of this stuff. So maybe this scene over here sets up this scene. So once we see that, we go, oh, that's a setup for this. Setup payoff. And maybe this scene is a callback to that scene where it's a repeat of that in a way that makes sense in the story. Okay? Something like that. And maybe there's a subplot that's introduced and resolved. Okay? Maybe the story world is revealed to be a dream within a dream or something like that. Or other questions are resolved right at the end, and the story has meaning, and so on and so on and so on. These various meanings of things, that is, these coloured lines represent the connections, links, the set-ups, pay-offs, the tensions and resolutions, all of which play out in our coherence-seeking system-one brain as we witness the simple linear sequence of events of the flow of time. And they all need to add up to something that gives it meaning at the end. And this represents the coloured stuff, represents the programming intended to be experienced by your system one brain of the screenwriter putting it together. That's where the screenwriting happens. Example of this thing you haven't seen, Pearl by Patrick Osborne, one of our Google Spotlight stories in 2016. It was a hit at the Tribeca Film Festival, won Emmy, Annie's Oscar nomination. And it's a story about a musician who lives out of a car with his young daughter, travelling across the United States, and here's the moment in the story when he decides to give up his dream of being a traveling musician because he has a little daughter and she needs to go to school and she has to have a decent life. So he settles down, lives a poor life, and this is much later in the story, years later, when she picks up his guitar and she's rebellious and angry, and she decides to go on the road as a musician following her father's dream. Mae hi'n cymryd ei gitar, mae hi'n rhyfeddol ac yn anghyffredin, ac mae hi'n penderfynu mynd ar y rôl fel cerddoriaeth yn dilyn fathur ei fathur. Ac yn olaf, maen nhw'n cyd-drethio ar y diwedd. Ac dyma sut roedd Patrick wedi gwneud y stori yn ystod y byd cyffredin yn Google, yn ei ysgrifennu fel dyn fad ar y ddrwg. Ac mae'n stori ymrwymiol, on'n gweithredu rhwng ddwy faterion. Roeddwn i wedi gwneud y sgwrs hwnnw, mae'r tri act ar y tu ôl. Mae'r llinellau yma'n gweithredu sy'n ymwneud â sut y dylem fod yn teimlo ar wahanol rhanau o'r stori. Mae'n dweud nad oes angen i chi ddeall yr amser y ddrama ei fod yn ei wneud. Ond roedd y syniad hwn yn cael ei ddod yn ddodd. Roedd y sgrinfroedd yn ymwneud â gwneud y pethau hynny yn mynd i fyny a'u droi gyda chynnydd. Felly, eto, mae'r sgain yr ydym yn eu gweld, y stwff sy'n chwarae yn ein chyllid, dyma'r hyn y mae'r ysgrifennydd yn meddwl amdano. neu i ddweud yn wahanol, mae'r stwff ar y sgrin yn y sgwrn a'r stori is the light show and the story is happening in our heads. Okay? Clear enough. This is what people like Norman Holland call psychodynamics. And we shouldn't confuse those two things. Famous quote from Alfred Hitchcock. Film creation is based on the exact science of audience reactions. That's what he's talking about. There's another film reference that I'm going to ignore here. It's The Great Escape. For another time. So let's go to immersive stories quickly. So if you're immersive, you're inside the story world, right, there's the possibility of interactivity. It's real time. It's happening. Why are you inside it? What is your agency? What can you do in the world of a story? Are there choices you can make? So why can't we go nonlinear? But interactivity isn't necessarily nonlinear storytelling. All of our Google Spotlight stories were linear stories, actually, with some interactivity isn't necessarily nonlinear storytelling. All of our Google Spotlight stories were linear stories, actually, with some interactivity. I'll tell you about that in a second. And in these stories, mostly we're using interactivity, passive interactivity, to tell essentially linear stories with many branches for managing your attention in the immersive world. It's a horror when you're a filmmaker and you have control of the screen and suddenly you give that control up and you say you put the audience inside the story world and you have to tell them where to look. How do you do that? That is the chief problem and managing your attention with interactivity is one of the goals. Here's an example of one of our most interactive stories. It's a very interactive story, very, very simple. It's called Piggy, directed by myself and my old friend Mark Oftedahl. And I'm going to show you a bit. It's very, very simple but very, very complex to give you an idea of what this stuff does. And this is the closest we got to the audience interacting directly with a digital character, altering the linear story with some variations. This is about a self-conscious pig who goes jogging. And he wants a cake. But there's only three things. There's you, a pig, and a cake. There's nothing else. And as long as you're looking at the pig or the cake, he can't have the cake because he's embarrassed. That's the drama. So he won't have the cake while you're looking at him. I'll just show you the beginning. This is a recording of an immersive interactive story. Schubert. And there's lots of things like eye tracking, so wherever you are in the Sixth Degree of Freedom VR, he looks actually at you, so you feel that you're actually interacting with him. He notices you. He sees the cake. He wants the cake. Strangely, he goes back at the cake again. Nothing to see here. So his task is to distract you so that he can have the cake while you're not looking. But you're looking. So nothing to see. He's going to pretend. And it goes on and on and on like this. It's a series of frustration, gags. The idea was, could we tell a comedy with giving the audience the control of timing? I'll interrupt that, and here's somebody watching the show with their VR headset and same show you've just seen. Busted. Busted. Okay. So there's an interaction there. This was written like this. There are, as I said, three different things. If you're're following the pig if you're looking at the cake or these Orange lines if you're not looking at anything if you're looking to space, that's the only other option, right? And depending on those things different things happen. That's how we designed the story and it was instantiated created like this The tech was look at triggers places in the in the world connected to the character or to the world, where if you look, different branches are taken in the graph of the story. It's a finite state machine, a directed 8-cyclic graph. Game designers love this stuff. It has a 3-act structure, actually. It is not a choose-your-own-adventure. The hero of this story is the pig, right? Just flavors of the same story. You can do what you like, but Piggy is a consistent, coherent character. We know what he wants. He doesn't get it, by the way. And your choices do have an impact on him, whether or not he gets the cake at any given time. And it's a very simple idea idea but just for this simple story with only this is the state graph of all the different possibilities it's ridiculously complicated and took us months to generate right more interestingly and I'm gonna run out of time so I better rush choose your own adventure choose your own adventure that is when you are deciding how the story goes, actively, intentionally, when the audience makes conscious choices. As Rashid likes to say, be careful, you might not get the right one. Things can get complicated. Just ask Steven Soderbergh, who, with his mosaic show with Saren Stone, I once met him in a green room in Future of Storytelling in New York. He was just complaining about the enormous amount of stuff you have to generate to make a truly interactive story. And Sam Barlow with her story from 2015. These are good titles. I want to talk about Charlie Brooker's Netflix episode Bandersnatch from 2018. Has anyone seen this? Okay. So some of you understand. This is basically binary choices at different parts of a story that you see on the screen on Netflix and they figured out you could use the remote to say, just like we did with Mae'r dewisau binariol yn gwahanol rhan o straeon y gwelwch chi ar y sgrin ar Netflix. Fe wnes i ddod â'r ffwrdd i ddweud, fel y gwnaethom gyda Iron Man yn gynharach, mae'r rhain yn ystod y stori. Mae'r stori yn sef ystod y Brifysgol y 1980au, lle mae Stefan yn ysgrifennu gêm fideol o ran llyfr ysgrifennu eich hun, a ddod yn ddifrifol i'r ysgrifennwyr. Mae'r cynulleidfa'n gwneud dewisiadau gyda ffwrdd TV. an adventure book, which drove the writer insane. And the audience makes choices with the TV remote. Some of them are just retail choices like what he has for breakfast. But others are real dramatic life and death choices that actually change things. You could kill him or not. And to write this stuff, you have to deal with a lot of complexity. He has five different endings. And you get to start again, as you must, in all nonlinear stories, because why do it just once? The whole point is you can go in different directions. And the actual story graph of this actually looks like this. These are all the various choices you can make while watching this on the screen. And I want to give you, before we run out of time, an idea of how complex this really gets. And I'm sorry I'm going to rush through this. Imagine this is your story graph. This is not a story graph of any specific story, just a complex graph. Okay? This is there are no loops in this. It's acyclic, which makes it much simpler. And each node in this graph is a piece of action, right? It's a piece of story. Now, if you experience this, you can go one of these directions. Let's say these are three examples. There's a red way to go through, a blue way to go through, a green way to go through. And notice that each one of these is a linear story experience because life is one damn thing after another. That's how we experience life. So whichever way you go through this nonlinear story, it'll always be a linear experience. And the idea is to construct this whole complexity in such a way that each possibility is a story and ideally a good one. And just how difficult is that to do? Right? Let's try not to give the audience bad choices. You never choose just one path, right? So how many possibilities are there in this graph? If you put a one at the end, the three possible endings of this, so there's the beginning over here, and there's the endings over there, and you walk backwards, and you add up to how many does this one point at? You just add the numbers up in the ones that it's pointing at. This is one method of just doing the numbers. And you go backwards and backwards like this. So here, for instance, this node is going to... There's one there, two there, put three there, and so on. You walk all the way back to the beginning, and this gives you the result that there are 75 different ways to go through this relatively simple graph. So there are 75 stories in this construction. This is a simple one with three different endings. And remember, the nodes in this graph are fixed pieces of action. So in these 75 stories, for instance, the blue story and the green story, that has a shared node. You go through that several. In either of those stories, you go through the same piece of action. So what does that mean for what happens in that sequence, sequence x that I just noted? Because everything in a real story has context, as we were talking about earlier. sydd wedi'i nodi. Oherwydd mae popeth mewn stori gwirioneddol yn gweithio â'r cyfnod, fel y ydym yn siarad amdano. Mae'r bwysigrwydd o beth sy'n digwydd yn dibynnu ar beth rydych chi wedi'i weld a beth rydych chi'n mynd i'w weld. Felly, a yw'r un peth o weithrediad hwn yn mynd i'w olygu yr un peth yn y ddau stori gwirioneddol? Peidiwch efallai. I ddeall hynny stories, probably not. To understand that better, there's another way of doing these numbers. If you put a 1 at the beginning, and you write the sum of the numbers of each node going forward, so ignore how this works, but this is these are the different ways that you can get into each one of those nodes, okay? So you march through until you get there are 21 ways to get to that ending 27 ways to get that ending in 20 So that adds up to 75 there are 75 ways through this graph And if you add those numbers there that shared no there are nine different ways that you could get there nine right so so each of that's that's Felly, mae'r trefn o'r 75 ffyrdd gwahanol. Ac yn y canol hwn, mae'r peth o'r gweithrediad hwn yn rhaid i'w gweithio mewn nain gwahanol fersiwn o'r stori a gwneud yn dda. Ac yn sylweddol bod hyn yn y llwybr lineaol o ddigwyddiadau, nid yn siarad am y peth hwn. Ydych chi'n cofio hynny? Dyma'r hyn mae'r ddysgu'rwyddoedd yn meddwl wrth geisio seilio pethau ac i wneud pethau'n gweithio. flow of events, not even talking about this thing, do you remember that? This is what the screenwriter is thinking, trying to set things up and make things work. Imagine adding that level of complexity to the linear experience. It gets crazy. It will blow your mind. Right? So I'm going to and it's no surprise then that this story is about Bandersnatch, it Ac nid yw'n amwys i unrhyw un i fod yn ystod ystod y stori. Mae Bandersnatch wedi cyflawni flwyddyn i'w ysgrifennu ac mae'n ystod un o ddynion sy'n trwbwng iawn. Nid ydym yn amwys i Stefan i'w clywed ei siarad â'i therapeutaidd a dweud, Nid wyf yn cael cymryd unrhyw beth. Mae'n dod i'r cofnod ei bod yn cael ei gyfweli gan rywun yn y dyfodol. Ac mae hi'n dweud, rydych chi'n dechrau gwneud llwythyn, And she says, you're starting to disassociate, which is what the shrink would say, because the associative coherent interpretation of life cannot work for this character. And she gives him medication. I'm actually from a family of shrinks, so I jive with this. So there are different ways to go through this story, and Charlie Brooker understands this and which is why he makes the stories and the meta stories about a character who has no moral agency in his life at all actually. He's basically crazy. So Charlie Brooker commits to this contradiction and says this is about a crazy guy. There's even a meta story where the whole thing is just a show. And everything is revealed to be a story within a story. Now, just as the author of the Bandersnatch book in this story went mad, Stefan goes mad in various versions of this story. And we are encouraged to drive poor Stefan mad to get to the end. That's what we're encouraged to do in the way it's written. Into paranoid psychosis to help him finish his job of making the game. To satisfy our curiosity of how the thing ends. And so from the traditional perspective of the storytelling, he's not the protagonist, he's our puppet. And we are making him go crazy, so what does that say about us, who are driving the choices? Dystopian incoherence of an interactive story like Bandersnatch is a natural fit. And I'll just finish with giving you... It's just very difficult to do really well. If you're interested in this category, by the way, Sambalo turned us on to a lecture given by Brian Moriarty at the NYU Game Center back in 2015 called I Sing the Story Electric. It's really great. Game Center back in 2015 called I Sing the Story Electric. It's really great. And he talks about the origins of chooser and adventure stories all the way back to Renaissance England in the theater and points out Kino Automat, the Czech 1967 first machine implementation of a branching narrative in the Czechoslovak pavilion yn y 67 Expo yn Montreal, a wnaeth y cyflwyniad y ddiddordeb o agensi, y graff stori gwirioneddol, yn edrych fel hynny. Ac mae'n siarad am ddewis eich llyfrau addysg eich hun. Un o'r rhai fwyaf yn Raiwela gan Julio Cortázar, ysgrifennydd Argentynol yn 1963, neu Hopscotch yn Saesneg, a gafodd ei gyhoeddi yn 1966. Mae'r Ddemain yn Hupfspil, writer in 1963 or hopscotch in English in it was published in 66 I the German is hoop hoop spiel right your children's game that you jumping up and down and you can read the chapters in almost any order that was the construction I tried to read this book once it's difficult my recommendation to you is another example of craziness this great book book here, which I picked up in the airport a couple years ago, The Never-Ending Tori, The Life of Boris Johnson. It is this choose from 350 million endings by Ian Hollingsworth. It's a satire also leaning into the difficult incoherency of choose your own adventure stories and it's satirizing the Boris Johnson's meaningless political nihilism which resulted in the tragedy of Brexit. And it's hilarious and it really works it's a good choose your own adventure. Stories are structures that give meaning to events. That's what I think they are. And remember, protagonists make dramatic choices. The real choices in the story are the protagonist's choices. And if you are making the choices in a story, if you are making the choices, it's about you. It must be about you. There's a strong statement there to end on. Thank you. I'm done. Thanks a lot. I'm quite impressed. I need a second time to watch it. The good thing is we have an archive, so if you want to see it again or want to tell somebody, yeah, they can watch it again, which helps, I guess. Sorry I had to rush through that too much material, as usual. Yeah. Do we have questions in the audience? Yeah, I have a question. Thank you for the great talk. I was wondering if you think that these, let's say, relatively new ways of storytelling, in the context of the history of storytelling, are going to impact the way we view cinema and view television? And will there be more interactive experiences in theatres? Or are we just, for lack of a better word, lazy? I don't think it's lazy, first of all, and the big answer is yes. I think we will figure out how to deal with the complexity of interactivity in a coherent way, but it is complex. So it's been the dream and intent of very smart people to figure out how to, in a context like this, a large audience with a screen or everyone in a some kind of immersive space to have an interactive experience where you can affect what's happening and of course there's a very big difference between everybody in a room affecting something and you as an individual affecting something but if I'm telling you a story as a narrator around a campfire. I'm looking at you and you're looking at me. Live theatre does this and our dream is to create systems that do this in a useful way that makes sense. Not just to mess with you but to ideally make something with meaning that is relevant to you specifically. And so I I don't know, there are many different things to talk about, but yes, I think that's where it's going. But it's a complex journey. Over there we have a question. Thank you, Jim. Thanks for a great talk. Thank you. I'm wondering what's the prime motivation for you or for these new worlds of interactivity to deal with the observer, listener, the one who watch, in terms of this possibility to decide. Typically when following a story, we can enjoy this possibility to step out of ourselves and to get a chance from someone else to enjoy ourselves in a different way. If taking the advantage or the risk to decide about how the story is going to develop, that's a difference. It's different agency, yes. Is there a primarily business motivation in that to come up with new strategies, how to attract people, or is it more experimental? It's appealing to believe that giving agency choice to audiences makes something better as a story. We want agency in our lives. So it's appealing to think that if I'm allowed to make choices, it can be a stronger experience for me. I'm not sure that's true. Maybe it is, maybe it isn't. My motivation is to whatever the experience that I create for you, and if I do give you choices, I don't want to be responsible for giving putting you in a situation where you can make bad choices that make the experience crappy. The motivation is how can I understand how to make your experience as the audience good in almost all cases? By good, I mean coherent, meaningful. And that's the motivation, to understand these complexities, to say what do I have to do as the creator of the situation, the world you're in, to make sure that you're not left wandering around having what I would call just an experience, not a story, if I'm interested in storytelling. The premise is I'm interested in storytelling. Somehow the experience you have should be a story, and then we can debate what a story really is. But as far as I understand a story, if I want you to experience a story, there are certain things I have to provide for you that make it more likely, at least, that you will have a story experience in an immersive, interactive context. And that already opens up a huge array of really interesting problems, and we're just beginning to address some of those. Except for the fact you don't own me. Of course I don't own you. You're just the audience. But any creator of any book or film or play or any experience that is a story is out there to communicate something, I think. Right? Why am I making anything? Over there is another question. Yes, there's a question from our YouTube livestream. They are saying, what an interesting excursion into storytelling. What do you think about interactive storytelling games like Detroit Become Human, where the user can directly influence the course and the ending of the story? And do you think this is the way to go regarding immersive VR stories? Thank you very much for that question. I'm afraid I'm ignorant of the specific example. So I don't know the particular example given. Again, if you're going to choose the ending, the question is, does that ending that you choose make sense of what has happened before? And I, by all means, have choices. But if the things you choose to do don't create a whole coherent experience, then I don't think you're necessarily experiencing a story. So this is a very interesting dilemma or interesting problem to make that more likely at least. You're in the realm of probabilities. Let's help stories happen. Thank you. So, yeah, I would say if there is no further question, is there one more? Ah, there's one, there's one, so... Could you hand it over? Thanks a lot. I would say last question maybe, depending on the time. Yeah, thanks for the talk. I was just wondering why you talked about the illusion of choice when you can influence a story or the development of a character. To give you actual choice as you have in the real world, that's an enormous complexity. That's that graph that I showed when things get modular and you're world building and you're essentially simulating worlds. If you really are a free agent in a simulated world, you really are a free agent in a simulated world, the degree of choice you have must correspond to how rich in choices the world is, right? How complete a simulation of whatever that world is, is it? Do you really, if you open this door, is there something behind it? Or is it just a facade? So most experiences that are constructed by human beings even with machines that give you choices give you the illusion of choice because we can't support true freedom in simulated worlds. It's too complex. complex. Thanks. Thank you. So maybe one more question if you're willing to. Sure. And then we end this session. Yeah, when it comes to actually designing an interactive story, how can you wager the choices so that one choice doesn't detract from the overall experience? Do you have any tips for that? It's an excellent question and it again gets to the heart of the matter. You have to assume that one choice is better than another in terms of your whole experience. When I say better, I mean does the whole experience move you, mean something, is dramatically strong. And you... In most stories that I know that work mostly what happens is there are various choices of how to essentially get to the same place that their variations depending on you and your and your and your and your style and your interests but eventually taking you on a journey that will end up in the place that was intended as a meaning by the author or authors. And so it's about finding something that appeals to you. And when you're doing passive interaction, when you're dealing with an immersive story with VR and you're firing things depending on where you're looking, it's your attention that we follow. It's about listening to you. Like when I'm doing a live show. Where are you looking? What are you interested in? My job is to make what I'm doing very interesting. I'm up here talking to you and And I should be interesting enough for you to follow me if you're not following me I should I'll run over and get in front of you and try and get your interest back So just to interact with you basically. Yeah, I Think we should Thanks Bonnie get on. Thank you very much. Thanks one more time, Jan Piccaro. So we will switch the computers for a second. So I try to give a brief introduction. I really can say just a brief one. If you want the detailed information about our speakers, please have a look at our homepage where the biographies are in much more detail. But I want to give a brief overview. So we get to another way of storytelling which is based upon lots of big data sets and archives and it will be presented by bonnie mitchell who is a media artist and professor at the bowling green state university in digital arts in ohio usa and mitchell's artwork explores spatial and experiential spaces or data sets and ask the question about our physical social and cultural and psychological environment throughout animation interaction and immersion you have been currently director of the acm seagraph history archive you are now right now sorry and you have been the history chair of the sePH 5th edition and you worked out an installation which you will present today to give us an idea about this way how to interact with archives, right? Thank you. Bonnie Mitchell. Hello, first of all, I want to start with the fact that in the past, 1995 in particular, I won an honorable mention at Ars Electronica, but I never went. And I was young, stupid, didn't understand the significance of Ars Electronica, and this year is my first year here. In the early 90s, I was doing net art, worldwide collaborative pieces that were really focusing on my study that was investigating influence, the influence of the visual image and how experiences with visuals coming into your system affect your outcome. So it really doesn't have a whole lot to do with what I'm going to be talking about, but in some ways it does. My interest in influence and how you can use visual imageries and visual experiences to transform somebody's self. So let's go into the past. So I'll be talking about past, present, and future. So the past. Who am I? What have I done? Well, first of all, I started off as an animator, as a craftsman working a lot with fiber arts, and also as a computer science programmer. We are just a position. So how can I take all of these things that I love doing and create experiences that change people's perception, emotionally move them. So rather than talking about story and cognitive effects of the experience, I'm going to be talking about the emotional effects of using animation in immersive experiences. So I began to take my animations and create these spaces by sewing these semi-transparent scrims and creating spaces that you had to walk into. You were completely surrounded by the animation. into, you were completely surrounded by the animation. And I am kind of repurposing or liberating animation from the screen. And one of the things that I became very interested in is engaging the audience through their physical actions of walking through the space, touching things, interacting with things. I began working with a composer and we integrated sound, but it wasn't just to create something that was fun. We really began to investigate what we were doing in regards to art and design and animation and sound principles. So I'm just gonna show little clips of these. And I'm gonna jump right to the part right here. So I created these projection screens. Sorry about the quality, it's very low quality capture in the dark back then but people pick up one of these screens they walk into the space they have to collect the poetry that's falling from the ceiling and so you'll see people interacting with each other trying to make sense of this poetry some some of the poetic text must be collected by three or four or five people together working together Oh, thank you. Here he comes. This is fun. I had no idea he was going to have so much fun here. Oh, here he goes. Squares. Whoa. And there you can see that. And so I began to investigate what was happening to the individuals in these spaces. And I began to think about psychology I've always been interested in psychology I'm not a psychologist but a lot of my work deals with psychologically altering the participant in very deliberate ways and so I am diving right into those principles that I talked about earlier. How does color, how does the direction of a line affect people psychologically? And so this particular piece, I'll give you just a quick context and then I believe I shouldn't spend too much time in the past, we need to get to the future. to the future. But essentially you're walking through the space, there's sensors all along the floor. You're stepping on these various sensors and they are transforming the visuals and the audio. So let's watch. Experiential extremism is an immersive interactive installation that uses time-based visual and sonic events to control emotional and physical responses the project explores abstractly the concept of extremism in today's society focusing on aspects that entice people to gravitate toward thrill-seeking. A participant's physical presence within the space determines the piece's evolution, density, and intensity, thus affecting the viewer emotionally and psychologically. Extremism has gained popularity in various cultures worldwide over the past decade. Extremists and thrill-seekers continuously push the envelope of possibilities in sports and other leisure activities. Experiential extremism explores the concept of extremism by focusing on the emotional responses that occur before, during, and after extreme activities. So there was a 20-hour drive back from a very large installation of this in Miami, Florida. And along that drive, myself and my collaborator, the Eleni Lelios, composer, we started discussing a new project and we wanted to do the exact opposite. Instead of it being extreme, because it really was an extreme experience to create that, and it was an extreme experience for the participants, we wanted to go the exact opposite direction. And so we wanted to immerse somebody in a space where they were confronted with contemplative questions. And so let's play a little bit of this one. And I'll tell you a funny story about this. After the participant sits down, an abstracted human figure emerges. This animated, ghost-like, ethereal image moves slowly toward the participant. The figure represents the self, a friend or loved one, a spiritual guide or other virtual presence. What does life mean to you? to you. There was a problem with this piece and that was we had the seemingly endless trail of individuals, these particle people that would come up and ask you questions. We had like 56 questions and so people didn't perceive a beginning, middle, and end. It seemed endless. So at the opening, the premiere opening, somebody sat in that seat and wouldn't get out for two hours. And we're like, other people want to try it. And he says, but it's not over yet. And so essentially what we did is we really transformed their perception of time. I'll be talking a little bit about that later. And the other thing that I've done is a number of pieces that have explored this concept of the self and the mirroring of the self and how people really enjoy putting themselves into the experience in more than just their their bodily presence but their actual image so let's just play a little snippet of this one and sometimes they would bring their boom boxes you know that's that's what they call them back then and dance in the space you can't hear the music playing, but there's music playing in the background and they're dancing around. I never anticipated this when I made the piece. And it just became a social space for people to gather and see themselves in various ways. So I began to think about this idea of immersing people in abstract spaces because oftentimes they would come out and they're like, I have no idea what just happened. They were constantly looking for meaning. They're trying to create connections with experiences that they had in life. But I wasn't giving those connections. I wasn't giving an avenue to get to those connections because of the pure abstraction. So I began to play with images that bordered on representation. But then I would capture them and bring them back into this abstract kind of reality that was happening. And another thing that I play with a lot, because I am interested in psychology and the effect of these images, is tension and release. So I want you these images is tension and release. So I want you to look for tension and release. I use this a lot in my work. This is just a little snippet. I'm going to get this. So those of you that are working with animation, think about that. Think about the fact that in order for something to sound really loud, you have to juxtapose it with silence. And so juxtaposing the slow periods of your animation with something that's very abrupt, chaotic, is going to make that chaotic element seem more intense. And so I played with that quite a bit. Now, another thing that I've done, which is bringing me to the time tunnel, but there's a couple more things in the past that led me to this. And that is I began transforming spaces in much more three-dimensional ways where you had to walk through the space. and I'm bringing actual objects from the real world into these spaces. Believe me, I'm only showing you a small clip of the number of works that I've done, but I've played with moss and rocks and glass and trees and all kinds of these elements that I bring into the space so that there is a little bit of, oh, I recognize this space. And then I bring them into this abstract sort of space where it's affecting them emotionally instead of on a cognitive level. And I'm dealing with environmental issues, as you can see in this piece And here's another one where what I'm doing is I'm working with data to create the experience of you walking into a pod and being completely submerged in the data. And the concept here, there's always a concept it is never just a pretty picture the concept is that your shadow is obscuring the data meaning that our our we choose to ignore what is happening around us and so people go in there now a a very unexpected thing happened. So what's happening is I've got the data from 1880 all the way up to the present at the point this was shown. And the data would slowly, you know, go up, up, up. This is CO2 emissions. We've got sea level rise, I had a couple more pods dealing with climate change data, and what would happen is when it got all the way up to the present, it would drop down to 1880, and I never planned that that would be the highlight of the experience, is when it actually plunged down back to its 1880 state. So let's just take a tiny peek at that one. I'll jump around. Negative impact on the earth when the visualization reaches its peak in the present day and drops down suddenly to the 1850 to 1880s levels. The piece consists of three animated pods, each analyzing and visually representing data as it changes from the late 1800s to the present. When the viewer enters the pods, their shadow overwhelms the data and obscures it, metaphorically representing their mental scotoma. Okay, finally, let's get to the present. But actually, I have to honestly say it's now the past. But I was put on this panel of art and industry. I'm like, oh, I'm an academic. I'm not really an industry person. But actually, I am working with people very closely in industry through the SIGGRAPH organization. This is an organization that consists of computer scientists, academics, researchers, artists. It's very, very diverse, brings together people at an annual conference in Asia and also in North America once a year, but it also has a number of year-round activities involved in the History Committee and Digital Arts Community Committee, and they focus on the future. But the problem was they were having their 50th conference celebration so they chose me based on this proposal that I would create an immersive experience where people could walk back 50 years and experience the changes that have happened over time. And so it has a very rich past and I had access to a lot of data. So that worked out pretty good. Okay. So now how did I have access to all this data? Well in the early 90s I became concerned that the art shows were not being documented well, so I started making websites, 1994 all the way up to 2006, but they're all different, and, you know, they really didn't have a consistent design and kind of, you know, reflected the size of the graphics and the limitations of the technology at the time. So in 2020 I started a new one. Actually it was more like 2016 I started a new one and it was the digital art archive with one consistent interface and then in 2020 it transformed into hey let's archive everything for SIGGRAPH. Now SIGGRAPH is as large as Ars Electronica and less focused on creative stuff and so you've got a lot of technical materials. We have all the technical papers from industry. We have technical papers from academics. So I'm collecting these papers, I'm creating this archive. I have a software development background. So we're programming all of these interactions between the data. We have got a section with experiences. I do recommend that for any artist or researcher, go check it out. There's a wealth of information. We have artifacts in there, photographs of artifacts. We have got profiles of individuals and all their contributions to the SIGGRAPH community. And because of necessity, I started a physical archive. And this is where I work every day, more so than my actual academic office. I'm there almost every day. And so they chose me to put together this celebration. And my idea was, hey, let's create immersive experience, but also let's kind of dig through all of these artifacts and create an experience that would be informative, experiential, and inspirational. So I'm dealing with this huge convention center and I'm telling you there were probably 50 different mock-ups done in CAD of what this space could be, but this is what it ended up being, is this huge corridor. And now I had thought of it kind of like IKEA where you go up and down aisles and you know maybe a little less linear but that's the space they gave me so we make it work right and and so one of the important things is I wanted visualization artists to be able to look at this past and in different ways and present their work so visualization posters and we had Everardo Reyes visualize the art shows. There's a couple more over here that are taking a look at the art shows over time. And then I wanted to take those artifacts and present them to the public. So I had all these showcases. That's my administrative assistant, Ella Fruzen. She's amazing. So we brought boxes and boxes of CD-ROMs and ribbons and lots of swag. All the posters, having to deal with the posters, get them scanned, digitally retouched them, all the publications. So all of this material was actually just physical books that had to be scanned and designed and then we put on a graphic hardware display and that is probably about one tenth of what you're seeing there. So we have all these head mounted displays and graphics boards from the past. And I wanted to bring back the pioneers, the people that actually developed the software and the algorithms and the ideas for what we take for granted today. And so I had four panels that focused on various aspects of computer graphics history. This was magic to me to bring these people back. And I also had a section called Blast from the Past and we were taking a look at the materials from the past that really stood out, that we thought were kind of transformative. And we had a number of people come back and show the stuff that they were showing in the 90s. Reconstructing old robots. We had playable games over here from the 70s, 80s. I don't think actually 70s, more like 80s, 90s. And we also... Now, I can't take full responsibility for this, but we had a retrospective animation screening that was magical. And so we got permissions from people that were creating animations in the 70s and the 80s. And it was, you know, it was just amazing. A funny thing happened is I was asked to do a podcast with Mark Lavoie from Stanford, who I believe is at Google now, and a guy named Dan Sisson. I'm like, who's Dan Sisson? And so backing up a little bit, everybody said, oh, you're in charge of the 50th. We gotta have Pixar's walking teapots. And I was like, but that's not SIGGRAPH, right? That's Pixar. And I don't have any control. I don't know anybody there that could give us permission to do that. So this Dan Sisson guy in the podcast, as I find out, is the designer of the teapots. And so I said, Dan, would you like to do a display of the Pixar's 20 years of these walking teapot toys? And he said, absolutely. And so that was a highlight also. So now let's get into the art and industry aspect. So essentially, we had this problem. SIGGRAPH is really obsessed with the future. I was told young people don't like history. Okay. So how are you going to make it palatable for the young people? And so I said, well, okay, you want us to really focus on the future as well as the past. Okay. So I am going to create an installation of postcards similar to this postcard collection that was created around the turn of last century, and they asked artists to predict the future, to illustrate what they thought the future would look like in a hundred years. Well, we're talking 50 years of SIGGRAPH, so hey, let's go in 50 years. So the problem was, I put out a call for participation. I really got minimal response. I put out a second call. The results, not good. So I contacted one of my alumni. Well, he was actually a graduate student of mine years ago, and he's working in AI. I said, hey, let's have AI predict the future, right? Why not? And so I worked with him, he created a number of prompts and I, you know, we kind of selected from many, many postcards, but there was actually one submission from a human being that was really quite good and so we included that and it became a game. Pick the one that was created by a human. And people would go through and I saw these two guys standing there and I said, so what do you think? Which one was created by a human? And they immediately pointed to this piece. And I was like, how do you know? Somebody told you, right? And they said, oh no, no. It's the text right here on the word honey. AI would never do it like that. And so I just thought that was incredibly interesting. Now, across from that display, we're going to move across the hall here, you see a sneak peek of the time tunnel. So this is at the entrance of the time tunnel. Perfect. We have a robot. And this robot was designed and built from scratch by my colleague, Yamin Shou. And I asked Yamin, I knew he was a robotics artist, and I said, hey, can you create a robot? You know, and as time was going on, he was like, I should just buy the robot parts. And I went, no, it's SIGGRAPH. You need to design it yourself. And he's so glad that he really worked night and day to create this robot all 3d printed parts and so essentially the idea that I presented to him is I want to connect people from the present to people from the past I want people to walk up to this robot and the robot takes a picture of you, does an AI facial analysis and connects you to another person. Maybe it's based on your smile or the structure of your face, but it connects you to another person and then it gives you their name. And my intention was that you go, wow, it connected me to Ed Catmull. Who the heck is he? I'm going to look him up, right? And that is not what happened at all. So, so essentially people gathered around this robot and. Just take a look at what's going on here. The robot appeared to be very sentient, so people started to play with it. They were kind of freaked out by it. They were, you know, entranced. This was the first art show chair of SIGGRAPH, Darcy Garbarke. And it found herself, a younger herself. And so people were really aiming to try and reconnect with a younger self. And so, let's get to the time tunnel. Now the time tunnel involved a lot of people, way more than I could put in this list. But I proposed the project, invited a number of collaborators to join the team, and I have to tell you there are a couple people that just absolutely stood out. We have one in the room. But Kenny, is, uh, Kang Yong Kim, was just absolutely amazing and saved the day on the technical aspects of this. This is Kenny over here. And Wei Hao is in the room. Raise your hand, Wei Hao. Came to the rescue at the very end to rescue the idea, because the idea was to take a lot of this data from the archive and represent it on this immersive tunnel on all the walls. And yeah, it was a little challenging. And we also have Alexa, she's now at DreamWorks, and she did a lot of the floor projections, and this is Yamin with his robot. And so essentially when you enter into the space, you are immersed with the materials on both the walls. This is all stitched together. Well, it's not really stitched together. It's all running real time. And you're also able to interact with floor projections. So we had to build this space. It was quite a chore. Took a professional company that normally puts on concerts. So this is, you know, I don't know, one ton, two ton, materials for concerts. And they had to have this screen custom-built in Italy so there it's one continuous piece of material. And so it took quite a bit of labor to put this together and to get all of these images to kind of overlap in a way that you never saw the overlap. Let's just watch a little video here. So as you're walking through, you're seeing the crew kind of putting things together. Now, we had some dancers come in the night before it opened, and they kind of danced in the space. This was interactive. We had musicians involved, so you could do these HTV vibe kind of sensors, and it would affect the music. And I want to show a few particular pieces that took from the archive. Now the idea was taking materials from the archive. So over here we have Kenny and Graham who took the images from technical papers and floated them in various ways. These are also interactive. And then I'm going to go to Weyhouse's work. And Weyhouse, hopefully during the questions and answers, if anybody has any questions about this piece, please ask because he came in and he took the references from the technical papers from the archive and traced the lineage of citations in this really beautiful visualization. Now you can see that what was happening is we have this content that is directly taken from the archive, but then it's artfully juxtaposed upon each other, so we have very representational materials and I should pause to say that none of this would have happened if it wasn't for Autodesk. Autodesk came in, we got the largest sponsorship in the history of SIGGRAPH to put this on. So because of their connections, we were able to get really amazing footage from Weta and ILM and a number of industry partners with Autodesk. And so you're going to see a lot of those kind of floating around the space. So we have this kind of combination of art and representational materials. of art and representational materials. And over here, you can see that they're using masks to kind of integrate the material together. So there was software that did that, that was developed by University of California Santa Barbara students, and it was really quite beautiful, that integration. But the problem was that we also had materials like this, which is a timeline of 50 years of computer graphics. It has over 2,000 entries. These are spinning cubes, and each cube can spin an infinite number of times with different materials, has images. That was a massive chore to create. This is all algorithmically created as well. And took 40 pioneers gathering, lots of Zoom meetings to put this together. And so the issue was a number of the pioneers that I had showed earlier were like oh I did that I actually developed that and so they're walking through but then it artwork would obscure it and so they were actually getting quite annoyed by that they wanted a special screening of just the timeline so this was a magical moment where you, on the spur of the moment, we just said, hey, at five o'clock, meet down here, and we'll have a 15-minute, non-interrupted screening of the timeline. Well, the problem was, we got about 200 people that showed up. I thought maybe one or two, five, 20. And the fire marshal was freaking out, too many people in the space, and they didn't go away. They were so immersed in the past. It was really a magical moment, I have to say. So I'm gonna play just a little bit of this. So I mocked this whole thing up in After Effects, but it clearly became a problem in that to change over this data, to do it in a manual way would just not work. So Ryan Laney, who owns a really amazing company that uses AI to sort of do deep fakes for good, replacing faces on people that are recounting things that if they were recognizable might be assassinated. So it's very real-life materials. He jumped in with his technical skills and his artistic skills as a director and rewrote all of the materials in an algorithmic way. And so now I go to the floor projections. It was very important that, to me anyway, that people be able to traverse the space without really knowing that their body was affecting the space and then coming to realize that that happened. And so Kenny and Alexa pretty much focused on the materials that were projected down on the floor with another one I'm going to show in a minute that was a larger team. And so people were walking around, interacting with this floor projection and discovering Easter eggs hidden in here, little bunnies and other icons from the history of computer graphics. of computer graphics and you see a number of people that are playing with each other and dancing and you know if I was to do it again I would make it much much larger I thought it was going to be much larger but when I got there that that was it and then let's go to the next one. Again, I wanted to connect people from the present to the past, and so I created this game where you catch people falling down from the ceiling, just like the first piece I showed, right? And so you're going to recognize these little projection screens, but this time we wanted to track these projection screens and know exactly where they were and what they caught. And so this was definitely an amazing team of people. We had lots of Zoom meetings and a lot of last minute coding, but it really was the highlight. And I got quite a bit of resistance on, you know, nobody's going to like that. They loved it the most because it was interactive. It allowed them to use their body and it was very game-like. And so they had to catch people within a category. Let's catch all the conference chairs. Let's catch all the artists. This sort of thing was happening. So we have trackers above them that are noticing when those circles, those circles were very important. And Yamane actually wrote the code to detect the circles. But then we had to recognize the colors within the circles to determine whether they caught the right category or not. Now this experience created a lot of memories for some people that had been around at this conference for many years. For the younger people, it was very inspirational, motivational. I didn't even talk about a lot of aspects of this. For instance, since I have a minute or two, I will say that, again, I was told that young people don't like history. Now how many young people in here love history? Yeah! I said, no, that's wrong. That is wrong. There's a lot of people that love history and they're young. My students love history, especially the history of something that they're really engaged with, virtual reality, right? The 3D modeling, 3D rendering. And so, essentially, I got this guy who did his dissertation in 1969 come and tell stories. And he always had a gathering of 20 or more people around him, mostly young people, and he was such an engaging storyteller. It was just magic to watch what was happening. And I didn't really plan that. I just thought his presentation, his display of these old artifacts was really interesting. So, hey, you know, let's have him involved. And he came with his family, with his son and his grandson and a bunch of chairs. And they just sat there all week, told stories. It was really magical. So now let's jump to the future. So me as an artist, first of all, where do I go with this? This was 2023. Took me a year to recover, I have to say. It was a lot of work. I was doing this while I was teaching. But I have to say that it was very inspirational. And so I'm still interested in transforming spaces. And this is just a couple of works that I've worked on. So working with the materials in unusual ways, still very interested in that. But I also feel like that timeline is very incomplete. And so we have a web version of that timeline. This is actually not for public consumption, but if you want to check it out, you can. So I really feel as though I can make that timeline much more interactive where you can go into the space even in a web-based version that is very accessible to people. And so this is my goal, is to create more immersive spaces that make people think about the past, the present, and the future, and recontextualize history in a way that it does sort of become a story, it does become memories, inspiration and a pathway to a better future. Thank you. Yeah, so thanks a lot for your presentation. I would have a question. Do you know the archive of the Ars Laetronica? Absolutely. So I guess there are some similarities or things that are interesting if you look at it because it's really from the beginning to now and it's very dense, I would say, from the data structure. Well, there's a few things I didn't tell you. I was probably born hyperactive, but that's beside the point. I do a lot of things and one of the things I do is I work on a project called connecting worldwide archives and Juergen's kind of involved in this as well and we put on a symposium called the well a summit on new media art archiving and so we've been working with Christina Ragnar for many years and we're having a workshop at ZKM in February on connecting archives. So if you're interested in that, we're looking for new ideas using new technologies, new methodologies, and you know conceptual approaches to connecting Ars Electronica's archive to SIGGRAPH's archiveive and I'm also co-director of IZEA's Archive, that's the International Symposium on Electronic Arts. And AIDA, the Archive of Digital Arts is involved, the FILE Festival Archive is involved, Memo Doc is involved, so we've got a lot of small and very large archives of new media art and we all want to play together as a big family. Okay, cool. So do we have some questions in the room? Otherwise I would ask one. One thing you tried to mention is you had this concrete, this very detailed timeline with all the works. Is there a way or is there thought about a way that someone can really get to that things in particular to grab out something out of the archive or was it always kind of how to say rough? Like it looked I don't know how how easy is it to navigate through this archive? This would be the question. And that really gets to my future goals. So, I think there are some aspects of the experience that were incredibly successful, but understanding the timeline of computer graphics in such an experiential way, I didn't think offered the possibility to really dive deep into the material. So I really want to rethink that. And I want people to be able to grab a little nugget and go deeper and deeper and deeper into that. And yeah, it wasn't even interactive at that point. You know, it was algorithmically created, but you couldn't actually touch it. Now the web-based version, you can at least click on it and spin the cubes however you want to get to what you want. But I really want to be able to go deeper and connect you to different archives that might have information related to that. The Ars Electronica archive, the Ada archive, whatever. And I also know that because of the rapid creation of the content of that timeline, there are some biases. We had some people there that put like, oh, we got to put this, version one, version 1.1, version 1.2. And so there's just maybe it's a little heavily biased in certain directions. So I'm gonna be having meetings with the pioneers and we're gonna be going through this material and really sorting out what are the highlights that should be here, what's missing, right? There's a lot of things missing. So if you wanna to get involved with you know helping to create a more let's say a broader view of the history of digital arts please talk to me because I know it was a little bit North America centric on the content that was added we definitely have a lot of amazing European history and Asian history that needs to be added to this archive other questions hello hi thank you for your presentation. It was lovely. Where can we find your exhibition information? And when do you think you'll have this new exhibition? Well, I don't know when the new exhibition of the timeline will be because I'm just in the weeks ago, the organization gave me a Vimeo channel. So now I can start putting, you know, video materials. And I did mention that we had this retrospective exhibition of animations and we got permissions to show these animations on the website in the Vimeo channel and so just in the next month I'm working with an individual from NASA's visualization lab and he just uploaded five of them five or six while I was here so I haven't really had a lot of time to deal with it but we're going to be connecting all those old animations as well to the archive. It's very much an archive in progress. One thing that I didn't mention that it was in the slides is in the year while we were building up to this immersive time tunnel, we added 20,000 new entries into the archive. And I just want to commend my team because I could never have done that without them I I would just like overseeing this but they did the amazing hard work of researching this material scanning it OCR ing it copy and paste a thousand times you know it was very very hard work and so we couldn't have done it without them and Wei Hao's contribution was incredibly valuable. So thank you Wei Hao. Thank you. Another question in the back here. Thank you. And what I was wondering just besides is that you have this kind of exponential branching I I guess, too, right? I mean, this is what happens right now. If you say something like AI art or something like that, then it's really makes it kind of an inflation. Yeah, well, that branching structure is very inevitable. I mean, one idea spawns another idea, spawns another idea, and I think that was very well represented in that branching structure that you saw of Weyhouse work, where he took one technical paper and who cited that paper, okay? And then who cited the papers that were citing that paper, and it just kept branching. And so that was really beautiful, and I hope to have more of that type of work in the future iterations of this. It's challenging. Citations are easy, right? Because they're recorded. We can find that information. But how do I know that you were influenced by my artwork? I don't know that. So we're dealing with the easy stuff, even though it doesn't seem so easy. We're gathering documentation that's already there, but now we want to take that next step within the next year of finishing up gathering all the stuff that we know has been published, it's out there, and start to think about ways that we can incorporate AI and maybe more data analysis of the content. We've got 50,000 abstracts. What do you do with that? And so if you're interested in data visualization, again talk to me because we are very interested in having you get involved. And it is a very large team, international team that does this. So please don't give me the credit. It is really the team. So there we have another question. Thank you for your presentation and for your work. I have a very practical question. Is there already kind of navigation system, an engine, a tagging system in this amazing world of experts and all our high sets of data? Is there a tagging system? Like a navigation, search engine, tagging system, how people can dive into our archive work? Oh, yes. We deal with metadata. That's the hard work is we try and gather as much metadata as possible. All the images have metadata embedded in the image itself. And so if you take an image off of the archive, you're not going to lose the context or who the artist is and all of that. It's still...it's embedded in there. We have relationships. So I'm a software developer as well as an artist. And so I wanted this to be that all the data was interconnected. And so the potential is there for amazing data visualizations. Right now, it's like every day I go in and it's like, okay, we're missing all of this. We need to scan all these books. So it's very practical right now, just let's get the data in there. But the tagging system that we're adding to the materials takes a lot more time, but I know it's going to be worth it. Okay? And right now, we do have different ways through the menu that you want to look at all the artworks that deal with VR. Click on that, you know? But I think we could do way better, for sure. So I would say if there are no further questions, over there. Hello. I have a question which might sound like criticism, but it's really not. It's just out of curiosity. Because you were talking about the postcards you did where you didn't receive any entries or not a lot of entries that had potential and then you used AI for it. Because like in the artist community, a lot of people have a lot of criticism about, like big companies who have a lot of budget using AI for stuff. Did you get like any criticism at the exhibition about using AI for the postcards and not commissioning artists for it? Okay, so I'm going to clear up a misperception. People feel like SIGGRAPH has a lot of money. So I go into this situation thinking, this is fabulous, I'm going to be able to do all this great stuff, and they said, you have no budget. Zero budget. I'm sorry, we forgot to put it into the budget. We did the budget three years ago, and you have no budget. I was like, fundraising is not my expertise, I don't like doing it, you know. But it was through the efforts of a number of team members, really Kenny and Gustavo spent all night every week, well, not every weekend, but it was a real crunch time and, you know, putting together promotional materials and the UCSB students did a walkthrough and my student did a walkthrough. So we had all kinds of stuff to kind of show the idea, but it was Autodesk that came to the rescue. I think it would have just been a bunch of posters on the wall or something if it wasn't for Autodesk because it was challenging. We had no money. And why did we not get submissions? Because it was a new program. Everybody knows you need to submit to the animation program. You need to submit to the emerging technology program. But submitting to the history program, what's that? Right, it happens once every 25 years. Nobody really knew. And even though they did a little promotion, it was just a new venue. And, you know, in retrospect, I could have sent it out to the Ars Electronica list and you guys would have done an amazing job. I sent it out to a few lists, but the responses were not that interesting and so it was an interesting conceptual idea to ask AI to predict the future so I don't mind at all that the twist in my intention you could see that a lot of my intentions sort of changed I shouldn't say it changed but let's just say that I intended it to be one thing And you could see that a lot of my intentions sort of changed. I shouldn't say it changed, but let's just say that I intended it to be one thing, and then it turned out to be something else, but it was beautiful. And I think the postcards really turned out fabulous. It really did show off the aesthetic of AI, because you could see aesthetic similarities, even though we said, okay, do it in the style of a charcoal drawing, do it in the style of, you know, you could still tell it was AI really. And that was okay. We're intended for that, you know, that was okay. So furthermore questions? No, seems we got it. Thanks a lot, Bonnie Mitchell.