Timmy Ghiurau is an innovation leader at Volvo Cars, and Carlos Calva is an AI and augmented reality entrepreneur. Both of them are experts in virtual reality, augmented reality, game engines, and other 3D interactive technologies, with experiences spanning design, engineering, and business. Timmy originates from the worlds music, fashion, and culture; while Carlos has been building projects for clients that include NASA and multiple branches of the US military.

We had this conversation in front of a small live audience at Timmy’s workspace, the Open Innovation Arena at Volvo Cars – a multi-disciplinary platform within the company that makes preparations for a diverse range of futures and great uncertainties.

The themes of our conversation, recorded at the end of 2021, were the hot topics of the year: the metaverse, AI, and NFTs. These trends dominate conversations in tech today. The metaverse is expanding new graphics and interaction technologies to a massive role in our lives. NFTs embody value, identity, and symbolism that is digital-native (and metaverse-ready). AI promises immense convenience and creative possibilities in the digital world.

But how far have we come towards realizing these promises today, and what is on the horizon? How do these technologies serve the needs and desires of human societies – like safety, prosperity, diversity? How are established companies, like Volvo, integrating these trends into their business? In a wide-ranging conversation, we tried to answer these questions with our stories and insights.

Special thanks to David Peterson, who kindly stepped in last minute to help us record the video for this episode.

The transcript below has been edited for clarity, brevity, and readability.

How did the two of you meet? What are you doing together these days?

Timmy Ghiurau: The VR/AR world is still quite small. At all the conferences and exhibitions, it's pretty much the usual suspects.

It was when we did the Varjo presentation with Volvo, where we presented our project... We met at an expo, really briefly. Carlos really wanted to try it out. He told me about the aerospace cases, and asked a few complex questions... Like, OK, so you know a bit more than the average people at the conference... We started to talk.

And then, when we, with Volvo, went to an MIT hackathon... I was super jet-lagged, sitting on a bench. I was looking for coffee. And Carlos sits next to me: "hey, remember me?" And then, we started to talk about AI and avatars...

Carlos was competing at the hackathon. We, from Volvo, were there as judges or mentors. So we tried to see how we can combine haptics, and eye tracking, and haptic gloves, and the suits, and everything, combined to create the ultimate AR/VR experience.

Since then, we started to jam.

What do you think Carlos? Is he telling the truth?

Carlos Calva: When I got there... I was fascinated by the XR-1 from Varjo. It was just an amazing VR headset. It was the first headset that made me believe that pass-through may be an actual way to approach AR, better than, you know, wave guides or other type of tech.

For our audience – pass-through is when you have a VR headset which is covered in the front, but has cameras looking outwards. And then you feed the video from the cameras into the person's eyes, so you have what they call augmented reality. Do I have it right?

CC: Absolutely. But the challenges of pass-through... In order to get that video to feel real... You have issues with depth perception, and also the latency. It can get very tricky, because that can make people feel dizzy. It can make it hard to navigate a real environment. And you wouldn't necessarily want that imposed video to make you, kind of, slower in the real world.

The first time I saw the headset, actually, with one of the founders of the company... I told him, "so, how much do you trust the latency?" And he just grabbed a bottle of water, tossed it right at my face – with the only prototype they had in the US! And I wasn't even thinking about it – I just grabbed the bottle. And I was like, "oh no, no, you trust your product!" Because he wouldn't have done that otherwise.

So when I met Timmy... I had brought a Teslasuit, which is a haptic suit. It allows you to feel digital objects within VR, and interact from that more physical perspective. And I just saw the XR-1, and it was my only and first chance, at that point in time, to get one to play with and try to dev something.

So I went to him, I'm like, "Hey, think about it: You have the best headset that I can think of right now. I have these haptic suits. My buddy has haptic gloves. Let's try to do an experience that explores, what is the holodeck that we can actually get today? What is premium VR, in 10 years, going to be like?"

It was just trying to put these pieces together. Because this equipment, at the time, was very expensive. It was not even for a prosumer market, it was more for a research market. So it was just this idea of, let's put it together, because we have the puzzle pieces today, here at MIT... Let's see what can we come up with.

So both of you had different pieces of technology at your disposal, through the work that you were doing at the time. And then you thought, "hey, why not put this together?" That's super exciting.

Based on the experiences that you have, with the two of you, I really wanted to talk about this thing that we call the metaverse. This is the hot topic of the day. Everyone is talking about this. One of the largest companies in the world has changed its name to capture the trend.

First of all, how do you personally define what this thing called the metaverse even is? And what are you actually doing about it, at this moment?

To give you an example, what I tell people is that the metaverse is about taking all the computer graphics and interaction technologies that we've been developing for years in the context of entertainment – games, animation, arts... Taking that and using that for serious purposes, for business. Just like you enter a virtual world and play World of Warcraft or Minecraft, you can now go in there and have a business meeting, or design a car, or do city planning... So it's really fertile ground, especially for design, and that's very exciting to me.

What is your language to describe what's happening here? What are you excited about in the space? And what are you doing about it?

TG: The metaverse doesn't necessarily need to be really connected to VR/XR, but it is a transition. It is where the whole VR/XR area was dreaming to be.

It has a lot of definitions – everything from being the next internet, to bringing the physical and digital worlds to a level where it kind of has the same value. And you, as a person, having digital value as your avatar or representation in that space.

It is connected with the transition to web 3.0, the new creator economy... Some people say that metaverse is the next society... But if we take it from the human-centric point of view, and also the infrastructure, it does come back to the devices and technologies that enable it – hardware, modes of interaction – until we get to the experiences and so on.

It's not that a specific company can own the metaverse. It's almost like saying that only Disney does theme parks. You know, it's like, theme parks everywhere – you can have a Universal theme park, you can have Liseberg here in Sweden...

In automotive we played with virtual content since the 80s. We had big cave systems, domes, and magnetic field VR headsets that were really clunky. For automotive, to do a transition like that, it's quite easy, because we do rely heavily on CAD models, 3D models... It is part of our work when we design and develop cars.

Where we are today, in the Innovation Arena, we are looking beyond the car. We are using these tools to design better cars, make more efficient processes, here and there... In R&D, in research, you know, by being able to evaluate the concepts early on... But we also need to look in the future. Can we actually play with these topics? Can we actually focus on storytelling, on experiences?

I'm going to get back to some of those topics. There's some super interesting points that you made. What's your take on it Carlos?

CC: The way you frame it is interesting, but it goes a little bit targeting enterprise. And the metaverse can be both – social or gaming, and enterprise.

If you go back to game design, and you talk about things like World of Warcraft... You have these nodes, right – you have World of Warcraft, Diablo... You have these different ecosystems that are split from each other. And then, the metaverse is almost like this mesh that can connect them all. It aims to have this easier transition between one node to another. That's why, you know, it's the "metaverse" – it's kind of playing on this multi-verse type of construct. Where you can actually create an economy that does not lock your entities, your assets, or what you're building one of those places into a singular entity.

And that's one of the issues that has happened... At VRDays we had a conversation about, what are the potential risks with the metaverse? And it was about, what happens when you spend a lot of time in a game, and you basically build your own persona there – you gather a lot of assets, you have an emotional attachment to those, because you've actually put some of your life and time in to get them. And what happens when a company just seamlessly says, "because we're not making a profit anymore, we're going to deprecate this system. We're basically going to erase any progress you have." It actually has a massive impact, from an emotional perspective, because that world becomes part of your identity, of who you are.

So I think the metaverse is this platform that allows us to have constructs – whether these are enterprise, purpose-driven, or just gaming, or social... But it's meant to be this mesh, that allows us to have a safe haven, of saying, "OK, this theme park is closing. I'm gonna take all of my souvenirs to another one, and I can have them there." Which almost forces companies to think about a different type of collaboration – such as, you know, using blockchain protocols to track digital assets, and how can we transfer them. And we're going to be covering some of those topics later.

So I think the metaverse is a little bit higher than just a use case construct. It's more of an infrastructure play, that allows us to build on top of, for different reasons.

What are some projects out there today? It could be your own projects, but it could also be other people's projects... What are some of them that are actually getting these results, realizing some of these promises today, or beginning to realize them? Can you give concrete examples of how this is happening out there?

TG: I think there are a few platforms for gaming, like Decentraland or Sandbox, where they actually work on the infrastructure, and working with the creator economy.

There is also Tony Parisi, who's like a vice president at Unity, working on a marketing aspect. And he has a really good blog on defining, like, the seven layers... They are actually categorizing, which companies are active in which of these layers – in the tools, in the infrastructure, in the devices...

But I think Decentraland and The Sandbox have a really interesting model.

CC: Yeah, one of my favorites is Decentraland.

I mean, Decentraland comes from the age of ICOs. For those that are not crypto enthusiasts, or haven't been following that for too long: ICOs was "initial coin offerings," which is similar to alternative coins that we see today... People would be able to come up with a white paper for a purpose-driven coin... So Decentraland started by creating this virtual world, that would basically leverage the value of their cryptocurrency on the available amount of land; as the finite, kind of tangible asset, the value was placed upon. And you would be able to purchase one of these parcels, by first purchasing their cryptocurrency.

In the beginning, I was surprised. It was a pretty good white paper. But I saw their Unity repo and it was, kind of like, are you guys kidding me... And as the ICO market exploded, we saw it collapse, we saw the typical bubble burst, that was in financial markets...

So. in the beginning. you actually looked into the engineering of the project, the code, and you didn't think much of it?

CC: It wasn't impressive from a technical perspective. And I think that shows a lot about the development of startups... It's that whole concept of, don't wait to be perfect, just get it out there, and test it.

Over the past six years or so they've grown to become this very interesting project, where now you have an ecosystem of people working there. They actually have a casino where they were hiring people to act, as the avatars within the casino, to provide services. Atari created a virtual arcade. An NFT company that recently got acquired by Nike – their offices are right besides the Atari arcade. So you start to see this micro-economies, these micro-verses on their own. start to flesh out. And how an economy just starts to, basically, emanate out of that – out of the interactions between people... It's not just people pouring money into that, but you actually now have some individuals who can work from home, directly serving other users, as part of one of these establishments that have been built in this place.

Similar to Decentraland, The Sandbox is a very similar experience. They got a pretty big amount of funding coming through, and they also have a crypto token – which obviously went higher up in value, with the whole Facebook-turning-into-Meta...

Those are really interesting projects to follow.

I will zoom into some of those topics that we were talking about now – especially what you said about architecture, interoperability, and value in the digital space.

This trend of everything becoming digital translates to new kinds of objects which are native to the digital. They are born in the digital.

I actually remember, Timmy, we were talking about this, last year, when we recorded another conversation for this university course I was teaching... We were talking about how to design virtual reality. You were suggesting how the laws of physics don't really apply: we don't need gravity, for example, in virtual reality. Now, it seems the law of economics also don't apply. Digital assets have broken the charts this year. Bitcoin, NFTs, digital art... Everyone is talking about these things, and they have outperformed every other investment out there – despite the hot debates about the actual underlying value.

To bring this back home: we're sitting today in the innovation department of Volvo Cars. A company like Volvo has a lot of touchpoints with this economy. You actually invest in other companies and innovations out there. You have branding and marketing functions, which must be in touch with the culture of the day. You offer services like car sharing, where the core functionality actually relies on digital identity and digital ledgers.

My question is: At Volvo and the industry at large, in the ecosystems that you're involved in, what are you taking seriously when it comes to these trends around digital-native constructs like NFTs?

Are we looking at a future where Volvo will be selling NFTs of digital cars for our avatars to drive in the metaverse? I'm sure you have discussed this somehow in the company. What I'm curious about is: when you talk about this, is this a joke that you laugh about? Or is it like an actual business possibility that you're evaluating?

What are the things that you laugh about? What are the jokes? What are the things that you take seriously? And how do you actually discern between the two? What is the framework that you bring in here?

TG: You know, we talk about the Volvo Vision: "the freedom to move in a personal, safe, sustainable way." When we started the Innovation Arena, we were playing with the thought, "what about the freedom not to move?" And this was before COVID. We were laughing back then.  The thought of, what if that can be another track that we're looking at...

We're thinking about virtual meetings as an aspect of sustainability. A lot of people, for instance at VRDays, were saying that the metaverse more sustainable... Sustainable fashion, sustainable transport, and so on... It's not necessarily like that. Some of these digital experiences, they have a bigger footprint – huge data centers, and servers that are running somewhere under the ocean, that consume a lot of energy, for you to play Decentraland...

CC: People usually forget about those...

TG: They forget that whole chain...

But, ideas with NFTs... If we are about to look into it, of course we need to take the "is it safe, is it sustainable..." How how can we, Volvo, as a company, make it safe? Like, a safe metaverse... Cybersecurity, and aspects like that... Then, Scandinavian, Swedish values; ethics, being considerate with one another, empathy...

We spend more time in the digital world, on our phone, in meetings. It's like 10 to 12 hours, 15 hours for some people, per day. There's no surprise that L'Oréal comes up with a virtual makeup line, where you pay a subscription for your Zoom meetings. It has a value. You spend more time in the digital world.

So we are looking at some areas. Not necessarily only for marketing and brand. If we are about to try out new territories, metaverse is one of them.

CC: Non-fungible tokens, fundamentally, address a big issue within digital assets, which is ownership. Not just ownership after a transaction, but authoring – who is the actual, original author?

Today you can very easily know, which one is the real Mona Lisa. And the reason why that specific painting – as opposed to a really good copy – is the one that holds the value is because it's the original one. And there are different ways to trace, why this is the original one; different methodologies that we can use to test it. But with digital, prior to NFTs, what could we really do to know that you were the original person who uploaded the photo? Of course, we can look at the metadata, and we can do some kind of forensic assessment, to understand which one was the original. But that's not feasible at scale. People are not going to be digging and learning these methodologies. So we now take this construct of blockchain technology, to almost create a digital fingerprint – a way to assess whether you actually hold the original version of it, and not a screenshot or just a replica of it.

That gives the creators a lot of power because they can now directly monetize their digital assets. They don't necessarily require this crazy infrastructure to go and do it. But we're also seeing prices of the so-called gas fees – which is what you need to release an NFT within a blockchain – just go up, which also makes it really prohibiting for certain people. It's not like a 15-year-old can just go and make a pixelated art type of thing, and just go and put it up somewhere. They now require a certain degree of investment. There needs to be a belief from that individual, to be able to get that digital art piece within that ecosystem.

I think that's very powerful. Because we do see digital ownership, especially in a construct like the metaverse. If we look at it from a more immersive perspective, so VR... We see Meta and other companies talking about, your VR hub, your VR house... The thing with personal space is personalization. You want to feel like it's yours. So you add small things, here and there. And being able to have that thing that makes you feel unique, that makes you feel like it's truly your little corner within the metaverse, your micro-verse... That's where the NFT has become really interesting.

Obviously, right now, they have exploded. There are trends, within that niche, that we could argue, "are these sustainable?" But something that Timmy and I were talking about is, "what makes NFTs art?" Not just a construct of beauty within a canvas... If you look at some of the really successful projects, like Bored Ape Yacht Club... It's just these faces of apes, right, that people are using all over Twitter, and their profile photos... But what makes people actually attracted to it? Is it just the craze in value, that you can make some money out of it? I actually think it goes beyond that.

If you look at them, they have hundreds or so of characteristics, that were procedurally generated. This in its own that means, there is a digital beauty within that algorithm. They tried to run and do these 10 000 prints, over and over, until they liked it. You cannot just have a single artist do 10 000 designs, and then say, "I don't like them completely, let's do it again!"

And then they play to this, kind of, pop culture essence. You have the little twisted cigarette. You have the mohawk. You have the Terminator eyes. So they're playing within culture. We have subgroups, right – we have the rockstars, and the punks, and the pop culture... They are really tapping on making a community, that now has this identity, or unique identifier. It's digital, but also it's unique. You know it's the original one. And people are even using it as a part of the ability to enter a club, right? It's like a way of validation – which could also go wrong, by the way, but I think that's the interesting part of it.

TG: The sense of belonging, and having those perks... Because one aspect with NFTs is those smart contracts, where you can actually program and change dynamically the perks and access to things, which creates communities.

I was talking with you guys, that I've been to Slush in Finland, in Helsinki, at this startup conference. And I was surprised how they were dealing NFTs. It's almost like a fashion statement. In order to get into a club, or an investor meeting, you had to have this NFT on your phone, and you have to show it at the entrance. And people were trading, and their value went like crazy, and it's like, "wow, this is actually happening!"

You know, one thing is to read on Forbes or Business Insider about it, but one thing is to actually see it in action. And I was really surprised how advanced this kind of community-based... Not even a sense of belonging... But humans do have this need of association, you know, like the group identity aspect.

Like, I'm often related to a specific fashion style, or that I am a VR nerd, or that I work for Volvo. Those are some aspects that people know me for. You know, that I have the round glasses, and so on. So I want those elements to be represented in the virtual world as well. Therefore I would pay for my glasses on my virtual avatar, so I'm recognizable – if I want to be recognizable. If I want to be a fox, so people wouldn't know who I am, that's possible as well.

I was reading [The Embedded Entrepreneur* by Arvid Kahl] recently... Talking about marketing and entrepreneurship... He was talking about the difference between an audience and a community. I think these NFTs, these projects that you were talking about, have a lot of community value in them.

The difference between them is: an audience is the people that show up to see a particular person, a particular performer, or a particular work of art. Like, the Mona Lisa has an audience. We go to the museum to see it.

A community is a group of people that show up for each other. So you're not there, necessarily, to look at the painting, but to hang out with the other people that enjoy the painting.

It's an interesting play on putting the community at the real center of it, and having this digital-native, pure community value constructs. That's very interesting to me.

CC: That is one of the reasons why it may achieve sustainability. Because, one of the biggest questions that everyone has with NFTs is: Is it sustainable?

Do you mean in terms of the projects themselves enduring, or do you mean in terms of environmental sustainability?

CC: Environmental sustainability is a completely different question. I don't think many of the blockchain protocols that these technologies are riding on were created with the intent of such a high frequency in trading, which then means you need to create more blocks on a blockchain, to keep track of these... So that's a different question. I mean sustainability as a market play, or as an economic play.

At the end of the day, monetary systems are nothing but belief systems. When you have a community that is large enough, that believes in the system – whether it's enforced on them by, say, a government, right... So they changed from trade of salt and spices to, you know, a coin-based system that was based on alloys, and then based on paper, and based on just a digital number in your bank account... Then, very similar to that, if you have enough communities that keep the flow of this new type of asset, as a means to actually achieve trade, and that it has a value in it...

That's what can make it sustainable: if enough people actually believe in it. Not for the means of just getting a quick buck out of it... But if people actually manage to establish it as a belief system – where I do believe having my personalized avatar, or a painting in my VR home, or be part of just a community that goes to explore these museums within the metaverse constructs like Decentraland or Sandbox – then, I think, that adds some long-lasting value. That can then create this new node of an economy.

It may not be as exponential in growth as it is today, but it may still continue to hold some value, and then become its own component of that virtual economy.

I sometimes have these conversations with people who are not as technically or technologically interested as the two of you. The way I like to explain why cryptocurrencies, for example, have value, is that – as far as I can tell – these are the only kind of money we have in the world, which is not backed by violence.

What does that mean? Think about: why does the money of a state have value? Why does the Swedish krona or the American dollar have value? It's because, if you live in that place, you're expected to use that money. You have to keep your books, your accounts – if you're a business – in that money. You have to pay tax in the currency of the state in which you live. And if you don't do that, they will literally come and take you away. You will go to jail, if you don't pay your taxes. If you refuse to use use that money, you will not be able to exist in that society. Ultimately, at the very end of the play, you will be physically forced to use that money. And that's why they have value. That's how the value is preserved.

But cryptocurrencies, like bitcoin, are purely positively motivated. The value is purely based on people's belief, and it has actual value. I say to people that bitcoin, maybe, is the only currency in the world that has real value. It's the most real currency out there.

Before we move on to some of the other topics that I have in my notes... Are you personally – or as part of some of the companies that you work with, some of the ecosystems that you're part of – doing anything that capitalizes on these ideas, at this moment, that you can talk about? (I know that you guys do a lot of things that you can't really talk about on the air...)

TG: There are some projects around NFTs or virtual real estate. I'm heavily involved in virtual fashion – like, people ask for my expertise.

What attracted me to automotive was that it was a super traditional industry. And I wanted to see how can we bring these new emerging tools, and the creative mindset, to disrupt the industry. For me, it was surprising to see some of the slow processes, and so on. But then, when you look at architecture or fashion, those are even more conservative. In fashion, to move from pen and paper to an iPad was like... It's still hard to convince designers that they can do so much more on an iPad. Then, you know, from iPad to 3D, to VR... That was a big step for them. On an advisory level, it took years to convince... But it comes down, also, to universities; that if they teach these tools...

I was a bit frustrated. Why is it so hard also for automotive to move from clay models to VR, and so on? And I realized, that knife and the clay model, and that whole process...  If you switch from that physical craft of carving, of drawing, and you go completely digital; it's like this dissonance. It kind of feels like it loses its value in the process.

CC: Humans are multi-modal beings by nature. From the moment of birth, we spend a lot of time literally looking at our limbs, and learning how to use them. And whenever you lose that haptic feedback... We cannot just replace it with the vibration of a controller.

When you get these artists that do craftsmanship... They can sometimes even close their eyes and continue to go just by the feel of the blade, going through the clay, just making those lines... You don't get those feelings. There is a sense of craftsmanship that is lost. Which means, there is also a sense of control that you are losing.

There are great tools for modeling, and there are great tools for sculpting in immersive experiences. But, nonetheless, you don't have the same finesse. The craftsmanship that you get from the analog world is not directly transferable. It's almost like you need to learn how to use these tools again, to get to that level of finesse. And that's what creates this dissonance. It makes it hard for people to just jump from one to another.

Lately I've been advising an architecture firm that is based in Mexico City. It is an amazing company. It started in Mexico City a few decades ago, and it's grown to LA, and now Monaco. They do designs for mega yachts. The interior design – it's beautiful work.

One of the things that we're talking about is, how they do these amazing experiences on rendering, to show the work they're going to do. And then they want to pass it on to VR. They want to use things like Unreal or Unity to do hyper-realistic rendering in real time. And the pipeline is so different. You have these 3D artists, that are working on platforms that are CAD-driven, making hyper realistic renders, that simply end up spending way more time, trying to replicate their work in immersive platforms for real-time rendering. It just doesn't justify... They just fall back onto their craft. I think that's one of the issues. It's that people tend to miscalculate the effort that it takes, to move from one medium to another.

When I got deeper into VR or AR, I just saw it this is another screen. It's another medium. It's just that the delivery system is different. But when you move from a medium to another one, there is going to be affordances that you have to deal with.

I think there is tools, and there is methodologies, that have been created over the past 3–4 years, where we started to see design tools in VR. Because VR, 6–7 years ago, was predominantly, either you know how to code, or you can't do anything. Now we have designer tools, and now we're getting to the more finesse, kind of easier to use workflows that allow these people to, first explore, and then be able to bring, partially, their workflow into it.

What are those tools? Can you name some of them, so we can actually go and check out, maybe learn some of those?

CC: Earlier today, for example, you were asking me about the difference between developing for the HoloLens 1 vs. the HoloLens 2. In the HoloLens 1 days, it was very complex. You had to do a lot of dirty, low-level coding work to get menus to work, to just upload a 2D sprite into a game engine...

Just to upload the program into the HoloLens, that was...

CC: Today we have these workflows with, like, MRTK, which is a mixed reality toolkit that Microsoft created, that now they've also opened to other platforms like the Oculus Quest and, you know, other VR platforms.

You have all of these libraries of interactions and menus, and how to use them. Now they've integrated, for example, Figma, which is probably the most used UX/UI tool right now. That means, me, as a creator, within a 2D construct, can actually take part of my work, leverage that, and test it out.

Then we saw things like Reality Composer from Apple – which is basically this UX prototyping tool, which allows me to very quickly have an idea in AR, without having to go to a game engine. On my phone, take a couple of snaps, get a 3D model down, and just place it somewhere. And do a few sequences – when I tap a button it appears here, if I tap this other button it moves around. And like that, you can get an idea: is this concept worth pursuing past, you know, just pen and paper? Because you are seeing it in the medium.

We also have volumetric tools. A volumetric capture that – if you have a phone with a LIDAR sensor – you can use, to actually take a 3D model of an object, quite seamlessly with your phone. We were actually playing with it a couple of nights back, with some friend of ours that are in the industry. And they hadn't really seen the process. But you can now actually take a 3D model of, you know, this sofa, crop it, pass it into USDZ, which is Apple's proprietary format... And then, within five minutes, I can send it to someone else. And without having to download an app, when they tap on a link, it will open their browser, the camera, and they can see that 3D version of it.

And that's the stuff that people are not aware of. Because it is a complete different workflow. But today we can explore these, you know, easier pipelines that we didn't have – not even two years ago!

TG: And again, to the workflow – there are other tools like Tvori, or Gravity Sketch, and so on, where they specifically didn't try to mimic real workflows, as I was saying. It's like multiplayer – you can meet, collaborate. You can bring in your own models, create user journeys... We had our design department trying it out.

When you're trying to mimic a real process too much, like just doing it in VR, it loses its purpose. So you're actually doing double work. The whole purpose, with VR, is to break rules. Flip a car in the air, make it as big as possible, get inside the exhaust pipe, or be an element in the battery and try to understand how it works... You can do so many things. We were talking before, where I want to draw in the air, and I leave the pen here, and I can pick it up back from the air... That whole flow enables you to do so many other things – instead of, you know, mimicking the real processes or laws that apply to to the physical world.

We talked about uh the digital world. We talked about digital-native constructs like NFTs, being in this world with your avatar, and these objects...

Perhaps the ultimate digital-native construct is a digital mind, or a digital person – also known as artificial intelligence. I know that both of you are interested in this topic, and working on it as well. What are some of your projects, where you are utilizing AI? And exactly what kinds of models, algorithms, data sets does that involve – particularly in connection to these these topics of digital worlds, 3D, virtual reality, the metaverse, and so on?

TG: Since I come from the gaming industry, I always had to create those NPCs – like non-playable characters, you know... They are like these AI systems that you interact with, you know, when you play against the computer; or they are part of the story, and so on.

To program those complex behavior trees that respond to your input... I was always fascinated about it. And some of those elements, of course, they translate, for instance, to simulation – for self-driving car training and so on, where you want to capture, to see various scenarios like traffic scenarios, on the highway, and the parking lot... So you can transfer some of those gaming principles, and even using the same tools for simulation...

Then, the other element... I was always fascinated about human behavior, and how little it takes to fool the human brain. You know, to make it not realistic, but believable. That's why, for me, with Varjo, when we were doing the project...

Varjo is the company that makes the super high-end VR headsets, right?

TG: Varjo does, like, the most advanced headset in the world. We've been collaborating with them since they were like an early startup, and Volvo invested in them.

What was interesting there was to see, like... We placed a virtual car. You could see your own reflection, and shadows, and lights in it. And then we had the same real car next to it, at a conference in Copenhagen. And I could actually make both disappear and appear. So people were starting to be confused. Because they thought, that's the real car! Because they touched it. But when they saw that it disappears, in the same way as the virtual one disappears, and it appears again... They started to be confused. And every time I asked them, "OK, now you can step in the car..." Me, knowing that it's virtual content... I would just go through the door, and just sit in the car, because I don't get haptics. But for them, when I opened the door, they were ducking, and trying to get away, not touching the virtual content.

Can you tell me a bit more about what the actual hardware is that actually makes this happen? Like, a million cameras in the room?

TG: Again, coming back to the VR with passthrough, which only has two cameras, and it has a depth sensor... And nowadays, the latest models have LIDAR, so you can have occlusion. I can place the virtual object in front of the real car, or behind the real car... So it's aware of the real space. That, in itself, allowed us to capture the reflections and the lights of the room, and apply it on the material of the car.

We even took it to the next level. We were actually driving a real car with the headset on. And then, you know, swipe models and so on. I have now a few colleagues that are doing, like, really crazy experiments with it – for research, for design reviews... It's like, well, if we make a "car Tinder" for designers... So when they draw, they can actually swap between their car concepts, and make decisions – "OK, we go with this dashboard, we go with this screen..." Doing that real time, without having to go back to the lab all the time, and take a decision meeting, and then download the new CAD file, modify it... I think that, in itself, can save a lot  of time.

But if you transfer it from efficiency and utility, to the explorations they are doing... There, with AI... Like, playing with human senses – eye tracking, capturing behavior, interaction, prediction... So have this contextual design, where you have the things at the right time, at the right place. Data-driven. That a really interesting aspect. I always like to play with the senses – like sound, touch, and how you combine them...

This reminds me of something which I want to share with you, and you can maybe tell me, what's wrong with me. Recently I drove a modern car. I have been driving cars from, maybe, 15 years ago. And recently I drove a very recent model, like this year's model car. It was actually a Volvo.

So I sat in this car, I drove for a few minutes. Here's how it felt: I felt like this car really wants to drive itself. It really doesn't want me there. I'm only there as a driver because I'm legally required to be there – like, it's illegal for me not to be there; so I have to be there, I have to keep it going... But it really, really wants to drive itself. And it resents me for sitting there. It's like a teenager, you know, resenting its parents for keeping it under control. That's how I felt – like I was the parent of the car, and it hates me for just sitting there, and watching every move that it makes.

But I really love to drive, you know. How do you, I guess, make the decision to have a car that feels this way? Versus, engineering it, making it, for people who want to drive themselves. Is there some kind of market data, some kind of consumer trend that is going on, which is driving the industry towards this direction? Or was that an anomaly that I experienced?

TG: I think it's a mix of answers... There's always been this strife for efficiency. And we, as humans, by design; we're not efficient – like in bigger systems and bigger scales...

Also, when it comes to safety aspects... Most accidents happen because of human error. And if you want to have this safe society, and so on... And I have some brilliant colleagues working on these aspects like system level safety, like orchestration...

So I have to, kind of, give up my own pleasure, in order to integrate with the society, and the traffic, and the entire system?

TG: You can find pleasure in it, in different ways. You can use your time in a different way.

For me, that's where I had dissonance. Because we see how much investment is done in this industry, as a whole, for people just to scroll Instagram... You know, the car drives itself, so you can scroll Instagram or buy NFTs... Does that bring value to humanity?

I get much more value out of driving a car that feels good, like feeling connected to the road, and having that sort of physical feeling... But I think I'm rare – maybe like five percent of people think this way...

TG: You can go to a test track and drive one. Still have it for your entertainment...

Or I could just drive an old car, I guess... That's what I'm enjoying more.

TG: For a while.

Yeah, that's true. I've actually been contemplating, maybe I should get a car... And if I get the car, I'm definitely getting, like, a V8 or something, you know. Because, within our lifetime, that thing is not going to be legal anymore. There will be a point in our lifetime where you're not driving a V8, V12 sports car with a with a 6-liter gasoline engine, or whatever. Those are going to be extinct at some point.

Anyway, Carlos, man, talk to me about AI.

CC: Alright, so, AI... I'm going to touch on the cognitive dissonance topic, to loop into AI.

So, funny experience with the demo that Timmy was talking about... When they were setting up the booth for Varjo at AWE – which is this conference in 2019, an augmented reality or augmented world expo, right, so it's AR and VR – they asked me for help. When I had the headset on, to put markers as to where the car was going to be...

And cognitive dissonance, if you are not really vested in that topic, is what happens in your brain when a schema of knowledge – a set of rules that you have around the topic – is challenged. So, for instance, if you grow up in a place where there is only black cats... The day that you see a yellow cat, you're going to have this brain short circuit going, like, "wait, is that a cat?" It actually makes the conscious part of your brain – your executive function – just, kind of, centralize more energy onto analyzing that, until you get the validation: "yes, it is a cat."

And then, that can lead to – if all of the dogs in your town are black – "are there also yellow dogs, or are there green or white dogs, and cats?" So it kind of brings you in this loop that takes you out of what you were paying attention to, to begin with.

You get types of technology in AR that have more dissonance triggers... It was very interesting to see that, when they asked me, "hey, can you move and go to the back of the car, to just put the other marker..." I actually, instinctively, walked around the virtual car. Because the reflections were so good, that my brain was like, "there is an object!" Timmy and someone else laughed at me, and said, "what are you doing?" Because the cable wasn't long enough. They're like, "just walk through it!" And I was like, "dude, there is a car!"

Back in the day, I visited a lab at ETH Zurich – the university in Zurich, which is like a super good technical university. They had a VR lab. I put one of their headsets on, and I was trying out some of the applications that they were doing.

They were doing something called redirected walking. This is where you play with the way that you track the person, and mirror the movement in the virtual environment; so that you can have a small space, but when you walk, it feels bigger. So they can keep you walking for, like, hundreds of meters, in a room that is very tiny even. Because it subtly nudges you to, like, go left, or go right. It's called redirected walking. I'm not doing a good job explaining it, but people can look it up.

I put this on. I was in this castle kind of place, on the side of a cliff, with like a view over the ocean or something. But it's a cliff – there's a point where the ground, the virtual ground, is over, and the cliff starts, and then there's like empty space below your feet... Obviously, you're in the room – so if I keep walking, I will just be standing on the ground in the real world, and I will be floating mid-air in the virtual world.

They said, "hey, just go, walk there, it's super fun!" I couldn't walk there. I couldn't walk off the edge of the cliff, in virtual reality. I physically, literally could not do it.

CC: Yeah, it challenges people, in those kind of edge scenarios, to just jump. And you will see that dissonance moment, where their brain expects them to continue falling, and then they just kind of land on their feet. And they look at you – or, well, they look through the experience – like, "what is happening?" Because that's something that your brain is just not used to, right? Your brain is used to – from the moment of you being a child – just observing an object rolling off a table, and it follows through a trajectory... You know that an object that goes off the resistance path will eventually fall, due to gravity.

Kids don't understand the construct of gravity, and the value of it, and how to calculate it... But their brain has, already, this construct of the trajectory that it's going to follow. And unless there is something to hold it, it will continue until it gets to some other surface that will hold it... Those are the really deeply rooted constructs that we have of reality. And that's what I wanted to use...

To loop back to AI, so, let's use the example of Decentraland – since, you know, there is context for the audience already. Why are we hiring people to act as a waiter or a server, when, you know, as game designers, we would assume it's fairly easy to create an NPC to do that? I mean, what are the 5–10, or even 100 routines that you need there? People are going to buy a drink, or buy chips, exchange them...

It's simply because, humans, or the real world, has this entanglement that creates emergence. Entities that have a consciousness, or an affective system, or a certain degree of awareness... We are never in the same state, right? Like, something can happen, that makes me feel upset, and that changes, a little bit, my next interaction with someone...

We haven't gotten to a point today, where we can actually embed that into the current models we have for NPCs. If we're exploring a world where we want people to have this sustained engagement... What happens when you play games like GTA. How long can you play with the NPCs that act as cops, before you get bored and you need to go and play with the gangsters, right? You eventually get to see the entirety of that decision tree, and its limitations; and it's very quick.

That's, kind of, the dissonance that I would have gotten, if I was looking at the Volvo experience through something with a smaller field of view, or that didn't give me those reflections. I would really quickly catch those limitations. So, I think the area that is really interesting here to explore is: the metaverse is not just about going for a play session, like Fortnight, that has, you know, a finite amount of time... Or like World of Warcraft, where you have a quest... But, I think, it's more about recreating a world where people could – if they wish to – have an indefinite amount of gameplay. And to get to these engagements, we require NPCs that go beyond these normal constructs of just decision trees. The real world is fun because there is emergence. There are things that we can control, that just happen and pop, and that makes it interesting. It happens through self-diverging agents, that interact with each other constantly.

I know that this is actually part of your work; that you're involved in developing these kinds of agents, these kinds of AIs. What kinds of models, data sets, technical infrastructures do you actually utilize for this?

CC: We're first looking into replicating models from affective neuroscience, from behavioral psychology, and psychology models that are used to depict personalities in people – so like, the Big Five personality model, and different psychometric assessments...

In terms of the AI component behind it... First we're looking at things like GANs – generative adversarial networks – that are used for deepfake technology, for example, or speech synthesizing. Because, at the end of the day, one component is being able to create a system that has this divergence in an agent. So, making it feel slightly happier, or slightly more upset... But then, we still need, you know, the front-end version of it – the animation or the video which then uses these types of neural networks. So these are intricate systems that don't just use one type of data sets, or one type of neural networks. You have, for visuals, usage of things like GANs. You have other types of constructs, like transformers, for NLP (natural language processing) models.

In terms of data sets... It really depends on, what is the specific thing you're looking for? For instance, to get the construct of emotional valence, we were taking data from Twitch streamers, because you have a really good reaction loop. They are not just playing a game and reacting to the gameplay – which you can also kind of measure, what was the stimuli that made them have a reaction – but also the chat, right? So, that one is very interesting. We worked with some Twitch streamers. They gave us the recorded streams that they play on YouTube, to be able to start to see, what were their triggers? What are the words that people may say in the chat, that actually make them have a mood swing? What changes their affective state? And that's very interesting, because those types of anchor points, that made us feel one way or another, have a lot to do with our development as humans, throughout childhood, and, you know, the teenage years.

I wish we could go very deep into this subject. Maybe we can sit down and do another episode about this. But we are approaching the end of our time, that we have allocated at the company. So I want to ask a final question to both of you, which comes back to the themes that we were talking about today.

When we were brainstorming topics together with you, for this panel, we found three concepts that stand out – with regard to the world that we want to live in, with regard to how we can create positive outcomes. And these were: safety, prosperity, and diversity.

These new technologies – the metaverse, the AI, the NFTs... – they bring a lot of opportunities on these fronts, but also dangers. Like violations of privacy (such as I was suspecting that you might have done with the Twitch people), or amplifying unfairness and inequalities in the world... In your work, and in the industry and the ecosystem around you, what is the priority level for these concerns? And what concretely are you doing about them?

TG: Since I'm attached to a company and its values, and it's something I subscribe to – because that's why I work here – I think, as I said with NFTs, if it's not safe or sustainable, how do we make it safe or sustainable? Or should we even touch the topic?

There is always that trade-off, as I said, that strife for efficiency, the greater good, that you can present to the user. So they know, that this amount of data collected can actually contribute to this greater good. For instance, make safer cars, because you record the driving behavior. Things like that.

It's always at high priority. We were talking, a lot of times, about art and curatorship – like deciding what not to do, what not to put on the table. A lot of these topics are fun, but it's always interesting to decide and say, "OK, this is probably what we should not do..."

Especially at the Innovation Arena, we're trying to have this principle that, whatever we do, can translate, for instance, on a human-centric level, society level, and planetary level. Like, safety: how can society safety look like, and planetary safety look like? Those things can be instantiated through different artifacts – they don't need to be cars, or car-related... Planetary safety is sustainability, right – like our efforts in that direction...

When we talk about AI and the metaverse, it's like, how do you bring those sets of principles or rules through the experiences, to the narratives? What is the story that we are telling, that can actually make those places better places, where people actually desire to be, or take part – like the communities that we were talking about...

CC: From my perspective, I think AI is one of the biggest minefields when it comes to ethics and and safety. Safety is a primary concern to me, to the people I work with.

When we enter the construct of the metaverse, the only thing that is actually in the metaverse, that belongs to us, and that can get affected, is our psyche. It's our mental health. So, I think we're we're entering a very interesting age, where we already see the impact of people spending a lot of time in there, with their digital personas as the front. People engaging extensively with social media, and the negative impacts that it has – especially on young people...

For us, it's very important to be careful as to how we use these technologies, to try to diminish – because we will never be able to stop it completely – but to try to diminish the potential bad players from targeting people, with the technology derivatives from what we're working on...

Part of that is how can we turn the table, and use deep fakes for good. So, for example, I want to collect your data, as to what you react to, not because I want to try to target you in a better way; but simply because, today, we live in an age of, kind of, fast food of technology. You can, kind of, go to McDonald's and Burger King, and kind of remove pickles or add onions – meaning that's kind of what UX is. We're still just guiding people for, what is the best purpose of our product, as to what we believe is the best end result of what they need, or where their need is going to be met. But the reality is, if I don't understand you as an individual – not just as a component of a subgroup, not just as a bucket within a psychometric, kind of, population; then I don't really know how to service that. And if we can actually understand you better, and get to empathize a little bit more with you, your current state... Just as simple as this: if you are down today, can I actually stop pushing ads to you, just not make you loop into something more negative... Can I give you content that I know has a higher chance of actually making you feel, if not better, at least amused? At least get to that point...

We don't do that today. But that also has the ethical challenge of, do I want people to read my emotions? Well, what if we don't know it's you? What if I use deepfakes to, in real time, actually change your face to someone who is completely synthetic? And then just, you know, put you in a completely unidentifiable way, that cannot tag you to the real user? So I think there is a lot of constructs that we need to explore...

I said this at VRDays: if we stopped thinking about the user, and actually realized that the user is people... And if we see them more as people, and not just the user of my product, and my use case, and my solution; but how does that actually impact them as people, as human beings. beyond the use of the product?

Incredible. These are such interesting ideas... Unfortunately we don't have as much time as I would like to today, to go super deep in them. But I will do a bit of homework after this live session. I will try to find a few resources about the things that you're talking about, and insert them into the notes that we have on our website, and that accompany the episode on Youtube, and the podcast.

Thank you so much guys. This has been super enjoyable for me. I really look forward to having many more conversations with both of you – on-camera, and off-camera as well. Thank you very much for having me today in your super interesting space, the Innovation Arena at Volvo Cars.

Links and Resources

• Apple's Augmented Reality (AR) Creation Tools
• ARCAR: On-Road Driving in Mixed Reality by Volvo Cars by Florin-Timotei Ghiurãu, Mehmet Aydın Baytaş, and Casper Wickman
• AWE
• Big Five Personality Traits on Wikipedia
• Bored Ape Yacht Club
• Decentraland
• Figma
• Generative Adversarial Network (GAN) on Wikipedia
• Gravity Sketch
• Initial Coin Offerings (ICOs) on Wikipedia
• LIDAR on Wikipedia
• Microsoft HoloLens
• Mixed Reality Toolkit (MRTK)
• Natural Language Processing (NLP) on Wikipedia
• Redirected Walking on Wikipedia
• Slush
• Teslasuit
• The Embedded Entrepreneur* by Arvid Kahl
• The Sandbox
• Timmy Ghiurãu – Innovating in Virtual Reality (conversation with Mehmet Aydın Baytaş)
• Tony Parisi's Blog
• Tvori
• Unity
• Unreal Engine
• Varjo XR-1 announcement
• VRDays

*: Affiliate link – at no extra cost to you, we earn a commission if you purchase from these links.