Ken Perlin doesn’t hesitate to say that he’s more than just a scientist. Along with being a professor of computer science at New York University and directing the NYU Games For Learning Institute, he said he’s writing, doing art, doing engineering— “I’m doing all sorts of things.” His blog, too, is polymathic, with posts that range in tone from deep to whimsical. The root of his passion could very well be his blog’s tagline: “Because the future has just started.”
One way he’s already contributed to that future is helping to create, with his colleagues at NYU, the “Holojam,” an untethered virtual reality headset that enables a “community-created collaborative 4-D spacetime sculpture.” With Maria Lantin, the director of the S3D Centre at Emily Carr University of Art + Design, he’s going to facilitate a dance rave party where everyone may be thousands of miles away. “That’s a good step,” he says, “for understanding future reality.”
In our conversation, it became apparent how carefully Perlin chooses his words. The reason he speaks of “future reality”—rather than “virtual reality”—is because he thinks the latter will simply be the former: VR, he says, will permeate our lives and culture just like writing has and hand-held computers still are. The everyday will become magical.
The video interview plays at the top of the screen.
Interview Transcript
What got you interested in virtual reality?
We are all interested in VR, every single one of us. We just may not call it that. When I was little, I read Harold and the Purple Crayon, which was written well before I was born. It’s about a little boy, named Harold, who has a purple crayon. He starts to draw and he creates entire worlds: That’s virtual reality. Kids who grew up, say 15 years ago, grew up reading J.K. Rowling’s Harry Potter books—all about this stuff: It goes back centuries. No matter how far you go back, everybody is interested in the things we see in our dreams, and if we could do them, what would we do. I don’t think that being interested in this is a question of technology. You use whatever available tools to explore the possibilities in our heads—whether it’s novels, or theater, or movies, or putting on headsets.
HTC Vive and Oculus Rift seem to be the leading headsets in the field. Are there any big differences between them?
I think that they’re very similar to each other, so I don’t think there is any decisive difference in the hardware. I think the key difference is going to be that the companies [that own these systems] Valve versus Facebook are probably going to focus on different kinds of virtual reality content. Basically, there are two markets right now for VR: There is the market for the gamers, who are extremely valuable to the industry because they pay upfront for content; and then there’s everybody else—the people who would use things like Google Cardboard—who are a much larger market but are generally expecting things to be free; sort of the Google kind of users. Both Vive and Oculus—and also the Sony Morpheus, for the PS4—are going for that first group, because it helps to monetize everything.
Gamers tend to like the ability to have realism when they want it, so that’s part of what’s expected. But I don’t think realism actually sells content—good game play, good experience, good interaction, good design, sells content. So it’s necessary to show that you can do a high level of realism. But that’s really at the end of the day, not what’s essential.
You’ve created the Holojam VR headset. How’s that different?
The Holojam is using the technologies that are available for virtual reality, but its goals are not virtual reality. Its goals are to prototype what will be the ordinary future—when people are wearing first glasses, then perhaps contact lenses. We start with the idea that you should not have wires trailing. All of the high end game platforms so far require you to be tethered to a wire. For social interaction—everyday life—that’s pretty much a nonstarter; for gameplay it makes sense, but not for understanding what the future of everyday reality will be like. We can walk around in our everyday lives, and we’re not thinking, I’m playing a game, or I’m doing something high-tech. We’re just thinking, I’m going to the store, I’m meeting with my friends, I’m shopping, I’m doing everything. So we want to understand how people interact with each other in that future that’s coming, when visual things will be able to float in the air between us because the technology will enable it. The ordinary and banal is much more important than any one particular thing, whether it’s playing games or seeing movies or any one activity. On the one hand, it’s ordinary and banal; on the other hand, it’s our lives.
Will VR technology take us away from reality?
People seem to shift their definition of reality depending upon which thing they disapprove of in any given year. Whatever technology we ever develop—whether it’s musical instruments, or bicycles, or pens, or anything to do with VR—the big question is: Can a human being use it effectively with the brains we have and the bodies we have? I call that, anything that does that well, a natural interface. What genes we have to work from doesn’t change in scale of civilization. Statistically speaking, a child is born with pretty much the same average bag of tricks as any other child—we have the same brain capability, the same physiological body. There’s variation between individuals, but it’s not as though the human species is biologically evolving in such a small timeframe. So we’re stuck with these brains and these bodies.
Can virtual reality be an art form?
All the literary arts interact with each other—so the movie based on the novel shows you different aspects of the novel, and there are things you can get from reading the novel that you’ll never get in the movie, and vice versa. I think Ed Catmull [president of both the Walt Disney and Pixar Animation Studios], was completely right when he pointed out that virtual reality is not a form of movie, it’s something else. Just as movies are not a form of stage theater, and poetry is not a form of novel, and vice versa. We don’t know yet how it will play out. We haven’t gotten our great VR artists; we haven’t gotten our W.D Griffiths and Eisenstein’s yet. They just haven’t happened because it’s too soon. And when we do, those people, who are the brilliant content makers and designers for this emerging media, will start to tell us how literary forms manifest in this kind of media.
Could virtual reality transform performance arts, like improvisational dance?
I’m doing collaborations with some interesting people, for example Maria Lantin’s group at [the Emily Carr Institute for Art and Design]. We’re asking questions like: If dancers were able to see anything, because we give them visual enhancement—including, for example, seeing what’s happening behind them or what’s happening on the other side of a solid wall—would they be able to collaborate in a more enhanced way with each other, just the way that, say, jazz musicians now can collaborate, with musicians that are behind them, because they can hear the music? Since there’s such a strong connection between the philosophy and practices of improvisational dance and of improvisational music, it would be interesting to add these tools to dance to see if it can start benefitting from them.
Have you ever let a crowd of people use Holojam simultaneously?
We had done a very successful showing of our Holojam technology at SIGGRAPH 2015, on August 10th in Los Angeles, in which hundreds of people came through and experienced seeing themselves and each other as virtual avatars in physical space, walking around without any wires and painting in the air together, as a collaborative medium. So we decided that we wanted to continue to work on projects that would push our technology forward and would help our students to understand how to do productions for this new medium. Since it was kind of trite to do Halloween, we decided to just move one day later and do a project inspired by the Mexican Day of the Dead, which as you probably know is a very happy occasion, because it’s the day, November 1st, when the spirits rejoice because we remember them. So we placed everybody in the spirit world, virtually, in Holojam, and when you put on the headset, and you grab the hand controllers, you see yourself and others as a ghostly spirit in the spirit world: You’re travelling down a parade float. The visual iconography was inspired by all of the very rich art that’s developed around Dia de los Muertos, the Mexican Day of the Dead, and we created a celebratory experience for people to share together.
I had mentioned Maria Lantin’s lab in Vancouver, Emily Carr—we are working with her and the center for digital media in Vancouver to create a dance rave party. The reason that we want to push in this direction is that this will allow us to take our Holojam technology and create a free moving experience, where the other people in the room might actually be 3,000 miles away. That’s sort of a good next step for understanding future reality.
How does the philosopher Sartre’s play No Exit inform your views of VR?
The characters in No Exit are characters in Hell, and they’re in Hell because they’re rotten people. Their punishment is to spend eternity in a room with each other, and it’s only Hell because they’re stuck with other rotten people. So it’s a kind of existential irony that I’m in Hell only because of who I am. His larger point—when the character says, “Hell is other people,”—what Sartre was getting at is: None of us are alone, none of us are completely defined by our own view of ourselves. We’re constantly bouncing off other people and looking at other people as a mirror of us. Our very sense of who we are is intertwined with what we see when we see other people look at us. The reason that I like to say the Holodeck is other people is that I think we shouldn’t just go off and develop technologies: We should understand what is this interaction between people that we’re trying to support—specifically, not just information passing but emotional affirmation, our views of each other, our ability to relate to each other—all of those are central to any communication technology.
How will the Holojam lead to a new evolution of language?
We have these hands, we have visual ideas—should we go find a whiteboard and start scribbling on it, or should we use the space between us and start making gestures and see things here? It’s my hypothesis that when little children grow up in a world where anything can appear between people in the air, because we will have these technologies that are worn, then language will evolve and those children will start using that to communicate with each other in richer ways that augment their speech. We build, we layer—written language hasn’t gone away: We simply change our preferred way of expressing it. Texting and tweeting and all of these text-based social media are making use of the same written literacy—they’re just using it with a different physical modality.
Will AIs be treated the same as humans in virtual reality?
Because there is very strong empirical evidence that people on a physiological level react differently when they believe they’re interacting with a real human being or they believe they’re interacting with a bot, non-player character, AI, I do not think that, in general, society will accept a blurring of the distinction. People will always want to know: Is that a real person or is that a computer program? That will influence how we design these future enhanced realities. I think if you ask the question: If a flesh-and-blood child is in danger, or a computer program is in danger, which one do you save? I think everybody will answer the same way and I think that will always be true.
What would you be if you weren’t a scientist?
I’m not doing any one thing now. I’m writing; I’m doing art; I’m doing science; I’m doing engineering; I’m doing all sorts of things. So I don’t see any of those things as disconnected from any of the others. When I was a teenager, I remember I would ask my dad sometimes a question that he thought was beside the point, and he would say, “So do you want to go to Brooklyn or by bus?” I think that this question of, “What would you do if you were not this?” is a little bit like that—we are all all of those things. So I don’t ask, “If I didn’t have my left hand, what would I do with my right hand?” It’s not a useful question for me.
