These Tricks Make Virtual Reality Feel Real

I am standing a few feet away from a conference table, about to test something called a “Holojam.” It is made of head-mounted displays, motion capture cameras and many lines of code. Affixed to the edges of the table, as well as to my hands and ankles, are little plastic balls, which are used to track things in the virtual environment. Like QR codes, they help the software interpret the table as a table.

Ken Perlin, who directs the Media Research Lab at New York University, describes it to me as the “first community-created collaborative 4-D spacetime sculpture.” Then he flips the switch. The space I am standing in is a kind of forest clearing at night, with a bright moon illuminating tall trees. As I move my head, the view subtly warps. “To give you freedom of movement,” he says, “we had to make everything more cartooney. The moon and the trees are like billboards, not 3-D. We decided it was more important to have immediacy and presence than high fidelity.”

But isn’t fidelity what makes a virtual reality feel real? I want wind to whistle past as I walk in full Doppler splendor. Sunlight should gently dapple on the grainy surface of a wood table, and objects should possess the heft and feel that they really do. Many in the field buy into this kind of supremacy of detail. Others, though, believe it’s a fetishization that obscures how effective virtual reality isn’t just about verisimilitude. As with the way we experience reality itself, it’s also about the way we lie to ourselves.

Perlin knows about the verisimilitude angle. Early in his career, he worked on the movie Tron, now known for its seminal use of computer-generated animation. Dissatisfied with the digital sterility of visual effects, he began investigating ways to create “controlled random processes,” which led to more realistic depictions of things like smoke and fire. Today his influential textural effect is called “Perlin noise.”

But Perlin is skeptical of the merits of an indefinite push toward more visual detail. While the emphasis in commercial and gaming virtual reality has been on boosting graphical resolution, there is not necessarily a one-to-one correlation between pixel density and realism. “People who are myopic,” says Perlin, “don’t think they’re living in a fantasy world.” The human foveal region itself, says Perlin, maxes out at around 60 pixels per linear degree—smaller than that, he says, “and you’re falling between the cells.” The mistake of many recent computer games, Perlin argues, is to confuse realism with believability.

This, in essence, describes what many academic researchers see as the current, game-driven state of virtual reality. “It’s never going to be real, any more than a movie is real,” Perlin argues. Evan Suma, a research assistant professor at the University of Southern California’s Institute for Creative Technologies (Palmer Luckey, the founder of Oculus, is an alum), says he worries that virtual reality, once the novelty has subsided, may reach a plateau; because, as he says, most of the experiences being created, while graphically sophisticated, are “couch-locked.” “A lot of the magic for me is lost when you’re just sitting down and moving.”

We could walk through an infinite kingdom in our living room.

When we walk, by contrast, we get more spatial cues from the environment, and we are able to pay more attention: It’s a cognitively richer experience. Research shows, for example, that children who walk to school can create more robust maps of their routes than those who are driven.¹ “Much of our sense of reality has to do with proprioception,” Perlin argues. “Even if my hand were invisible,” he says, reaching for a pen on the table, “I would have no problem picking this up, thanks to that notion of body sense, of balance, of the sensation at the bottom of the feet.” To know space is to move one’s self through space.

It is now easy enough, using motion capture, treadmills, and other technologies, to create canned versions of walk-through virtual environments. But how, absent creating some sprawling, Borgesian one-to-one mapping of the actual world, do you create a virtual space that can be realistically walked through and yet is many times larger than one’s living room?

You delude yourself, of course. Scientists are exploiting the natural inaccuracies in people’s own proprioception, via a technique called “redirected walking,” to create the perception of space where none exists. Frank Steinicke, a professor of Human-Computer Interaction at the Department of Informatics at the University of Hamburg, is one of its pioneers (he is also recognized, unofficially at least, to have spent the most time of anyone on the planet—24 hours, with short breaks—in an immersive environment, wearing an Oculus Rift head-mounted display). With redirected walking, Steinicke tells me, users can sense they are exploring the twisting byways of a virtual city when in reality they are simply walking in circles inside a lab.

Without getting feedback on our heading, Steinicke says, in most cases we begin to drift, even retracing our path, after as little as a few dozen meters. The phenomenon has been noted in people who wander endlessly lost in the forest or deserts—particularly when the sun is not visible—even as they hardly cover any ground at all.² In an experiment in which people were blindfolded and left to wander for 50 minutes, they began walking in surprisingly small circles of less than 20 meters in diameter. Virtual environments themselves present a kind of perceptual distortion: For “reasons that are not really known,” says Steinicke, even when virtual spaces are modeled one-to-one with real environments, subjects routinely underestimate distance.

Redirected walking goes one step further, exploiting a discrepancy that arises as we walk and rotate our gaze: We can confuse the field of “optic flow” that comes our way as corresponding to a curved path, even when walking straight. In experiments, subjects immersed into a virtual world did not notice when their physical 90-degree turn translated into just an 80-degree turn in virtual space. Compression also went unnoticed in forward walking speeds, distance covered, and path curvature. By stacking various kinds of misdirection, an infinite virtual space can be mapped onto a confined real space. We could walk through an infinite kingdom in our living room.

The perceptual sleight of hand, which comes largely due to vision’s outsize role in our perceptual array, can be applied to more than just walking. As Suma notes, even hand motions can be redirected, through what he calls “haptic retargeting.” By “warping” the virtual environment—or the sense of one’s virtual body in it—researchers have been able to decouple what a subject’s hand seems to be doing from what it is really doing by, again, playing on our visual dominance.

He describes a “fun demo”—dubbed the “one for three illusion”—in which a user is instructed to pick up and stack cubes in a virtual environment. Users are first shown three real cubes on a table. As they don a virtual reality headset, two of the real cubes are taken away—though three cubes still exist in virtual space. As users reach for one of the now-illusory cubes, a subtle visual warping causes them to correct their hand trajectory in the real world, so that they reach toward the one remaining real cube. “You don’t ever realize that you’re only ever touching one object,” says Suma. Users, As reported by a team at Microsoft Research, users found the task “repetitive”—until they took off the headsets and saw what was really happening.¹

As with the way we experience reality itself, it’s also about the way we lie to ourselves.

Not only can researchers create inconsistencies between virtual and real spaces, they can make the virtual spaces themselves impossible. Suma has created two seemingly separate virtual rooms that somehow occupy the same space, and are larger on the inside than they appear from the outside—not unlike Dr. Who’s Tardis. He has found, in experiments, that subjects only begin to notice violation of the rules of Euclidean space at high levels of overlap. “They don’t exist in the real world,” he says of his impossible spaces. “There’s no reason why our perceptual system should have developed a sensitivity to this.”

On the other hand, there are visual cues that can’t be messed with. We’ve evolved to pay close attention to cues like the horizon line, because they help us orient ourselves. Even very subtle manipulations of a visible horizon line or a stable view of a ground plane—essential for proprioception—cause people to get unstable. “If you can’t get it exactly right” in virtual reality, Suma says, “you’re better off hiding it.”

But there would not have been a selective pressure to detect non-Euclidean angles or warped spaces. By aiming at these soft targets in our visual system, scientists can trick us into believing we’re in some pretty remarkable spaces. The real key to this make-believe, though, may be in letting one person trick another.

Back in his conference room, Perlin suddenly slips on his own head-mounted display, and he—or, rather, a large, bright, animated figure that looks like some failed entry for an Olympic mascot competition—appears across the clearing from me. He waves his hand, tracing a line that seems to hang in the air. And then, with eyes no more lifelike than a snowman’s two pieces of coal, he looks at me.

This is a curiously powerful moment. Locking eyes with the simple avatar unleashes that warm, almost familiar prickling of human connection. What do you want from me? What can I do for you? Just the fact that someone else is in the virtual environment underscores the sense of “social presence,” as media scholar Carrie Heeter has called it. “If other people are in the virtual world, that is more evidence that the world exists.”³ The act of recognition inside a shared space validates both your existence and the existence of the space. A lack of recognition denies those things, not unlike the Hollywood trope in which a person, who has not yet learned he is a ghost, tries in vain to get other people’s attention.

We are social mimickers. When we see our peers accepting the reality of an environment, we are encouraged to accept it too. Our behavior in virtual space is no different. While much of the current energy in virtual reality research is channeled into getting things to look real as possible, Perlin suggests that even with heightened visual verisimilitude, virtual reality will still feel solitary unless a compelling social presence is achieved. “The holodeck,” says Perlin, “is other people.”

We even take the form of our avatars as a social cue, reverting to real-world human dynamics quickly in virtual environments. In the so-called “Proteus Effect,” people begin to quickly adopt the subtle psychological dynamics associated with their particular avatars. As a Stanford University study noted, “Participants who had more attractive avatars exhibited increased self-disclosure and were more willing to approach opposite-gendered strangers after less than 1 minute of exposure to their altered avatar.”⁴ As William Bricken argued, “psychology is the physics of virtual reality.”⁵

Realism will be one of the first qualities to be discarded as we make virtual space more realistic.

It should be little surprise that I was arrested by Perlin’s cartoon eyes: Eye contact is one of our most potent non-verbal communication forms, in the real world or not. What some have found in virtual worlds, however, is that eye contact per se is less important than head movement—which can also signal acknowledgement and intentionality. When Perlin first began to set up the Holojam, a few years ago, his cartoonish characters had no faces, just square boxes atop bodies. “One person shrugged,” says Perlin. “Even though you couldn’t see the shoulders, you could only see this box, you knew it was a shrug because we only move our head like that when we shrug.”

Did Perlin look realistic to me? No. But faced with the problems of the “uncanny valley,” maybe a cartoon is better: It stops us from thinking about whether some- thing is real and just lets us feel that it is. The cognitive scientist Mel Slater defines the “plausibility illusion” in virtual reality as the sense “that what is apparently happening is really happening (even though you know for sure it is not).”⁶ We can become invested even in blocky, 1980s-arcade representations of people. Slater conducted a set of experiments in which a low-fidelity virtual character was “shocked” while people watched. As the shocks to the hapless avatar grew in intensity, the onlookers grew increasingly anxious. “Bugs Bunny is not realistic,” Perlin says, “but believable.”

Perlin’s Holojam environment—which is currently rendered off of a smartphone to be mobile—is only bound to get more realistic, graphically, as head-mounted display technology and processing power continues to improve. And when high-quality untethered body tracking—he says there’s “currently no commercial market”—arrives, virtual reality can begin to become a social space. “By allowing people to walk around,” as opposed to what he calls the “Wall-E model,” he argues that “we can get to the important questions: What’s going to be the impact on people when they interact with each other in a world that’s been transformed?” It may very well transform the unreal into the real.

We shouldn’t be too surprised that realism will be one of the first qualities to be discarded as we make virtual space more realistic. After all, we do it in the actual (real) world all the time. Casino entrances use curved paths so people don’t slow to make perpendicular turns, drawing gamblers “further into the ‘mousetrap,’ ” as described in anthropologist Natascha Dow Schüll’s book Addiction by Design. So-called casino “atmospherics” like background music, meanwhile, are carefully modulated to sit just below the threshold of consciousness—to do otherwise, one industry expert quoted by Schüll notes, “restores … your cognitive state to where you can make rational decisions.” The whole effort, as one casino designer describes it, is to create for gamblers a “different plane on which they lose their sense of reality.”

There are even slivers of virtual reality creeping in. Devices like slot machines have been made more immersive with virtual reels, liberating them from physical laws of probability and putting players under the illusion that they can win more often than they actually can.

Retail experiences can also be designed to confuse. Alan Penn of University College London has described the meandering pathway through an Ikea store as intentionally confusing: “As you move through the showroom you rapidly become dissociated from everyday life—all views of the external world are obscured and even one’s sense of direction is removed by the twists and turns of the route. At the same time you are shown stage set glimpses of what your life might be like. The effect is more or less subliminal.”⁷

So while Steinicke, Perlin, and Suma are operating on digital photons, and retail and gambling designers are not, there is a point of intersection in their work: Whatever space we find ourselves in requires our active interpretation, and allows for somebody else’s active obfuscation. It wouldn’t feel real without it.

Tom Vanderbilt writes on design, technology, science, and culture, among other subjects.

References

1. A. Rissotto, A. & Tonucci, F. Freedom of movement and environmental knowledge in elementary school children. Journal of Environmental Psychology 22, 65-77 (2002).

2. Souman, J.L., Frissen, I., Sreenivasa, M.N., & Ernst, M.O. Walking straight into circles. Current Biology 19, 1538-1542 (2009).

3. Heeter, C. “Being There: The Subjective Experience of Presence” MIT Press (1992).

4. Yee, N. & Bailenson, J. The Proteus Effect: The effect on transformed self- representation on behavior. Human Communication Research 33, 271-290 (2007).

5. Markley, R., ed. Virtual Realities
 and Their Discontents Johns Hopkins University Press, Baltimore, MD (1996).

6. Slater, M. Place illusion and plausibility can lead to realistic behavior in immersive virtual environments. Philosophical Transactions B 364, 35-49 (2009).

7. Penn, A. The complexity of the elementary interface: shopping space. Proceedings, 5th International Space Syntax Symposium, Delft, Holland (2005).