November 2016 (In 2 weeks by team of 5) @Entertainment Technology Center, CMU
R O L E
Producer, Interaction Designer, Artist
O V E R V I E W
Our final experience, fAiRy, is an Mixed Reality 3D platform game using Microsoft HoloLens which interacts with both real and virtual worlds. In this experience, the player will use the latest XBOX S wireless controller to help the little girl to follow the path that made with books and collect light orbs, and finally become a fairy. Our team made fAiRy successfully into the Fall Festival 2016 at the Entertainment Technology Center, Carnegie Mellon University.
Hello, Spatial & Diegetic UX
HoloLens brought me to a new design space. I can look and walk through the whole content from inside-out. Transferring 2D thinking into 3D space is not easy, but this project gives me a great opportunity to learn and practice it.
Designing for Magical Moments
At the outset of the project, we all knew it was going to be a huge challenge, and the biggest question was just how exactly we could go about designing for a new and different experience. No one had an answer. It took us four days to figure out what kind of experience we wanted to bring to our guests and how we were going to design it with the best use of the HoloLens and its technology.
I just wanna make you feel the magic.
Figuring out Our Requirements & Limitations from Our Audiences & Environments.
● What kind of experiences are we going to bring to our guest?
This project is specifically designed for 2016 ETC Festival. Every year during the festival, there would be more than 300 people at all age groups coming to this event in just three hours. Therefore, we need to shorten the time of the experience in each turn to allow more people to try our experience. Also, the ETC festival is aiming to show and make everyone feel the power of the emerging tech, so we expected to create an entertaining experience that could make people feel the magic and the future in a very short time.
After analyzing the situation, we knew we need to bring to our guest a:
● fast (efficient & maximizing the turns), but
● a magical (emergent & feeling the future) experience.
Started by playing with our new toy.
Understanding the Technology by Reading Up on Existing Resources & Trying Demos
● How to achieve the basic goals by making the best use of AR/MR & HoloLens?
Since there is no real standard, everyone is still exploring. So our team started from doing research on other people's projects and trying demos that showed off the capabilities of the HoloLens. In this way, everyone got a better understanding of this technology and knew what exactly our platform is good at.
Based on the technical research, we understood and summarized the top features of HoloLens. Compared with the traditional AR product, HoloLens could enhance the experience by its strong localization, responsive interaction, and various user inputs.
By doing the research and trying the demos, we got a better understanding of the technology and its capabilities about designing for an:
● immersive (seamless & bridging the physical and digital worlds by using
the perception capability ), and
● highly involved (interactive & increasing the engagement by using the
various inputs) experience.
So, Any possibilities?
Using the Design Framework to help us Brainstorming.
Based on the goals we decided, our team started to brainstorm. In order to help us envision what elements we wanted to include in our experience including what design methods we wanted to utilize, I used the framework Table of Elements by 8ninths as a reference so that we can know the possible options available to us.
We were trying hard to find that best point to connect two worlds. Based on the MR Design Table and the considerations of the two weeks working time, the limitations from the Festival, and the skillsets of the team, we decided to build a world by using real objects which are very easy to find in the school, like books, cups or any tools. And having a virtual character which can be interacted with in this world seamlesslly.
● Who is the Character? Why it comes to the real world? What could it do here?
The idea was vague but it gave us some wiggle room while also being narrow enough to focus our attention to creating magic moments. But for now, we needed to make it more clear and concrete. We need a story. After brainstorming, I continued by sketching out some of the most promising ideas to distill them down and also to get feedback from others.
The whole experience was aiming less than 90 secs in each turn, so we could not do a very complex storytelling experience. In other to keep the pace fast but invovling, we tried to make the background story simply, providing a whole atmosphere instead of being a heavy role in the entire experience.
Once upon a time, a cute little girl was coming out from a fairy tale book and trying to collect light to become wings to fly.
After we got this more concrete idea, our team opted to dive right into thinking about what potentially interactions could support our story and experience. I sat down with the programmer and created our own Table of Elements. Based on our goals and story, we included a lot of elements we wanted to do at first. And in the process of our project growing, we added or removed more items depending on if we could best use it in the project or not.
Time to make magic, but...
Taking a Deep Dive into Mechanics & Level Design.
I felt it was so hard to do the mechanic and map design part since we had to combine and balance both the real and virtual parts into the game seamlessly. Our experience is a world-locked experience. So there would be unpredictable things happen, like: our guests might walk around in the open space, and look at the real props and the virtual girl in different angles intentionally. It would cause the occlusion problem.
● How to avoid these intentional and unintentional behaviors from ruining the magic moments? And in other words, how to make a proper use of these behaviors, giving people the feeling of freedom but still in our hands?
So we need to consider each detail carefully to avoid those offset problems. In order to get inspirations and find the best solution , we did a lot of research on the level design of different 3D-platformer games, especially Super Mario 3D World.
Our tabletop is a square. So we tried to design the path as an S shape with different heights for different parts so that we can try to lead and indirectly control the player to look in different angles to find a good way for him/her to help the girl to jump over the gaps. Then, I made lots of boxes with different sizes for testing and finally found the best way to set our landscape. The last step was to choose the books with different solid color covers as our real objects instead of boxes and rearranged the orders and shapes to make them looks good based on the original positions.
Feel the magic in the moment.
Adding More Magic Feeling by Using the Table of Elements.
After building our real world prototypes, we started to build 3D models in Maya and build the corresponding world in Unity. So the little girl can stand on the real books and jump over between the books seamlessly. Besides this, in order to bridge the gap between two worlds, we got a really good idea from the Super Mario 3D World. Why don't we use the bridge to bridge the gaps. We found we could do the virtual bridge concept. It is not only about the path, but also shows we "bridge" both of the real and the virtual worlds. After trying and testing the prototype again and again, we wrote the best shader script to show the virtual bridge at the right time. We felt so excited, it is really like magic.
So, we almost got our worlds. Then we wanted to think about the step by step interactions in this end to end experience. Drawing interaction map and workflow diagram based on our Table of Elements was the best way for us to see through technology and decide what we want.
● How to control and interact with the little girl? Could we make emotional connection between the girl and our guest?
We wanted the control of the experience as easy as possible since the guests came from all the age groups. Therefore, our decision was made heavily focused on using gamepad as well as body movement to interact with the real and virtual world.
Also, We had to think of interactions that would leverage the freedom and mobility that the real world environment provides to our guests. The interactions concept map helped us define potential interactions for all the pieces we needed to include. Here in the basic interaction map, we basically included any ideas our team had on how users could interact with specific pieces.
Our Interaction Map
Besides the basic action interaction, our team also paid lots of efforts on building emotional connection between the guests and the character. Here is the emotion interaction design we decided.
At last, make it look nicer:)
Because our game is called fAiRy, so we decided to use massive particle effects to create the feeling of a sweet fairy tale. I helped programmer to design the most visually appealing particles: orbs, lights, stars, fogs and etc. For the bridge, we made it with ice texture, imitating the feeling of Frozen.
We know we make it!
Look at the little boy, his facial expression is the best evidence to say We Make It!
The festival was great. Almost everyone made a stop by our room and tried to play it, but time was limited. Really appreciate for the people coming to our world and talking to us. Your smile is the best encouragement to us! Thank you!
Design for the human instinct.
Successful MR experience will take full advantage of human reactions to guide users through experiences. Considering in every details include subtle changes in lighting, differences in size and color as well as any discernible motion can be extremely powerful when smoothly guiding user behavior.
Be visual and magical whenever you can.
Mixed Reality is the science of magic. It can bring breathtaking moments that are incomparable to anything today in any other medium. If there’s a way that could cause user delight, we should go for it. Making things visually and creatively could surprise your guest in a more beautiful way.