There’s plenty to talk about when it comes to best practices for designing gesture-based applications.  On the blog, we’ve covered a range of topics related to building your application: how augmented reality can be used as a feedback system, the role of ergonomics when designing a practical UI, and how to leverage our Gesture Authoring Tool to create custom gestures in your apps. . .just to name a few.

Future posts will continue to offer ideas and feedback on how to design intuitive applications.  But in this post we offer a few practical tips to consider when designing and testing the usability of your gesture or motion sensing application.

Where to begin? Start with the experts

When we talk about usability, we are referring to the “ease of use and learnability” for your application, game or interface.

There are many excellent resources available to help you with your usability testing.  We recommend a few here but there are obviously many, many more out there in Google just waiting for you!

• Usability 101: Introduction to Usability from the Nielsen Norman Group
• Ten Usability Heuristics, also from the Nielsen Norman Group
• Brian Pagán’s “New Design Practices for Touch-free Interactions” in UX Magazine
• Designing Gestural Interfaces by Dan Saffer

8 Principles for Designing Gesture-Based Applications & Interfaces
(and a few common mistakes to avoid)

We’ve gathered up a few key points to keep in mind as you get started building your applications. These are principles that we’ve learned along the way, or perhaps reflect mistakes we’ve either made ourselves or have seen others make when building gesture-based apps.  While this list isn’t intended to be a comprehensive, it does touch on foundational points to keep in mind.

1. Precision is a good thing. . . up to a point.
   When designing the UI for a gesture-based app, it’s better to fall on the side of larger buttons making it easier for users to correctly make a selection.  When you require a great deal of precision it can quickly lead to user fatigue. Our last blog post touches on exactly this point – imagine using your finger instead of a mouse to select a specific “cell” in Excel.  Frustrating, no?
   
   

   Design for fingers & hands, not a “mouse”

2. Don’t model your application on existing user interfaces.  Instead, build on the strengths of gesture and motion tracking technology.
   Traditional user interfaces are based on a WIMP model – windows, icons, menus, pointer. As mentioned above and in our last post, your finger is not a mouse!  So, rather than trying to turn it into one, instead leverage the 3 dimensionality of the hand. For example, imagine an online store where you can pick up an object (how about a shoe!) and turn it around by rotating your hand.
3. Avoid actions that require your users to lift their hand above the height of their shoulder.


Try to avoid awkward poses. It should be fun to interact with your app!

You don’t need to have a degree in physical therapy to design gesture-based applications, but it does help to give some thought to ergonomics and how human bodies move naturally.  It can get pretty tiring pretty quickly, and can even be challenging for some users to have to lift their arms high.

Co. Design featured the work of “Curious Rituals” to call out the unnatural postures we make to conform to technology. Gesture recognition is a way to free us from these awkward poses.

4. Do break up activities into small, short actions.
   Try to keep actions relatively brief with rest periods in between rather than having users move all over the screen.  John Pavlus recently wrote an article for the MIT Tech Review on whether gestural computing breaks Fitts’s Law.  You may want to review a brief primer on Fitts’s Law when considering menu placement – users want to be able to navigate from selection to selection as quickly as possible.
5. Not all users are right-handed.
   You’ll have to make a judgment call as to whether to design your application to be equally useable for both right and left-handed users. The good thing is that if you work with Omek’s Grasp then you are ensured accurate detection of left vs. right hands. This Grasp feature can prove useful if you are creating an application intended for multiple users sitting next to each other.
   
6. Gestures have cultural connotations.
   Gestures can have different meanings in different cultures so be conscious of context when designing your app. For example, the “thumbs up” and “peace” signs both have positive connotations in North America but quite the opposite in Greece and areas of the Middle East, respectively. It may be a good idea to do a quick check on gestures before going live in other countries.
7. Design your interface to keep users within the “Effective Interaction Zone”.
   Try to avoid placing elements (menu selections) too close to the edge of the FOV of the camera. If you do, users may not realize that their hand has moved outside of view of the camera, causing a frustrating experience for them when their selection doesn’t work.
8. Test your design with hands and people of all sizes.
   Test, test and test some more before going live. That’s a motto that we try to stand by. And the broader the types of people that you have to test (i.e., tall vs short, large vs. small hands, different sized arms) the better! That way you’ll be sure your app and interface will work just as well for a basketball player as it will for a small-handed lady (like the author of this blog post).

Interested in creating your own close-range gesture-enabled experience? Sign up to be notified of our upcoming Grasp beta release.

Have the following questions been keeping you up at night?

• What is embedded vision all about?
• What are practical things I should know if I want to incorporate gesture recognition into embedded systems?
• What are some of the different technologies used to create depth maps?
• Why is 3D technology better than other technologies at solving certain computer vision problems?

If you live in the Bay Area (or plan to be there April 25^th) then you’re in luck! Omek Interactive CTO, Gershom Kutliroff, will be one of a few esteemed speakers represented at the upcoming Embedded Vision Alliance Summit.

Embedded Vision Summit 2013

What exactly is the Embedded Vision Summit? According to the source, it is “a technical educational forum for engineers interested in incorporating visual intelligence into electronic systems and software.” The Summit is part of the larger DESIGN West event being held next week.

Quick Details:
April 25, 2013
San Jose Convention Center
San Jose, California, U.S.A.
In Conjunction with DESIGN West
Register Now!

Why should you attend?

I caught up with Gershom before he leaves for his trip and asked him to provide us with a few highlights from his presentation. Check in on our blog after April 25^th for more details from his talk, including videos showing off what you can accomplish using depth data from 3D cameras vs. a standard RGB camera.

Sneak Preview: Gershom’s Talking Points

In addition to touching on the questions listed at the beginning of this post, Gershom will offer analytical thinking about 3D data, explaining how it is inherently different from 2D data. He will explain how these differences drive the need for different algorithms to be constructed in order to understand the data that comes from depth sensors.  Gershom argues that these algorithms are the basis for a more fundamental paradigm shift that has broad implications which go beyond the depth sensor.

Using a case study to illustrate his points, Gershom will show how these 3D-specific algorithms cascade down the value chain. Ultimately, he argues, different software libraries and different hardware architectures will be required — ones which are optimized to support these new algorithms.

Gershom will provide you with key insights into how algorithms for depth cameras are constructed. These  ideas will help form the basis of thinking on how to best design software systems and construct your hardware architecture so that it is optimally designed to run these new software library systems.

What does that mean for you? Well, whether you are a software developer building 3D-based applications or a hardware manufacturer interested in learning how to better support emerging 3D cameras, this session is for you.

Sign up today (for free! space permitting): http://www.embedded-vision.com/embedded-vision-summit-registration

We hope to see you there.

This is a common suggestion we hear from clients when first discussing possible use cases for close-range gesture recognition. It makes sense and on first glance seems like an easy extension of how things work today. But how well does this work in reality?

Actually, gesture as plain mouse replacement, does not work very well. Gesture recognition can be incredibly intuitive, easy, and fun – but, only when it is implemented with care and thought to the end user experience. This means building in from the start with a wide range of design considerations on how users interact with touchless systems.

Most current user interface paradigms are not designed for gesture. Commonly available Graphical User Interfaces (GUIs) were not intended for hands and fingers to manipulate the objects inside of them. They were designed within the paradigm of a hand-controlled mouse that operates in the screen of the user. Let’s take a closer look.

Your finger is not a mouse.

Take a look at the cursor that appears on your screen. Put your finger next to it and you may be surprised to see how small that cursor actually is. Imagine trying to “maximize” the Word document you are using and accidentally you end up selecting the “close” function and are hit with a dialog box asking you if you want to save your document. Windows 7 is just not conducive to touch.   

A mouse isn’t very expensive.

It may be challenging to make a convincing business case to an OEM to incorporate gesture recognition into their manufactured devices for the sole purpose of replacing the mouse. Does the additional cost of a 3D depth camera justify the added value? Especially when the substitute is a low-cost mouse that a user can pick up for just a few dollars.

The mouse works pretty well for what we want it to do.

The on-screen cursor that represents your mouse offers an accurate way to navigate through most of the tasks that you want to perform on a computer today. They are responsive, fast and familiar. By now, the mouse is ubiquitous and has become the most intuitive way for most people to interact with a GUI system. A lot of people would question, why replace something that isn’t broken?

Gesture adds value, without replacing the keyboard and mouse.

At Omek, we see gesture as a means to enhance your experience when interacting with your computer. We don’t think of gesture vs. keyboard + mouse as a binary choice, where you must choose between having gesture OR your keyboard + mouse. Rather, we see gesture adding another important dimension to how you interact with your device. For application developers, gesture can provide real context and understanding of a user above and beyond what’s known when he is simply using a mouse.

I can imagine a scenario in which I lift up my hand from my keyboard in order to access a “parallel desktop”. In this parallel desktop I am able to pause or change the music I’m listening to with the simple flick of my wrist. Once my hands rest back on the keyboard I immediately return to writing this blog post.

Design the interface to match the input method.

In order for the scenario above to work, the “parallel desktop” should be designed into the system from the start. To create a successful product, the UI should align with how a user interacts with the product, in order to provide the user with the right and continuous feedback he needs.

Tablet computers provide a perfect case study to illustrate this point. Many may have forgotten, but Microsoft first began releasing tablet computers well before the iPad, all the way back in 2000. But it wasn’t until Apple released the iPad that we really saw tablets take off.

From the first glance it is pretty evident that early Microsoft tablets were not designed for touch.  Instead, Microsoft extended the current Windows operating system and put it into a tablet.  Since the operating system wasn’t designed for fingers, users had to use a stylus to make selections.  The icons were small, not easy on the eyes, and not optimized for touch.  In addition, there was the issue of “screen coverage”. Dan Saffer, Interaction Designer and author of Designing Gestural Interfaces, reminds us that since our fingers are attached to a palm, your hand often can get in the way and cover the screen while you are trying to make a selection, creating a frustrating experience for any user.

The right UI can mean the difference between success and failure of a product.

With the recent release of Windows 8, we’ve seen sales of 1.5M Microsoft Surface tablets, with demand beginning to take off in the marketplace. The Windows 8 operating system represents an entirely new UI, designed with “touch” in mind and thus better suited for tablet and mobile devices. It features “swinging slabs”, a boxy design, and a horizontal layout.

Essentially, in order for Microsoft to realize success of their touch-based hardware devices, they needed to redesign their software-based interfaces to support that mode of interaction.

Will gesture ever replace the mouse?

YES! We do believe that gesture will take on a more prominent role in the computing experience and may ultimately fully replace the mouse. Gesture, when implemented well, can be a much more natural way of interacting with the many devices we have in our lives.

It won’t happen on its own, though. Software and hardware companies are already in the process and must continue to rethink and redesign the UI experience with gesture in mind from the start. We will see the best results if hardware and software companies plan for gesture from the concept stage all the way through to release. Once we have integrated systems that are designed for gesture will we see more widespread adoption of gesture-based interfaces.

Gesture control still plays a critical role in interface design and is being incorporated into a wide range of platforms. From infotainment control in your cars, to rehabilitation and physical therapy feedback systems using gesture, we are seeing more and more companies finding innovative uses for 3D gesture control. Authors Shel Israel and Robert Scoble are working on a book (titled The Age of Context) which explores the ever-growing role that contextual sensors are playing and will play in our lives, and what it means for us. More and more we are seeing multi-modal capabilities being rolled out, working to seamlessly integrate voice recognition, gaze detection, and touchless control.

In the meantime, our UX Studio and design team continue to test, learn and share thoughts through meaningful articles on our blog describing how to create truly interactive, intuitive interfaces that incorporate gesture. Stay tuned to the Omek blog to learn how to drive wide adoption of gesture in today’s interfaces. And sign up below to be notified of our upcoming Grasp beta release!

Grasp is in many ways the “magic” that enables us to transform our ideas into reality.  You may already be familiar with our Beckon SDK, which provides full-body skeleton tracking from distances of 1-5 meters.  Well, Grasp is our “close-range” counterpart.  It is a middleware solution and full set of tools for hand & finger motion tracking and gesture recognition from distances as close as 10cm.

But it is so much more! With Grasp, we took a unique approach in order to solve the question of close-range motion sensing.  We built a sophisticated and detailed solution, because it is our belief that you need technology complexity in order to achieve user simplicity.

Sign up now!

Interested in how you can get your hands on Grasp?  Currently it is in closed alpha testing but we are rapidly gearing up for a beta launch.  You can sign up to be notified of the release and be one of the first to try out these tools.

Below, we highlight several key features that are unique to our Grasp offering, all of which we leveraged when designing the Practical UI to ensure we ended up with a genuinely intuitive interface.

Cross-Camera, Software-only Motion Sensing and Gesture Solution.

While there have been quite a few exciting recent product developments, the gesture recognition market is still quite young with many new innovations underway.  At Omek, we want to enable you to take advantage of the latest advancements in camera technology.  To that end, our solution is cross-platform – we are working closely with different hardware providers to support their 3D sensors. What does this mean for you?

• Support for depth cameras to offer you the most robust experience. With our eye towards usability, we want to provide developers with the most cutting edge technology to create meaningful user experiences.  At Omek we believe that 3D cameras have the power to change the paradigm of how we interact with our devices. Read our prior blog post on why we work with depth cameras rather than 2D cameras.
• Seamless integration into your personal computing devices. One of our goals at Omek is to help you simplify your life; not complicate it by adding more peripherals!  That’s why we focus on supporting cameras that will be incorporated into your device, whether it’s an All-in-One PC or the dashboard of your car.

Full Hand Skeleton Model vs. 6-Point Tracking.

Unlike other close-range solutions available, Grasp creates a full 3D skeleton model of the hand, complete with 22 joints. Rather than simply assigning tracking points to each of your fingertips, we offer developers a complete model of the hand, opening up a broad set of possibilities to create a range of experiences.

Our approach offers significant advantages to ensure more robust tracking.  Why?  Well, the hand has several degrees of freedom, making it difficult to track. Think about it – you can open or close your hand, cross your fingers, curl your fingers, or rotate your palm. Your hand can take on many different configurations. By using a hand model, though, we are able to tackle many different scenarios, including self-occlusions, rotations, or closed / curled fingers.  The hand skeletal model effectively constrains the movements of the fingers to only actual possible configurations your hand could feasibly make.

• Applying it to the Practical UI. Take for example, the pinch and rotate gestures used when taking a book off the shelf and opening it up to see what’s inside. First, we have to recognize the pinch – not too difficult.  It becomes more complicated, though, once the hand rotates.  A lot of information all of a sudden becomes occluded, making it difficult to maintain continuous tracking.  The fingertip points are no longer in the field of view of the camera.  Using Grasp, however, we can define that we are seeing the back of the hand and can instruct the application to continue tracking.

No Calibration Required.

When you develop your application using Grasp, your users will be able to immediately interact with your interface without having to calibrate to get started, providing a smoother and more dynamic introduction. Essentially, behind the scenes and invisible to anyone using Grasp, we calibrate the skeleton model to different hand dimensions and continue to auto-calibrate during the duration of the use of the application.  This ensures we are adjusting our tracking to all variety of users, whether they are kids or adults, those with small hands or someone with long, thin fingers.

When you walk up and begin using our Practical UI, regardless of your hand size or shape you are guaranteed to have a smooth, robust, and instantly-tracked experience.

Hand Detection + Classification.

Unlike other systems, we detect hands by searching for real hand features rather than simply looking for motion in the scene or using a skin color model. This helps us ensure that we are tracking what we want to be tracking – hands and fingers.  It allows us to robustly remove false positives and instead, quickly detect hands as soon as they appear in the scene, so your users won’t get frustrated by the application not working as they expected it to.

• Differentiate between Right and Left hands. Combined with the full skeleton model of the hand, we offer the ability to detect whether the camera is tracking a right vs. a left hand, even in the case of poses that are non-trivial to detect.   

Designed for Usability

A major benefit to having an in-house UX Studio is that we have actual developers working with our SDK from the first stages of its creation. We leverage this feedback loop for both early testing and to ensure that our tools are designed with developers and their specific needs in mind. Our tools are easy-to-use and process much of the technical components “behind-the-scenes”, allowing you to focus on creating meaningful applications, rather than trying to solve computer vision problems.

In Summary….

Are you a developer seeking to create a gesture-based game?  Perhaps a medical device company looking to design a touchless interface? Or, an automotive company reinventing the in-vehicle-infotainment system? Whatever system you would like to create, Omek offers the most advanced, cutting-edge and user-centric tools to help you power your gesture-based experiences.

Sign up now to be notified of our upcoming Grasp beta:

Part 2 of the Series: Designing a Practical UI

In this post we explore the challenges of providing an effective and useful “feedback” within gesture-based systems and offer our thoughts on how augmented reality can be implemented as a supporting tool for creating an intuitive and engaging interface.  Read on to learn how we came to the approach implemented in the video below:

Background

A feedback system is the application’s method of communicating with users to: prompt them to take an action, inform them that a given task has been understood, and assure them that the system is aware and responding to their interaction. Feedback can come in the form of sounds, animations, color changes, highlights, or textual messages such as instructions, pop-ups, balloons, etc…

Omek’s UX Studio has spent much of the past six years researching best practices for gesture based HMI.  One of the most important lessons we have learned time and again is that an effective feedback system can make the difference between a frustrating and confusing experience to one that naturally guides the user through an application, in a fun and successful way.

Gesture-based systems require their own form of feedback

What works well in one modality often does not work as well when applied to another modality. This is a learning that we cannot emphasize enough. There are attributes that are unique to gesture-based systems, and how you design your application should take into consideration these attributes:

• No tactile feedback: This one may seem obvious but has important implications. Unlike with most interfaces, gesture-based systems are “touch-less”. They lack physical, tactile feedback. For example, with a mouse or keyboard there is the haptic response when you push down and release a key. In a gesture-based system, you need to find the appropriate means of letting a user know that they have performed a given task.
• Invisible interaction space: The interaction space in a gesture-based system refers to the effective field of view of the camera (check out the photo below). Without feedback, a user has no way of knowing if he is being “seen” by the application. Check out our last blog postfor more detail on this topic.
  

  Defining the Interaction Space

•  Even the best-designed systems sometimes fail: Yes, on occasion even the most accurate of tracking systems may “lose” a user, for example, because of occlusions. When this does happen, the application should fail gracefully to keep frustrated reactions to a minimum.
• No standardized gestural language: We’ve referenced before the article that Don Norman wrote on gesture wars for the magazine Core 77. For example, something as simple as the action of “selecting” can be interpreted in many different ways. One user may hover their hand over a button to select, while someone else may try to push their hand forward, mimicking the action used with a physical button.

Experiments in Feedback

For the Practical UI app we tested out a number of different ways to provide users feedback throughout their experience, from the first interaction all the way through to the end. Below we share examples of a few different feedback methods, their pros and cons, and considerations to keep in mind when you apply these in your own applications.

1. The Traditional: Your hand as a “cursor” on the screen, similar to a mouse pointer.  In this scenario, your hand (or, more likely, your finger) becomes the mouse cursor on screen giving you visual feedback of where the “cursor” is at all times.

Here, a user’s finger becomes the mouse pointer

Pros:

• Extension of the current paradigm, thus requiring little explanation.  From both an applications design and a user perspective, this seems like the most natural way to provide feedback in a gesture-based system. Users easily relate to this method of feedback since it takes such a familiar form (who hasn’t used a mouse before?). During testing, we found that users almost expected this to be the means of interacting with the application. There’s no manual or additional guidance you need to offer to users when they are getting started.
• Constant feedback given to the user.  With a “cursor” method, the user always knows where his hand is in relation to the screen, providing invaluable information.

Cons:

• Easily leads to user fatigue. The work here is placed on the user, who has to be very accurate in his selection. Users have to hold an extremely steady hand in order to ensure they make the intended selection. All that steady hand-holding will quickly lead to fatigue and frustration. Bernd Plontsch just wrote an excellent post on exactly these challenges when trying to implement this using a Leap Motion.
• Applying Fitts Law in a NUI world. When the hand becomes a cursor in a gesture-based interface, a significant amount of thought must be put into the design and layout of the interface to ensure that users can quickly navigate from one selection to the next. John Pavlus interviewed UX Expert, Francisco Inchauste, on exactly this topic.

2. Active Regions: Highlighting selection buttons on screen when a user’s hand hovers over them (no cursor). 

Jinni Demo: The “selected” box is defined by a white outline

Check out a real-life example of this in our gesture-controlled media selection  demo we created in partnership with Jinni for CES 2012.  Instead of a cursor appearing on screen, when a user’s hand moves over the selection options, the selection buttons activate indicating that the user is in the “area” of a specific selection.  The specific feedback may be manifested by buttons lighting up, enlarging, or creating a “shadow”.

Pros:

• No issue of “accuracy”. This significantly lowers the issue of user accuracy that we saw above with the cursor method.  As long as a user is in the general area of the selection button, he is able to make a selection. This means that the tracking feels much more stable for end users – they don’t see the small jumps and shakiness the tracking data creates.
• Fewer “false” selections. Bigger buttons mean less chance for error (see Fitts Law).  By creating bigger, more clickable areas it is much easier for the user to focus on their target, point at it and reach it.

Cons:

• Changes in the interface design. This feedback method requires that you construct your interface to fit this approach, creating from the get-go large “selection buttons”.
• No constant feedback for users.  In contrast to the mouse cursor approach, users here only know whether their hand is selecting a certain icon or not.  The feedback received can be somewhat vague, without offering guidance on how far your hand must move to reach the next icon. 

3. 3D Hand Model: Creation of a Hand Avatar. Animating a 3D model of a user’s hand and turning it into a “hand avatar”. Every joint in the user’s hand is directly mapped to the model’s joints. Imagine the extension of a user’s hand into the application.

A user’s hand is transformed into a robot arm, showing all of the user’s joints

Pros:

• Very detailed feedback.This example offers a clear physical representation of your hand on screen, representing each and every one of your actual movements.  It’s almost as though your hand has extended into the application on screen.  There are two options for physically demonstrating this:
  • Mirror – where the screen reflects a mirror version of your hand
  • First person – the user sees their hand on screen the way they see their hand in front of them
  • Creation of an immersive world.  You can simulate collisions and physics in order to interact with virtual objects as if they were real – pick them up, push them, squash them etc.  You become the puppeteer – moving a virtual you in a virtual world.

Cons:

• Issue of the “uncanny valley”.  A hypothesis widely used in robotics and animation, the issue here references the delicate balance that must be struck between creating a virtual hand that users will find engaging to utilize.
• Very sensitive to tracking issues. Since most of the data from the tracking system is used all of the time, any error will be instantly seen.  Essentially, it runs the possibility of introducing noise to the experience even though the actual points of interest (such as the index finger) are stable on their own. This issue can arise in any tracking system, even the most accurate ones.
• Requires a certain amount of control. It sounds strange but we are so used to controlling interfaces on a 2D interaction space that having your 3D hand inside the screen full control can actually make for a bulky and awkward experience.
• High sensitivity to the perspective of the interface.  Since this is a virtual world the hand might be rendered in a different perspective than what the user is used to, resulting in a disorientating experience for most of the users we tested.

Our current model for the Practical UI: Using Augmented Reality as a Feedback Method

An augmented hand means rendering the hand’s image in each frame and overlaying the user interface on top of that. You use your own hand virtually represented on-screen as the pointing device and the feedback system becomes very responsive and actually fun and intuitive for most users (See part 1).  Why?

“Silhouette” of a user’s hand created using Omek’s Grasp to abstract just the hand

• Incredibly intuitive system.  A user raises their hand to get started and immediately sees it reflected on screen, requiring almost no explanation on how to use the system.
• Constant feedback. The user always understands if the camera is tracking them or not based on whether their hand is being shown on the screen, providing quiet reassurance to the user. Moreover, the user has accurate and constant feedback for his position in space.
• Use of all three dimensions. As the user’s hand moves closer or farther from the camera, their virtual augmented hand inside the application becomes larger or smaller, respectively.  The user can easily understand their area of influence inside of the screen
• No pointer required, thus offering the ability for lower latency.  Every tracking system has a smoothing algorithm to provide accurate data. In this instance, however, we aren’t rendering an actual pointer since the user’s hand becomes the pointer.  Therefore, using “behind-the-scenes” calculations we were able to remove a lot of the smoothing thus eliminating latency in the application.

There are, however, a few things to keep in mind:

• First, it requires rendering of a constant full screen video stream at 60 fps, which can have impact on performance.
• This one is sneaky: what part of a user’s hand is considered the selector?  Is it your index finger?  The palm of your hand?  More about this topic in an upcoming post.
• Pay attention to the details: we didn’t use a classic augmented reality hand.  Instead we created almost a silhouette of the hand by cutting the hand out of the background.   Rather than seeing a user’s face and arm, all you see is a subtle hand.  This doesn’t just provide a more aesthetically pleasing interface; it also reduces the distraction level for the end user, allowing them to focus more on the experience.
• Finally, you will have to design the application so that the hand is visible in all circumstances. For example, when the hand is behind an element (i.e., a button or menu) it becomes obscured and the feedback is lost.  Alternatively, if you render the hand above everything else you run the risk of blocking elements on screen (not to mention the fact that it is strange to interact with elements that are behind your hand). We addressed this issue by rendering the hand twice – one rendered opaque in the background and one as an outline in the foreground, thus solving both problems.

In Brief…

Gesture Recognition is an amazing technology that allows the user to interact with devices on his own terms. But it is an entirely new paradigm that requires a different approach to the design and feedback systems.  If you simply extend traditional approaches that work well for a different modality (say, touch), you’ll find almost always that it doesn’t fit for gesture.

“Feedback” in a gesture-based system should be subtle yet constant, informative yet fun, and always intuitive.

Gestural interfaces and 3D sensors offer us new way of interaction with machines, computers and applications. As designers we need to keep in mind those do’s and don’ts in order to create clear and responsive feedback systems.

Part 1: The Evolution of the Arc Menu

In the first article of the series, Omek UX Studio’s Creative Director Yaron Yanai and Lead Designer Shachar Oz talk about designing the application’s menu system called the Arc Menu.

Jump directly to Part 2, featuring how we used Augmented Reality to provide “feedback” to users.

The final product: a virtual bookshelf you can interact with using your hands

It is the mission of Omek’s UX Studio to explore new ways of interaction with computers using gesture and motion control. UX Studio team members research user experiences as the technology is being developed in order to inspire developers and create better tools for using this exciting new means of control.

For CES 2013 we decided to create a demo that shows off a 3D content browser: essentially, being able to view your books, music and pictures as three dimensional models. You can “pick them up”, look at them from all sides, open them and compare them – all using natural gestures, with no teaching required. The demo we created shows how to use this application for one’s own library of books, this can easily be extended to an online retail application, providing customers the ability to “try out” products virtually, right in their own home. Consumers can “try out” actual items, compare them with similar items, and then purchase them.

The “Practical UI” as we’ve called it, was built from the ground up with the intention of deploying gesture recognition to control every aspect of the app. The ease of use of the tool is the result of several months of development and user testing.

Making Sense of Gestures

At the start of designing the application, we defined several interactions we planned to create:

1. Menu Navigation and selection: navigating between different collections
2. Collection Navigation: navigating in 3D once inside a collection, i.e., panning and zooming
3. Object Selection and Manipulation: Picking up an object, pulling it closer to see more details, rotating it in 3D to view it from every angle
4. Object Comparison: Picking up two objects, one in each hand, and compare them visually by rotating each of them
5. Player: Looking inside an object, i.e., opening a book, playing a record

The challenge

Create an application with the functionality to perform all of the gestures and interactions listed above in an intuitive and comfortable fashion. Ensure that we are always providing a responsive, engaging, convenient, and most of all, fun, experience. And finally, showcase the potential of gesture recognition to enable a more dynamic interface with better control of three dimensional objects in a virtual world.

When we began the design process, we leveraged the Studio’s vast experience of building gesture-based applications to ensure we avoided a few of the classic pitfalls people make when translating standard mouse + keyboard or touch paradigms to a 3D environment.

• Boundaries: The user must always know whether or not she is being tracked, i.e., being “seen” by the camera
• Location feedback: Provide constant feedback on where a user is located so they can know how & where to move in order to get to their desired selection.
  • One simple option is to place a cursor on the screen the way a regular mouse does, and have it follow the user’s finger. There are a lot of disadvantages to this method, however, because it requires the user to be very accurate leading to increased fatigue and frustration.
• Item selection: How does a user make a selection? There are several methods for selection using gestures, however a simple “click” is not one of them since there’s no button to click.
• Fatigue, responsiveness and accuracy: We found out that these mechanisms are closely tied together and fatigue, being one of the major drawbacks of this technology, is probably the problem we spend the most time on.

This first article in our series touches on our approach to designing an engaging, easy-to-user menu system, we’ve called: The Arc Menu.

The Menu System

The first challenges we addressed were Boundaries and Location Feedback. Often, boundaries are defined by a small “live feed” frame at the bottom of the screen and an “out of boundaries” alert when the user approached the edge of the field of view of the camera.

This time, however, we decided to try something new: we stretched the camera’s feed (the depth data) to fill the entire screen, so that it became the background of the entire application. This way the user is receiving real-time feedback on whether his hand is inside the FOV of the camera. It acts as if the application is a mirror of the user’s hands. When we tested it out, people’s reactions were very positive. We ended up liking it so much that we let it shape many other aspects of the application.

Figure 1: Stretching the live feed over the entire screen

We decided to give the application an augmented reality feel, where the hand itself becomes the “pointer” instead of a traditional cursor. This required the creation of a specialized tracking system to “understand” what the user is pointing at. The application needed to support most hand configuration scenarios in order to keep the application intuitive; we designed for:

• index finger pointing
• full hand pointing
• middle finger pointing
• And more!

While we were testing the input system, we started designing the menu. We started out with a four item menu system: 1) Books, 2) Music, 3) Photos and 4) Friends. We kept in mind a few key learnings from our previous experiences building menu systems designed for long range environments:

• Buttons must be relatively large in order to be selected easily
• Menu selections must be placed sufficiently far apart to avoid false selection
• Menu buttons must enlarge on hover in order to avoid flickering at the edges

We started off with a simple design: a linear horizontal distribution, with four buttons spread across the width of the screen.

Figure 2: Horizontal Menu

So, what did we find out from our initial user testing?

The good:

• The Augmented Reality style selection worked great for all of the test subjects in terms of intuitiveness and responsiveness. There was no need to explain how to use the interface since a user could immediately see their hand and whether it was being tracked. There was almost zero latency since the hand itself was the cursor and there was no need for accuracy since the buttons were big and you didn’t need to have a tiny cursor point at a specific point

The not-so-good:

• Fatigue was high, even though the movements are relatively small
• When a user moved her hand in a horizontal line across the screen, her hand would obscure parts of the menu and the screen
• Right-handed users found it difficult to select items on the far left side of the screen

Small, incremental changes sometimes aren’t enough.

We started off by making small changes to the design to tackle the issues we identified during user testing.  First, we limited the menu to the right half of the screen only.  User testing showed that the experience was slightly better but not good enough.

We realized that the best way to deal with fatigue was to enable users to rest their elbow on a table or the arm of their chair. Our tests showed that for the first two buttons on the right side, the users had great results. Fatigue was minimal even after several minutes of interaction. To reach the two buttons on the left, however, the users had to raise their elbow, which brought back the problem of fatigue. This proved to be true even when we put all four buttons in a stack formation or a square formation. We kept coming up against the issue that only some of the points were convenient to select without having to bend the wrist or elbow in an uncomfortable way.

Build Upon a User’s natural movements.

We took a step away from the computer and just observed the natural movements of our bodies.  We had a few test users sit comfortably in front of their computer and had them move their hands horizontally and vertically, without lifting their elbow from their desk.  Visualizing these movements as lines it quickly became clear that the joints in our body don’t naturally move in a straight line, but rather as an arc.  So why not build the menu into the shape of an arc?

Figure 3: Visualization of users’ natural hand movements

We gathered a wide set of examples of hand movement “arcs” from people of all sizes in order to create a “standard” Arc Menu that would work across a wide range of users.

We tested the new Arc Menu and the results were especially positive. The interaction was intuitive and fun, while fatigue was low even after several minutes of continued use. All of the buttons on the menu were equally accessible and it worked perfectly with the input system.

Finishing Touches

To ensure an elegant experience, we designed the application so that the arc menu only appears when needed. We accomplished this by folding the arc into a single button on the top right that unfolds automatically when a user hovers over it.

Figure 4: Final design of the arc menu

What’s next for the Arc Menu?

1. Extend the experience to more than 4 buttons
2. Make the arc size adjustable according to the screen’s size or even user preferences
3. Make the arc flip for left-handed people

Conclusion

Close range interaction is very different from long range gesture-based experiences. Although the tracking is much more precise and responsive, we still face similar issues, such as fatigue.  And in close-range, these issues are often felt immediately and get worse over time.

When the user rests his elbow on a table or arm, fatigue is much lower and interaction can last minutes or even more. This however limits the mobility of the user’s hand to pivot around the elbow. All of this information led us to create an arc-like menu which was not only an answer to a problem but actually proved as a very useful and even fun experience extending the amount of time the user can work in front of the computer.

And if you’re interested in signing up for our upcoming Grasp beta…just click on the link below.

Thanks and stay tuned for the next chapter.

The applicability of gesture technology is wide-reaching and our meetings included representatives from a diverse set of industries, many of which also showcased gesture-based applications on the show floor:

• Personal Computing Devices such as All-In-One computers and laptops are having gesture added to create more intuitive and hands-free interaction with our devices, and an optimal Windows 8 experience

• Automotive companies are already showing prototypes of vehicles with gesture-based systems incorporated into the dashboard and beyond.
  • Read about what the car of the future will look like according to the Sunday Times Driving, with a shout-out to Omek
• Smart Televisions with gesture built right in for the next generation of control
• Casinos and Gaming companies seek ways to excite and delight their customers and are finding gesture as a great way to engage players

We didn’t have a booth this year so in order to get a peek at our latest demos folks had to have scheduled a meeting with us beforehand. But what surprised us the most during our days at CES was the number of walk-ins we had! Luckily our suite included an extra room so we were able to accommodate just about everyone who came by, but it made for some interesting juggling.

This year we saw a number of 3D hardware options featured at CES, including the pmd CamBoard Nano, the PrimeSense Capri, the Leap Motion and the SoftKinetic-Creative camera. Our team in Israel is diligently working on developing our Grasp close-range software to support the various sensors that will become available in the coming months.

Want in? Be sure to sign up to be notified of our upcoming Grasp beta release here.

“Omek’s gesture control demo was easily the most impressive at the show”

We received great feedback on our demos and are working away on creating additional videos so those who didn’t make it to CES can check out what we’re working on. In the meantime, you can check out a couple of teaser trailers below showcasing how we believe gesture and motion tracking can transform the user experience to one that is more natural, engaging and fluid.

We would like to extend a big thank you to all of the people who visited us and to the journalists who featured us! We continue to be energized and excited about the opportunities that lay ahead.

You can check out a few of the nice words written about us in various publications on our News page. HZG9WZN9KCGZ

It’s been a very busy year for us at Omek and we’ve made great strides in continuing to develop and release gesture recognition software and developer tools to create new, touchless experiences. With CES 2013 right around the corner we couldn’t be more excited about the opportunity to showcase the results of our efforts.

We continue to be inspired by the market opportunities and consistent interest in close range interaction. And Omek’s Grasp solution is groundbreaking when it comes to addressing consumer and customer needs. At CES 2013, we will be there to demonstrate responsive, robust, accurate hand and figure tracking like you haven’t seen before! In addition, we will be presenting our demos which showcase the deep research and learnings we’ve gleaned on how to create best-in-class experiences using gesture for a broad set of industries. Come check out our solution for intuitive control of Windows 8 with just a wave of your hand! Or experience virtual 3D control and modeling when you develop clay using just your hands. The opportunities are limitless.

If you want to see this and much more, book a meeting with us today. Call us at +972-72-245-2424 or email info@omekinteractive.com. We look forward to seeing you in Vegas!

2D RGB cameras are now a standard component in nearly every laptop or all-in-one computer, and have a lower price point than 3D sensors. But, while they are well-suited for capturing images and video for recording or display, they have limitations when applied to the development of gesture-based systems.

At Omek, our goal is to enable the creation of natural and intuitive applications that incorporate motion tracking and gesture recognition. 3D sensors offer tangible advantages to reach this objective. For example, they can perform under a wide array of environmental and lighting conditions. 3D cameras also have the ability to detect fine, nuanced movements. Finally, with the cost of these sensors falling, it’s likely that consumers will start seeing these cameras appear in their personal computing devices as early as 2013.

The rest of this post will highlight a few of the key challenges that arise when we attempt to translate “human vision” into “computer vision” and how 3D sensors are better equipped to handle these issues.

RGB vs. Depth Images
Let’s start off by defining how a depth sensor differs from its 2D counterpart.

An RGB camera provides a 2D image that depicts the color of each point within its field-of-view. A depth sensor provides a 3D image that reports the distance from the sensor of each point within its field-of-view. It provides additional data on the “z” coordinate, or, in other words, on the distance between the camera and every point within the field of view.

Here’s an example of the same image, rendered both as a color image (on the left) and as a depth image (on the right). In the depth image, lighter shades represent points closer to the camera, and darker shades represent points that are farther from the camera.

A depth image can be used to construct a 3D model of its subject. In the case of a gesture-enabled application, this will usually be a 3D model of a hand or a person to enable them to control an application. This 3D model is necessary both for 3D display and for gesture recognition.

Translating Human Vision into Computer Vision
Seeing and comprehending what we see, comes so naturally to humans, that we often overlook what a subtle and sophisticated task it is. And translating “human vision” into “computer vision” is quite complex.

However, 3D depth sensors are much better equipped than their 2D counterparts at overcoming some of the inherent challenges found in computer vision.  We’ll highlight a few of them here.

Separating the “Object” from its Background
In the context of computer vision, we will be talking about “objects” – i.e., whatever it is we are tracking.  In the case of Grasp, it is hands and fingers, while with Beckon we are tracking a full human skeleton.

The first task a computer vision-based application performs in order to interpret objects is to differentiate between the object itself and the “background”. If an application relies solely on a 2D color image in order to find an object’s contour, it can be “confused” if there are similar colors in both the object and the background.

In the example above, you can see that in the color image, the man’s white sweater merges into the white background, making separation of his upper body from the background difficult. On the other hand, using the depth image on the right, this separation becomes a trivial task.

Blocking
A related problem is when one object or part of an object blocks another object from view. On the right you can see an example of this, when the person’s arm moves in front of his body.  Similar colors and textures can make it hard to separate the two objects. Using depth data, however, the software can approach this challenge much more easily.

Differences in Lighting
Different light intensities and angles can radically change the appearance of an object, as demonstrated in the example below – it’s the same face, photographed under different lighting conditions.

This can contribute to making it difficult to separate objects from their background.  Or an application may not correctly identify the same object under different lighting conditions. It can also cause parts of the object to be missed altogether, as in the left part of the person’s face in the left-hand image.

This is a case where it would also be difficult to collect depth data, unless the sensor has an independent light source. As a result, most 3D sensors come with their own light source, usually infrared light.

Interpreting Size
Interpreting the size of an object is nearly impossible, based on a 2D image alone. Small objects that are close to the camera may appear the same as large objects that are far from the camera.

Since a depth image provides precise distance values for each point on the object, the object’s size can be easily extrapolated from the depth image. Size interpretation is important in order to differentiate between users and track each user continuously. For instance, a child who is close to the sensor cannot be confused with an adult who is far from the sensor.

Handling Object Orientation and Motion
While an application is tracking a non-stationary object, the object may change both its position and its orientation in relation to the camera, sometimes abruptly. The application must still identify the object as the same one that was tracked before the change.

This is especially true for human bodies, which may change posture as well as position and orientation.  By matching a theoretical 3D model (for example, a skeleton of a hand) to real-time 3D position data, an application can maintain a continuous representation of a moving object.

Summary
Although 2D color cameras are sufficient to provide some types of visual input, at Omek we believe that 3D sensors have significant advantages in the domain of motion tracking and gesture recognition.  Omek products use 3D depth sensors because we’re convinced they enable superior motion tracking and gesture recognition enabling you to create natural and engaging user interfaces and applications.

The C# Extension hides low-level details and allows you to perform complex management functions with minimal code. The bottom-line result is shorter, more readable code, flexible application control, and faster development.
Here are some examples of high-level management features in the Beckon C# Extension:

• Player Selection Strategy – and by “player” we mean any person identified as controlling your Natural User Interface application. The C# Extension allows you to configure the automatic strategy that determines how “players” are selected. This could be according to people’s distance from the sensor, whether people control application pointers, or whether people perform a certain trigger gesture.
• Pointer Management – the C# Extension provides convenient classes for mapping people’s hands to pointers that control your application. You can also determine the number of pointers, and set the strategy for choosing the player or players who control pointers.
• Monitoring Person Location – the C# Extension lets you monitor the location of your players according to areas that you define on the “playground”. See an example in this picture. You can determine the green area as the desired location for person tracking, and produce a warning if a person enters the red area or beyond.

But sometimes it’s the little things that make a difference. Here are a couple of examples of C# Extension features that will shorten your code and your debug cycles:

• C# Events – instead of adding code to poll for alert and gesture events, the C# Extension supports registering C# Event objects for handling events via callback.
• Central Configuration – the C# Extension provides a central configuration object where you can configure all Beckon session parameters. You can also read configuration from an external text file, so you can reconfigure your session without recompiling your code.

We hope you’re convinced! If you’re developing a .Net application, the easiest way to access Beckon functionality is by using the Beckon C# Extension.