Usability

Apple’s new Next Great Interface

Posted on January 13th, 2012 by admin

3D, Innovation, News, Software, Technology, Usability

—

On January 12, 2012, the US Patent & Trademark Office published a patent application from Apple that reveals an exciting new 3D GUI for iOS mobile devices. The new UI will work with proximity sensor arrays and will respond to hovering gestures. The Crazy Ones in Cupertino have been working on advanced 3D GUIs for some time now. We first learned of a major 3D GUI project back in 2009 that involved using head tracking technology. Then in early 2010 we learned of Apple’s first project relating to a 3D GUI for iOS devices. Later that year Apple 3D multifunctional widgets and over time revealed advanced 3D and hovering based gesturing for CAD users on an iPad. With twenty times the GPU power coming to iOS devices over the next year, Apple appears to paving the way for a new 3D GUI for mobile devices in the not-too-distant future.

Apple’s Patent Background

Modern computer operating systems often provide a desktop graphical user interface (“GUI”) for displaying various graphical objects. Some examples of graphical objects include windows, taskbars, docks, menus and various icons for representing documents, folders and applications. A user could interact with the desktop using a mouse, trackball, track pad or other known pointing device. If the GUI is touch sensitive, then a stylus or one or more fingers could be used to interact with the desktop. A desktop GUI could be two-dimensional (“2D”) or three-dimensional (“3D”).

Modern mobile devices typically include a variety of onboard sensors for sensing the orientation of the mobile device with respect to a reference coordinate frame. For example, a graphics processor on the mobile device could display a GUI in landscape mode or portrait mode based on the orientation of the mobile device. Due to the limited size of the typical display of a mobile device, a 3D GUI could be difficult to navigate using conventional means, such as a finger or stylus. For example, to view different perspectives of the 3D GUI, two hands are often needed: one hand to hold the mobile device and the other hand to manipulate the GUI into a new 3D perspective.

Apple’s Wild Solution

Apple’s latest invention is a wild one indeed. The invention covers a 3D display environment for mobile device that uses orientation data from one or more onboard sensors to automatically determine and display a perspective projection of the 3D display environment based on the orientation data without the user physically interacting with (e.g., touching) the display. In some implementations, the display environment could be changed based on gestures made a distance above a touch sensitive display that incorporates proximity sensor arrays.

In some implementations, a computer-implemented method is performed by one or more processors onboard a handheld mobile device. The method includes generating a 3D display environment; receiving first sensor data from one or more sensors onboard the mobile device, where the first sensor data is indicative of a first orientation of the mobile device relative to a reference coordinate frame; determining a first perspective projection of the 3D display environment based on the first sensor data; displaying the first perspective projection of the 3D display environment on a display of the mobile device; receiving second sensor data from the one or more sensors onboard the mobile device, where the second sensor data is indicative of a second orientation of the mobile device relative to the reference coordinate frame; determining a second perspective projection of the 3D display environment based on the second sensor data; and displaying the second perspective projection of the 3D display environment on the display of the mobile device.

Apple’s 3D Environment for Mobile Devices

Apple’s patent FIG. 1A shown below illustrates a perspective projection in a 3D display environment of a mobile device using orientation data derived from onboard sensors.

In some implementations, the mobile device includes display 102 presenting display environment 104. For example, the mobile device could include a touch-sensitive surface that could receive touch input and gestures. The display environment could be a 3D GUI having a floor (106), back wall (108), ceiling (110) and sidewalls (112a and 112b). One or more two-dimensional (“2D”) or 3D objects could be presented in the display environment. In the example shown, 2D object (118) is pasted to the back wall and 3D objects (114 and 116) are resting the on floor.

Apple’s proposed display environment could be a perspective projection of a 2D desktop or homepage for an operating system or application. The perspective projection could be conceptually understood to be the view from an imaginary camera viewfinder. The camera’s position, orientation and field of view could control the behavior of a projection transformation or camera transform.

Shifting Camera Views

Apple states that the “camera view” of the display environment changes when a viewer rotates their mobile device about a gyro sensor axis. If no gyro sensor is available another sensor such as the magnetometer or accelerometer could be used.

In some implementations, the display environment is determined based on the current orientation of the mobile device as indicated by one or more onboard sensors. For example, the display environment could be displayed when the viewer holds their mobile device with the display directly facing them. In this orientation, the camera view is directly facing the back wall. As the user rotates their mobile device either clockwise or counterclockwise, the camera view is moved towards either side wall.

More particularly, as the user rotates their mobile device clockwise (about the X axis of rotation), the camera view moves toward sidewall 112a. Likewise, as the user rotates their mobile device counterclockwise, the camera view moves toward sidewall 112b. As the user rotates their mobile device clockwise (about the Y axis of rotation), the camera view moves toward the ceiling. As the user rotates their mobile device counterclockwise (about the Y axis), the camera view moves toward floor.

By rotating mobile device about the X and/or Y axes, the viewer is easily able to change the view of their display environment as desired to view various icons or documents as presented on the display.

Angular rotation about the X and Y axes could be measured or estimated from data provided by gyro sensors, accelerometers, magnetometers or any combination of sensor data that could provide an estimate of the orientation of the mobile device relative to a reference axis of rotation.

Apple Introduces a “Snap To” Feature

Apple’s patent FIG. 1B shown below illustrates a perspective projection in a 3D display environment using orientation data derived from onboard sensors, and including a “snap to” feature for changing camera position in the display environment. Note what was on the left wall of patent figure 1A is now positioned in the center back wall of the 3D environment in patent figure 1B below. As the user rotates their device, a new view of their display environment could be seen and snapped into place so that they could interact with it.

When using the mobile device, there are physical constraints on the amount of any particular rotation. If the viewer rotates their mobile device about the X or Y axis too far, the viewer will no longer see their display environment 104.

Exemplary Process

Apple’s patent FIG. 2 shown below is a flow diagram of an exemplary process 200 for determining perspective projections in a 3D display environment using orientation data derived from onboard sensors.

Multiple GPUs and a 3D Graphics Rendering Engine

In some implementations, the process could begin by generating a 3D display environment for a display of a mobile device. The 3D display environment could be displayed using one or more graphics processing units (e.g., NVIDIA GeForce 330M) and a 3D graphics rendering engine, such as the open source object-oriented graphics rendering engine (“OGRE“).

Yet a development this week may prove to be the more likely GPU candidate to power Apple’s proposed 3D GUI for mobile devices. At the Consumer Electronics Show in Vegas this week, Imagination Technologies presented their new PowerVR G6200 and G6400 GPU IP cores – which may be able to offer up to 20 times the performance of smartphones currently on the market today. We discussed advances coming from Imagination Technologies in our December 12, 2011 report titled “Apple Pounds out a New Round of Camera Related Patents.” Being that today’s 3D GUI patent relies heavily on utilizing the cameras built-into Apple’s iOS devices, the developments presented in Vegas this week are very relevant.

3D Environments for a Full Range of Devices

Some examples of mobile devices that will be able to implement Apple’s proposed 3D environment include but are not limited to: a handheld computer, iPod touch (a personal digital assistant), an iPhone, an iPod, iPad, network appliance, a camera, an enhanced general packet radio service (EGPRS) mobile phone, a network base station, an iPod, a navigation device, an email device, a game console, or a combination of any two or more of these data processing devices or other data processing devices.

The War over the Next Great Interface will be with us for a Decade

In October 2011, Microsoft revealed their upcoming Metro UI for portable devices and made a pitch at how it would be so much better than Apple’s use of static icons. Well, just when Apple knew that Microsoft would be trying to copy iOS in some fashion, we now see Apple’s revelation of how they may take a future iteration of iOS to the next level. Time will tell of course if Apple gets this 3D GUI out to market in a timely manner. But at the moment, it looks very promising indeed. In the bigger picture, the war over the next great interface for larger tablets, TVs and our vehicles are still up for grabs. But between Apple’s advanced work on 3D graphical user interfaces as well as audio-centric user interfaces like Siri, it sure looks like Apple has the wind at its back.

Apple’s patent applicationwas originally filed in Q3 2010 by inventor Patrick Piemonte. For more on Apple’s 3D related patent applications, see our Archives.

Other Noteworthy Patent Applications Published Today

Apple has designed and invented a new power adapter with an arm mechanism that is manufactured from a single piece of material. The plug arm could include a plug operative to extend into a wall socket, an elongated plate coupled to an end of the plug such that the plug extends from a first surface of one end of the plate, and a pin coupled to the opposite end of the plate and extending from the opposite surface of the plate.

That’s a very nice description from Apple, but it’s still an odd power adapter to me. Yet perhaps the super geeks amongst us could make sense of this (ha!) – and let us know below in our comment section.

For more information, see patent application 20120005897.

[via Patently Apple]

— Read more

Top Interaction design books and links

Posted on November 29th, 2011 by admin

Interaction, Interactive, Usability, UX

No Comments

—

Here are some recommendations for books and links that should be essential reading for everyone interested in interaction design, visual design, usability, and much more. Feel free to comment, add and criticize!

BILL MOOGRIDGE
William (Bill) Moggridge, an industrial and interaction designer, is co-founder of the Silicon Valley-based design firm IDEO and the current director of the Smithsonian Institution’s Cooper-Hewitt, National Design Museum in New York. He designed what was the first laptop computer, the GRiD Compass. He advocates applying a user-centered design process in product development cycles and also works towards popularizing interaction design as a mainstream discipline. Read his books Designing Interactions and Designing Media

BILL BUXTON
William Arthur Stewart (Bill) Buxton is a Canadian computer scientist and designer. He is currently a Principal researcher at Microsoft Research. He is known for being one of the pioneers in the human–computer interaction field. Read his book Sketching User Experiences and visit his website Billbuxton.com

BOOKS:
About Face 3
This completely updated volume presents the effective and practical tools you need to design great desktop applications, Web 2.0 sites, and mobile devices. You’ll learn the principles of good product behavior and gain an understanding of Cooper’s Goal-Directed Design method, which involves everything from conducting user research to defining your product using personas and scenarios. Ultimately, you’ll acquire the knowledge to design the best possible digital products and services. Buy this book on Amazon

Simple and Usable
In a complex world, products that are easy to use win favor with consumers. This is the first book on the topic of simplicity aimed specifically at interaction designers. It shows how to drill down and simplify user experiences when designing digital tools and applications. It begins by explaining why simplicity is attractive, explores the laws of simplicity, and presents proven strategies for achieving simplicity. Remove, hide, organize and displace become guidelines for designers, who learn simplicity by seeing before and after examples and case studies where the results speak for themselves. Buy this book on Amazon

User-Centered Interaction Design Patterns for Interactive Digital Television Applications
Technology is meant to make life easier and to raise its quality. Our interaction with technology should be designed according to human needs instead of us being required to adapt to technology. Even so, technology may change quickly and people and their habits change slowly. With the aim of supporting user acceptance of iTV, the focus of this book is on the usability of iTV applications. A method for developing interaction design patterns especially for new technologies is presented for the first time. The main characteristics covered in this new approach are: systematic identification of recurrent design problems; usability as a quality criterion for design solutions; integration of designers into the pattern development process including identification of designers’ needs, and iterative evaluation and optimisation of patterns to encourage designers to accept and use them; usability testing to identify proven design solutions and their trade-offs; presentation of specific design guidelines. Buy this book on Amazon

Don’t Make Me Think
Five years and more than 100,000 copies after it was first published, it’s hard to imagine anyone working in Web design who hasn’t read Steve Krug’s “instant classic” on Web usability, but people are still discovering it every day. In this second edition, Steve adds three new chapters in the same style as the original: wry and entertaining, yet loaded with insights and practical advice for novice and veteran alike. Don’t be surprised if it completely changes the way you think about Web design. With these three new chapters:

Usability as common courtesy — Why people really leave Web sites
Web Accessibility, CSS, and you — Making sites usable and accessible
Help! My boss wants me to ______. — Surviving executive design whims

Buy this book on Amazon

The Inmates Are Running the Asylum
Imagine, at a terrifyingly aggressive rate, everything you regularly use is being equipped with computer technology. Think about your phone, cameras, cars-everything-being automated and programmed by people who in their rush to accept the many benefits of the silicon chip, have abdicated their responsibility to make these products easy to use. The Inmates Are Running the Asylum argues that the business executives who make the decisions to develop these products are not the ones in control of the technology used to create them. Insightful and entertaining, The Inmates Are Running the Asylum uses the author’s experiences in corporate America to illustrate how talented people continuously design bad software-based products and why we need technology to work the way average people think. Somewhere out there is a happy medium that makes these types of products both user and bottom-line friendly; this book discusses why we need to quickly find that medium. Buy this book on Amazon

LINKS
LukeW Ideation + Design
Interakt.nu (in swedish)
UX Design @ smashingmagazine
Boxes and Arrows
Use It

[Thanx to Jonas Norberg, Sai-Kit Cheung and Indir Topcagic]

— Read more

Who do you want to be today? Realtime face substitution is awesome!

Posted on September 20th, 2011 by admin

Animation, Campaign, Creativity, Innovation, Interactive, News, Software, Technology, Usability, Web

No Comments

—

FaceTracker is a C/C++ API for real time generic non-rigid face alignment and tracking.

Goal:
Non-rigid face alignment and tracking is a common problem in computer vision. It is the front-end to many algorithms that require registration, for example face and expression recognition. However, those working on algorithms for these higher level tasks are often unfamiliar with the tools and peculiarities regarding non-rigid registration (i.e. pure machine learning scientists, psychologists, etc.). Even those directly working on face alignment and tracking often find implementing an algorithm from published work to be a daunting task, not least because baseline code against which performance claims can be assessed does not exist. As such, the goal ofFaceTracker is to provide source code and pre-trained models that can be used out-of-the-box, for the dual purpose of:

1. Promoting the advancement of higher level inference algorithms that require registration.
2. Providing baseline code to promote quantitative improvements in face registration.

Features:
• Real time: ranging from 20-30 fps (depending on processor, compiler and use of OpenMP)
• Generic: designed to work for most people under most conditions
• No training required: a pre-trained model is provided
• Detection based initialisation: no user intervention required
• Automatic failure detection: requires no user re-initialisation
• Camera or video input

Download:
FaceTracker is available for download (for research purposes only). The library includes the C/C++ API, example code for interfacing with the API, a pre-trained model and documentation. To download it, please send an email to Jason Saragih(Jason.saragih@csiro.au).

The Tracker:
The code requires OpenCV 2.0 and the provided model was trained using the MultiPIE database. The tracker is based on a modified version of the constrained local model described in:

J. Saragih, S. Lucey and J. Cohn, “Deformable Model Fitting by Regularized Landmark Mean-Shifts”, International Journal of Computer Vision (IJCV)

People Using FaceTracker:
Some people are using FaceTracker to do some really cool stuff:

• FaceOSC
• A music Video!
• Face Projection
• Being John Malkovich
• pkmFace

Videos:

Follow this link to see some examples of difficult YouTube videos that were processed using FaceTracker with the pre-trained model. There was absolutely no user intervention in any of the videos.

[Via FaceTracker]

— Read more