THAW: Hybrid Interactions with Phones on Computer Screens

Sang-won Leigh, Philipp Schoessler

THAW is a reinvention of smartphones as a magical hand tool that intervenes into computer screens. The phone now becomes a link between the physical and digital spaces, translating our physical gestures into exertions on digital content in larger display spaces. It acts both as a physical interface and as an additional graphics layer e.g. a physical token that lets us intervene “in” a larger screen and interact with its displayed graphics content, or a Magic Lens “through” which we can see certain information or content.

Technology: the system allows a collocated large display and small handheld devices to seamlessly work together. It tracks accurate position of a smartphone placed on or over any screen by displaying a 2D color pattern that is captured using the smartphone’s back-facing camera. The proposed technique can be implemented on existing devices without the need for additional hardware.

Created using iPhone 5s, MacBook Pro, openFrameworks.


Finalist, Experimental Category, 2015 Fast Company Innovation by Design Awards. Sep 14 2015 [Link]
Sang-won Leigh, Philipp Schoessler, Felix Heibeck, Pattie Maes, and Hiroshi Ishii. THAW: Tangible Interaction with See-Through Augmentation for Smartphones on Computer Screens, TEI 2015, 2015
Gizmodo, The 7 Most Important UI and UX Ideas of 2014, Dec 30 2014 [Link]
UIST 2014 People’s Choice Best Demo Honorable Mention, Oct 2014
Sang-won Leigh, Philipp Schoessler, Felix Heibeck, Pattie Maes, and Hiroshi Ishii. THAW: Tangible Interaction with See-Through Augmentation for Smartphones on Computer Screens, UIST 2014 Demo, 2014






We can drag and drop multiple files at once, using the phone as an advanced clipboard

Knowing the phone’s location turns it into a context aware tool. It can be used as a x-ray glass or a dial input for intuitive image editing

The phone acts as a physical boundary that a digital character can interact with

We can use the phone as a filter that alters the behavior of digital contents

A smartphone is used as a magic lens that makes underlying information visible

We can manipulate digital contents on the phone, and seamlessly exchange them between devices

The phone becomes a physical handle for controlling digital contents on the computer screen