Envisioning an open source multi touch table top surface window manager

May 10th, 2012

Lately there’s been some some excitement about the new Samsung SUR40 table which ships with the latest version of Microsoft Surface. Cool piece of hardware indeed, if I could afford it I would put it into my wish list. Another device, another SDK, another user experience. Am I the only person who is getting sick of porting applications for different platforms?

For the past 5 years the “apps” world has taken over. We have created sophisticated games and applications for desktops, consoles and now we are spending so much time porting all of these applications to the coolest new device. First it was the iPhone, then Android, then tablets arrived, oh my, my textures are not big enough, the screen is a different resolution, this device has retina display, this one supports a compass, this one doesn’t, now there’s Apple TV, hey, Windows Phone 7 just arrived, learn one more SDK… this will happen once more with Surface. Yes, at the core it runs on Windows, but will still need to learn to use a Surface SDK.

As a way to develop software, is this sustainable? I mean, how many more devices will we encounter in the next few years? How many more different APIs will a developer have to learn to get an application to run on multiple systems?

Some people argue that web apps are the answer and I agree with them for the most part. A native layer (doesn’t matter of small) is always needed in a platform, but I think it should be as small as possible. Making web apps using HTML5, a Javascript (or Dart?) Framework and CSS is nice! Dealing with C++, Objective-C, or even C# in comparison is much more difficult! How do you deal with things like accelerometer, compass, gps, etc? Create an API layer that you can invoke from Javascript, so that you can call native functions through a scripting layer. Phonegap does it already.

Google is going in the right direction with ChromeOS and Chrome. It’s going to take some time before web browsers catch to speed with things like WebGL and full HTML5 support, but it’s going to happen eventually! Even Mozilla with Boot2Gecko is going the right direction, and is the clear example that a functional (I didn’t say fully functional, yet) device can be built on top of a browser layer.

Can the same idea (browser layer + small native layer on top) be applied to table top surfaces? I say yes.

What is the issue with a table top device? There’s no up, down, left or right orientation. Every window must be able to be displayed in any orientation, because multiple people might be interacting from different sides of the table. A small native layer must take care of handling rotation, transformation and translation of “windows” across the display. Once that’s done (along with a few other tweaks), everything (system menus, widgets, applications, configuration panels) can be designed at the browser level. A small Javascript API is injected into the browser level to call functions like CloseWindow or OpenNewWindow and the result is quite impressive.

This application I developed runs on Linux as well as Windows (and with some tweaking should work on Mac too). It supports the TUIO protocol (which is a standard for multi touch devices) and runs on 100% free software. I plan to release an alpha version of this program (I haven’t chosen a definitive name for it yet) sometimes late during this summer. In this screenshot you can see a several applications running at the same time, none of which took me longer than a few minutes to load onto the platform because they were already working on a regular web browser.

Here you can see an instance of YouTube as well as a view of the menu I created. The nice thing about the whole application is that you are not stuck with one UI. If you know HTML and CSS you can make changes to the whole UI in a few minutes.

Browser layer + small native layer. If we are going to make development for devices easy and enjoyable, I think this is the right direction. Stay tuned for updates on this project!