Browser TUIO interface

We have been asked on a few occasions how you might utilise multi-touch gestures in a browser application under Mac OS X so they work the same as when using iOS.

The problem is that OS X doesn't pass its system-level touch events to web browsers as HTML5 / javascript touches. So while these pages will work with an iPad (and similar) it can't work in OS X, even when using Apple’s magic trackpad.

Therefore, when performing a gestures on a browser application the gestures is processed by the browser dialog and not by the browser application and therefore gestures cannot be used to control touch-enabled web pages.

However, to cater for this requirement, we have undertaken some experimental work with a browser extension to offer multi-touch within a browser that utilises WebSockets and the TUIO interface.

When using this extension with UPDD Commander it is important to switch the Browsers' configuration in Commander to utilise the TUIO configuration via the Presets option.  UPDD Commander TUIO interface has an option to dispatch TUIO events using WebSocket protocol which is also on by default.

If you are using the old stand alone UPDD TUIO server this also has an option to dispatch TUIO events using a WebSocket interface which is on by default. This change is in UPDD TUIO Server 2.2.0. This uses the same port numbers that the regular TUIO server(s) are configured to use.

The extension is available for Firefox and Chrome.

The Firefox extension was rejected from inclusion in their add-on site on the grounds that it was too specialized. The good news is that you can still install Firefox add-ons from downloaded files so the add-on is available here. Clicking the link will cause Firefox to announce that it prevented the site from asking you to install software, but you can select the "Allow" button and proceed with the installation.

When requesting this extension in Chrome you may see a warning that "this extension can read and change all your data on the websites that you visit". Chrome shows this warning because the extension has permission to modify individual web pages (i.e. by injecting a script that allows touches to work). If you have concerns you can view the source code to the extension and confirm it's not malicious. We hope that it also helps that there's a legitimate company behind this extension rather than an individual.

The extension can be configured to select which server and port number to connect to.

By default it uses 'localhost' and port 3333 which matches the default settings for UPDD TUIO which means that both UPDD TUIO and the extension should work together without any additional configuration.

In the unlikely event that you need to adjust these settings...

Chrome - First open the Extensions tab by picking More Tools > Extensions in Chrome's main menu. Click "Details" under Touch-Base TUIO Bridge. Then finally click "Extension Options".

Firefox -  Open the Extensions page for Firefox (available from its main menu) and select the Preferences button listed next to our extension.

Once installed you can test with an appropriate touch-enabled website. During testing we tested with : (no longer seems to be working)

You can touch the screen and have it paint trails that follow your touches as seen in this dual touch demo:

When using these extensions with UPDD Gestures its likely you won't want UPDD Gestures to invoke any actions from touches in the web browser since the page will be processing touches directly.

In that case it's probably better to run UPDD TUIO without Gestures running, or remove the browser from the list in UPDD Gestures and then manually disable all of the actions for gestures that will interfere with the browser.

UPDD Commander TUIO interface is configured at an application level and automatically disables gestures at the TUIO client application level.

The extension currently has a few limitations:
  • It can't distinguish between which display a touch comes from, so all touches are assumed to come from the display containing the browser window.
  • It can only receive touches from one source, meaning it's not possible to have multiple browser windows receiving touches from different touch devices simultaneously.
  • There's no option to set touches to be scaled -- all touches are processed as unscaled -- though such an option could be added fairly easily if required.