We have been asked on a few occasions how you might utilise multi-touch gestures in a browser application under Mac OS X so they work the same as when using iOS.
The problem is that OS X doesn't pass its system-level touch events to web browsers as HTML5 / javascript touches. So while these pages will work with an iPad (and similar) it can't work in OS X, even when using Apple’s magic trackpad.
Therefore, when performing a gestures on a browser application the gestures is processed by the browser dialog and not by the browser application and therefore gestures cannot be used to control touch-enabled web pages.
However, to cater for this requirement, we have undertaken some experimental work with a browser extension to offer multi-touch within a browser that utilises WebSockets and the TUIO interface.
When using this extension with UPDD Commander it is important to switch the Browsers' configuration in Commander to utilise the TUIO configuration via the Presets option. UPDD Commander TUIO interface has an option to dispatch TUIO events using WebSocket protocol which is also on by default.
If you are using the old stand alone UPDD TUIO server this also has an option to dispatch TUIO events using a WebSocket interface which is on by default. This change is in UPDD TUIO Server 2.2.0. This uses the same port numbers that the regular TUIO server(s) are configured to use.
The extension is available for Firefox and Chrome.
The Firefox extension was rejected from inclusion in their add-on site on the grounds that it was too specialized. The good news is that you can still install Firefox add-ons from downloaded files so the add-on is available here. Clicking the link will cause Firefox to announce that it prevented the site from asking you to install software, but you can select the "Allow" button and proceed with the installation.
When requesting this extension in Chrome you may see a warning that "this extension can read and change all your data on the websites that you visit". Chrome shows this warning because the extension has permission to modify individual web pages (i.e. by injecting a script that allows touches to work). If you have concerns you can view the source code to the extension and confirm it's not malicious. We hope that it also helps that there's a legitimate company behind this extension rather than an individual.
The extension can be configured to select which server and port number to connect to.
By default it uses 'localhost' and port 3333 which matches the default settings for UPDD TUIO which means that both UPDD TUIO and the extension should work together without any additional configuration.
In the unlikely event that you need to adjust these settings...
Chrome - First open the Extensions tab by picking More Tools > Extensions in Chrome's main menu. Click "Details" under Touch-Base TUIO Bridge. Then finally click "Extension Options".
Firefox - Open the Extensions page for Firefox (available from its main menu) and select the Preferences button listed next to our extension.
Once installed you can test with an appropriate touch-enabled website. During testing we tested with :
http://borismus.github.io/mobile-web-samples/browser-ninja/
https://www.paulirish.com/demo/multi (no longer seems to be working)
You can touch the screen and have it paint trails that follow your touches as seen in this dual touch demo:
When using these extensions with the old UPDD Gestures application (superseded by UPDD Commander) its likely you won't want UPDD Gestures to invoke any actions from touches in the web browser since the page will be processing touches directly.
In that case it's probably better to run UPDD TUIO without Gestures running, or remove the browser from the list in UPDD Gestures and then manually disable all of the actions for gestures that will interfere with the browser.
UPDD Commander TUIO interface is configured at an application level and automatically disables gestures at the TUIO client application level.
The extension currently has a few limitations:
- It can't distinguish between which display a touch comes from, so all touches are assumed to come from the display containing the browser window.
- It can only receive touches from one source, meaning it's not possible to have multiple browser windows receiving touches from different touch devices simultaneously.
- There's no option to set touches to be scaled -- all touches are processed as unscaled -- though such an option could be added fairly easily if required.
Application based Browser functionality
In development environments where you might be using an embedded HTML interpreter, such as WKWebView, you may also wish to receive touch data into your application.
We are not sure what sort of security restrictions is imposed within the page when running with WKWebView, but since it's not a browser specific application we think it should be more lenient.
One possibility to receive touch events is to try using this TUIO JavaScript library linked on the main TUIO page:
https://github.com/fe9lix/Tuio.js
You could also check out the source code for our TUIO browser extension and see whether that could be adapted to work in your webpage.
The extension source is not publicly available, but it's not protected and can be downloaded and viewed from within a browser.
Multiple monitor support
A customer wanted two touch monitors to use the browser TUIO extension simultaneously. They 'Identified' both touch monitors in the driver but the TUIO extension processed all touches on monitor 1.
Unfortunately, the TUIO extension can only listen for TUIO touches from a single TUIO server, so there's no way to get it to distinguish between touches on two monitors in the same web browser.
However, what is possible is to use two web browsers, one for each monitor. We suggest using Chromium, Opera or Brave, all of which are also Chrome-based browsers, or Firefox. All of these are already configured to work with UPDD Commander browser presets. ( Safari can't be used since we don't have a TUIO extension for it.)
Whichever browser it is, install the relevant UPDD TUIO plug-in, In the TUIO extension settings configure one browser to use port 3333 and the second to use port 3334.
Now in UPDD Commander, find the "Convert touches to TUIO" gesture under Browsers / Safari and Chrome web content, click its advanced options button, and check "Configure TUIO servers separately for each device". Set one device to use port 3333 and the other to use port 3334. Now touches from each monitor should go into one browser specifically, and you can arrange the browser windows accordingly.
The customer reported this worked well.