Clickable UI element feature

Application dialogs are created with visual UI elements, such as buttons, display areas, sliders etc.

If applications conform to certain Apple development guidelines, then the Accessibility API can be used to get a list of these UI elements, their attributes and locations within the dialog.

Commander makes use of these UI elements in two ways;

Directing a click gesture to the nearest clickable control, allowing for accurate selection of small controls, as described in this document.
Determining if the gestures is being performed over an area within the dialog that might have a specific gestures associated with it, such as a scroll area, where a drag gesture can be configured to scroll rather than perform a "click and drag" action. This usage is described here.

In some applications, the accessibility API will stall if the app does not respond with a list of UI elements and times out after 250 ms. This will cause latency issues with touches within the application. Click UI element processing will have to be disabled in these cases, as discussed below.

Enabling Click Nearest feature

As of Commander version 1.4.1, Sep 22, Disabling UI elements for an app or app group prevents the 'click nearest' feature from being used within that app". The reason for this change is that, since disabling UI elements in an app is generally about fixing latency issues related to Commander using the Accessibility API, then it would make no sense to allow the "click nearest" feature in an app that has UI elements disabled since it makes even heavier use of the Accessibility API.

Some MacOS controls are small and can be difficult to accurately select with a touch screen, especially when using a large stylus such as a finger, so we have introduced an option for appropriate click actions to locate the nearest clickable UI element within a defined pixel radius of the point of touch and click it.

This option is defined at both individual click action definitions and as a global setting to set the default value for all click actions.

Gesture defining Click action

Advanced Options

This feature is enabled by default as it is a very useful feature and improves touch usability, especially when dealing with small controls not designed for touch interface.

However, please note with this enabled, the gesture software deliberately redirect tap and press touch locations that occur near a UI control element.

This option is very useful for selecting small UI controls difficult to accurately touch, such as the dialog close, minimise and maximise UI buttons, as shown, and similar desktop and application controls:

Notes when using this feature:

This feature will activate any GUI button within a certain radius, even under the dialog being touched, and this behaviour is by design. The click UI feature will consider every UI element within a radius of the touch, regardless of what window it is in and what window contains the touch. This caters for the situation whereby a click is required on a button near the edge of a window, like for example the close or minimise buttons. If the nearest UI element is restricted to only the window underneath of the actual touch, then touching just slightly above the top edge won't result in any of those buttons getting clicked, and it's pretty easy to miss. We understand that not restricting the search to within the touched window can sometimes yield undesirable results. Further, GUI buttons can be enabled or disabled and Commander restricts clicks to enabled buttons only. In our tests, this feature really improved the touch experience and makes the selection of small buttons and UI elements a lot easier., hence this feature is enabled by default.
We believe there are a few improvements we could make which may come in future releases. Specifically, this feature doesn't work as well when browsing the web in Safari, as Safari reports practically every element in the browser window as being clickable, even when they aren't. We think we could improve its detection of ‘real’ clickable items to make it work properly in Safari, including finding text links.
We also briefly tested in Logic Pro X and found that it works partially as Logic Pro X supports the Accessibility API, but it reports odd information about some of its UI elements, similar to Safari. We think we could code round these issues and may do so in a future release.
This feature is searching for the closest clickable UI element using the Accessibility API but in some cases there can be a considerable delay in some applications reporting the UI information. This was especially observed in the case with many games, including those created with Cocos2D-X. To cater for this we have added a timeout waiting for the information and by default it will time out after 250 ms. For applications that are slow to respond to UI element queries, you can set UI element detection disabled.
If searching for a nearest GUI element on an application that is totally unresponsive or frozen (perhaps because it crashed), then the request will never be fulfilled. Apple's Accessibility API gives up after six seconds, but in that time Commander will be frozen and unresponsive.
Finally, this feature won't readily work in any applications that do not support Apple's accessibility API, such as Chrome, Firefox, and Qt applications build using a Qt version prior to 5.2. It'll only recognize a small subset of their UI elements, like the menu bar.

Disabling Click Nearest Feature

Given that this setting, although very useful, can distort the point of touch, or cause a delay to touch if the application does not respond to the UI Query, it may be desirable to disable this feature.

The Commander log file will indicate of the application does not respond to queries about its UI elements fast enough for UC to be able to detect them thus causing a delay:

> 2021-10-24 11:53:44.946 Error: UI element task timed out
> 2021-10-24 11:53:44.946 Error: applicable gesture search timed out

> 2021-10-24 11:53:44.946 Error: Warning: search for nearest clickable UI element took too long"

It can be disabled at the system system-wide level in the Touch Gestures, setting dialog or at an individual application level.

Application level

At an application level you have two options, either disable the Clickable UI Element function for a given gesture or, if the function is timing out, disable the function for the entire application:

Disable for a specific gesture

Add the application to UPDD Commander
With the app selected, find any gestures that have "click" as their assigned action, especially "One Finger Tap"
If these gestures are inherited, double click them to copy them into the selected application
Expand the gestures and click the "Advanced Options" button
Check "Override application wide-settings" and then uncheck "Click the nearest clickable UI element

Disable for the entire application

Add the application to UPDD Commander
With the app selected, select the application settings and disable the feature: