Tag: accessibility

Accessibility in GTK 4

The big news in last weeks GTK 3.99.3 release is that we have a first non-trivial backend for our new accessibility implementation. Therefore, now is a good time to take a deeper look at accessibility in GTK 4.

Overview

Lets start with a quick review of how accessibility works on Linux. The actors in this are applications and assistive technologies (ATs) such as screen readers (for instance, Orca), magnifiers and the like.

The purpose of ATs generally is to provide users with alternative ways to interact with the application that are tailored to their needs (say, an enlarged view, text read out aloud, or voice commands). To do this, ATs need a lot of detailed information about the applications UI, and this is where the accessibility stack comes into play—it is the connecting layer between the application (or its toolkit) and the ATs.

Applications and ATs talk to each other on the accessibility bus, which is a separate instance of a D-Bus session bus, using the interfaces described by the AT-SPI project. The UI elements of the application are represented on as objects on the bus that implement somewhat abstracted interfaces, such as Text or Value. Applications emit signals to communicate changes in the UI, and ATs can call methods on the objects to get information or make changes (e.g. change the current value of a Value interface to move the GtkScale that it represents).

What has changed

In GTK 2 and 3, this was done in an awkwardly indirect way: GTK widgets have auxiliary accessible objects that are implementations of ATK interfaces (translation 1: GTK ➙ ATK). These are then turned in AT-SPI objects (translation 2: ATK ➙ AT-SPI) that are represented on the accessibility bus by adapter code in at-spi2-atk. On the other end, ATs then use pyatspi to convert the AT-SPI interfaces into Python objects (translation 3: AT-SPI ➙ Python).

This multi-step process was inefficient, lossy, and hard to maintain; it required implementing the same functionality over at least three components, and it led to discrepancies between the documented AT-SPI methods and properties, and the ones actually sent over the accessibility bus.

In GTK 4, we are simplifying the application side by cutting out ATK and at-spi2-atk. Widgets now implement a GtkAccessible interface that lets them set a number of roles, states, properties and relations that are more or less directly taken from the WAI-ARIA spec published by the W3C. The AT-SPI backend for GTKs accessibility API then takes these ARIA-inspired attributes (and the knowledge of the widgets themselves) and represents the widgets as objects on the accessibility bus, implementing the relevant AT-SPI interfaces for them.

This is a much more direct approach, and matches what Qt and web browsers already do.

Application API

Here are the highlights of the accessibility API that you are most likely to run into when using GTK 4 in applications:

Setting an accessible role. A role is a description of the semantics of a widget, and ATs will use it to decide what kind of behavior should be presented to their users. Setting a role is a one-time operation, which means it has to be done at widget creation time, either in class_init, or during instance initialization:

gtk_widget_class_set_accessible_role (widget_class,  
                                      GTK_ACCESSIBLE_ROLE_BUTTON);

Updating a widgets acccessible state or properties. This should be done whenever the widget’s accessible representation changes:

gtk_accessible_update_property (GTK_ACCESSIBLE (widget),
                       GTK_ACCESSIBLE_PROPERTY_VALUE_MIN, minimum,
                       GTK_ACCESSIBLE_PROPERTY_VALUE_NOW, value,
                       GTK_ACCESSIBLE_PROPERTY_VALUE_MAX, maximum,
                       -1);

The GTK reference documentation has an overview of the accessibility API, which includes guidance for application developers and widget writers.

What’s next ?

For GTK 4.0, we are focusing on completing the AT-SPI backend. But with the new API and the backend separation, we have a clear path towards making accessibility backends for other platforms, which is something we want to look into for subsequent GTK 4 releases.

On Linux, we want to work with other stakeholders on modernizing the AT-SPI interfaces to finally overcome the CORBA legacy that is still visible in some places. A part of that will be moving away from an accessibility bus toward peer-to-peer connections between the application and the ATs; this would enhance the security of the accessibility stack and plug a hole in the sandboxing used by technologies such as Flatpak.

In the future we want to introduce tools to ensure that application developers will be aware of missing accessibility annotations, such as providing a label attribute, or a labelled-by relation, to icons and images in their UIs; or ensure that every UI element is correctly represented in the accessible tree. We already have a test backend for the GtkAccessible interface which can be used to write unit tests and verify that roles and attributes are updated where necessary.

GTK Hackfest 2020 — Roadmap and accessibility

Between January 28th and January 31st, the GTK team held what’s now the third hackfest in Brussels.

The main topics of the hackfest were:

  • the schedule for the next development snapshot of GTK4
  • the missing features blocking the release of GTK 4.0
  • the current state of the accessibility support in the toolkit

The first two items occupied the most of the first two days of the hackfest; you can read the GTK 3.98 release announcement for what we’ve been working on for the past 300 days since the 3.96 release. The missing features are:

  • Event controllers for keyboard shortcuts
  • Movable popovers on Wayland
  • Row-recycling list and grid views
  • Animation API

and all of them are being worked on in topic branches. The keyboard shortcuts branch has recently been rebased, and it’s in the process of being documented and cleaned up; the movable popovers is also being reviewed after a few iterations. The last two remaining branches are fairly sizeable, and will require some more iterations to get them right—with the animations API currently being mostly a prototype.

The final topic of the hackfest was the largest, and was a discussion long overdue.

GTK’s accessibility support was added as part of the GTK 2.0 release by the Sun Accessibility Team; it depends on the abstract data types provided by ATK (the Accessibility Tool Kit), which are then implemented concretely in GTK classes like GtkWidgetAccessible, or GtkEntryAccessible. Each widget has an “accessible” object associated to it, which is either automatically created by GTK, or can be provided by application code when subclassing a GTK widget. Non-widget types can also have accessible objects associated to them—the most notable case is the set of cell renderers for tree views and combo boxes. Underneath it all, sits AT-SPI, a protocol that is used by AT—Accessible Technologies, like a screen reader—to consume the data provided by applications. Typically, ATs will use a library like libatspi to deal with the protocol itself.

The main issues with the existing stack are:

  • there’s a lot of indirection caused by the existence of ATK; any new feature or bug fix needs to be defined inside ATK and then implemented into GTK and libatspi
  • ATK was written in a very different environment, and while it has seen a few deprecations, it shows its age in the assumptions it makes—like global coordinate spaces—and in its design
  • there’s a certain overlap between AT requirements and requirements for GUI testing that end up creating friction in the API design
  • the stack has fell in disrepair since the Sun accessibility team was disbanded; most of the ongoing work is still pretty much happening in the AT space (like Orca) and in web browsers
  • the entire stack was written when CORBA was a thing, and then ported to DBus in time for GNOME3; the protocol, though, is not really efficient and requires lots of roundtrips to move around small amounts of data, instead of having bulk operations and notifications

The last point is also the reason why we need a separate accessibility bus in order to avoid spamming the session bus, and making everything slower as soon as the accessibility support is enabled. A separate bus means that we need to poke an additional hole in any sandbox, and still lets everything that connects to the accessibility bus potentially snoop into what happens in every application.

Finally, GTK only supports accessibility on Linux; there is no support for macOS or Windows, which means applications written in GTK and ported to other platforms are not accessible to ATs there. As we expose ATK in our API, adding support for accessibility features on other platforms would require bridging ATK, creating further complexity.

As we want to redesign and update the accessibility features in GTK4, we need to understand what are the requirements for existing consumers of the accessibility stack, and what kind of use cases we need to target. For that, we asked Hypra, a company dedicated to the development of accessible solutions based on free and open source software, to help us.

Hypra developers are familiar with GNOME, and have been working on the Linux accessibility stack. Their clients cover a wide gamut of accessibility users, so they are in the best position to describe what kind of ATs are in actual use on a day to day basis.

There are a wide range of tools and functionality that have to be provided by different layers of the stack, from the toolkit to the compositor; application developers must also have access to the tools necessary to provide proper support to ATs, as they have a much better idea of what their applications should look and behave than the toolkit.

Over the course of two days we have identified a plan for moving forward:

  • drop ATK from the stack, and have GTK talk the AT-SPI protocol directly; this is similar to what Qt does from the toolkit side, and it makes it easier to both expand and verify eventual protocol changes
  • clean up the AT-SPI protocol itself, updating it where needed when it comes to using DBus more efficiently
  • drop the global accessibility bus, and have ATs negotiate a peer-to-peer connection to each application
  • make ATs ask the compositor to gather global state, like key shortcuts, instead of talking to applications that would then have to ask the windowing system—if that’s possible—or return invalid data when it isn’t
  • decouple GUI testing from accessibility
  • write widget and application authoring guides for application developers, and provide validation tools that can be used as part of the build and CI process to check if UI elements have the correct accessible description and links

There are more information available on the wiki for the notes and the roadmap, and we have already scheduled an additional check point meeting for this summer.

There’s a lot of work to be done, but we have now a much clearer idea of the scope and deliverables for such a redesign. If you want to help making things happen faster, feel free to join the effort; you can also make a donation to the GNOME Foundation.

The GTK team would like to thank the GNOME Foundation for the sponsorship for the venue and the attendees, and the fine folks at Hypra for joining the hackfest and explaining use cases and the current state of the accessibility stack, as well as helping out on the development side.