Author: Emmanuele Bassi

Accessibility improvements in GTK 4.14

GTK 4.14 brings various improvements on the accessibility front, especially for applications showing complex, formatted text; for WebKitGTK; and for notifications.

Accessible text interface

The accessibility rewrite for 4.0 provided an implementation for complex, selectable, and formatted text in widgets provided by GTK, like GtkTextView, but out of tree widgets would not be able to do the same, as the API was kept private while we discussed what ATs (assistive technologies) actually needed, and while we were looking at non-Linux implementations. For GTK 4.14 we finally have a public interface that out of tree widgets can implement to provide complex, formatted text to ATs: GtkAccessibleText.

GtkAccessibleText allows widgets to provide the text contents at given offsets; the text attributes applied to the contents; and to notify assistive technologies of changes in the text, caret position, or selection boundaries.

Text widgets implementing GtkAccessibleText should notify ATs in these cases:

Text attributes are mainly left to applications to implement—both in naming and serialization; GTK provides support for common text attributes already in use by various toolkits and assistive technologies, and they are available as constants under the GTK_ACCESSIBLE_ATTRIBUTE_* prefix in the API reference.

The GtkAccessibleText interface is a requirement for implementing the accessibility of virtual terminals; the most common GTK-based library for virtual terminals, VTE, has been ported to GTK4 thanks to the efforts of Christian Hergert and in GNOME 46 will support accessibility through the new GTK interface.

Bridging AT-SPI trees

There are cases when a library or an application implements its own accessible tree using AT-SPI, whether in the same process or out of process. One such library is WebKitGTK, which generates the accessible object tree from the web tree inside separate processes. These processes do not use GTK, so they cannot use the GtkAccessible API to describe their contents.

Thanks to the work of Georges Stavracas GTK now can bridge those accessibility object trees under the GTK widget’s own, allowing ATs to navigate into a web page using WebKit from the UI.

Currently, like the rest of the accessibility API in GTK, this is specific to the AT-SPI protocol on Linux, which means it requires libraries and applications that wish to take advantage of it to ensure that the API is available at compile time, through the use of a pkg-config file and a separate C header, similarly to how the printing API is exposed.

Notifications

Applications using in-app notifications that are decoupled by the current widget’s focus, like AdwToast in libadwaita, can now raise the notification message to ATs via the gtk_accessible_announce() method, thanks to Lukáš Tyrychtr, in a way that is respectful of the current ATs output.

Other improvements

GTK 4.12 ensured that the computed accessible labels and descriptions were up to date with the ARIA specification; GTK 4.14 iterates on those improvements, by removing special cases and duplicates.

Thanks to the work of Michael Weghorn from The Document Foundation, there are new roles for text-related accessible objects, like paragraphs and comments, as well as various fixes in the AT-SPI implementation of the accessibility API.

The accessibility support in GTK4 is incrementally improving with every cycle, thanks to the contributions of many people; ideally, these improvements should also lead to a better, more efficient protocol for toolkits and assistive technologies to share.

We are still exploring the possibility of adding backends for other accessibility platforms, like UIAutomation; and for other libraries, like AccessKit.

The GTK Documentation

As you may have noticed, there have been various changes in the GNOME developer documentation website, as of late. These changes also affected the API references for GTK and its core dependencies.

What has changed

The main change is that GTK moved to a new documentation tool for its API reference and ancillary documentation, called gi-docgen. Unlike the previous documentation tool, gtk-doc, gi-docgen uses the introspection data that is generated by GObject-based libraries to build the API reference. This has multiple benefits:

  • gi-docgen is simpler to run and integrate within an existing library, as it only has a single project description file and relies on the introspection data for everything else; additionally, it can be easily included as a Meson sub-project
  • gi-docgen uses Markdown everywhere, instead of DocBook
  • gi-docgen is considerably faster, as it does not perform an additional source code parsing step; it does not have the bottleneck of the XML to HTML conversion via xsltproc; and it does not have to parse Devhelp files to fix cross-references to other libraries after generating the reference
  • gi-docgen can infer much more information about an API, as it has access to the entire introspection data for a library, including its dependencies; this allows the automatic generation of more accurate and consistent documentation, instead of relying on humans to do this job
  • gi-docgen generates stable URLs for all the API entry points and additional documentation, which means it’s easier to link to and from it without using obscure references
  • the default documentation template is usable on different form factors and layouts; it respects the dark theme options on web browsers that support it; and provides an in-tree live search functionality that does not depend on third party services
  • gi-docgen can also be run out of tree—this will come in handy later

Outside of these improvements, using the introspection data as the source for the documentation has additional benefits: it keeps us honest in the type of API we expose to non-C users; and it makes the C API reference closer to the reference in other languages that consume the same data.

GTK4, Pango, and GdkPixbuf have been migrated to this new tool, and while we were at it, we also reviewed the documentation to improve its consistency and accuracy—especially for the older sections of the API.

The new API references can be used offline through Devhelp 41, which will be released next September alongside GNOME 41.

Online documentation

The canonical online location for the GTK references is now docs.gtk.org. There you will find the API references for:

The API references for GTK3 and ATK have been moved to docs.gtk.org as well.

The docs.gtk.org website is generated by the GTK CI pipeline, so it is always up to date with the state of the repository; thanks to gi-docgen supporting out of tree builds, the website can also generate documentation for various libraries that have not been ported to gi-docgen yet, like GLib, GTK3, and ATK.

Known issues

Of course, with any large change come side effects.

The main issue is the change in the URLs for the documentation; existing documentation referencing locations on developer.gnome.org will have to be fixed. Thanks to the GNOME system administrators we have some redirects in place, and there are ideas on how to improve them without creating an unmaintainable mess of static redirections.

The new documentation website is in the process of being indexed by various search engines; the more you use it, and link to it, the easier it will be for the new references to raise in ranking. In any case, we strongly encourage you to use the search feature: simply press ‘s’ to start searching for symbols and types, or even content inside the extra documentation pages.

Unfortunately, GLib’s introspection data has some issues, given how low level the C API is; we are working on improving that, which will have an impact not only in the documentation but also in the overall bindability of the API in other languages.

The documentation for GLib, GObject, GIO, and GTK3 is also still written for gtk-doc; this means that cross-links in the documentation may not work; the content may not be rendered as nicely; or there can be redundant paragraphs. This will be fixed in the future, both by changes in gi-docgen (wherever possible) and by updating the documentation inside the libraries themselves. This will also improve the language bindings documentation, as they consume the same introspection data as gi-docgen. Help in this effort is very much welcome.

GTK 4.2.0

GTK 4.2.0 is now available for download in the usual places.

This release is the result of the initial round of feedback from the application developers porting their projects to GTK4, so it mostly consists of bug fixes and improvements to the API—but we also added new features, like a new GL renderer; various improvements in how the toolkit handles Compose and dead key sequences; build system improvements for compiling GTK on Windows and macOS; and a whole new API reference, generated from the same introspection data that language bindings also consume.

For more information, you can check the previous blog post about the 4.1 development cycle.

The NGL renderer

Thanks to the hard work of Christian Hergert, the NGL renderer is now the default renderer for Linux, Windows, and macOS. We’ve had really positive feedback from users on mobile platforms using drivers like Lima, with noticeable improvements in frames per second, as well as power and CPU usage; the latter two are also going to impact positively desktop and laptop users. The NGL renderer is just at the beginning: the new code base will allow us even more improvements down the line.

For the time being, we have kept the old GL renderer available; you can use export GSK_RENDERER=gl in your environment to go back to the 4.0 GL renderer—but make sure to file an issue if you need to do so, to give us the chance to fix the NGL renderer.

Input

Matthias wrote a whole blog post about the handling of Compose and dead keys input sequences, so you can just read it. The dead key handling has seen a few iterations, to deal with oddities and workarounds that have been introduced in the lower layers of the input stack.

There is one known issue with the handling of dead acute accents vs apostrophes in some keyboard layouts, which is still being investigated. If you notice other problems with keyboard input, specifically around Compose sequences or dead keys, please file an issue.

Portability

One of the goals of GTK is to have a “turn key” build system capable of going from a clone of the Git repository to a fully deployable installation of the toolkit, without having to go through all the dependencies manually, or using weird contraptions. You can see how this works on Windows, using native tools, in this article from our friends at Collabora.

Additionally, we now ensure that you can use GTK as a Meson sub-project; this means you can build GTK and all its dependencies as part of your own application’s build environment, and you can easily gather all the build artifacts for you to distribute alongside your application, using the toolchain of your choice.

Documentation

One of the most notorious issues for newcomers to GTK has been the documentation. Application developers not acquainted with our API have often found it hard to find information in our documentation; additionally, the style and structure of the API reference hasn’t been refreshed in ages. To improve the first impression, and the use of our documentation, GTK has switched to a new documentation generator, called gi-docgen. This new tool adds new features to the API reference, like client-side search of terms in the documentation; as well as nice little usability improvements, like:

  • a “Copy to clipboard” button for code fragments and examples
  • a visual hierarchy of ancestors and interfaces for each class
  • the list of inherited properties, signals, and methods in a class
  • a responsive design, which makes it easier to use the API reference on small screens

An API is only as good as it allows developers to use it in the most idiomatic way. GTK not only has a C API, it also exposes a whole API for language bindings to consume, through GObject-Introspection. The new documentation uses the same data, which not only allows us to cut down the built time in half, but it also generates common bits of documentation from the annotations in the source, making the API reference more consistent and reliable; finally, the C API reference matches what language binding authors see when consuming the introspection data, which means we are going to tighten the feedback loop between toolkit and bindings developers when introducing new API.

Pango and GdkPixbuf have also switched to gi-docgen, which allows us to build the API reference for various dependencies through our CI pipeline, and publish it to a whole new website: docs.gtk.org. You’ll always find the latest version of the GTK documentation there.

Odds and ends

Of course, alongside these visible changes we have smaller ones:

  • performance improvements all across the board, from GLSL shaders used to render our content, to the accessibility objects created on demand instead of upfront
  • sub-pixel positioning of text, when using a newer version of Cairo with the appropriate API
  • a responsive layout for the emoji chooser
  • improved rendering of shadows in popover widgets
  • localized digits in spin buttons
  • improved support for the Wayland input method protocol
  • improved scrolling performance of the text view widget

The numbers

GTK 4.2 is the result of four months of development, with 1268 individual changes from 54 developers; a total of 73950 lines were added, and 60717 removed.

Developers with the most changesets
Matthias Clasen 843 66.5%
Emmanuele Bassi 124 9.8%
Timm Bäder 87 6.9%
Christian Hergert 33 2.6%
Jakub Steiner 24 1.9%
Benjamin Otte 21 1.7%
Chun-wei Fan 15 1.2%
Alexander Mikhaylenko 14 1.1%
Fabio Lagalla 10 0.8%
Bilal Elmoussaoui 8 0.6%
Carlos Garnacho 6 0.5%
Ignacio Casal Quinteiro 6 0.5%
Michael Catanzaro 6 0.5%
Emmanuel Gil Peyrot 5 0.4%
Xavier Claessens 4 0.3%
David Lechner 4 0.3%
Jan Alexander Steffens (heftig) 4 0.3%
Kalev Lember 3 0.2%
wisp3rwind 3 0.2%
Mohammed Sadiq 2 0.2%
Developers with the most changed lines
Matthias Clasen 38475 42.6%
Emmanuele Bassi 15997 17.7%
Christian Hergert 13913 15.4%
Kalev Lember 9202 10.2%
Timm Bäder 5890 6.5%
Jakub Steiner 2397 2.7%
Benjamin Otte 902 1.0%
Chun-wei Fan 783 0.9%
Ignacio Casal Quinteiro 717 0.8%
Fabio Lagalla 292 0.3%
Marek Kasik 267 0.3%
Alexander Mikhaylenko 254 0.3%
Emmanuel Gil Peyrot 232 0.3%
Simon McVittie 214 0.2%
Jan Tojnar 83 0.1%
wisp3rwind 74 0.1%
Jan Alexander Steffens (heftig) 65 0.1%
Carlos Garnacho 62 0.1%
Michael Catanzaro 61 0.1%
Ungedummt 60 0.1%
Developers with the most lines removed
Emmanuele Bassi 8408 13.8%
Jakub Steiner 1890 3.1%
Timm Bäder 493 0.8%
Simon McVittie 203 0.3%
Emmanuel Gil Peyrot 146 0.2%
Chun-wei Fan 43 0.1%
Jan Tojnar 26 0.0%
Alexander Mikhaylenko 25 0.0%
Jonas Ådahl 17 0.0%
Luca Bacci 13 0.0%
Robert Mader 4 0.0%
Chris Mayo 3 0.0%
Bartłomiej Piotrowski 2 0.0%
Marc-André Lureau 2 0.0%
Jan Alexander Steffens (heftig) 1 0.0%
Tom Schoonjans 1 0.0%

Who Wrote GTK4

GTK 4 has been a colossal, multi-year development endeavor that started in October 2016 and ended in December 2020. Now that the 4.0 release is finally out, it’s time to look back to the incredible amount of work done by hundreds of contributors over these four years.

Back in 2016 we were definitely a bit optimistic on the time table, and thought we would be able to release 4.0 in three years, by the end of 2019. The plan was to start by changing the rendering pipeline of GTK, by moving it to a retained graph of operations that could be submitted to the GPU, as opposed to the immediate mode rendering that we had since the very beginning of the toolkit, and which survived two major API cycles—first by abstracting Xlib drawing commands, and then by moving to Cairo operations. Of course, we also knew we wanted to improve other sub-systems, like input and the windowing system API, to move away from X11-isms and towards a design more in line with the requirements of Wayland (and other windowing systems). What we got, after all was said and done, is a deep redesign of the internals of the toolkit, as well as a different programming model that favors more delegation through ancillary objects, and fewer leaky abstractions and deep type hierarchies; additionally, we pared down the exposed internals, to ensure that the toolkit, and the applications using it, will be more maintainable in the future. The downside is that GTK is less of a “meta toolkit”, whose internal state can be poked at from the outside while expecting to work across multiple releases; that approach was, in the long term, unsustainable given the available resources, and left us unable to optimise or improve the internals of GTK, to the detriment of every user.

We don’t expect the next major development cycle to take this much time, but you know as they say: no plan of operations extends with any certainty beyond the first contact with the main hostile force. The important thing, though, is that we still see potential for improvement; and while we want to celebrate the release of GTK4, as well as everyone who contributed to it, we also want to communicate that the plan outlined in 2016 is still very much in effect. We already started collecting items for a GTK5 development cycle in our GitLab project page, and we plan to have more discussions with various stakeholders for future work in that direction.

Some statistics

Starting from commit 4cce6104 on October 5th, 2016:

  • Total commits: 16852
  • Developers: 256
  • Employers: 10
  • Lines added: 1053492, removed: 1053542 (delta: -50)

The most interesting thing is that, for all that work, we ended up with a total of 50 lines of code removed from the existing code base, which includes tests and documentation.

Per year breakdown

Year Developers Commits Lines added Lines removed Delta
2016 50 1422 64014 152103 -88089
2017 84 3446 186024 166198 19826
2018 111 2332 129726 140526 -10800
2019 85 3244 200476 207623 -7147
2020 109 6384 470907 386761 84206

We only had a couple of months in 2016, and the largest effort between October and December was the removal of the deprecated API, and various changes to cope with the renaming from GTK3 to GTK4. Nevertheless, GSK initially landed in November/December. On November 21st, GTK 3.89.1 was released, the first development snapshot of the new GTK 4 API.

In 2017, GSK had with multiple iterations over its API: new render nodes were added to implement CSS drawing primitives, and widgets were moved to the GtkSnapshot API from the old Cairo rendering primitives. Additionally, the new clipboard API was introduced, with a stream-based implementation. On March 31st, GTK 3.90.0 was released, followed by 3.92.0 in October.

In 2018, things seem to slow down, but mostly because we moved to GitLab. Suddenly, we had access to a CI pipeline and merge requests, and we became progressively more confident in our development process; lots of work happened in development branches, especially large refactorings, like the renaming of GdkWindow to GdkSurface; the move to GtkGesture objects and the removal of per-widget event signals; or making leaf classes not derivable. GTK 3.94.0 was released in June.

In 2019, development pace picked up once again:

  • widget transformations
  • layout managers
  • constraint layouts
  • no more GtkMenu
  • scope object for GtkBuilder
  • GtkText widget
  • GtkNative and GtkRoot interfaces
  • per-widget actions
  • simplified GdkSurface sub-types

Plus, of course, lots of performance and functionality improvements in the rendering pipeline. GTK 3.96.0 was released in May.

Finally, we arrive at 2020:

  • new drag and drop API, based on event controllers
  • new macOS backend in GDK, replacing the old Quartz one
  • new accessibility API, dropping ATK
  • no more GtkContainer
  • no more GtkRadioButton
  • simplified GdkDevice API
  • new keyboard shortcuts API
  • new expressions and filter models
  • new model-based list and tree widgets

GTK 3.98.0 was released in February; and, finally, 3.99.0 in July, as the first true beta towards GTK 4.0.

 

Developers

Top 20 by commit
Matthias Clasen 6519 (38.7%)
Timm Bäder 3229 (19.2%)
Benjamin Otte 2596 (15.4%)
Emmanuele Bassi 1094 (6.0%)
Carlos Garnacho 494 (2.9%)
Daniel Boles 383 (2.3%)
Alexander Larsson 313 (1.9%)
Jonas Ådahl 167 (1.0%)
Chun-wei Fan 162 (1.0%)
Christian Hergert 158 (0.9%)
Jakub Steiner 134 (0.8%)
Piotr Drąg 132 (0.8%)
Руслан Ижбулатов 120 (0.7%)
Alexander Mikhaylenko 93 (0.6%)
Rico Tzschichholz 78 (0.5%)
nana-4 66 (0.4%)
Christoph Reiter 62 (0.4%)
Tim-Philipp Müller 60 (0.4%)
Mohammed Sadiq 57 (0.3%)
Olivier Fourdan 42 (0.2%)
Top 20 by changes
Matthias Clasen 620248 (36.8%)
Benjamin Otte 466996 (27.7%)
Timm Bäder 187516 (11.1%)
Emmanuele Bassi 165354 (9.8%)
Alexander Larsson 53065 (3.1%)
Carlos Garnacho 27227 (1.6%)
Christian Hergert 26964 (1.6%)
Руслан Ижбулатов 21760 (1.3%)
Jakub Steiner 18388 (1.1%)
Jonas Ådahl 12824 (0.8%)
Chun-wei Fan 12518 (0.7%)
Daniel Boles 12371 (0.7%)
Lapo Calamandrei 9995 (0.6%)
Christoph Reiter 8391 (0.5%)
Alexander Mikhaylenko 4936 (0.3%)
Tim-Philipp Müller 3932 (0.2%)
Rico Tzschichholz 3108 (0.2%)
William Hua 2900 (0.2%)
Jason Francis 1908 (0.1%)
Peter Bloomfield 1727 (0.1%)

As usual, Matthias towers over every other contributor, both in terms of sheer number of commits and in terms of code changes.

Employers

By commit
Red Hat 13706 (81.3%)
(Unknown) 1185 (7.0%)
GNOME Foundation 1095 (6.5%)
GNOME 571 (3.4%)
Purism 93 (0.6%)
Canonical 84 (0.5%)
Centricular 75 (0.4%)
Endless 20 (0.1%)
Collabora 14 (0.1%)
Intel 6 (0.0%)
Novell 3 (0.0%)
By changes
Red Hat 1415771 (84.0%)
GNOME Foundation 165355 (9.8%)
(Unknown) 58220 (3.5%)
GNOME 33442 (2.0%)
Purism 4936 (0.3%)
Centricular 4205 (0.2%)
Canonical 3347 (0.2%)
Novell 336 (0.0%)
Intel 222 (0.0%)
Collabora 208 (0.0%)
Endless 121 (0.0%)
By contributor
(Unknown) 199 (72.4%)
GNOME 32 (11.6%)
Red Hat 22 (8.0%)
Endless 6 (2.2%)
Collabora 4 (1.5%)
Centricular 3 (1.1%)
Novell 3 (1.1%)
GNOME Foundation 2 (0.7%)
Canonical 2 (0.7%)
Intel 1 (0.4%)
Purism 1 (0.4%)

While Red Hat employees affect the most changes, we still have a very large number of contributors from the community—both using their GNOME email address and using their work address even when contributing on a personal level. We also have multiple contributors from various companies in the GNOME and free software at large ecosystems, signalling a healthy community around the project.

Comparing numbers with GTK3 is a bit more tricky:

  • the GTK 2 → 3 development phase only took about a year and change, instead of four years
  • we were still using the old GNOME infrastructure, which meant fewer, larger patches, and a higher barrier to contribution in the form of Bugzilla

Nevertheless, the amount of contributors more than doubled compared to 2011, with almost three times the amount of unaffiliated contributors.

¹ – Paid staff of the GNOME Foundation.

² – Contributors using a gnome.org email address with no clear employment status.

Building and testing GTK

… or: how GTK developers check their work on the toolkit.

Since GNOME’s collective move to GitLab, GTK has taken advantage of the features provided by that platform—especially when it comes to its continuous integration pipeline.

In days of old, the only way to check that our changes to the toolkit were correct was to wait until the Continuous build bot would notify us of any breakage on the main development branches. While this was better than nothing, it didn’t allow us to prevent breakage during the development phase of anything—from features to bug fixes, from documentation improvements to adding new tests.

These days, the CI pipeline available in GitLab is run on every branch and merge request, long before the changes reach the public development branches used by everybody else.

Topic branches and merge requests

When developing a topic branch against the GTK 4 main development one, we run a CI pipeline that starts with a simple coding style check for the changes applied in the branch. The style check uses clang-format, which is often good enough for the GTK coding style; the coding style has a few “special” caveats, and clang-format can raise false negatives and false positives. For that reason, the style check is allowed to fail, but contributors and reviewers are strongly encouraged to check the logs in case of failure.

Once the style check is passed, we run the build phase, which currently contains three separate jobs:

  • a Linux debug build, using a Fedora container
  • an MSYS2 build on Windows
  • a Linux release build

The Linux debug build is pretty standard fare.

The MSYS2 build catches any issue with a GNU toolchain on Windows.

The release build is necessary to ensure that we don’t rely on side effects of the debugging code we have in place during development.

All of these jobs run the GTK test suite.

We publish the tests reports both as a JUnit file, taking advantage of GitLab’s support; and as an HTML report, stored as a pipeline artifact. This makes it easier for us to check what failed and what succeeded.

Ideally, we want to add more environments:

  • Linux builds based on other mainstream distributions
  • a Windows build using the MSVC toolchain
  • a macOS build, once the GDK backend is fixed

After the build and testing jobs pass, we step into an analysis phase. We run the Clang static analysis tool on GTK’s code base and generate a report. In the near future we could also run sanitizer tools like UBSan and ASan; fuzzying tools for our parsers, like GtkBuilder and CSS; or tools that verify that our UI definitions contain the appropriate accessibility descriptors.

Just like the tests, we publish the analysis reports as GitLab artifacts for review.

Once the analysis phase is passed, we build the API references, and check the result so that newly added symbols are properly documented.

Finally, we have manual CI jobs to build Flatpak bundles for the GTK demo application; the widget factory; and the icon browser. This allows designers to immediately test changes in Adwaita, or newly added widgets, without necessarily building GTK from a scratch on their systems.

Mainline development branches

Once the CI pipeline for a topic branch/merge request passes, we can merge the changes into the main development branch with a certain level of confidence that the code is correct and does what we want.

The main development branch runs the same pipeline as previously described, except that the Flatpak jobs are not manual any more—thus is always possible to test locally the current bleeding edge of GTK. Additionally, the documentation is published online, so it’s always up to date.

The GTK CI pipeline

What about GTK 3?

In the GTK 3 branch we have a simpler pipeline that runs the following jobs:

  • a full Meson debug build on Linux and Windows/MSYS2, for both static and shared libraries artifacts, on the current stable versions of Fedora and Debian
  • a full Meson release build on Linux, which also generates the API reference
  • an Autotools build on Linux and Windows/MSYS2
  • an optional Autotools distcheck build on Linux

The Autotools jobs will be in place for as long as GTK 3 supports Autotools. Ideally, we want to add other jobs for macOS and Windows/MSVC, taking advantage of the Meson build.

The GTK3 CI pipeline

Once the GTK 4 CI pipeline reaches a certain level of features and stability, we’re going to backport it to GTK 3, so we can be even more confident that the current stable branch does not regress.

For more information, you can check the GTK repository:

GTK Hackfest 2020 — Roadmap and accessibility

Between January 28th and January 31st, the GTK team held what’s now the third hackfest in Brussels.

The main topics of the hackfest were:

  • the schedule for the next development snapshot of GTK4
  • the missing features blocking the release of GTK 4.0
  • the current state of the accessibility support in the toolkit

The first two items occupied the most of the first two days of the hackfest; you can read the GTK 3.98 release announcement for what we’ve been working on for the past 300 days since the 3.96 release. The missing features are:

  • Event controllers for keyboard shortcuts
  • Movable popovers on Wayland
  • Row-recycling list and grid views
  • Animation API

and all of them are being worked on in topic branches. The keyboard shortcuts branch has recently been rebased, and it’s in the process of being documented and cleaned up; the movable popovers is also being reviewed after a few iterations. The last two remaining branches are fairly sizeable, and will require some more iterations to get them right—with the animations API currently being mostly a prototype.

The final topic of the hackfest was the largest, and was a discussion long overdue.

GTK’s accessibility support was added as part of the GTK 2.0 release by the Sun Accessibility Team; it depends on the abstract data types provided by ATK (the Accessibility Tool Kit), which are then implemented concretely in GTK classes like GtkWidgetAccessible, or GtkEntryAccessible. Each widget has an “accessible” object associated to it, which is either automatically created by GTK, or can be provided by application code when subclassing a GTK widget. Non-widget types can also have accessible objects associated to them—the most notable case is the set of cell renderers for tree views and combo boxes. Underneath it all, sits AT-SPI, a protocol that is used by AT—Accessible Technologies, like a screen reader—to consume the data provided by applications. Typically, ATs will use a library like libatspi to deal with the protocol itself.

The main issues with the existing stack are:

  • there’s a lot of indirection caused by the existence of ATK; any new feature or bug fix needs to be defined inside ATK and then implemented into GTK and libatspi
  • ATK was written in a very different environment, and while it has seen a few deprecations, it shows its age in the assumptions it makes—like global coordinate spaces—and in its design
  • there’s a certain overlap between AT requirements and requirements for GUI testing that end up creating friction in the API design
  • the stack has fell in disrepair since the Sun accessibility team was disbanded; most of the ongoing work is still pretty much happening in the AT space (like Orca) and in web browsers
  • the entire stack was written when CORBA was a thing, and then ported to DBus in time for GNOME3; the protocol, though, is not really efficient and requires lots of roundtrips to move around small amounts of data, instead of having bulk operations and notifications

The last point is also the reason why we need a separate accessibility bus in order to avoid spamming the session bus, and making everything slower as soon as the accessibility support is enabled. A separate bus means that we need to poke an additional hole in any sandbox, and still lets everything that connects to the accessibility bus potentially snoop into what happens in every application.

Finally, GTK only supports accessibility on Linux; there is no support for macOS or Windows, which means applications written in GTK and ported to other platforms are not accessible to ATs there. As we expose ATK in our API, adding support for accessibility features on other platforms would require bridging ATK, creating further complexity.

As we want to redesign and update the accessibility features in GTK4, we need to understand what are the requirements for existing consumers of the accessibility stack, and what kind of use cases we need to target. For that, we asked Hypra, a company dedicated to the development of accessible solutions based on free and open source software, to help us.

Hypra developers are familiar with GNOME, and have been working on the Linux accessibility stack. Their clients cover a wide gamut of accessibility users, so they are in the best position to describe what kind of ATs are in actual use on a day to day basis.

There are a wide range of tools and functionality that have to be provided by different layers of the stack, from the toolkit to the compositor; application developers must also have access to the tools necessary to provide proper support to ATs, as they have a much better idea of what their applications should look and behave than the toolkit.

Over the course of two days we have identified a plan for moving forward:

  • drop ATK from the stack, and have GTK talk the AT-SPI protocol directly; this is similar to what Qt does from the toolkit side, and it makes it easier to both expand and verify eventual protocol changes
  • clean up the AT-SPI protocol itself, updating it where needed when it comes to using DBus more efficiently
  • drop the global accessibility bus, and have ATs negotiate a peer-to-peer connection to each application
  • make ATs ask the compositor to gather global state, like key shortcuts, instead of talking to applications that would then have to ask the windowing system—if that’s possible—or return invalid data when it isn’t
  • decouple GUI testing from accessibility
  • write widget and application authoring guides for application developers, and provide validation tools that can be used as part of the build and CI process to check if UI elements have the correct accessible description and links

There are more information available on the wiki for the notes and the roadmap, and we have already scheduled an additional check point meeting for this summer.

There’s a lot of work to be done, but we have now a much clearer idea of the scope and deliverables for such a redesign. If you want to help making things happen faster, feel free to join the effort; you can also make a donation to the GNOME Foundation.

The GTK team would like to thank the GNOME Foundation for the sponsorship for the venue and the attendees, and the fine folks at Hypra for joining the hackfest and explaining use cases and the current state of the accessibility stack, as well as helping out on the development side.

Constraint layouts

What are constraints

At its most basic, a constraint is a relation between two values. The relation
can be described as a linear equation:

target.attribute = source.attribute × multiplier + constant

For instance, this:

Can be described as:

blue.start = red.end × 1.0 + 8.0

Or:

  • the attribute, “start”, of the target, “blue”, which is going to be set by the constraint; this is the left hand side of the equation
  • the relation between the left and right hand sides of the equation, in this case equality; relations can also be greater than or equal to,
    and less than or equal to
  • the attribute, “end”, of the source, “red”, which is going to be read by the constraint; this is the right hand side of the equation
  • the multiplier, “1.0”, applied to the attribute of the source
  • the constant, “8.0”, an offset added to the attribute

A constraint layout is a series of equations like the one above, describing all the relationships between the various parts of your UI.

It’s important to note that the relation is not an assignment, but an equality (or an inequality): both sides of the equation will be solved in a way that satisfies the constraint; this means that the list of constraints can be rearranged; for instance, the example above can be rewritten as:

red.end = blue.start × 1.0 - 8.0

In general, for the sake of convenience and readability, you should arrange your constraints in reading order, from leading to trailing edge, from top to bottom. You should also favour whole numbers for multipliers, and positive numbers for constants.

Solving the layout

Systems of linear equations can have one solution, multiple solutions, or even no solution at all. Additionally, for performance reasons, you don’t really want to recompute all the solutions every time.

Back in 1998, the Cassowary algorithm for solving linear arithmetic constraints was published by Greg J. Badros and Alan Borning, alongside its implementation in C++, Smalltalk, and Java. The Cassowary algorithm tries to solve a system of linear equations by finding its optimal solution; additionally, it does so incrementally, which makes it very useful for user interfaces.

Over the past decade various platforms and toolkits started providing layout managers based on constraints, and most of them used the Cassowary algorithm. The first one was Apple’s AutoLayout, in 2011; in 2016, Google added a ConstraintLayout to the Android SDK.

In 2016, Endless implemented a constraint layout for GTK 3 in a library called Emeus. Starting from that work, GTK 4 now has a GtkConstraintLayout layout manager available for application and widget developers.

The machinery that implements the constraint solver is private to GTK, but the public API provides a layout manager that you can assign to your GtkWidget class, and an immutable GtkConstraint object that describes each constraint you wish to add to the layout, binding two widgets together.

Guiding the constraints

Constraints use widgets as sources and targets, but there are cases when you want to bind a widget attribute to a rectangular region that does not really draw anything on screen. You could add a dummy widget to the layout, and then set its opacity to 0 to avoid it being rendered, but that would add unnecessary overhead to the scene. Instead, GTK provides GtkConstraintGuide, and object whose only job is to contribute to the layout:

An example of the guide UI element

In the example above, only the widgets marked as “Child 1” and “Child 2” are going to be visible, while the guide is going to be an empty space.

Guides have a minimum, natural (or preferred), and maximum size. All of them are constraints, which means you can use guides not just as helpers for alignment, but also as flexible spaces in a layout that can grow and shrink.

Describing constraints in a layout

Constraints can be added programmatically, but like many things in GTK, they can also be described inside GtkBuilder UI files, for convenience. If you add a GtkConstraintLayout to your UI file, you can list the constraints and guides inside the special “<constraints>” element:

  <object class="GtkConstraintLayout">
    <constraints>
      <constraint target="button1" target-attribute="width"
                     relation="eq"
                     source="button2" source-attribute="width" />
      <constraint target="button2" target-attribute="start"
                     relation="eq"
                     source="button1" source-attribute="end"
                     constant="12" />
      <constraint target="button1" target-attribute="start"
                     relation="eq"
                     source="super" source-attribute="start"
                     constant="12" />
      <constraint target="button2" target-attribute="end"
                     relation="eq"
                     source="super" source-attribute="end"
                     constant="-12"/>
    </constraints>
  </object>

You can also describe a guide, using the “<guide>” custom element:

  <constraints>
    <guide min-width="100" max-width="500" />
  </constraints>

Visual Format Language

Aside from XML, constraints can also be described using a compact syntax called “Visual Format Language”. VFL descriptions are row and column oriented: you describe each row and column in the layout using a line that visually resembles the layout you’re implementing, for instance:

|-[findButton]-[findEntry(<=250)]-[findNext][findPrev]-|

Describes an horizontal layout where the findButton widget is separated from the leading edge of the layout manager by some default space, and followed by the same default amount of space; then by the findEntry widget, which is meant to be at most 250 pixels wide. After the findEntry widget we have some default space again, followed by two widgets, findNext and findPrev, flush one against the other; finally, these two widgets are separated from the trailing edge of the layout manager by the default amount of space.

Using the VFL notation, GtkConstraintLayout will create all the required constraints without necessarily having to describe them all manually.

It’s important to note that VFL cannot describe all possible constraints; in some cases you will need to create them using GtkConstraint’s API.

Limits of a constraint layout

Constraint layouts are immensely flexible because they can implement any layout policy. This flexibility comes at a cost:

  • your layout may have too many solutions, which makes it ambiguous and unstable; this can be problematic, especially if your layout is very complex
  • your layout may not have any solution. This is usually the case when you’re not using enough constraints; a rule of thumb is to use at least two constraints per target per dimension, since all widgets should have a defined position and size
  • the same layout can be described by different series of constraints; in some cases it’s virtually impossible to say which approach is better, which means you will have to experiment, especially when it comes to layouts that dynamically add or remove UI elements, or that allow user interactions like dragging UI elements around

Additionally, at larger scales, a local, ad hoc layout manager may very well be more performant than a constraint based one; if you have a list box that can grow to an unknown amount of rows you should not replace it with a constraint layout unless you measure the performance impact upfront.

Demos

Of course, since we added this new API, we also added a few demos to the GTK Demo application:

A constraints demo
The constraints demo window, as part of the GTK demo application.

As well as a full constraints editor demo:

The GTK constraints editor demo
A screenshot of the GTK constraints editor demo application, showing the list of UI elements, guides, and constraints in a side bar on the left, and the result on the right side of the window

More information

Layout managers in GTK 4

Containers and layout policies have been a staple of GTK’s design since the very beginning. If you wanted your widget to lay out its children according to a specific policy, you had to implement GtkContainer for handling the addition, removal, and iteration of the child widgets, and then you had to implement the size negotiation virtual functions from GtkWidget to measure, position, and size each child.

One of the major themes of the GTK 4 development cycle is to delegate more functionality to ancillary objects instead of encoding it into the base classes provided by GTK. For instance, we moved the event handling from signal handlers described by GtkWidget into event controllers, and rendering is delegated to GtkSnapshot objects. Another step in that direction is decoupling the layout mechanism from GtkWidget itself to an ancillary type, GtkLayoutManager.

Layout Managers

A layout manager is the object responsible for measuring and sizing a widget and its children. Each GtkWidget owns a GtkLayoutManager, and uses it in place of the measure() and allocate() virtual functions—which are going away. The gist of the change: instead of subclassing a GtkWidget to implement its layout policy, you subclass GtkLayoutManager, and then assign the layout manager to a widget.

Just like in the old GtkWidget code, you will need to override a virtual function to measure the layout, called measure(), which replaces the get_preferred_* family of virtual functions of GTK 3:

static void
layout_measure (GtkLayoutManager *layout_manager,
                GtkWidget        *widget,
                GtkOrientation    orientation,
                int               for_size,
                int              *minimum,
                int              *natural,
                int              *minimum_baseline,
                int              *natural_baseline)

After measuring, you need to assign the size to the layout; this happens in the allocate() virtual function, which replaces the venerable size_allocate() virtual function of previous GTK major versions:

static void
layout_allocate (GtkLayoutManager *layout_manager,
                 GtkWidget        *widget,
                 int               width,
                 int               height,
                 int               baseline)

On the more esoteric side, you can also override the get_request_mode() virtual function, which allows you to declare whether the layout manager requests a constant size, or if one of its sizes depend on the opposite one, like height-for-width or width-for-height:

static GtkSizeRequestMode
layout_get_request_mode (GtkLayoutManager *layout_manager,
                         GtkWidget        *widget)

As you may notice, each virtual function gets passed the layout manager instance, as well as the widget that is using the layout manager.

Of course, this has bigger implications on various aspects of how GTK widgets work, the most obvious being that all the complexity for the layout code can now stay confined into its own object, typically not derivable, whereas the widgets can stay derivable and become simpler.

Another feature of this work is that you can change layout managers at run time, if you want to change the layout policy of a container; you can also have a per-widget layout policy, without adding more complexity to the widget code.

Finally, layout managers allow us to get rid of one of the special cases of GTK, namely: container child properties.

Child properties

Deep in the guts of GtkContainer sits what’s essentially a copy of the GObject property-related code, and whose only job is to implement “child” properties for types deriving from GtkContainer. These container/child properties exist only as long as a child is parented to a specific class of container, and are used for a variety of reasons—but, generally, to control layout options, like the packing direction in boxes and box-like containers; the fixed positioning inside GtkFixed; or the expand/fill rules for notebook tab widgets.

Child properties are hard to use, as they require ad hoc API instead of the usual GObject one, and thus require special casing in GtkBuilder, gtk-doc, and language bindings. Child properties are also attached to the actual direct child of the container, so if a widget interposes a child—like, say, GtkScrolledWindow or GtkListBox do—then you need to keep a reference to that child around in order to change the layout that applies to your own widget.

In GTK’s master branch we got rid of most of them—either by simply removing them when there’s actual widget API that implements the same functionality, or by creating ancillary GObject types and moving child properties to those types. The end goal is to remove all of them, and the relative API from GtkContainer, by the time GTK 4 rolls out. For layout-related properties, GtkLayoutManager provides its own API so that objects are created and destroyed automatically once a child is added to, or removed from, a widget using a layout manager, respectively. The object created is introspectable, and does not require special casing when it comes to documentation or bindings.

You start from deriving your own type from the GtkLayoutChild class, and adding properties just like you would for any other GObject type. Then, you override GtkLayoutManager‘s create_layout_child() virtual function:

static GtkLayoutChild *
create_layout_child (GtkLayoutManager *manager,
                     GtkWidget *container,
                     GtkWidget *child)
{
  // The simplest implementation
  return g_object_new (your_layout_child_get_type (),
                       "layout-manager", manager,
                       "child-widget", child,
                       "some-property", some_property_initial_state,
                       NULL);
}

After that, you can access your layout child object as long as a widget is still a child of the container using the layout manager; if the child is removed from its parent, or the container changes the layout manager, the layout child is automatically collected.

New layout managers

Of course, just having the GtkLayoutManager class in GTK would not do us any good. GTK 4 introduces various layout managers for application and widget developers:

  • GtkBinLayout implements the layout policy of GtkBin, with the added twist that it supports multiple children stacked on top of each other, similarly to how GtkOverlay works. You can use each widget’s alignment and expansion properties to control their location within the allocated area, and the GtkBinLayout will always ask for as much space as it’s needed to allocate its largest child.
  • GtkBoxLayout is a straight port of the layout policy implemented by GtkBox; GtkBox itself has been ported to use GtkBoxLayout internally.
  • GtkFixedLayout is a port of the fixed layout positioning policy of GtkFixed and GtkLayout, with the added functionality of letting you define a generic transformation, instead of a pure 2D translation for each child; GtkFixed has been modified to use GtkFixedLayout and use a 2D translation—and GtkLayout has been merged into GtkFixed, as its only distinguishing feature was the implementation of the GtkScrollable interface.
  • GtkCustomLayout is a convenience layout manager that takes functions that used to be GtkWidget virtual function overrides, and it’s mostly meant to be a bridge while porting existing widgets towards the layout manager future.

We are still in the process of implementing GtkGridLayout and make GtkGrid use it internally, following the same pattern as GtkBoxLayout and GtkBox. Other widgets inside GTK will get their own layout managers along the way, but in the meantime they can use GtkCustomLayout.

The final step is to implement a constraint-based layout manager, which would let us create complex, responsive user interfaces without resorting to packing widgets into nested hierarchies. Constraint-based layouts deserve their own blog post, so stay tuned!

Testing Discourse for GTK

For the past 20 years or so, GTK used IRC and mailing lists for discussions related to the project. Over the years, use of email for communication has declined, and the overhead of maintaining the infrastructure has increased; sending email to hundreds or thousands of people has become increasingly indistinguishable from spam, in the eyes of service providers, and GNOME had to try and ask for exceptions—which are not easy to get, and are quite easy to be revoked. On top of that, the infrastructure in use for managing mailing lists is quite old and crumbly, and it’s unnecessarily split into various sub-categories that make following discussions harder than necessary.

After discussions among the GTK team, with the GNOME infrastructure maintainers, and with the GTK community at large, we decided to start a trial run of Discourse as a replacement for mailing lists, first and foremost, and as a way to provide an official location for the GTK community to discuss the development of, and with, GTK—as well as the rest of the core GNOME platform: GLib, Pango, GdkPixbuf, etc.

You can find the Discourse instance on discourse.gnome.org. On it, you can use the Platform and Core categories for discussions about the core GNOME platform; you can use the appropriate tags for your topics, and subscribe to the ones you’re interested in.

We’re planning to move some of the pages on the wiki to Discourse as well, especially the ones where we expect feedback from the community.

We’re still working on how to migrate users of the various mailing lists related to GTK, in order to close the lists and have a single venue instead of splitting the community; in the meantime, if you’re subscribed to one or more of these lists:

  • gtk-devel-list
  • gtk-app-devel-list
  • gtk-list
  • gtk-i18n-list

then you may want to have a look at Discourse, and join the discussions there.

Report from the GTK hackfest in Brussels

Thanks to the GNOME Foundation, various GTK developers were able to meet in Brussels right after FOSDEM, for one of our yearly hackfests.

The main topics of the hackfest were:

  • recap the work that landed into the master branch in the past 6-12 months, in order to have everyone on the same page
  • discuss the features still in flight in separate branches, assess their state of completion, and identify blockers
  • figure out what are the blockers for the first release of GTK 4.0

Hackfests allow us to have this kind of discussions with a large bandwidth at our disposal, compared to online communication channels, so they are very important for the project.

You can see the full agenda on the wiki, and we’ll make sure to write articles on the biggest items on it.

The largest items of the discussion were the introduction of new list models and list/grid view widgets; a unified key handling API; the decoupling of layout management policies from containers, and the introduction of constraint layout management; the possibility of merging widgets from libhandy, to allow for writing applications responsive to form factor changes; the switch to a purely declarative menu description API, and the removal of public menu widgets; adding 2D and 3D transformations to GtkWidget; implementing an animation API that applications can consume.

  • list models and list/grid widgets — we’d really like to retire GtkTreeView and GtkIconView, but the existing replacements, GtkListBox and GtkFlowBox, are not performant enough when scaling to very large and dynamic data sets. We need better data storage types, that can be composed to perform operations such as mapping, filtering, and sorting, but can also avoid iterating over all the elements when sizing and drawing widgets. Benjamin Otte already added various models to GTK, and is working on a list and a grid view widgets that can efficiently display their contents. Benjamin and other GNOME application developers are in the process of identifying various stakeholders for  a separate hackfest specifically for gathering more requirements and getting feedback on the new API.
  • unified key handling API — now that we moved all our pointer and touch input handling away from events and towards gestures, we want to do the same for key handling, like key bindings, mnemonics, and accelerators. The overall design is based on triggering actions, and allow introspection of all the “shortcuts” currently available to the GTK inspector, for ease of debugging. There is a development branch already available.
  • layout managers — in GTK 3, layout is imposed by containers on their children; we want to be able to decouple that from widgets and move it into a separate delegate objects hierarchy. Layout managers allow us to reduce the complexity of writing new widgets; they keep the layout code in a separate, non-derivable type; and they allow us to simplify the toolkit internals to the point that we might even make GtkWidget and instantiable type in the future. Layout managers are the first step towards adding constraint-based layout management to GTK, which do away with nesting boxes to create complex UIs. There is a development branch already available. For more information on constraint layouts, you can see the Emeus experimental library for GTK 3.
  • merging widgets from libhandy — Adrien Plazas gave an overview of what’s currently provided by libhandy, and what would be useful to have straight from GTK4 in the future. We discussed reactive layouts, and the ability express sizing with percentages, as well as possibly using constraints to get similar results.
  • declarative menus — GTK has iterated over different menus API over the years; from building menus out of widgets, to GtkUIManager, to GtkBuilder, to GMenu; we also moved to declaring the behaviour of pop up menus, in order to have the windowing system display them more accurately without exposing global coordinates. There’s a lot of overlap, but no clear winner, mostly because we still allow using widgets to build application menus and context menus. Fully switching to declarative style menus, adding new API to make them more expressive, and making GtkMenu and friends private implementations for the toolkit, would allow us to get things like being able to inspect all menus, even out of process; menus manipulable by plugin systems without necessarily creating widgets and keeping track of them; avoiding positioning bugs. There is a full strawman proposal available on the wiki, and Matthias Clasen is working on switching context menus to GMenu in a development branch.
  • widget transformations — Sadly, Timm Bädert couldn’t make it to the hackfest, but we’ve been reviewing his development branch that adds 2D and 3D transformations to GTK widgets, and we’re very excited about it.
  • animations — one last thing we’d like to land for GTK4 is an animation framework for GTK widgets to replace the current generic “frame tick callback”. The model for it is the Clutter explicit animation API, which in turn was based on Core Animation and CSS3 transitions. This work is still in the design phase, but you can expect development branches for it to land soon.

Aside from the big topics, we also discussed various smaller ones:

  • improving performance and memory use; we want to expose the SysProf counters during the frame clock phases, so we can easily identify problems.
  • improving the test suite, especially when it comes to reporting failures; right now, we have to go through the CI failure log, but we’d like to publish proper reports using the GitLab CI infrastructure
  • replacing child properties with real GObject properties on ancillary objects, especially for layout managers; would make documentation, introspection, and usage clearer.
  • finishing the drag and drop rework, to get a more modern API.
  • adding a top-level interface for “window-like” objects—such as windows, dialogs, popovers, menus/popups—useful for establishing common behaviour, and removing hacks and complexity in GtkWindow.

And, finally, yes: we did remove the “plus” from GTK. ;-)