GTK 4.4

GTK 4.4.0 is now available for download in the usual places. Here are some highlights of the work that has gone into it.

The NGL renderer and GL support

The NGL renderer has continued to see improvements. This includes speedups, fixes for transformed rendering, avoiding huge intermediate textures, and correct handling of partial color fonts. After some help from driver developers, NGL now works correctly with the Mali driver. We are planning to drop the original GL renderer in the next cycle.

Outside of GSK, our OpenGL setup code has been cleaned up and simplified. We increasingly rely on EGL, and require EGL 1.4 now. On X11 we use EGL, falling back to GLX if needed. On Windows, we default to using WGL.

Our GL support works fine with the latest NVidia driver.

Themes

The included themes have been reorganized and renamed. We now ship themes that are called Default, Default-dark, Default-hc and Default-hc-dark. The Adwaita theme is moving to libadwaita.

Among the smaller theme improvements are new error underlines (they are now dotted instead of squiggly) and support for translucent text selections.

Input

Input handling has seen active development this cycle. We’ve matched the behavior of the built-in input method with IBus for displaying and handling compose sequences and dead keys. As part of this, we now support multiple dead keys and dead key combinations that don’t produce a single Unicode character (such as ẅ).

We fully support 32-bit keysyms now, so using Unicode keysyms (e.g. for combining marks) works.

Emoji

Our Emoji data has been updated to CLDR 39, and we can are looking for translated Emoji data by both language and territory (e.g. it-ch).

Debugging

The Inspector is now enabled by default, so debugging GTK applications should be a litte easier.

Windows

Apart from the WGL improvements that were already mentioned, we now use GL for media playback on Windows. A big change that landed late in 4.4 is that we use the WinPointer API for tablets and other input devices now, replacing the outdated wintab API. DND support on Windows is also improved, and the local DND protocol has been dropped.

The numbers

GTK 4.4 is the result of 5 months of development, with 838 individual commits from 71 developers; a total of 88133 lines were added and 63094 removed.

Developers with the most changesets
Matthias Clasen 456 54.4%
Benjamin Otte 82 9.8%
Emmanuele Bassi 48 5.7%
Alexander Mikhaylenko 35 4.2%
Chun-wei Fan 30 3.6%
Christian Hergert 18 2.1%
Luca Bacci 17 2.0%
Carlos Garnacho 10 1.2%
Bilal Elmoussaoui 10 1.2%
Florian Müllner 7 0.8%
Yuri Chornoivan 6 0.7%
Maximiliano Sandoval R 6 0.7%
Marc-André Lureau 5 0.6%
Marco Trevisan (Treviño) 5 0.6%
Pawan Chitrakar 5 0.6%
Piotr Drąg 4 0.5%
Timm Bäder 4 0.5%
Xavier Claessens 4 0.5%
Zhi 4 0.5%
Sebastian Cherek 4 0.5%

Text input in GTK 4

To wrap up the recent series of posts about input topics, lets talk about text editing in GTK 4.

The simple: shortcuts

Maybe you just need to handle a few keys as editing commands, for example Ctrl-z to undo. In that case, you can just use a shortcut with an action, and set it all up in your widgets class_init:

/* install an undo action */ 
gtk_widget_class_install_action (widget_class,
                                  "text.undo", NULL,
                                  my_undo_func);

/* bind Ctrl-z to the undo action */
 gtk_widget_class_add_binding_action (widget_class,
                                      GDK_KEY_z, GDK_CONTROL_MASK,
                                      "text.undo", NULL);

The complex: a text editor

When you need full text editing, the best approach is to re-use one of the ready-made widgets in GTK for this purpose: one of the entries, or if you need a full-blown text editor, GtkTextView.

If none of the existing entries fit your use case, you can also wrap your own GtkEditable implementation around a GtkText widget, and get all the hard parts of a text editing widget for free. The GTK docs explain how to do that.

The middle ground

But what if you don’t want an entry, but still need to let your users enter individual Unicode characters such as ñ or Å conveniently? I’ll let you come up with a use case for this (although I have one in mind).

One thing you can do is to use a GtkIMContext directly, and let it process key events for you. The way this works is that you attach a key event controller to your widget and connect an input method context to it:

controller = gtk_event_controller_key_new ();
gtk_widget_add_controller (widget, controller);

im_context = gtk_im_multicontext_new ();
gtk_event_controller_key_set_im_context (controller, im_context);

Now key events that reach your widget will be passed to the input method context. Connect a handler to its ::commit signal to receive the completed input:

static void
commit_cb (GtkIMContext *context,
           const char   *str,
           DemoWidget   *demo)
{
  pango_layout_set_text (demo->layout, str, -1);
  pango_layout_set_attributes (demo->layout, NULL);
  gtk_widget_queue_draw (GTK_WIDGET (demo));
}

...

g_signal_connect (im_context, "commit",
                  G_CALLBACK (commit_cb), demo);

You can connect a similar handler to the ::preedit-changed signal to provide the user feedback during preedit like GtkEntry does.

The complete example for single-character input can be found here.

More on input

I’ve written about input before (here and here), and more recently, Carlos and myself gave a Guadec talk about input-related topics (slides). In those writings, I have explained how dead keys work, and how you can type

<dead_acute> A

to produce an Á character.

But input is full of surprises, and I’ve just learned about an alternative to dead keys that is worth presenting here.

Background

First lets recap what happens when you send the <dead_acute> A sequence to GTK.

We receive the first key event and notice that it is a dead key, so we stash it in what we call the preedit, and wait for the next event.  When the next key arrives, and it represents a letter (more precisely, is in one of the Unicode categories Ll, Lu, Lt, Lm or Lo), we look up the Unicode combining mark matching the dead_acute, which is U+301 COMBINING ACUTE ACCENT, and then we flip the sequence around. So the text that gets committed is

A <combining acute>

The reason that we have to flip things around is that combining marks go after the base character, while dead keys go before.

This works, but it is a bit unintuitive for writing multi-accented characters. You have to think about the accents you want to apply from top to bottom, since they get applied backwards. For example to create an  with an acute accent on top, you type

<dead_acute> <dead_circumflex> A

which then gets flipped around and ends up as:

A <combinining circumflex> <combining acute>

A better way

To me, it feels much more natural to specify the accents in that order:

  1. give me an A
  2. then put a ^ on top
  3. and then put an ´ on top

The good news is: we can do just that! Keyboard layouts can use any Unicode character as keysyms, so we can just use the combining marks directly, without the detour through dead keys.

For example, the “English (US,  Intl, AltGr Unicode combining)” layout contains keys for combining marks. A slight hurdle to using this layout is that it does not show up in the GNOME Settings keyboard panel by default. You have to run

gsettings set org.gnome.desktop.input-sources show-all-sources true

to make it show up.

The combining marks in this layout are placed in a “3rd level”. To use them, you need to set up a “3rd level chooser” key. In the keyboard panel, this is called the “Alternative Characters Key”. A common choice is the right Alt key.

After all these preparations, you can now type A Alt+^ Alt+’ to get an   with an   ́ on top. Neat!

Adventures in graphics APIs

Various people are working on porting desktop virtualization UIs to GTK4. This typically involves virgl, and the GTK3 solution was to use GtkGLArea.

With GTK4, rendering is happening in GL anyway, so it should be enough to just wrap your content in a GdkTexture and hand it to GTK, either by using it as a paintable with GtkPicture, or with a GskTextureNode in your own snapshot() implementation.

dmabuf detour

This is a nice theory, but the practice is more complicated – the content is typically available as a dmabuf object, and with 4k rendering, you really want to avoid extra copies if you can help it. So we had to look at the available solutions for importing dmabufs as textures into GL without copies.

This turned into a quick tour through the maze of graphics APIs: OpenGL, EGL, GL ES, GLX, DRI, … the list goes on. In the end, it turns out that you can use EGL  to wrap a dmabuf into an EGLImage, and use the GL_OES_EGL_image extension to create a GL texture from it.

GLX to EGL

This works fine with our Wayland backend, which uses EGL. Unfortunately, our much older X11 backend has a GL implementation using GLX, and there doesn’t seem to be a way to get a dmabuf imported into a GLX context.

So we had to do a little bit of extra work, and make our X11 backend use EGL as well. Thankfully Emmanuele had an old unfinished branch with  such a conversion from a few years ago, which could be made to work (after some initial head scratching why it would not render anything – as always the case when doing GL work).

The solution

It turns out that importing dmabufs with EGL can be done outside of GTK just fine, so we don’t need to add Linux-specific API for it. To save you the trouble of writing such code yourself, here is what I’ve come up with.

After we had already decided to port the X11 backend to EGL, I learned that another possibility for importing dmabufs might be to use DRI3PixmapFromBuffer to create  an X11 pixmap from a dmabuf, turn it in a GLXPixmap and use glxBindTexImageEXT to make a texture.

Aren’t graphics APIs wonderful! :-)

Input, revisited

My last update talked about better visual feedback for Compose sequences in GTK’s input methods. I did not explicitly mention dead keys back then, but historically, X11 has treated dead keys and Compose sequences in exactly the same way.

Dead keys are a feature of certain keyboard layouts where you can hit a key that does not produce a character by itself, but modifies the next key you type. Typically, this is used for accents that can be combined with different base characters. For example, type <dead_acute> <a> to produce á or  <dead_acute> <o> to produce ó.

Traditionally, dead keys were really dead – you didn’t get any visual feedback before the final result appears. With the improvements described in the last update, we now show dead keys as they are entered:

That is a nice improvement. But as it turned out, not everybody was happy.

The shared treatment of Compose sequences and dead keys has some implications: one is that entering a non-existing sequence such as <dead_grave> <x> will produce a beep, and no output. That is acceptable for a Compose sequence that you explicitly started with the Compose key, but not so great when you maybe meant to enter `x.

The people who decided to use Compose sequences for dead keys foresaw the need to actually enter spacing accents every now and then, and added sequences such as <dead_grave> <space> and <dead_grave> <dead_grave> for producing a single ` character.

While that is a nice thought, it is still pretty inconvenient, since you need to type <dead_grave> six time to produce `‍`‍`, e.g. for entering code examples in markdown.

After thinking about this for a while and comparing what other systems do, we’ve made two changes, that will hopefully make dead keys as convenient to use as any other keys on your keyboard.

  • When a <dead key> <key> sequence does not match one of our Compose sequences, commit the individual keys
  • When a <dead key> follows another <dead key>, commit the first one, and treat the second as the beginning of a new Compose sequence

Together, this makes it so that typing <dead_acute> <a> produces á, typing <dead_grave> <x>  produces `x, and you only need to type <dead_grave> three times to enter `‍`‍`:

Much better!

GTK happenings

GTK 4.2 is due out in March – it will not be an enormous release, just incremental improvements. But besides the usual bug fixes and performance improvements, there are a few things that are worth calling out indvidually.

A new GL Renderer

Christian Hergert has been hard at work, creating a new GL renderer for GTK. The initial motivation for this work was the desire to improve our rendering performance on MacOS, where the GL drivers are not quite as forgiving as on Linux. Apart from that, starting over with a new renderer gives us a chance to apply all the things we’ve learned while working on our current GL renderer, and to reorganize the code with an eye towards future improvements, such as reordering and batching of draw commands.

The new renderer hasn’t been merged  yet, but it is closing in on feature parity and may well make it into 4.2. We will likely include both the old and the new renderer, at least for a while.

Popover Shadows

Ever since GtkPopover was introduced with its signature ‘beak’, popovers have clipped everything outside their border, since we needed the tip of the beak to be consistently placed. With the new xdg-popup based implementation in GTK4, we have an positioning protocol that is expressive enough to place popovers in a way that makes the ‘beak’ point where it is supposed to while allowing shadows underneath and around the popover. As with window shadows, the popover shadows are outside of the input region, so clicks go through to the underlying window.

This is a minor thing, but it may have a noticeable impact in giving the UI depth and structure.

Better Input

GtkIMContextSimple is the input method implementation that is built into GTK. It is used when we don’t have a platform method to use, such as the Wayland text protocol. GtkIMContextSimple only does a few things. One of them is to interpret hexadecimal input for Unicode characters, with Control-Shift-u. Another is that it handles Compose sequences, such as

<Compose Key> <a> <acute>

to enter an á  character.

Most Compose sequences start with that Compose Key, and the keyboard settings in GNOME 40 will include a way to assign a key on your keyboard to this function.

On the GTK side, we’ve addressed a few longstanding complaints with the Compose sequence support. Apart from its built-in sequences, GTK parses X11 Compose files. The format of these files is described in Compose(5), but until now, GTKs support for this format was pretty incomplete. With GTK 4.2, we’re improving this to

  • Allow sequences of up to 20 keys (previously, the limit was 7)
  • Generate multiple characters (notably, this allows Unicode Emoji sequences)
  • Support hexadecimal codepoints

These are nice improvements for people who make their own Compose sequences by editing ~/.Compose. But what about the rest of us? One traditionally difficult aspect of using Compose sequence is that you have to know the sequences by heart, and type them blindly. There is no visual feedback at all until the sequence is complete and the final character appears. A while ago, IBus improved on this by showing the characters of the incomplete sequence as underlined preedit text, similar to what we do for hexadecimal Unicode input.

After copying their approach for GTK, initial user feedback was mixed, mainly because the official glyph for the Compose Key (⎄) is a bit distracting when it appears unexpectedly. So I went back to the drawing board and came up with a different approach:

I hope this works better. Feedback welcome!

All of these input changes will also appear in GTK 3.24.26.

GTK 4.0.1

We all took a bit of a break after 4.0 and did some other things, but now it is time for GTK 4.0.1.

This is the first release after 4.0, and it naturally contains a lot of small bug fixes,  theme and documentation improvements, and the like. But there are a few highlights that are worth pointing out.

Better media support

Among the bigger advances in this release: we managed to make the gstreamer media backend use GL textures, which avoids bouncing frame data between gpu and cpu when using hardware acceleration for decoding, such as vaapi. This requires careful orchestration to bridge the differences in how gstreamer and GTK treat GL, but we managed to make it work in many cases.

Does this mean GtkVideo is now ready to support fully-featured media player applications? Far from it. It still just lets you play media from a file or url, and does not support multi-channel audio, video overlays, device selection, input, and other things that you probably want in a media player.

It would be really nice if somebody took the code in the GTK media backend and turned it inside out to make a GStreamer plugin with a sink that exposes its video frames as GdkPaintable. That would let you use gstreamer API to get all of the aforementioned features, while still integrating smoothly in GTK.

Better CI

In order to keep our new MacOS backend in working shape, we’ve started to set up CI builds for this platform, both for GTK itself, and for its dependencies (pango, gdk-pixbuf).

GTK 4.0

2020 has been a very long year. What better way to end it than with a major release!  Today, we released GTK 4.0.

GTK 4.0 is the result of a lot of hard work by a small team of dedicated developers. We will have a separate post to go over the statistics, but the short summary is that since the 3.89.1 release in November 2016, we’ve added over 18000 commits and made more than 20 development releases.

Congratulations and a big thank you to everybody who has participated in this effort, and in particular to Benjamin, Emmanuele, Timm, Carlos, Jonas and Christian!

What’s new

It is impossible to summarize 4 years of development in a single post. We’ve written detailed articles about many of the new things in this release over the past year: Data transfers, Event controllers, Layout managers, Render nodes, Media playback, Scalable lists, Shaders, Accessibility.  Here are some of the highlights, in visual form:

Media playback:

Drag-and-Drop:

Layout managers and transforms:

Scalable lists and grids:

Shaders:

What’s old

GTK 4 is now stable, and we consider it ready for consumption. That does not mean GTK 3 is dead – we will continue to support and update it for the foreseeable future (the latest release, 3.24.24, quietly went out a few days ago). It does mean, however, that GTK 2 has reached the end of its life. We will do one final 2.x release in the coming days, and we encourage everybody to port their GTK 2 applications to GTK 3 or 4.

How to get it

The source tarballs are available in the usual place. Binary packages should appear in major distributions soon.

The GNOME 40 release will have a number of applications ported to GTK 4.  If you want to try GTK 4.0 today, you can use the nightly flatpak builds of gtk4-demo and gtk4-widget-factory:

$ flatpak install https://nightly.gnome.org/repo/appstream/org.gtk.Demo4.flatpakref
$ flatpak run org.gtk.Demo4

If you are itching to port your application to GTK 4, our migration guide is available as part of the documentation.

How to support GTK

GTK could not be developed without the many volunteers who contribute bug reports, patches, translations or ideas. Thanks to all of you. We are also grateful to the GNOME foundation for supporting GTK with development resources, infrastructure, and travel assistance.

Donating to the GNOME foundation is a good way to support future GTK development.

What comes next

We are very thankful for all the early testers that have provided us with bug reports and feedback, which made this release much better. But we fully expect that there will be a quick 4.0.1 release to fix up the oversights and gotchas that only come to light after a .0 release.

Now that we have a 4.0 release, we need to bring along the library ecosystem to enable applications to use it. vte, webkit and gtksourceview are some of the most notable libraries that tend to be used together with GTK. We expect GTK 4 ports of these to be available soon.

If more serious issues show up, we will do a 4.2 release in time for GNOME 40, otherwise we might wait until the summer for that.

What the future will bring beyond GTK 4 remains to be seen. We have some fun things in the pipeline, but we would also like to hear what features application developers would like to see in GTK. Tell us!

A celebration

We will celebrate the 4.0 release with an (online) gathering this coming Friday. Feel free to drop by!

Accessibility in GTK 4

The big news in last weeks GTK 3.99.3 release is that we have a first non-trivial backend for our new accessibility implementation. Therefore, now is a good time to take a deeper look at accessibility in GTK 4.

Overview

Lets start with a quick review of how accessibility works on Linux. The actors in this are applications and assistive technologies (ATs) such as screen readers (for instance, Orca), magnifiers and the like.

The purpose of ATs generally is to provide users with alternative ways to interact with the application that are tailored to their needs (say, an enlarged view, text read out aloud, or voice commands). To do this, ATs need a lot of detailed information about the applications UI, and this is where the accessibility stack comes into play—it is the connecting layer between the application (or its toolkit) and the ATs.

Applications and ATs talk to each other on the accessibility bus, which is a separate instance of a D-Bus session bus, using the interfaces described by the AT-SPI project. The UI elements of the application are represented on as objects on the bus that implement somewhat abstracted interfaces, such as Text or Value. Applications emit signals to communicate changes in the UI, and ATs can call methods on the objects to get information or make changes (e.g. change the current value of a Value interface to move the GtkScale that it represents).

What has changed

In GTK 2 and 3, this was done in an awkwardly indirect way: GTK widgets have auxiliary accessible objects that are implementations of ATK interfaces (translation 1: GTK ➙ ATK). These are then turned in AT-SPI objects (translation 2: ATK ➙ AT-SPI) that are represented on the accessibility bus by adapter code in at-spi2-atk. On the other end, ATs then use pyatspi to convert the AT-SPI interfaces into Python objects (translation 3: AT-SPI ➙ Python).

This multi-step process was inefficient, lossy, and hard to maintain; it required implementing the same functionality over at least three components, and it led to discrepancies between the documented AT-SPI methods and properties, and the ones actually sent over the accessibility bus.

In GTK 4, we are simplifying the application side by cutting out ATK and at-spi2-atk. Widgets now implement a GtkAccessible interface that lets them set a number of roles, states, properties and relations that are more or less directly taken from the WAI-ARIA spec published by the W3C. The AT-SPI backend for GTKs accessibility API then takes these ARIA-inspired attributes (and the knowledge of the widgets themselves) and represents the widgets as objects on the accessibility bus, implementing the relevant AT-SPI interfaces for them.

This is a much more direct approach, and matches what Qt and web browsers already do.

Application API

Here are the highlights of the accessibility API that you are most likely to run into when using GTK 4 in applications:

Setting an accessible role. A role is a description of the semantics of a widget, and ATs will use it to decide what kind of behavior should be presented to their users. Setting a role is a one-time operation, which means it has to be done at widget creation time, either in class_init, or during instance initialization:

gtk_widget_class_set_accessible_role (widget_class,  
                                      GTK_ACCESSIBLE_ROLE_BUTTON);

Updating a widgets acccessible state or properties. This should be done whenever the widget’s accessible representation changes:

gtk_accessible_update_property (GTK_ACCESSIBLE (widget),
                       GTK_ACCESSIBLE_PROPERTY_VALUE_MIN, minimum,
                       GTK_ACCESSIBLE_PROPERTY_VALUE_NOW, value,
                       GTK_ACCESSIBLE_PROPERTY_VALUE_MAX, maximum,
                       -1);

The GTK reference documentation has an overview of the accessibility API, which includes guidance for application developers and widget writers.

What’s next ?

For GTK 4.0, we are focusing on completing the AT-SPI backend. But with the new API and the backend separation, we have a clear path towards making accessibility backends for other platforms, which is something we want to look into for subsequent GTK 4 releases.

On Linux, we want to work with other stakeholders on modernizing the AT-SPI interfaces to finally overcome the CORBA legacy that is still visible in some places. A part of that will be moving away from an accessibility bus toward peer-to-peer connections between the application and the ATs; this would enhance the security of the accessibility stack and plug a hole in the sandboxing used by technologies such as Flatpak.

In the future we want to introduce tools to ensure that application developers will be aware of missing accessibility annotations, such as providing a label attribute, or a labelled-by relation, to icons and images in their UIs; or ensure that every UI element is correctly represented in the accessible tree. We already have a test backend for the GtkAccessible interface which can be used to write unit tests and verify that roles and attributes are updated where necessary.

GTK 3.99.2

The GTK 3.99.2 release continues the topics from 3.99.1: api cleanup, new and polished demos, better documentation. You can see the details here.

One small note on the topic of documentation is that we are relying on some unreleased gtk-doc features. Therefore, we now include gtk-doc as a subproject in the gtk release tarball. If you are a distributor, don’t be surprised that building GTK installs gtk-doc tools now.

The big news in this snapshot is our work on exposing the power of the new GL-based rendering stack a bit more.

Warmup: Shadertoy

gtk4-demo includes a Shadertoy demo now.

The demo using a GtkGLArea widget to run GLSL snipplets that are compatible with the ones found on shadertoy.com. Many of the examples found there will work, if you paste them into the editor of this demo.

This is fun, but somewhat limited. The GLSL is confined to its ‘sandbox’, the GtkGLArea widget, which is using GL api to compile and use the shaders.

Shaders as first-class objects

This is not our first attempt to make a shadertoy lookalike. When we first looked at it, we thought that we would make a shader abstraction that applications could use. We put it to the side when it turned out that making it work across different renderers and backends would require us to write our own shader compiler—too much work.

But after our shadertoy success, we revisited the idea of shaders as first-class objects, with more modest goals: We use GLSL, and don’t attempt to make the shaders work with anything but the OpenGL renderer.

In 3.99.2, we now have:

With these pieces in place, we made a demo that shows various uses of shaders. It is maybe a bit overloaded, and some of the effects are a bit over-the-top, but it gets the point across: you can use shaders in your widgets.

 

What we haven’t done yet is adding widgets that have shader support built-in. The demo showcases a few likely candidates:

A shader paintable. As you may recall, GdkPaintable is a very flexible interface for anything that can ‘paint’. Shaders certainly qualify. The GskShaderPaintable in gtk-demo uses a shader without input textures to just produce pixels, and we add it to a GtkPicture widget to make it appear in the widget tree.

A shader bin. This is a very simple container that can use shaders to draw effects on top of a child widget. It works with shaders that take a single input texture (for the child widget).

A shader stack. This is a stack-like container that shows one of many child widgets, and uses a shader for the transition whenever the visible child changes. It works with shaders that expect two input textures (for the old and new active child).

Thankfully, making custom widgets is a lot easier in GTK 4 than it used to be, so the render node API should be enough to get you started on some fun experiments. You can of course take the gtk4-demo code as a starting point.

You can debug it

Apart from widgets, the shader support is fully integrated. The GTK inspector can handle shader nodes like any other render nodes, you can serialize them and e.g. load the resulting file in gtk4-node-editor:

If you need to see the input that GTK sends to the shader compiler,  setting the environment variable

GDK_DEBUG=shaders

can be helpful.

Whats next?

After this GL adventure we’ll now focus on landing more of the new accessibility infrastructure.