textures and paintables

With GTK4, we’ve been trying to find better solution for image data. In GTK3 the objects we used for this were pixbufs and Cairo surfaces. But they don’t fit the bill anymore, so now we have GdkTexture and GdkPaintable.

GdkTexture

GdkTexture is the replacement for GdkPixbuf. Why is it better?
For a start, it is a lot simpler. The API looks like this:

int gdk_texture_get_width (GdkTexture *texture);
int gdk_texture_get_height (GdkTexture *texture);

void gdk_texture_download (GdkTexture *texture,
                           guchar     *data,
                           gsize       stride);

So it is a 2D pixel array and if you want to, you can download the pixels. It is also guaranteed immutable, so the pixels will never change. Lots of constructors exist to create textures from files, resources, data or pixbufs.

But the biggest difference between textures and pixbufs is that they don’t expose the memory that they use to store the pixels. In fact, before gdk_texture_download() is called, that data doesn’t even need to exist.
And this is used in the GL texture. The GtkGLArea widget for example uses this method to pass data around. GStreamer is expected to pass video in the form of GL textures, too.

GdkPaintable

But sometimes, you have something more complex than an immutable bunch of pixels. For example you could have an animated GIF or a scalable SVG. That’s where GdkPaintable comes in.
In abstract terms, GdkPaintable is an interface for objects that know how to render themselves at any size. Inspired by CSS images, they can optionally provide intrinsic sizing information that GTK widgets can use to place them.
So the core of the GdkPaintable interface are the function make the paintable render itself and the 3 functions that provide sizing information:

void gdk_paintable_snapshot (GdkPaintable *paintable,
                             GdkSnapshot  *snapshot,
                             double        width,
                             double        height);

int gdk_paintable_get_intrinsic_width (GdkPaintable *paintable);
int gdk_paintable_get_intrinsic_height (GdkPaintable *paintable);
double gdk_paintable_get_intrinsic_aspect_ratio (GdkPaintable *paintable);

On top of that, the paintable can emit the “invalidate-contents” and “invalidate-size” signals when its contents or size changes.

To make this more concrete, let’s take a scalable SVG as an example: The paintable implementation would return no intrinsic size (the return value 0 for those sizing function achieves that) and whenever it is drawn, it would draw itself pixel-exact at the given size.
Or take the example of the animated GIF: It would provide its pixel size as its intrinsic size and draw the current frame of the animation scaled to the given size. And whenever the next frame of the animation should be displayed, it would emit the “invalidate-size” signal.
And last but not least, GdkTexture implements this interface.

We’re currently in the process of changing all the code that in GTK3 accepted GdkPixbuf to now accept GdkPaintable. The GtkImage widget of course has been changed already, as have the drag’n’drop icons or GtkAboutDialog. Experimental patches exist to let applications provide paintables to the GTK CSS engine.

And if you now put all this information together about GStreamer potentially providing textures backed by GL images and creating paintables that do animations that can then be uploaded to CSS, you can maybe see where this is going

Drawing in GTK+

The topic of how GTK+ draws the content of a window is a fairly complex one; it involves drilling down from GtkWidget, to GdkWindow, to Cairo, to the windowing system currently in use. This task can seem somewhat daunting, even for people that are familiar with the GTK+ API from an application development standpoint, so I decided to write down a quick introduction of how GTK+ draws, going from widgets, to windows, to surfaces, to native windowing resources.

How it starts

GTK+ always draws because something asked it to. This request may come from the windowing system — for instance, because the window manager presented your application window to the user, or because the user resized it — but more often it’ll come from a widget updating its contents. Let’s say, a progress bar going from 50% to 60%; or a label, changing its text; or a spinner, doing a new iteration. This request invalidates the backing GdkWindow of the widget — which usually it’s the GdkWindow of the top-level GtkWindow that contains the widget. Each invalidation carries with itself the region of the window to be invalidated (the “damage”), so that when we get to actually drawing, we know which parts of the window need to be updated, and we can avoid drawing outside of the damaged areas.

Race the clock

The first invalidation will start the “frame clock”; this clock is an object that keeps track of each phase inside a frame, like painting windows, laying out widgets, or processing the event queue. This allows GTK+ to be synchronized to things like the windowing system compositor, and to avoid performing unnecessary work that won’t be seen by the user — for instance, drawing something at 1000 frames per second when your display can only run at 60 Hz.

Once the clock reaches the “paint” phase, we process all the scheduled updates on a window; this will cause a GDK_EXPOSE event to be emitted. The GDK_EXPOSE event contains the GdkWindow that needs to be updated, and the union of all the invalidated areas. It’s important to note that, by and large, only top level windows will receive a GDK_EXPOSE event; for historical reasons, though, some widgets may apply a particular event mask that will cause GDK_EXPOSE events to be delivered to them as well. You should not write code that depends on that, and if you have legacy code ported from older versions of GTK+ 2.x you should really consider dropping the GDK_EXPOSURE_MASK from the event mask.

Rendering

GTK+ takes the window and invalidated region out of the GDK_EXPOSE event and figures out which top level widget they belong to. Once that’s found, GTK+ will begin the actual rendering process. First of all, GTK+ will ask the GdkWindow to create a buffer where to draw the contents of the window; the buffer is going to be clipped to the region that needs to be drawn, and will be cleared with the background color of the window. GDK will create a “drawing context” — a transient object that keeps track of things like OpenGL and Cairo drawing. Then, GTK+ will ask the widget to draw itself using a Cairo context. For leaf widgets this means drawing themeselves on that context; for container widgets, this additionally means recursing through all their children. At the end of this process, GTK+ will end the frame by telling GDK to take the buffer that contains all the rendered widgets and use it to replace the current contents of the window. GDK will then ask the windowing system to present the window to the user, whenever it’s more appropriate.

Changing History

The process outlined above has various caveats, and the code that deals with invalidation and validation of windows inside GDK is fairly complex; it also has a long history, which means that its API is littered by the headstones of ages past.

Before GTK+ 3.0, for instance, you were supposed to handle the “expose” events yourself, and create a Cairo context to draw on a widget by using gdk_cairo_create(); this has long since been unnecessary, because the GtkWidget::draw virtual function already provides us with a Cairo context with which to draw. The gdk_cairo_create() function, though, has been deprecated in GTK+ 3.22, and should not be used in newly written code; if you need a Cairo context you should create a similar Cairo surface, call cairo_create() on it, and then use the surface as the source for the Cairo context that GTK+ provides to you when drawing a widget. On the other hand, if you were using gdk_cairo_create() to draw on a top-level, native GdkWindow in response to a GDK_EXPOSE event then you should use the newly added gdk_window_begin_draw_frame(), gdk_window_end_draw_frame(), and GdkDrawingContext API instead.

Shaping Future

The internals of the drawing code in GTK+ have been progressively updated over the years, to cope with things like new windowing systems, as well as other rendering API. It’s fairly certain that they will change again, especially when it comes to improving the rendering performance. Many of the changes that may seem arbitrary are, in reality, stepping stones towards reducing the time spent inside the toolkit in each frame, and leave more time to the application logic.