Things I wanted to know about libxcb

I've wanted to look at libxcb for quite some time as the "saner API for X programming". In the end I managed to write enough code to have a functional X application, but the main problem is a lack of documentation, which is unfortunately common for open source project. And since I've written down some notes it would be shame not to publish them.

Generally libxcb gives you better control over the X connection but you pay by increased complexity that is needed to implement the same functionality as compared to classical Xlib. Happily there are xcb-util-* libraries that are helping a bit.

Debugging

You can actually get decent error reporting from libxcb but it's not that simple. First of all unless you handle errors manually and use the xcb_foo_checked() function variants, error checking is asynchronous and the errors are reported as events with response_type zero.

Another complication is that all the error details are encoded as numbers, so you have to translate these into strings. There is a nice xcb-util-error library that does exactly that, but unfortunatelly it's not present on most distros yet despite the fact that it has been developed in 2015. Happily it's just a small library. I've just downloaded the C sources and put them into the project directory so that it's linked with the program.

Example error handler is here:

void handle_event(xcb_connection_t *c, xcb_generic_event_t *ev, ...)
{
    switch (ev->response_type & ~0x80) {
    case 0: {
        xcb_generic_error_t *err = (xcb_generic_error_t*)ev;
        xcb_errors_context_t *err_ctx;
        xcb_errors_context_new(c, &err_ctx);
        const char *major, *minor, *extension, *error;
        major = xcb_errors_get_name_for_major_code(err_ctx, err->major_code);
        minor = xcb_errors_get_name_for_minor_code(err_ctx, err->major_code, err->minor_code);
        error = xcb_errors_get_name_for_error(err_ctx, err->error_code, &extension);
        printf("XCB Error: %s:%s, %s:%s, resource %u sequence %u\n",
               error, extension ? extension : "no_extension",
               major, minor ? minor : "no_minor",
               (unsigned int)err->resource_id,
               (unsigned int)err->sequence);
        xcb_errors_context_free(err_ctx);
    }
    case XCB_BUTTON_PRESS:
    ...

    }

    ...
}

I've commented out graphic context initialization in my application and got this error:

XCB Error: GContext:no_extension, CopyArea:no_minor, resource 31457283 sequence 47

Here you can easily see that the failure occured at xcb_copy_area() function and that the problem was wrong gc context parameter, which is much nicer than staring at black window and trying to figure out what went wrong.

Blitting a bitmap

With error reporting in place we can actually start writing actual code that draws on the screen. In order to do that we have to handle expose events and repaint particular part of the window. However before we can do that we have to figure out the format for the pixel data we want to send to the X server and unfortunatelly the information is scattered around in different structures.

Querying bitmap format

We start with a visual, which is a description of a pixel.

When window is created we pass a visual id to the xcb_create_window() function, which is usually obtained from a xcb_screen_t structure as screen->root_visual.

We have to match the visual id against list of all supported visuals to get the xcb_visualtype_t structure with the actual pixel description.

X server supports palette, grayscale and RGB pixels, which are called classes and are accesible via the visual->_class structure member, see constants XCB_VISUAL_CLASS_FOO. My application only supports RGB and I doubt that I will encounter anything else these days, so we skip rest of the formats.

Other mebers describe offsets for the RGB channels in case of RGB pixels, or number of palette entries for paletted mode, etc.

xcb_visualtype_t *visual_by_id(xcb_screen_t *screen)
{
    xcb_depth_iterator_t depth_iter = xcb_screen_allowed_depths_iterator(screen);
    int depth;
    xcb_visuatype_t *visual = NULL;

    for (; depth_iter.rem; xcb_depth_next(&depth_iter)) {
        xcb_visualtype_iterator_t visual_iter = xcb_depth_visuals_iterator(depth_iter.data);

        depth = depth_iter.data->depth;

        for (; visual_iter.rem; xcb_visualtype_next(&visual_iter)) {
            if (screen->root_visual == visual_iter.data->visual_id) {
                visual = visual_iter.data;
                goto found;
            }
        }
    }

    printf("Failed to match visual id\n");
    return NULL;
found:
    if (visual->_class != XCB_VISUAL_CLASS_TRUE_COLOR &&
        visual->_class != XCB_VISUAL_CLASS_DIRECT_COLOR) {
        printf("Unsupported visual\n");
        return NULL;
    }

    printf("depth %i, R mask %x G mask %x B mask %x\n",
           depth, visual->red_mask, visual->green_mask, visual->blue_mask);

    return visual;
}

The visual itself is not enough in order to create the pixel buffer, we also need a format, that describes how long is our pixel, it's common that pixels are encoded in RGB 8 bits per channel while pixel is 32 bits long with 8 bits of unused space. For that we actually have to locate pixel format.

int bpp_by_depth(xcb_connection_t *c, int depth)
{
    const xcb_setup_t *setup = xcb_get_setup(c);
    xcb_format_iterator_t fmt_iter = xcb_setup_pixmap_formats_iterator(setup);

    for (; fmt_iter.rem; xcb_format_next(&fmt_iter)) {
        if (fmt_iter.data->depth == depth)
            return fmt_iter.data->bits_per_pixel;
    }

    return 0;
}

With that we can calculate the size of image buffer as bpp_by_depth(c, depth) * w * h and we know that channels are organized as described in the visual type structure.

Note also that the format data structure has also a scanline_pad member, it's common for the image horizontal lines to be aligned to 32bit boundaries a for performance reasons. Here we omit the padd as in the most of the cases we get 32bit depth on modern hardware which is aligned by definition.

Preparing a backing pixmap

Now that we know the exact pixmap format we can start with the drawing code. There are actually two options how to send pixmap to the X server. Either we send it with xcb_put_image() or, if the server and application are running on the same physical machine, we can use SHM which about one order of magnitude faster than passing the data via socket. In both cases backing pixmap will be created, which is then copied over the window on the expose event with xcb_copy_area().

The basic code that does not use SHM looks like this, the window and pixel_buffer initialization is left as an excercise to the reader:

xcb_screen_t *screen;
xcb_window_t win;
xcb_gcontext_t gc;
xcb_pixmap_t backing_pixmap;

struct pixel_buffer {
    int w;
    int h;
    int bytes_per_row;
    void *pixels;
} buffer;

void init_gc(xcb_connection_t *c)
{
    uint32_t values[] = {screen->black_pixel, screen->white_pixel};
    gc = xcb_generate_id(c);
    xcb_create_gc(c, gc, backing_pixmap, XCB_GC_FOREGROUND | XCB_GC_BACKGROUND, values);
}

void init_backing_pixmap(xcb_connection_t *c)
{
    pixmap = xcb_generate_id(c);
    xcb_create_pixmap(c, screen->root_depth, backing_pixmap, win, w, h);

    init_gc(c);
}

/*
 * Note that without copying the data to a temporary buffer we can send updates
 * only as a horizontal stripes.
 */
void update_backing_pixmap(xcb_connection_t *c, int x, int y, int w, int h)
{
    /* Send image data to X server */
    xcb_put_image(c, XCB_IMAGE_FORMAT_Z_PIXMAP, backing_pixmap,
                  gc, w, h, 0, y, 0,
                  screen->root_depth, buffer.bytes_per_row * h,
                  buffer.pixels +
                  buffer.bytes_per_row * y);

    /* Copy updated data to window */
    xcb_copy_area(c, backing_pixmap, win, gc, x, y, x, y, w, h);
    xcb_flush(c);
}

And finally the event handling that is the same both for the SHM and non-SHM case:

void handle_event(xcb_connection_t *c, xcb_generic_event_t *ev, ...)
{
    switch (ev->response_type & ~0x80) {
    ...
    case XCB_EXPOSE: {
        xcb_expose_event_t *eev = (xcb_expose_event_t *)ev;
        xcb_copy_area(c, backing_pixmap, win, gc,
                      eev->x, eev->y,
                      eev->x, eev->y,
                      eev->width, eev->height);
        xcb_flush(c);
    } break;
}

Is that all?

No, not really, there are a couple of things that still needs to be taken care of.

First of all there is a maximum request size that X server can handle, my X server returns maximal request size about 16MB, which is mostly fine unless I had display resolution over about 2000x2000 pixels, in that case full screen window backing pixmap would be over the size.

Hence we have to make sure the request for xcb_put_image() is smaller than that and split the request into several vertical stripes if it's not. The code for that would look like:

void update_backing_pixmap(xcb_connection_t *c, int x, int y, int w, int h)
{
    size_t req_len = sizeof(xcb_image_request_t);
    size_t len = (req_len + buffer.bytes_per_row * h) >> 2;
    uint64_t max_req_len = xcb_get_maximum_request_length(c);

    if (len < max_req_len) {
        xcb_put_image(...);
    } else {
        int rows_per_request = (max_req_len<<2 - req_len) / buffer.bytes_per_row;
        int y_off;

        for (y_off = y; y_off + rows_per_request < h; y_off += rows_per_request) {

            xcb_put_image(c, XCB_IMAGE_FORMAT_Z_PIXMAP, backing_pixmap, gc,
                          buffer.w, rows, 0, y,
                      screen->root_depth,
                      rows * buffer.bytes_per_row,
                      buffer.pixels + y * buffer.bytes_per_row);

        }
    }

    ...
}

And it's a good idea to send the image updates in a strips anyways, which is by the way, how the original Xlib does it, because that way the user will see the window being repainted slowly rather than waiting for the one big update once the whole image was transmitted over network.

SHM Images

The X server extension is build on the top of the SysV SHM. The application creates a SHM region, attaches it to the server.

int attach_shm(xcb_connection_t *c, int w, int h, int depth, void **addr, xcb_shm_seg_t *shm_seg)
{
    size_t size = w * h * bpp_by_depth(c, depth);
    int shm_id;

    shm_id  = shmget(IPC_PRIVATE, size, IPC_CREAT | 0600);
    if (shm_id == -1) {
        perror("shmget()");
        return 0;
    }

    *addr = shmat(shm_id, 0, 0);
    if (*addr == (void*)-1) {
        perror("shmat()");
        shmctl(shm_id, IPC_RMID, 0);
        return 0;
    }

    *shm_seg = xcb_generate_id();

    xcb_void_cookie_t cookie;
    xcb_generic_error_t *err;

    cookie = xcb_shm_attach_checked(c, *shm_seg, shm_id, 0);
    err = xcb_request_check(c, cookie);

    shmctl(shm_id, IPC_RMID, 0);

    if (err) {
        fprintf(stderr, "xcb_shm_attach_checked() failed");
        free(err);
        return 0;
    }

    return 1;
}

There are actually two different SHM modes and that depends on where the X server keeps the image data.

If the server keeps the data in RAM the SHM query reply info has the shared_pixmaps flag turned on you can create a pixmap from the SHM segment with xcb_shm_create_pixmap() which then behaves as any other drawable. That means that we can pass it to functions such as xcb_copy_area().

However if the X server keeps the images in graphic card memory we cannot simply memcpy() a data between these buffers and coputer RAM and we have to use the xcb_shm_put_image() instead. If you are wondering yes this mode is still fast, it's just ugly implementation details that are leaking into the application API.

int query_shm(xcb_connection_t *c)
{
        xcb_shm_query_version_reply_t *rep;
        xcb_shm_query_version_cookie_t ck;

        ck = xcb_shm_query_version(c);
        rep = xcb_shm_query_version_reply(c, ck, NULL);
        if (!rep) {
        fprintf(stderr, "Failed to get SHM reply\n");
        return 0;
    }

    fprintf(stderr, "SHM verision %i.%i\n", rep->major_version, rep->minor_version);

    if (rep->shared_pixmaps)
        fprintf(stderr, "Pixmaps are shared\n");

    if (rep->major_version < 1 ||
            rep->major_version == 1 && rep->minor_version == 0)
        return 0;

    return 1;
}

And lastly but not least as the SHM operations are asynchronous we need notifications when the server has finished copying the data from the buffer so that we can refrain from modifying the buffer during that period. For that to happen we can ask the functions to send a completion event that will, later on, land in our event queue. Unfortunatelly the event ids for extensions, remember the SHM is extension, are dynamically allocated so the event type is computed from SHM extension first event plus the XCB_SHM_COMPLETION offset.

int shm_completion_type_id(xcb_connection_t *c)
{
    return xcb_get_extension_data(c, &xcb_shm_id)->first_event + XCB_SHM_COMPLETION;
}

SHM sad story

There is unfortunately no way how to figure out if it's safe to use SHM or not.

Most applications just attempt to do the SHM intialization and fall back to sending the image over a socket if that fails. Which mostly works, but it may also happen that X sever will pick up unrelated SHM memory with the same id on a remote machine. I soved that by a simple hack that checks the '$DISPLAY' variable and turns off SHM if it does not start with colon, which seems to work most of the time.

links

social