Movatterモバイル変換

Jamey Sharp Bart Massey
Computer Science Department
Portland State University
Portland, Oregon USA 97207-0751
{jamey,bart}@cs.pdx.edu

Abstract

The X Window System has provided the standard graphical userinterface for UNIX systems for more than 15 years. Oneresult is a large installed base of X applications writtenin C and C++. In almost all cases, these programs rely onthe Xlib library to manage their interactions with the Xserver. The reference implementation of Xlib is as old as Xitself, and has been freely available in source form sinceits inception: it currently is a part of theXFree86 [xfr] distribution.

Unfortunately, Xlib suffers from a number of implementationissues that have made it unsuitable for some classes ofapplication. Most notably, Xlib is a large body of code.This is of most significance on small platforms such ashand-held computers, where permanent and temporary storageare both limited, but can also have performancedisadvantages on any modern architecture due to factors suchas cache size. In addition, because of Xlib's monolithicnature, it is difficult to maintain.

The authors' prior work on the X protocol C Binding (XCB) isintended to provide a high-quality but incompatiblereplacement for Xlib. While XCB is believed to be suitablefor most new application and toolkit construction, it isdesirable to support the large installed base of legacy codeand experience by augmenting XCB with an Xlib-compatibleAPI.

This desire has led to the construction of a new library,the Xlib Compatibility Layer (XCL), that isbinary-compatible with frequently-used portions of Xlibwhile being significantly smaller and easier to maintain.Benefits are demonstrated for both existing and newapplications written for Xlib. In particular, thesignificant share of existing knowledge and written materialabout Xlib remains applicable to XCL. Also, XCL cansignificantly ease the migration path from Xlib to XCB.

1 The X Window System

The X Window System [SG86] is thede facto standardtechnology for UNIX applications wishing to provide agraphical user interface. The power and success of the Xmodel is due in no small measure to its separation ofhardware control from application logic with a stable,published client-server network protocol. In this model, thehardware controller is considered the server, and individualapplications and other components of a complete desktopenvironment are clients.

Development of X began in 1984, and it has become a matureand stable specification that many vendors have implementedfor their particular hardware and operatingenvironments. There is now a huge installed base of clientapplications: X is available for most modern computersystems, and is typically the default on UNIX systems.

To date, most client software for X has been built on top ofone or more libraries that hide various details of theprotocol, as illustrated in figure 1. Manyapplications are built using a GUI toolkit, such atXt [AS90], Qt [Dal01], or GTK+ [Pen99]. Thesetoolkits themselves, however, are almost invariably built ontop of Xlib [SGFR92], a library that provides C and C++language bindings for the X Window System protocol. It isalso not uncommon to build applications directly atop Xlib.

**Figure 1:**Xlib's role in the X Window System
$\begin{figure}\begin{center}\input{xlib-arch.eepic}\end{center}\end{figure}$

2 Xlib and XCB

The authors' recent work has included development of a new Xprotocol C binding, XCB [MS01], that is intended as areplacement for Xlib in new applications and toolkits. XCBhas a number of interesting features, but the XCB API isquite different from the Xlib API: XCB is not intended as aplug-compatible replacement for Xlib in existingapplications.

XCB and Xlib are both designed to be C library interfaces tothe X Window System network protocol for X clientapplications. However, XCB exchanges Xlib's numerousfeatures for smaller size and enhanced performance. XCB isa lower level API than Xlib: much of the built-infunctionality of Xlib (such as caching, display propertymanagement, and internationalization) is expected to insteadbe implemented as separate libraries above XCB, with XCBhandling only interaction with the X server at the protocollevel.

Some of the Xlib features omitted from XCBare important for getting reasonable performance or properbehavior from an X application. Nonetheless, we believe thatthese features donot belong in the core protocol library, and should insteadbe built on top of XCB.

Some of the differences between Xlib and XCB are worthy ofdetailed consideration.

2.1 Code Generation

The reference implementation of Xlib's core X protocolsupport alone consists of around 400 files comprising about100,000 lines of hand-written code. Though the X protocolhas a well designed mechanism for protocol extension, andthe XFree86 X server allows for reasonably straightforwardserver-side implementation of these extensions, Xlib has notmade the client-side task particularly easy.

In contrast, XCB provides a domain-specific language forspecification of the binary encoding of the core protocoland of the majority of extensions. Automated tools translatethese protocol descriptions into C implementations. Thedomain-specific language has significant advantages formaintenance, as well as for implementation of new featuresand extensions in XCB.

Protocol descriptions may be easily verified against thepublished specifications for the core X protocol andextensions. Experience with XCB has shown that a briefinspection of the XCB protocol description for a brokenrequest can quickly lead to a correct fix. This benefit isdue to the language making the structure of a request clear,while hiding implementation details in a single location inthe source: the protocol-to-C translator.

This encapsulation has also made possible the implementationof a number of useful features, including request marshaling(described next) and tracing of requests and events. Withouta common nexus for implementation of these features, theeffort required to implement them would have beenprohibitive.

2.2 Request Marshaling

For requests that pass to the server a list ofindependent items, such as a collection of arbitrary linesegments, Xlib provides a form of transparent batching knownas request marshaling. Marshaling allows client applicationsto draw individual lines, for example, from differentsections of code in rapid succession, without incurringoverhead from many similar request headers being sent to theserver with small payloads.

Most requests are unsuitable for marshaling, for any ofseveral reasons. If the request is expected to return areply, combining requests would cause too few replies to begenerated. Many requests send only a fixed-length datasection with their request header: in these cases, there isgenerally no room to place additional data. In some cases,requests are not idempotent, so that combining requestswould result in different behavior than sending the requestsindividually.SetClipRectangles is one example of thiscase. (In this paper, we will refer to X protocol requestsby the name given to them in the X protocol manual. We willrefer to Xlib or XCB functions by their function name: forexample,XSetClipRectangles.)

XCB also supports marshaling, but treats it as a requestattribute that may be specified in the description of thebinary encoding of the X protocol. Because of this design,marshaling in XCB is generically supported anywhere in thecore protocol or in any extension, although it should onlybe used if the conditions for correctness given above aresatisfied. In the reference implementation of Xlib,marshaling is only used in the places where it is mostlikely to help: for instance, it will marshal multiple callstoXDrawLine but not calls toXDrawLines. Whilethis is a wise optimization of programmer time when eachfunction is hand-coded, XCB's use of automated codegeneration techniques allows it to be more thorough.

2.3 Latency Hiding and Caching

XCB has an easy-to-use latency hiding mechanismintended to allow client applications to get useful workdone while waiting for information to return from theserver. In XCB, this is accomplished by returning only aplaceholder for an expected reply to a server request.Acquiring the placeholder takes no more time than would arequest that does not expect a reply. The application canthen convert this placeholder into the actual reply data atany time. When the actual data is requested the requestingthread may or may not block. Obtaining the reply datarepresented by the placeholder takes a small amount of timein most cases: it is quite likely that a request's replydata will already be available by the time the applicationasks for it.

The reference implementation of Xlib has a mechanism for thesame purpose, called an async-handler, based on a callbackmodel where one registered function after another is calleduntil one of the callbacks claims responsibility for thedata. However, use of Xlib's mechanism is considerably morecomplex.

Xlib also does extensive client-side caching. Implementationof various caches built on top of XCB is planned as futurework, but XCB does no caching. The assumption is thatcaching can be better managed by a higher layer, for examplea toolkit (such as Xt) or a convenience library. In someinstances, caching is actually undesirable, either becausememory is scarce or because an application needs to knowwhat requests it is generating.

2.4 Thread Safety

Following the successful application of threads inJava's Swing GUI framework, XCB has been designed from thestart with the needs of multi-threaded applications in mind,while still supporting single-threaded clientapplications. While Xlib was designed to supportthreaded applications, and while that support is notunusable, there are known race conditions that cannot beeliminated without changing the Xlib interface. Inparticular, if an application needs the protocol sequencenumber of the request it is making, Xlib forces it to makethe request and, without any lock held, query the lastsequence number sent. If another thread issues a requestbetween these two events, the first thread will obtain thewrong sequence number. XCB simply returns the sequencenumber from every request.

2.5 Interfaces

Xlib and XCL have different function signaturesfor each request in the protocol. In some cases, such asXlib'sXCreateSimpleWindow, the differences aresubstantial. Still, in many cases these are simplere-orderings of the same parameters, since both librariesprovide APIs for the same X protocol.The ordering of fields in the protocol specificationis optimized primarily for packing efficiency. Xlib uses aparameter ordering based on the parameter ordering of thereference library from version 10 of the X specification.Since X10 compatibility is no longer an important issue,XCB attempts to re-usethe existing X11 protocol documentation by keeping itsparameter order identical to the order of the fields in thebinary protocol specification.

2.6 Feature Comparison

Thanks to the modular design of XCB, applications needonly link against the small quantity of code that theyactually need. In contrast, Xlib is a monolithic librarysupporting a variety of disjoint feature sets. Somefeatures are not directly related to the core protocol:

3 XCL: Xlib Compatibility Layer

XCB has significant advantages over Xlib in someenvironments, especially small platforms such as hand-heldcomputers. However, there are more than 15 years' worth ofapplications and toolkits built using Xlib. For mostexisting software, the benefits obtainable by using XCB areoutweighed by the effort required to port to it. As analternative, the XCB design allows it to be used as areplacement ``lower layer'' for Xlib, efficiently handling Xprotocol interactions for existing software.

The Xlib Compatibility Layer (XCL) library is an attempt toprovide an Xlib-compatible API atop XCB. While Xlib doesattempt to layer portions of its implementation, thedivision between upper and lower layers in Xlib is onlyvisible upon careful examination of the library.Identifying this split is key to providing anXlib-compatible C library implementing the most commonlyused portions of Xlib as a layer on top of XCB, asillustrated in figure 2.

**Figure 2:**XCL's role in the X Window System
$\begin{figure}\begin{center}\input{xcl-arch.eepic}\end{center}\end{figure}$

3.1 Xlib Coverage

XCL provides a significant fraction of the functionality ofXlib. This includes adapters from Xlib's protocol requestfunctions to XCB's, implementation of specific predicatesfor searching XCB's event queue, and a re-implementation ofthe resource database.

Xlib's interfaces for text internationalization are notprovided by XCL. A significant portion of the code size ofXlib is dedicated to translating text strings betweenvarious character sets and encodings: XCL deals strictlywith glyph rendering using the encoding-neutral interfacesof Xlib. Currently, few Xlib applications and toolkitsactually use the internationalized interfaces. Given thedecreasing relevance of Xlib's font and text drawing supportand the predominance of ASCII encoded text, applications arenot expected to need international text renderingfunctionality from XCL in the future.

Xlib color management is not supported by XCL. The Xlibimplementation of this function is by way of integrating anexisting color management library: it would be difficult toeither duplicate or import this functionality in XCL. Inaddition, as discussed earlier, few existing applicationsuse the Xlib color management. Many applications use thenon-uniform RGB color space provided by Xlib by default.Most applications that do require more sophisticated colormanagement obtain this functionality through toolkit APIs orthird-party libraries.

XCL imports some of the caching implementation of Xlib, forexample the GCValues cache. In addition, some additionalcaching is planned, for example an Atom cache. However,substantially less client-side caching is performed by XCLthan by the reference Xlib implementation. This decision isa design tradeoff that may need to be revisited as moreperformance data becomes available.

3.2 Source vs. Binary Compatibility

Many applications and libraries that use Xlib have sourcecode available. In initially specifying requirements forXCL, it was deemed necessary that XCL be at leastsource-compatible with Xlib, but not necessarilybinary-compatible. This weaker requirement leaves open thepossibility of sacrificing compatibility of implementationdetails such as CPP macros and underlying data structures inorder to simplify and ease the implementation.

As it turns out, making XCL largely binary-compatible withXlib is straightforward. The majority of X applications canuse the shared library version of the current XCLimplementation as a transparent replacement for the Xlibshared library. Some applications or libraries use obscurefeatures or unpublished interfaces, and may thus need to berecompiled or adapted at the source level.

3.3 Using XCL

XCL is expected to be useful to a variety of differentaudiences. While each of these target uses is limited,together they cover a broad spectrum of X softwaredevelopment activities. This section enumerates andevaluates some of the expected uses of XCL.

3.3.1 Porting From Xlib to XCB

Applications or toolkits written for Xlib may be ported toXCB in stages using XCL. By simply includingxcl.hinstead ofXlib.h, code gains access to the XCBconnection structure, and can mix calls to XCB with calls toXCL. Thus software can evolve over time from being entirelyXlib based to being entirely XCB based, claiming the fullbenefits of XCB.

There are a few cases where the effect of mixing calls toXCL and XCB can be problematic. For example, attempts tomanipulate the same graphics context from both XCB and XCLmay have unexpected results and lead to client software errors.However, XCL is largely stateless, with most of its stateencapsulated by XCB. Most calls to either XCL or XCB shouldthus produce consistent and predictable effects.

3.3.2 Writing Threaded Code

Xlib is ``thread-safe'' in the sense that its internal stateinvariants are protected against simultaneous access bydifferent threads. However, the implementation of threadsafety in Xlib is problematic. Internally, a single globallock protects most Xlib state, leading to the possibility ofunnecessary interference between unrelated threads and aresulting loss of client efficiency. In addition, the Xlibinterface to threading is awkward.

Implementing threaded programs to the XCL interface has acouple of advantages. As noted in the previous section,calls to XCB's more thread-friendly interfaces can be usedto replace sections of Xlib code of problematic correctnessor efficiency. In addition, the finer-grained lockingoffers the potential for performance enhancement.

3.3.3 Leveraging X Knowledge

There are a wide variety of books available describingprogramming at the Xlib level, as well as web sites andother documentation. Since the XCL interface is a largeproper subset of the published interface of Xlib, all ofthese resources can still be used by those wishing to learnto write X applications. Those already familiar with Xlibprogramming can also avoid a steep learning curve whilegaining some of the advantages offered by XCL and XCB.

3.3.4 Understanding X Internals

The implementation of XCL may be instructive tothose interested in learning about the inner workings of theX Window System. The division of XCL into an upper layerconsisting of code borrowed from Xlib and a lower layerreplacing some of the more complex and confusing portions ofXlib with calls to XCB may help to illuminate the structureof Xlib and of X protocol bindings in general.

As another example, the transparent translations to theprotocol performed by XCB may provide opportunities forbetter understanding the runtime behavior of Xlib. Forexample, consider the Inter-Client CommunicationsConventions Manual (ICCCM) [Ros] that governsinteractions between applications and the desktopenvironment. The ability of XCB to accurately specify andprecisely report the ICCCM interactions of an Xlib windowmanager running atop XCL may be of great benefit in bothvalidating the window manager and increasing developerunderstanding of ICCCM.

3.3.5 Developing Toolkits

XCB's latency hiding mechanisms and simple design areexpected to be particularly useful to toolkit authors, whocan implement optimizations unavailable through Xlib. Thatexpectation applies also to this work. XCL allows toolkitauthors to implement XCB-based optimizations in portions oftheir toolkits needing high performance, while leaving therest of their Xlib-oriented code base intact. Better yet,toolkits may benefit from further performance gains as XCLand XCB are developed further to support caching and otheroptimizations.

3.3.6 Accommodating Small Environments

Until XCB becomes widespread on standard systems,applications and toolkits targeted exclusively to XCB willbe rare. On the other hand, applications and toolkitstargeted to Xlib comprise the bulk of available software.XCL supports these applications by implementing theimportant portions of Xlib in an extremely lightweightlibrary. The combination of XCL and XCB should provide aplatform for legacy X application execution significantlysmaller than Xlib. While development should eventuallyproceed towards new interfaces, support for legacyapplications in constrained environments can be a usefulfeature.

3.4 XCL Influences on XCB

While XCB is largely completed, its development continues inparallel with implementation of XCL. XCB as it currentlystands provides a good basis for the XCL implementation.However, in the process of defining XCL, some additionaldesirable XCB functionality has become apparent.

During the X protocol connection setup phase, a wide varietyof per-session server data is sent from the X server to theconnecting client. Xlib provides access to this data by wayof a plethora of convenience functions. While a cleanmechanism for accessing this information has yet to becompleted in XCB, a design using accessor functions,including iterators, is currently being implemented. In themeantime, XCB provides direct access to structure accessorsfor setup data. XCL hides the details of XCB's accessors bycopying the data into the XCLDisplay structure in anXlib-compatible form at connection setup time.

Another aspect of XCB that needs work is its X protocolerror handling. In the current implementation, XCB treats Xprotocol error responses and events similarly, placing errorresponses in the event queue as though they were events. Inaddition, XCB does not provide any callback mechanism forerror handling, so the only ways to discover that an errorhas occurred are to process all of the events that precedeit in the event queue or to search the event queue using theXCB API. The most significant impact of these XCB designdecisions on the design of XCL is that errors often arereported only after dozens of further requests have beenprocessed.

4 XCL Design

XCL copes with two principal APIs. On the client side, itprovides the published Xlib API. On the downstream side, itmakes calls to XCB. XCL never communicates directly with theX server: all of its requests and all server responses arerouted through XCB.

As discussed previously, XCL's overall architecture is atwo-layer affair similar to that of Xlib. Reuse of Xlibcode in the XCL upper layer eases verification of therequirement that XCL behave identically to Xlib. In fact,one of the implementation strategies used has been to copythe source for the relevant sections of the reference Xlibimplementation, remove sections deemed irrelevant to XCL,and replace them with XCB-based equivalent lower layer code.(Another strategy is discussed in section 4.3.)

It may seem that this implementation approach would make theimplementation of XCL relatively trivial, but that is notthe case. Different sections of Xlib are coupled togethertightly: for example, a significant number of utility andconvenience functions contain (sometimes slightly modified)copies of code needed to deliver requests to and acceptreplies from the X server. XCL makes explicit the idea thata lower layer (XCB) handles communication with the X server.XCL also allows other distinct libraries to handle otherjobs, such as caching or color management. In doing so, theinterface between the convenience functions and the serverhas been narrowed and standardized, facilitating future codereuse and maintenance.

XCL's current build process uses the Xlib header filesinstalled on the build machine for their definitions offunction prototypes and data structures as an aid tomatching the behavior specified for Xlib. This means that aslong as the same compiler is used, applications builtagainst these header files are sure to be binary-compatiblewith XCL, as long as they use only supported interfaces. Onthe other hand, it means XCL is tied to particular versionsof particular implementations of Xlib, a difficulty weencountered early in development. A more detailed analysisof Xlib headers is needed to identify those that should beincluded in the XCL source.

Another downside to reusing Xlib source in XCL is that Xlibsource is not always particularly easy to understand. Somedefects in XCL have been caused by misinterpreting thesemantics of the Xlib implementation. For example,XPending andXNextEvent were implemented incorrectlyduring early development of XCL. The referenceimplementation of Xlib includes functions_XReadEvents and_XEventsQueued: both may readevents from the X server, but_XReadEvents alwaysreads one or more events by blocking until some areavailable, while_XEventsQueued reads zero or morewithout blocking. As this distinction was missed in asuperficial reading, it was assumed that_XReadEventscould be used for both jobs. As a result, it was discoveredduring testing that test applications would freeze once awindow was mapped and the event loop was entered.

Since XCL reuses much of the source of an Xlibimplementation, there are a large number of small sourcefiles in XCL. Xlib has only one or two functions in most ofits source files, so that applications statically linkedagainst Xlib can draw in as little code as possible. Whilethis is a sensible plan for the large and monolithic Xlib,it makes less sense for the small and modular XCLimplementation. Future work for XCL includes organizingthe source appropriately for its environment.

It is worth noting that the license of the reference Xlibimplementation (the MIT license) allows development throughcode reuse. This licensing policy has made the current XCLimplementation possible. Had Xlib been developed under aclosed-source or limited-availability license, the task ofcreating XCL would have been much more daunting. Hopefully,making the source to XCB and XCL freely reusable will allowsimilar opportunities for future developers.

4.1XCBConnection andDisplay

Xlib uses aDisplay data structure to track a singleconnection to an X server and its state. XCB uses anXCBConnection structure for the same purpose. However,these structures have few similarities.

When the XCLXOpenDisplay function is called, it inturn callsXCBConnect to get anXCBConnection.It then sets up a newDisplay structure. In order tosupport applications that peek into theDisplaystructure, XCL copies a number of values from theXCBConnection into theDisplay, including the datafrom connection setup and the file descriptor associatedwith the server connection. In many applications, this datais part of the required binary interface: although Xlibprovides dedicated accessors for this data, the accessorsare implemented as CPP macros. It is therefore impossiblefor XCL to replace the accessors with functions that examineXCB's state directly without breaking binary compatibility.While binary compatibility is not a mandatory requirement ofXCL, it is reasonable to provide it when possible: thecurrent architecture permits this.

Early in the development of XCL, an important questionarose: how can anXCBConnection be associated with aDisplay so that XCL can retrieve the former from thelatter? The solution chosen is a standard one: XCLinternally manages a structure containing both theDisplay data and a pointer to anXCBConnection. TheXCBConnection pointer is used by internal operations,while the return value of XCL'sXOpenDisplay is apointer to theDisplay data.

4.2 XID Types

The X protocol identifies resources such as windows andfonts using XIDs, small integers that are generated by theclient rather than the server. In Xlib-based applicationsXlib handles generation of new XIDs as needed. XIDs aredeclared to be 32-bit integers by Xlib, using Ctypedef statements. This has the significant disadvantageof allowing a variety of type-safety errors: because C typesare structurally equivalent, it is easy to use a font XID,for example, where a window XID is required. To remedy thisproblem, XCB uses a distinct structure type for each kind ofresource, allowing the C type system to distinguish betweenthem and ensure that XIDs are used in appropriate contexts.

This causes some difficulty for XCL, however: XCL has todeal with and be able to convert between the two systems ofXID types. Conversion from XCB-style to Xlib-style types iseasy: the single member of an XCB XID structure is of thesame 32-bit integer type as the Xlib XIDs are derived from,and so a structure member access (effectively a typecoercion) is all that is needed to perform theconversion. Conversion in the other direction, however, isnot so simple.

C does not allow anonymous construction of structure-typedvalues. To work around this difficulty, XCL provides anumber of inline functions that declare a structure of anXCB XID type, initialize it with a value of an Xlib XIDtype, and return the structure by value. In principal, thiscould be a source of inefficiency. However, as the bitpatterns of the types are identical in both systems, a goodoptimizing compiler ought to be able to inline and thenentirely eliminate the conversion code. In fact, the GNU Ccompiler does a very good job at this.

4.3 Core Protocol Requests

With XCB handling actual communications tasks, XCL simplyneeds to manage the data it delivers between XCB and theclient application. Figure 3 shows theimplementation ofXInternAtom. The implementationdelivers anInternAtom request to the X server andextracts the atom XID from the reply data.

**Figure 3:**XCL implementation of XInternAtom.
$\begin{figure}\begin{quote}\begin{verbatim}Atom XInternAtom(Display *dpy,con...... = r->atom.xid;free(r);return (atom);}\end{verbatim}\end{quote}\end{figure}$

Most functions in XCL at this point are at least thissimple, if not simpler: when caching, color management,internationalization, and other miscellaneous features ofXlib are removed, the core protocol is the largest piecethat remains. Almost all of the code needed to implementthe core protocol is supplied by XCB. Functionality onlyloosely related to the X protocol should be placed inmodules separate from XCL itself, to facilitate maintenanceand avoid dragging large amounts of dead code along withevery X application.

Some XCL functions are even simpler to implement thanXInternAtom, as illustrated byXCreatePixmap infigure 4. As development of XCL hasproceeded, common structures became apparent amongst severalof the functions that had previously been hand-coded forXCL. In the current implementation, 43 of the roughly 120requests in the core X protocol are described using adomain-specific language. This language, related to thelanguage used by XCB, describes only the interface to Xlib.A translator to C, implemented in M4 [KR77], reads XCB'sinterface from XCB's description of the core protocol. Givendescriptions of both of the interfaces with which Xlibcommunicates, the translator can generate the correct codeto map parameters between XID type systems and functionparameter orderings.

**Figure 4:**XCL description of XCreatePixmap.
$\begin{figure}\begin{quote}\begin{verbatim}XCLREQ(CreatePixmap,XCLALLOC(Pixm...... unsigned int height,unsigned int depth))\end{verbatim}\end{quote}\end{figure}$

The same benefits expected from XCB's protocol descriptionlanguage (section 2.1) are also expected forXCL's interface description language: maintenance,implementation of new features and extensions, anddemonstration of correctness should all be simpler than forhand-coded implementations. The most immediate benefit isthat functions generated from this language are known to beimplemented to XCB's interface, and can be easily checkedagainst the published Xlib interface specification ratherthan against the Xlib reference implementation. Future workfor XCL thus includes extending this code generation systemto implement more requests, including extension requests.

4.4 Protocol Extensions

Since XCB can generate the code for all aspects of the Xprotocol defined by protocol extensions, implementation ofeach extension in XCL requires only a proper XCB descriptionof the extension, and any code needed to pre-processrequests to or replies from the server.

An example of an extension requiring more than a simpleprotocol description is the shared memory extension. Thisextension allows raw data to be transferred through sharedmemory segments when both the client and the server haveaccess to the same physical memory. XCB can only deal withthe X network protocol: shared memory is outside itsscope. Thus, an XCL implementation of the shared memoryextension would include code to access the shared memorysegment, while leaving to XCB the job of exchanging segmentidentifiers with the server.

4.5 Caching and Latency Hiding

XCL should have as low a latency as reasonably possible froma client call to the desired effect or response. There aretwo ways to accomplish this goal: hiding the effect oflatency from the client application, and removing sources oflatency by eliminating high-latency operations.

Caching is an instance of the latter strategy. Littlecaching is implemented within XCL itself, although graphicscontext values, for example, are cached using code borrowedfrom Xlib. Various cache modules will eventually be builtdirectly atop XCB: as these become available XCL will bemodified to use them. Until then, most requests through XCLwill produce protocol requests to the X server. This cansubstantially increase latency in client applications underXCL: fortunately the impact on client applications incurrent common environments is expected to be minimal.

XCB has latency hiding functionality, and it is desirable toextend this effect to XCL. Unfortunately, Xlib's interfaceis generally not conducive to this effort. In most caseswhere a reply is expected, Xlib's interface requires thecaller to block until the reply arrives from the Xserver. However, there are some cases where the interfaceallows for a potentially large number of requests to beprocessed in parallel.

XInternAtoms (the plural here is important) is onesuch case: given a list of atom names to map to atom IDs, itsends requests for all of the atoms to the server, and thenwaits for each of the replies. The Xlib version of thiscode, using the async-handler interface discussed insection 2.3, is significantly more complex thanthe XCL version built on XCB. XCL'sXInternAtomsimplementation contains considerably less code thanXlib's. (Although, to be fair, this is also because the atomcache has been decoupled from the library and is not yetimplemented).

4.5.1 Request Marshaling

Xlib's ability to marshal several independent client drawingrequests (such as line drawing requests) into a singleprotocol request is of minor importance for performance oncore protocol requests, at least on modern hardware.However, more recent X extensions, particular the Renderextension [Pac01a], depend on request marshaling forperformance. In these cases, marshaling opportunities arefrequent, and failure to do so may be expensive.

XCB provides request marshaling using a mechanismtransparent to clients. In XCB, the ability of a type ofrequest to be marshaled is considered an attribute of therequest, and is specified in the same protocol descriptionlanguage as is used to define the binary protocolencoding. Care has been taken to ensure that extensions andthe core protocol may be described with the same language,so request marshaling is available to any extension with nomore effort on the part of the extension implementor than anextra line of markup.

Because of the transparency of XCB marshaling, the primitivedrawing requests in XCL have been trivial to implement,while still performing competitively with their Xlibcounterparts. In fact, because only Xlib'sXDrawLine(singular) and similar functions marshal in the referenceimplementation, whileXDrawLines (plural) and othersdo not, some applications could theoretically experienceperformance gains just by re-linking with XCL: this could beespecially important over low-bandwidth or high-latencylinks, where marshaling in the core protocol has the mosteffect. Since marshaling is an intrinsic capability of XCBand not of XCL, all applications built on XCB gain theperformance benefits, not just clients built on XCL.

4.6 Threaded Clients

XCL's support for threaded applications is about ascomplete as the Xlib API will allow. Unfortunately, certainrace conditions are still possible, as discussed insection 2.4. Clearly, XCL cannot ensure thecorrectness of threaded programs written to the Xlibinterface. However, it can improve performance formulti-threaded applications.

Xlib uses a single lock to protect the entire contents ofitsDisplay data structure, meaning that for mostapplications all calls into Xlib are serialized. However,there is no reason in principle to disallow multiple threadsto access disjoint portions of the internal statesimultaneously: while one thread is accessing the eventqueue, another could be sending a request to the server, anda third could be querying the GC cache.

Currently, XCL allocates a single lock for the entireDisplay as Xlib does, but for protocol requests that lockis unused. XCB provides a separate thread-safety mechanismfor itsXCBConnection data that ensures that protocolrequests and responses are handled correctly. Any cachesimplemented atop XCB in the future can (and should) handletheir own data structures in a thread-safe fashion.

By leveraging XCB's thread-safety mechanisms, XCL providesfine-grained locking. This may allow multi-threadedapplications to take better advantage of the resources of asystem than the Xlib implementation would permit.

4.7 Events and Errors

Event and error handling represents a fairly significantportion of the implementation of XCL. Not coincidentally,this portion of the implementation is also one of the leastwell-specified and understood pieces of Xlib. Severalaspects of event and error handling warrant furtherconsideration.

4.7.1 The XCL Event Queue

XCB's event queue is accessed through simple traversalfunctions. However, this narrow interface is sufficient forthe XCL implementation of Xlib's event searchinterfaces. Xlib's predicated event retrieval functions mayall be implemented as traversals of the event queue,evaluating each event against some arbitrary predicate.

Internally, XCB has a generic linked list implementationthat it uses to track several distinct types ofinformation, including a list of replies expected from theserver. All users of these internal XCB lists must be ableto search for particular items: thus, when the predicatedevent queue code was added, its implementation added almostno new code to XCB. All of XCB's event queue managementimplementation combined amounts to only 20 lines of code.

Throughout the design and implementation of XCB and XCL,significant decreases in code size and improvements inmaintainability have resulted from the use of modular,layered architecture. Separating list implementations fromevent queue maintenance and event queue maintenance from XCLpredicate implementations provides a nice example of thisphenomenon. The modular, layered design providesflexibility, permitting such improvements asreimplementation of internal list interfaces atop new datastructures, or pooling of list nodes for reuse. This can beaccomplished without modifications to the rest of XCB orXCL, providing improvements transparently to all XCB and XCLclient applications.

4.7.2 X Protocol Errors

When XCB reports to XCL that an X protocol error has beenreceived, XCL passes the error off to an error handler asrequired by the Xlib API. The default error handler displaysa (somewhat) human-readable description of what went wrongand terminates the application: however, the handler may bereplaced by the client program with an arbitrary clientfunction. This mechanism is awkward to use: as a result,most existing Xlib applications exhibit poor behavior inresponse to protocol errors. Unfortunately, it is not clearhow to address that problem without changing the Xlib errorhandling API.

4.7.3 XCL and XCB Internal Errors

The reference implementation of Xlib performs variouserror checks on data coming both from Xlib's callers andfrom the X server. XCB simply delivers whatever data it isgiven. The primary benefit of this approach is that XCB maydeliver the data faster. An interesting side benefit is thatXCB allows for the creation of certain kinds of tools fortesting X servers by delivering bad input. Naturally, thetrade-off is that XCB does not particularly help a developerdebug a faulty client or server.

In most cases, XCL and XCB have identical failure modes toXlib, and XCL can return status codes identical to Xlib'swhen an XCL or XCB error occurs. However, in a small numberof instances, XCL can fail in ways that Xlib couldnot. First, XCL will detect failure in situations that Xlibdoes not. Second, there may be opportunities for the XCLimplementation to fail that were not present in the Xlibimplementation (as in theXDrawString example below).Unfortunately, Xlib has no particularly well-structured orapplication-honored mechanism for reporting errors. Thecurrent XCL implementation makes reasonable efforts tosensibly handle and report internal errors within the boundsof the Xlib API semantics.

For example,XDrawString has to break the string givento it into 254 character chunks, with a two-byte header perchunk. Xlib uses theDisplay's output buffer directly,while XCL usesmalloc to create a temporary buffer,subjecting XCL to potential out-of-memory errors. In XCL'scurrent implementation, we call the Xlib-compatible_XIOError handler whenmalloc fails.

Despite the fact that even the Xlib implementation ofXDrawString can fail (if the string provided has anon-positive length), the Xlib implementation always returnssuccess. It would be nice to develop better error handlingand reporting mechanisms for these cases within theconstraints of the Xlib API.

5 Results

The implementation of XCL is not yet quitecomplete. However, we have some promising preliminaryresults.

5.1 RXVT on XCL

The XCL development model includes an initial iterationimplementing enough of Xlib to support a singlereasonable-sized Xlib application. This is intended toserve as proof of concept, and as validation and evaluationof the design and implementation. The application we chosewas therxvt [rxv]terminal emulator, a more modern replacement forxterm.

Therxvt terminal emulator is related to the standardxterm utility in the same way that XCL is related toXlib. According to the manual,rxvt is ``intended asan xterm replacement for users who do not require featuressuch as Tektronix 4014 emulation and toolkit-styleconfigurability,'' with the benefit that ``rxvt uses muchless swap space.''

This similarity of purpose maderxvt an attractiveinitial target for XCL. In addition,rxvt is areasonably powerful X application that performs very usefulwork. Finally,rxvt exercises a large portion of thecore X protocol, including text, rendering and events.

XCL is currently complete enough thatrxvt, withoutany source code changes, may be linked against XCL ratherthan Xlib and will run correctly. In fact, the currentstable and development versions ofrxvt can be compiled againstXlib and then correctly executed against the XCL librarybinary. The first iteration of XCL development is thereforecomplete and successful. The remaining work is expected tobe largely straightforward, if time-consuming.

5.2 Gnomegw: GDK on XCL

As mentioned in section 3.3.5, one expecteduse of XCL is with toolkits originally written for Xlib. Totest the feasibility of this use, we selected one of thesimpler Gnome/GTK+ applications for trials on early versionsof XCL.gw is a graphical version of the venerableBerkeley ``w'' command used to list the userscurrently logged into a Unix system.

Thegw user interface contains a button, a list widgetwith a scrollbar, and a dockable menu bar. We found thatthese widgets need only a small subset of Xlib'sfunctionality. Getting the basic interface to run on XCLwas a simple matter: scrollbars scroll, buttons click, menusdrop down, and menu bars undock and re-dock. GTK+ andGnome/GTK+ applications are implemented using a windowsystem independent layer known as GDK. Since the XCLfunctionality necessary to getgw working covers alarge portion of the Xlib API subset used by GDK, there isreason to believe that a substantial fraction of the workneeded to port GTK+ in its entirety is already completed.

XCL is not perfect yet: some aspects of currentgwoperation atop XCL are visibly wrong, or trigger X protocolerrors that shut down the application.XPutImage andXSendEvent, among others, have been sources oftrouble. While fixing these problems may require substantialdebugging effort, no major technical barriers are expected.

5.3 Size and Performance

While the implementations of XCL and XCB are still subjectto change, it is nonetheless useful to take at least afirst look at the size and performance of the currentsystem. Here are some preliminary metrics:

5.3.1 Size

The size metric used is kilobytes of code in the textsection of the compiled object file. The text section needsto be stored both on disk and in memory, both of which arescarce on small systems. In addition, if the entire textsection can fit into the L2 cache of the target system(typically between 64kB and 512kB on modern machines), asignificant performance improvement might be possible fortypical applications making a mix of calls into the library.Including data and BSS size would not significantly impactthe reported measurements. All numbers are produced byexamining the text sections of either object (.o)files or statically-linked library (.a) files on anIntel x86 architecture machine. Our tests show that use ofthe optimizer of the compiler is a major factor in the sizeof both XCB and XCL. The numbers that follow for XCB andXCL are produced by analyzing the output of ``gcc-O2'': the unoptimized output is nearly 50% larger.

82% of the 79 object files currently comprising XCL havetext sections smaller than 512 bytes. The total text sectionsize of XCL is 28kB. XCB adds another 27kB, for a total of55kB.

Xlib, compiled from 417 source files, has a text sectionsize of 658kB. (According to Jim Gettys [Get01], muchof this is data used as lookup tables forinternationalization.) The subset of Xlib providing roughlythe same functionality as XCL has about 115kB of text. Thisis more than twice the size of XCL combined with all of XCB.In addition, it would be quite difficult to build a sharedlibrary version of Xlib containing just this functionality.

5.3.2 Performance

The XCL version ofrxvt currently seems to havetext display performance comparable to that of the Xlibversion. Executing ``cat /usr/share/dict/words'' on a1GHz Mobile Pentium III running Linux yields runtimes ofabout 1.3 seconds forrxvt 2.7.8 when linked witheither Xlib or XCL. Results are similar on a 700MHz Athlon,although XCL consistently exhibits about a 5% speedadvantage on this platform for this benchmark.

It is notable that, at about 30,000 scrolled lines persecond,rxvt is more than acceptably fast for anyreasonable use. Indeed, this is the normal case for Xapplications on modern hardware: it is extremely unusual tofind an application whose performance is limited byinteraction with an X server on the same host. In addition,Xlib is believed to provide near-optimal performance in mostsituations. For these reasons, performance enhancement isnot a primary goal of XCL.

6 Related Work

We know of no other efforts to design X protocol clientlibraries in C. There have been some independent efforts towrite such libraries for a variety of other languagesincluding Java [O'N02,Tse01], Common Lisp [SOC$^+$89],Smalltalk [Byr], and ML [GR93], as well as moreexotic languages such as Python, Erlang, and Ruby. Theseefforts have concentrated largely on providing naturalbindings for their target language, with performance, size,and compatibility with the Xlib API being at most secondarytargets.

The Nano-X [Hae01] GUI environment has someinteresting parallels to XCL. Nano-X is targeted atlightweight and embedded systems, and provides an APIroughly comparable to that of Xlib. However, Nano-Xsupports a variety of underlying rendering systems, only oneof which is the X core protocol. The X protocol support ofNano-X is intended primarily for development and debugging,and is not intended as a principal API for normalapplication use.

Over the years, Jim Gettys and others have put a significantamount of effort into improvements to the Xlibimplementation. Much of this work has been to increase Xlibfunctionality. More recently, Gettys has put some effortinto reducing the size of Xlib for use with Linux on theCompaq iPaq hand-held computer.

The authors' work on XCL was inspired to some extent by anaward-winning program from the 1991 International ObfuscatedC Code Contest [NCSB02]. Thisremarkable 1676 character C program by David Applegate andGuy Jacobson runs Conway's ``Game of Life'' cellularautomaton on an X root window. Its small size andremarkable performance provided a powerful hint of what ispossible with clever coding.

7 Status and Future Work

The XCL implementation is well underway: as noted in theprevious section, a useful test application (rxvt)links with and runs on XCL. When all Xlib protocol wrapperfunctions have been implemented in XCL, many moreapplications are expected to run without modification.

XCL is dependent on the XCB library implementation, which isnearly complete. In its current form, it provides themajority of the functionality needed for the XCLimplementation. Some work remains. For example, accessorsneed to be constructed for variable-element-length listssuch as those sent from the X server on connection setup,and XCB error handling needs further exploration.

Allowing a wider variety of applications to run on XCL isthe immediate focus as the project continues. One way toquickly support a large number of applications is to supportone or more GUI toolkits: preliminary results for GTK+ andGnome are promising, and we plan to further research thispossibility in the near future. As discussed insection 3.3.5, migrating toolkits to XCLmay be a good first step in eventually migrating them toXCB.

8 Conclusions

XCL, in conjunction with XCB, represents the first realalternative to Xlib since Xlib's inception more than 15years ago. As both are open source, the X developmentcommunity can examine both for their merits and producesoftware that is useful for a wide variety of platforms andapplications.

Much of the hype surrounding the development offreely-available UNIX software in recent years springs fromthe idea that open development and the use offreely-available source materials can produce high-qualitysoftware products that can then be used to bootstrap futuredevelopment in this style. XCL provides a nice example ofthis phenomenon, as well as being a useful tool in its ownright. In the immortal words of Hannibal Smith, ``I love itwhen a plan comes together.''

Availability

Current implementations of XCL and XCB are freely availableunder an MIT-style license athttps://xcb.cs.pdx.edu/.

Acknowledgements

The authors gratefully acknowledge the advice and assistanceof Keith Packard, Jim Gettys, and other X contributors inthe design and analysis leading up to XCL. Andy Howe hasplayed a major part in the implementation and testing ofXCL. Finally, Chris Demetriou was invaluable throughout thetough task of shepherding this paper.