Design Goals

  • Modularity. By making the language dependent parts loadable modules, we can speed up the development process by allowing teams working on different languages to work independently.
  • Font system and toolkit independence. As much as possible, the lowlevel parts of the system that encode language knowledge should be independent of the higher level parts. This will encourage re-use of the code in other projects and hopefully provide a cleaner design.
  • Comprehensiveness. Although we can't expect to cover every language in the initial implementation, the general framework should be good enough to provide correct rendering for a large set of languages.
  • Quality. Subject to the rendering system employed, the goal of Pango is not just legible rendering, but high-quality, correct rendering.

Architecture Overview

Language specific knowledge is broken off into loadable modules for each language. These modules are used to implement a low-level API that allows its users to properly handle all sorts of scripts without having language specific knowledge.

The Bigger Picture

Pango provides facilities, that are not necessarily easy to use. A toolkit builds on top of these facililities to provide:

  • Simple "draw string" rendering calls.
  • Text entry and label widgets that transparently support Unicode and multi-lingual text.

Other text handling facilities, such as a printing architecture, also build on top of Pango and provide simplified string-based APIs for their users.

Fonts

Internationalized text, as handled by Pango, makes large demands on the font system. Many features that, for western text are only of interest for high-quality typography, such as ligatures, and the selection of alternate glyphs for a character, are vital for rendering non-western languages. Also, a much larger range of glyphs are needed, and finally, when rendering multi-lingual text, one must be able to simultaneously encode multiple languages, so encodings that are limited to the character set for a single language are not useful.

The process of choosing a font is considerably more complex in a multi-lingual environment which supports multiple font rendering systems than it is in a more homogeneous environment. Several different font selection activities can be identified:

  • Specifying a font to use for a portion of text in a document.

  • Selecting a font to match a font in an existing document. It is generally desirable to allow the user to select whether they want to match exactly at the expense of appearance or match loosely to gain superior readability. A similar task is selecting a font on one medium to match a font on a different medium.

  • Selecting a font to use for a certain task; for example, selecting a font to use for a viewing on web page, or for a dialog. Note that when this is done interactively by a user, it is a somewhat different activity from when it is hardcoded into a program, because the user can select from the list of fonts actually installed on the system, while, when hardcoded into a program, the program must specify the font in a way that is generic for all systems.

    In either case, it may desirable to specify a font list instead of a font. A user may may want to use different fonts for different languages. Note that this usage of a font list is different from a common use of font sets in X where the a fontset includes the same font in multiple different encodings. If the same font exists in different encodings with different glyph sets, then it is merged in Pango by the font rendering technology, behind the scenes to appear as a single font. This is a difference between selecting fonts for a role where the text is unpredictable versus selecting fonts for a role where the text is predictable. When specifying a font to use for a portion of text in a document, a font list is not interesting because the user can simply choose a font that is suitable for the language of the text portion.

There are three types of font related objects in Pango. A FontDescription is an abstract description of a font. A FontList is a list of FontDescription, and finally, a Font is a the realization of a FontDescription as as particular font on the system.

The glyphs within a font are indexed by 32-bit integers which are opaque to the application. (In the case we mention above where we merge various fonts with different encodings into a single font, a portion of the encoding space would be used to index the fonts within the fontset, and a portion would be used to index glyphs within each font.)

A number of aliases are provided on a system-wide level that are used to get standard fonts with certain general characteristics. Programs should use these aliases for fonts that are specified internally. These aliases include "fixed", "sans", and "serif".

The process of turning a character string into glyph string requires knowledge of the glyph encoding of the font. For this reason, the Font object includes a method to retrieve an appropriate Shaper object as a function of Unicode character and language tag. The Shaper object is responsible for taking a string of characters and turning them into a string of glyphs. A particular implementation of a Shaper object is most commonly be shared between a group of related scripts for a particular font system. For instance, there would be a single shaper implentation for Indic scripts rendered with OpenType, and a single shaper for western (latin, greek, and cyrillic) scripts rendered using X fonts.

For more complicated scripts, such as the Indic scripts and Arabic, there may be a considerable amount of code used in the shaping process that is independent of the font system. In these cases, a two-level system can be used - the Shaper uses an AbstractShaper to convert characters to abstract glyphs and the Shaper then is responsible for converting those abstract glyphs into glyphs for the specific font.

The Layout and Rendering Pipeline

Layout and rendering in Pango involves several steps:

Itemization
The input string is broken into portions rendered with a consistent font, with a consistent language tag, and with a specific bidirectional embedding level.
Reordering
The items are reordered from logical order into visual order according to their bidirectional embedding levels.
Glyph Selection (Shaping)
The characters in each item are turned into glyphs.
Justification
The glyph strings created in the previous step are adjusted to fit the line-justification policies that are in place.
Rendering
The justified glyph strings are rendered in their final order onto the output device. This step is not, strictly speaking, part of Pango, although a rendering routine is included for X fonts Rendering is a specific to the output device.

Layout Objects

A caller of Pango could apply all of the steps in the above algorithm explicitely by themselves. However, that process is rather involved and would involve code duplication in various places. Therefore, a higher-level facility is provided in Pango for callers that do not need a highly-detailed level of contro. This is the PangoLayout object. A PangoLayout object represents a paragraph of text and is created by passing in a attributed text string. It handles all of the above steps internally except for rendering.

Text Handling

Text in Pango is represented, in most cases, as UTF-8 encoded strings. This representation, has a number of advantages as opposed to a fixed-width representation such as UCS-16:

  • Compatibility with existing Unix API's is maximized. The standard C library functions widely availabe now (such as sprintf()) continue to work.
  • Two sets of API entry points are not needed, since functions continue to take a char *.
  • The character set is extensible to the full range of ISO10646 without requiring escape mechanisms such as surrogate pairs.
  • UTF-8 requires no extra space for storing ASCII text, and has only a 50% penalty as opposed to UCS-2 for double-byte character sets.
  • UTF-8 is independent of byte-order

There are some disadvantages as well:

  • Other popular systems such as Microsoft products and Java have adopted UCS-2 as their encoding, necessitating conversions. (But UTF-8 appears to be the emerging standard for open-source applications.)
    • .
  • UTF-8 has a 50% penalty in space as opposed to UCS-2 for storing double-byte character sets.

Where individual characters are represented, they are reprented as 32 bit wide characters. This again provides forwards compatibility with the full range of ISO10646, and should incur minimal cost for local variables and parameter passing. This agrees with the type of wchar_t in the GNU libc library.

Offsets into a utf-8 string are represented as byte offsets, not character offsets. This is more convenient for processing, and although there is the problem of having invalid offsets into the data, note that given a string of Unicode text with combining characters, character positions may already be invalid, and break-iteration is needed to determine valid positions.

Conversion between character sets will be handled via via the iconv. A lot of new systems provide a decent implementation of iconv - noteably, GNU libc-2.1, and these systems can be used as "reference platforms"; for other platforms, it shouldn't be hard to write a simple table-driven iconv implementation that can handle the small amounts of data in a typical GUI reasonably efficiently. (various implementations of this are availale - e.g., Tom Tromey's libunicode, Bruno Haible's libiconv.)

Module System

There are roughly three separate groups of tasks that Pango involves. There are tasks that are independent of language and font system, such as:

  • Breaking the text into directional runs
  • High-level driver functions.

Then there are tasks that are dependent only on the language, such as:

  • Identifying word and character breaks within a string.
  • Converting from character strings to abstract glyphs when applicable.

Finally, there are tasks that depend both on language and the font system being used ,such as:

  • Conversion of characteres into glyphs
  • Placement of glyphs
  • Justification

The first set of tasks are performed within the core of Pango, the other tasks are performed within dynamically loaded modules.

API Design Principles

Pango is intended to be a cross-platform, cross-toolkit, low-level library. Convenience of the API is not the primary goal, althought it is a goal. Some of the principles that are used in designing the details of the API are:

  • Thread Safety It must be possible to use Pango in a threaded system. There must be no implicit state to operations, and if global variables are needed internal to Pango, they must be properly locked.
  • Memory Management All objects in the interface must e reference counted or copyable by value. All routines returning such a value will either return a new refcount or a new value.
  • Consistent Error Reporting Pango may be wrapped by systems with varying ways of reporting and handling error conditions. For this reason, runtime errors that do not indicate a programming mistake must be reported by means of a returned error indicator instead of an error message printed to standard error. Runtime errors that indicate a programmer error should generally also be reported to standard error, unless they should be trivially caught by a wrapper, in which case a error printed to standard error is appropriate. (It is not necessary to have a error return value indicating that a mandatory non-NULL value was passed in as NULL.)
  • Robustness Passing in invalid data should not crash Pango when at all avoidable. Diagnostics should be as explicit as possible.

Remaining Questions
  • Various maps are needed font description to font, from character, language, and font-system to shaper, and so forth. In 99% of all cases, only a single set of these maps will ever be needed. However, making these maps truly global poses a couple of problems. First, using global maps requires the library to have explicit locking to be thread-safe. Second, it could be considered ugly from an object-oriented design standpoint.


Last modified 22-Jun-2000
Owen Taylor <otaylor@redhat.com>