This specification specifies a Hypothetical Render Model (HRM) that constrains the presentation complexity of documents that conform to the Text Profiles specified in any edition of Internet Media Subtitles and Captions ([IMSC]).
The objective of the HRM is to allow subtitle and caption authors and providers to verify that the content they provide does not exceed defined complexity levels, so that playback systems can render the content synchronized with the author-specified display times.
The model is not intended as a specification of the processing requirements for implementations. For instance, while the model defines glyph cache for the purpose of modelling how the number of glyph drawing operations can be reduced, it neither requires the implementation of such a cache, nor models the sub-pixel glyph positioning and anti-aliased glyph rendering that can be used to produce text output.
Furthermore, the model is not intended to constrain readability complexity.
As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.
The key wordSHALL in this document is to be interpreted as described inBCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
The objective of the HRM is to allow subtitle and caption authors and providers to verify that the content they provide does not exceed defined complexity levels, so that playback systems can render the content synchronized with the author-specified display times.
Playback systems include desktop computers, mobile devices and home theatre devices.
Note
The HRM is not a new concept: a version of it has been included in all versions and editions of [IMSC]. This specification extracts the HRM into a standalone document to simplify maintenance. The First Public Working Draft of this specification essentially included the HRM as it was specified in [ttml-imsc1.2]. Substantive changes made since then are summarized inF.Summary of substantive changes.
IMSC Document Instances are typically authored by a first party and rendered by a second party. Unless both parties agree on the maximum complexity of aIMSC Document Instance, it is likely that:
anIMSC Document Instance authored by a first party will exceed the capabilities of thepresentation processor of a second party, resulting in an incomplete presentation where some subtitles or captions might be missing or might be presented too late;
a first party authors only extremely simpleIMSC Document Instances in an attempt to ensure a complete presentation across all renderers, resulting in a lower quality presentation;
a second party over-provisions theirpresentation processor in order to ensure a complete presentation of allIMSC Document Instances, resulting in increased run-time resource usage, code complexity, etc.; or
onepresentation processor becomes the de-facto reference to determine whether the presentation of aIMSC Document Instance will succeed, at the expense of other renderers and consistency of presentation.
As illustrated inFigure1, by defining a method (the HRM) to compute a proxy for the complexity of anIMSC Document Instance and specifying a complexity limit based on such proxy:
implementers can design theirpresentation processors to maximize the likelihood that they will be able to render correctly allIMSC Document Instances that do not exceed this limit; and
audiences can expect subtitle and caption presentation to match the authorial intention.
5.3Why is the HRM needed to limit complexity?
The HRM supplements the syntactic and structural constraints imposed in [IMSC] by imposing constraints on the contents of the presentation.
Because of the temporal and spatial variability of subtitles and captions across types of content, territories and languages, it is not possible to limit the complexity of anIMSC Document Instance using only average values.
An average-based constraint of 840characters per minute could be met in multiple ways, with different rendering complexities. Contrast two potential approaches:
In the first, 5characters are presented for a fraction of a second, followed by 835characters that are then presented for over 59 seconds. This generates a high rendering complexity for the 835characters, since there is only a brief time available to paint them.
In the second, 210characters are painted every 15 seconds, giving 15 seconds to prepare for the next presentation. This has a much lower rendering complexity.
The HRM achieves a more accurate representation of the complexity of anIMSC Document Instance at any given time by taking into account its past complexity in addition to its instantaneous complexity. The same approach is commonly used in video to limit bitstream complexity, e.g., the Hypothetical Reference Decoder (HRD) specified in [iso14496-10].
5.4How does the HRM measure and limit complexity?
The HRM defines a simple model for the rendering of subtitles and captions, and uses the time it takes to render subtitles and captions according to that model as a proxy for the complexity of the subtitles and captions. Rendering includes drawing region backgrounds, rendering text and copying text. Complexity is then limited by requiring that the time to render one subtitle or caption is shorter than the time elapsed since the previous subtitle or caption.
This simple model requires only a static analysis of theIMSC Document Instance, requires no fetching of external resources and does not require theIMSC Document Instance to be actually rendered. Several simplifying assumptions are made to achieve this. For example, the model assumes that eachcharacter is drawn independently, and accounts for that assumption being, in many cases, false, by assigning different render speeds for different scripts. In general the model is not intended to capture the actual time that an implementation takes to render subtitles and captions, but rather scale with it: a document that is twice as complex according to the model would require roughly twice as many resources to actually render.
5.5Where is the HRM used?
The HRM is typically used prior to distribution of theIMSC Document Instance to the end-user, as an integral part of authoring and as a quality check before distribution.
When the HRM is used, the consequences of anIMSC Document Instance exceeding the HRM limits depends on the context:
an authoring system might, for example, flag the specific point in time where the HRM limits were exceeded;
the ingest component of a streaming platform might outright reject theIMSC Document Instance.
The HRM is not intended to be used when theIMSC Document Instance is presented to end-users since:
end-users are not concerned with a technical complexity measure, just as they are not concerned with video bit-rate, but instead with whether the presentation is successful;
non-empty ISDs are processed using a simple double buffering model: while anon-empty ISD En is being painted into a Back Buffer by thePresentation Compositor, the previousnon-empty ISD Em is available for display in a Front Buffer; and
empty ISDs merely disconnect the Front Buffer from the display when presented, i.e., presenting anempty ISD is equivalent to presenting nothing.
The model specifies a (hypothetical) time required for completely painting anon-empty ISD as a proxy for complexity. Painting includes clearing the Back Buffer, drawing region backgrounds, rendering glyphs, and copying glyphs. Complexity is then limited by requiring that painting ofnon-empty ISD En begins no earlier than the presentation time of the previous non-emptynon-empty ISD Em and completes by the presentation time of En.
In contrast, there is no complexity involved connecting and disconnecting the Front Buffer from the display, and thus no complexity associated withempty ISDs.
Whenever applicable, constraints are specified relative toRoot Container Region dimensions, allowing subtitle sequences to be authored independently of theRelated Video Object resolution.
To enable scenarios where the same glyphs are used in multiple successiveIntermediate Synchronic Documents, e.g. to convey a CEA-608/708-style roll-up (see [CEA-608] and [CEA-708]), a Glyph Cache stores rendered glyphs acrossIntermediate Synchronic Documents, allowing glyphs to be copied into the Presentation Buffer instead of rendered, a more costly operation.
Note
The HRM permits a maximum rate of 12Intermediate Synchronic Documents per second. This is ultimately limited by theBDraw parameter and is intended to capture processing and presentation overhead. When converting a [CEA-608] signal to IMSC, it is therefore impossible to createIMSC Document Instances that generate anIntermediate Synchronic Document for every [CEA-608] packet, which are sampled at the video field rate. It is instead preferable to coalesce sequences of [CEA-608] packets into longer groupings, such as words, phrases, complete lines or paragraphs before creating anIMSC Document Instance, and let thepresentation processor perform any desired animation, e.g., typewriter effect.
Each of the termsPresentation Compositor, Glyph Renderer and Glyph Copier is defined by the algorithmic requirements defined for it in this specification.
at the presentation time of Em, where m is the largest non-zero value that is both less than n and is such that Em is not anempty ISD, if the presentation time of En minus that of Em is less thanIPD; or
at the presentation time of En minusIPD, otherwise.
Figure3 illustrates the rendering and presentation ofIntermediate Synchronic Documents, where the hatched areas indicate time spent drawing the associatedIntermediate Synchronic Document. For example, thePresentation Compositor begins rendering E1 at the presentation time of E0 since E1 is not anempty ISD. In contrast, thePresentation Compositor begins rendering E5 at the presentation time of E5 minus IPD since (i) both E3 and E4 areempty ISDs and the presentation time of E5 minus that of E2 is greater than IPD. Furthermore, E2 remains in the Front Buffer until the presentation time of E5 but is not presented while E3 and E4 are presented, during which time the Front Buffer is not available for display. Finally, thePresentation Compositor begins rendering E0 at the presentation time of E0 minus IPD since E0 is the firstIntermediate Synchronic Document.
The contents of the Back Buffer are transferred instantaneously to the Front Buffer at the presentation time of anon-empty ISD En, making the latter available for display.
The Front Buffer is:
disconnected from the display while anempty ISD is being presented; and
connected to the display, otherwise.
Note
It is possible for the contents of the Front Buffer to never be displayed. This can happen, for example, if the Back Buffer is copied twice to Front Buffer between two consecutive video frame boundaries of theRelated Video Object.
BDraw effectively sets a limit on fillings regions - for example, assuming that theRoot Container Region is ultimately rendered at 1920×1080 resolution, aBDraw of 12 s-1 would correspond to a fill rate of 1920×1080×12/s=23.7×220pixels s-1.
For a region Ri in withtts:extent="250px 50px" within aRoot Container Region withtts:extent="1920px 1080px",NSIZE(Ri) ≈ 0.00603.
NBG(Ri) is the total number of elements within the tree rooted at region Ri that satisfy the following criteria:
the element is either aregion,body,div,p orspan; and
the opacity of the computed value oftts:backgroundColor is not0.
Note
An element and its parent that satisfy the criteria above and share identical computed values oftts:backgroundColor are counted as two distinct elements for the purpose of computingNBG(Ri).
Note
Theset element is not included in the computation ofNBG(Ri). While it can affect the computed values oftts:backgroundColor, it is removed duringIntermediate Synchronic Document construction.
9.Paint Text
In the context of this section, aglyph is a tuple consisting of (i) onecharacter and (ii) the computed values of the following style properties:
tts:color
tts:fontFamily
tts:fontSize
tts:fontStyle
tts:fontWeight
tts:textDecoration
tts:textOutline
tts:textShadow
Note
In the case where a property isprohibited in a profile of [IMSC], the computed value of the property specified in [ttml2] can be used.
Note
The Hypothetical Render Model defines a one-to-one mapping betweencharacters andglyphs (using the definition of glyph from this document). While a one-to-one mapping betweencode points and glyphs (using the definition of glyph from [i18n-glossary]) is common in some scripts (such as the Latin script), the actual relationship is more complex. Some scripts, such as Arabic, use different glyphs for a given character, depending on its position in a word. Some scripts require combining marks or use a sequence of code points to form a glyph. Cases exist where a given sequence of code points can have different glyph representations depending on context. This complexity is accounted for by reducing the performance of the Glyph Cache for scripts where a one-to-one mapping is not the general rule (seeGCpy below).
if an identicalglyph is present in the Glyph Cache, copies theglyph from Glyph Cache to the Back Buffer using the Glyph Copier and flag theglyph asretain; or
otherwise renders (using the Glyph Renderer) theglyph into the Back Buffer and Glyph Cache, and flags theglyph asretain.
NRGA(gi) does not take into account decorations (e.g. underline), effects (e.g. outline) or actual typographical glyph aspect ratio. An implementation can determine an actual cache size needs based on worst-case glyph size complexity.
purge from the Glyph Cache allglyphs not flaggedretain; and
remove theretain flag from all remainingglyphs in the Glyph Cache.
ItSHALL be anerror if the sum ofNRGA(gi) over allglyphs flaggedretain in the Glyph Cache is at any time larger than the Normalized Glyph Cache Size (NGBS).
Note
The abbreviation NGBS reflects the name of the Glyph Cache from earlier editions of the specification.
Unless specified otherwise, the following table specifies values ofGCpy,Ren andNGBS.
Normalized glyph copy performance factor (GCpy)
Script property, as defined at [UAX24], for thecharacter of gi
GCpy
Latin,Greek,Cyrillic,Hebrew orCommon
12
any other value
3
Text rendering performance factor Ren(Gi)
Script property, as defined at [UAX24], for thecharacter of gi
Ren(Gi)
Han,Katakana,Hiragana,Bopomofo orHangul
0.6
any other value
1.2
Normalized Glyph Cache Size (NGBS)
1
Note
WhileDURT(En) is not affected, the choice of font by thepresentation processor can increase actual rendering complexity at time of presentation. For instance, a cursive font might select different glyphs for a givengrapheme (in order to maintain joining or for the start/end of the word) even in theLatin script. Conversely the rendering of scripts that fall in theany other value category can in practice achieve performance comparable to, say, theLatin script.
Setting a Normalized Glyph Buffer Size effectively sets a limit on the total number of distinctglyphs present in any givenIntermediate Synchronic Document En. For example, assuming a maximum Normalized Glyph Buffer Size of 1 and the defaulttts:fontSize of1c are used, the font size relative to theRoot Container Region height is 1/15 , and the maximum number of distinct glyphs that can be cached is 1÷(1÷15)2=225 glyphs.
GCpy effectively sets a limit on animating text. For example, assuming that theRoot Container Region is ultimately rendered at 1920×1080 resolution and no regions need to have background color painted (so only aCLEAR(En) operation is required for the normalized drawing area for theIntermediate Synchronic Document), aGCpy andBDraw of 12 s-1 would mean that a group of 160glyphs with atts:fontSize equal to 5% of theRoot Container Region height could be moved at most approximately 12 s-1 ÷ (1 + ( 160 × 0.052 )) = 8.6 times per second.
Ren(Gi) effectively sets a limit on the text rendering rate. For example, assuming that theRoot Container Region is ultimately rendered at a 1920×1080 resolution, aRen(Gi) of 1.2 s-1 would mean that at most 120glyphs with a fontSize of 108 px (10% of 1080 px andNRGA(gi) = 0.01) could be rendered every second.
In a system whereIMSC Document Instances are expected to conform to the Hypothetical Render Model, anIMSC Document Instance that does not conform to the Hypothetical Render Model might negatively impact accessibility during presentation of theIMSC Document Instance and its associated content.
B.2User customisation of presentation
This specification does not attempt to model any additional complexity forpresentation processors that might arise due to the user customisation of presentation, for example as described by [media-accessibility-reqs]; such user customisation is not defined by [IMSC].
Implementers ofpresentation processors that support user customisation of presentation should ensure that those processors are able to presentIMSC Document Instances that conform to the Hypothetical Render Model, even if the customisation effectively increases the complexity of presentation.
C.Privacy and Security Considerations
This section is non-normative.
C.1General
This specification has no inherent security or privacy implications.
The algorithm defined within this specification is used for static analysis of a resource. This specification does not define any protocol or interface for obtaining such a resource, and it does not define any interface for exposing the results of the analysis. No personal or sensitive information is processed as part of the algorithm, other than any such information that might happen to be part of theIMSC Document Instance being analysed. No information is exposed by the algorithm to any origin. No scripts are loaded or processed as part of the algorithm and no links to external resources are dereferenced.
C.2Implementation considerations
Implementers of this specification should capture and meet privacy and security requirements for their intended application. For example, an implementation could, when reporting on anerror encountered during processing of anIMSC Document Instance, include a section of the content of anIMSC Document Instance to elaborate the error. If that content could include sensitive or personal information, the implementation should ensure that any such output is provided using appropriately secure protocols. No such reporting is defined or required by this specification.
D.Error Reporting and Exception Handling
This section is non-normative.
D.1Error Reporting
This specification does not define how, or even if,errors should be reported.
For example, an implementation could stop on the first error encountered, or continue to process theIMSC Document Instance and report every error. Or an implementation could exit with an appropriate status code without reporting any details at all.
D.2Exception Handling
This specification does not define any runtime exceptions, or how such exceptions should be handled.
The editor wishes to especially acknowledge the following contributions by members: Nigel Megitt (British Broadcasting Corporation) and Atsushi Shimono (W3C).
The editor also wishes to acknowledge Cyril Concolato (Netflix), Michael Dolan (Invited Expert) and Paul Londino (Warner Bros. Discovery) for contributing content producing implementations to the implementation report.
In order to allow short (less than 100 ms) gaps between subtitles, which is common practice, the complexity of presentingempty ISDs has been reduced to zero: instead of being drawn into the Back Buffer, anempty ISD merely disconnects the Front Buffer from the display while it is presented.
