US20140040357A1 - Optimized key frame caching for remote interface rendering - Google Patents

Abstract

Frames of user interface (UI) graphical data can be remotely rendered more efficiently at a client during a remote session with a server by utilizing graphical data cached at the client to prevent re-sending data to the client that was sent in previous payloads. By using cache memory to remember recurring frames of similar UI data and delta encoding to correct areas that are not similar, encoded payload sizes are greatly reduced. When a cached frame is closer to a new frame than the current frame, the cached frame is selected and delta commands are derived from differences between the selected cache frame and the new frame. The differences between the cached frame and the new frame are then encoded as delta commands and the cache commands and delta commands are sent to the client which receives and applies the commands to render the new frame.

Description

CLAIM OF PRIORITY
• This application is a divisional application of U.S. patent application Ser. No. 11/870,960, entitled “OPTIMIZED KEY FRAME CACHING FOR REMOTE INTERFACE RENDERING,” by Brian McDowell, filed Oct. 11, 2007, which is incorporated herein by reference in its entirety.
BACKGROUND
• Existing network services provide remoting to a thin client, very thin client or other machine having limited resources. In client/server applications using a thin client, the client is designed to be especially small so that the bulk of the data processing occurs on the server. For example, remoting to a client may provide interface graphical data to a thin client which is a low-cost, centrally-managed computer usually devoid of CD-ROM players, diskette drives, and expansion slot. A remote session allows a user to login to a server through the thin client. For example, a server may provide user interface graphics to render at the client in response to user input received at the client and provided to the server.
• The efficiency of previous remoting methods used for remote sessions for thin clients does not provide a viable solution to remoting issues involved with rich interfaces. In particular, when remoting a user interface with rich graphics to a thin client, previous remoting methods do not process updates to a user interface with the speed and performance that users have come to expect from modern network services. Even on a very capable client, the richness of an “entertainment centric” user interface does not correlate to efficient remoting. For example, with a normal desktop user interface, a simple command to fill a rectangle in the user interface frame can be sent to fill the background white before text is rendered using basic glyphs. However, if the user interface is a richly rendered UI with the background generated by a designer using a drawing program, photo-realistic frame and text that is anti-aliased using advanced rendering techniques to blend with that background, there are no simple commands which can be remoted to the client to achieve the resulting graphics in the frame. As a result, the frame data itself must be sent to the client. Even with modern bulk compression techniques, transmitting bitmapped graphics for a user interface frame can be expensive in terms of network bandwidth and client processing resources.
• The technology described herein pertains to encoding graphical data using caching for the purpose of remoting user interfaces in real time between a host and client over a network. The user interface is composited into graphic bitmap data on a frame by frame basis on the host server, encoded by the host server into a payload and sent over a network to a client that decodes the payload and renders it on a display device. The encoding method can use pixel accurate delta encoding and optimized cache utilization to reduce the network payload and keep the network and client CPU loads to a minimum. The host maintains a copy of a “current frame” and based on a “new frame” determines a minimum set of commands and payload needed to transform the current frame into the new frame. The transformation may be achieved using intelligent caching logic and delta encoding commands.
• This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1A is a block diagram of an embodiment of a system for remoting a user interface.
• FIG. 1B is an example of a current frame rendered in a user interface.
• FIG. 1C is an example of a new frame rendered in a user interface.
• FIG. 1D is an example of how move commands are applied to a frame.
• FIG. 1E is an example of a difference frame is derived from a current frame and a new frame.
• FIG. 1F shows three examples of user interface current frames.
• FIG. 1G shows three examples of user interface new frames.
• FIG. 1H shows three graphical representations of dirty rectangle payloads.
• FIG. 1I shows three graphical representations of delta encoded payloads.
• FIG. 2A is a flowchart of an embodiment of a method for establishing a remote session.
• FIG. 2B is a flowchart of an embodiment of a method for encoding and transmitting a payload by a server.
• FIG. 3 is a flowchart of an embodiment of a method for generating new frame graphical data.
• FIG. 4 is a flowchart of an embodiment of a method for generating motion hints by a render engine.
• FIG. 5 is a flowchart of an embodiment of a method for determining if an auto caching process should be performed.
• FIG. 6 is a flowchart of an embodiment of a method for performing cache logic.
• FIG. 7A is a flowchart of an embodiment of a method for performing motion encoding logic.
• FIG. 7B is an example of rectangle movement between a current frame and a new frame.
• FIG. 7C is an example of rectangle clipping.
• FIG. 7D is an example of a modified rectangle movement between a current frame and a new frame.
• FIG. 7E is an example of reconfiguring overlapping dirty rectangles within a user interface.
• FIG. 8A is a flowchart of an embodiment of a method for verifying motion using a threshold.
• FIG. 8B is an example of a current frame and a new frame depicting a moving rectangle.
• FIG. 8C is an example of a current frame wherein a motion command and delta encoding are applied.
• FIG. 8D is an example of a current frame wherein delta encoding is applied.
• FIGS. 8E-8F are examples of data in a current frame and new frame used to verify whether to apply motion commands.
• FIG. 9 is a flowchart of an embodiment of a method for performing delta encoding.
• FIG. 10A is a flowchart of an embodiment of a method for encoding raster segments for RGB color data
• FIG. 10B is a flowchart of an embodiment of a method for encoding raster segments for alpha data.
• FIG. 11 is an example of a payload format.
• FIG. 12 is a flowchart of an embodiment of a method for performing decoder logic on a client.
• FIG. 13 is a flowchart of an embodiment of a method for verifying a payload.
• FIG. 14 is a flowchart of an embodiment of a method for processing payload motion commands.
• FIGS. 15A and 15B are examples of managing a frame buffer with a frame cache.
• FIGS. 16A and 16B are examples of managing a frame buffer used as a virtual frame cache and a frame cache.
• FIG. 17 is a flowchart of an embodiment of a method for processing payload cache commands.
• FIG. 18 is a flowchart of an embodiment of a method for processing payload RGB delta commands.
• FIG. 19 is a flowchart of an embodiment of a method for processing payload alpha delta commands.
• FIG. 20 is a block diagram of an embodiment of a computing environment for implementing the present technology.
DETAILED DESCRIPTION
• Graphical data is encoded for remoting user interfaces in real time between a server host and client over a network. The server provides graphical interface data to the thin client. The graphical data is rendered at the client and comprises a current frame. The client transmits user input to the server and the server determines whether the input causes a change in the user interface graphics. The graphics change may result in a frame of data to be rendered and remoted to the client. Remoting the interface graphics may include one or more of performing caching logic, motion encoding and delta encoding with respect to the current frame presented by the client and a new frame rendered as a result of the user input. Cache frame commands, motion commands and delta encode commands are embedded in a payload and sent to a thin client. The thin client receives the payload, decodes and parses the payload, and performs commands contained in the payload to the current active frame cache on the client to render the new frame of frame data.
Overview of Encoding and Remote Rendering
• Upon determining a user interface state change has occurred, through user interaction or other event, the server generates a new user interface frame. The new frame is to replace a current frame provided by the client as a result of the user interface state change. In some embodiments, the server provides a list of rectangles that indicate areas of the frame where changes have occurred since the last frame, the rectangles indicating areas of the last frame which are no longer valid. In some embodiments, the server identifies frame objects that move from a first position in the current frame to a second position in the new frame rendered by an render engine. A motion hint and other graphical data can be created for each moving object. Collision detection and other motion hint reducing techniques can be performed to reduce the number of hints for a frame update. The collision detection process detects and prevents motion hint from overwriting each other. The reduced set of motion hints is used to generate motion commands that are applied to a copy of the current frame maintained on the server. The server then determines the differences between the modified current frame and the new frame and encodes the difference as difference or “delta” data to a payload, using the provided invalidation rectangles to reduce the search area for possible “delta” data between the frames.
• Once the delta encoding is complete, the server sends the encoded non-colliding motion commands, delta commands and other commands to the thin client. The thin client receives the commands and the difference data, applies the motion commands first and then the delta commands to the current frame. As a result, the client processes the current frame to match the new frame reflecting the changed user interface state.
• In some embodiments, the present technology may also utilize frame caching to efficiently render a frame at a client. A server may instruct a client to cache frames of data determined to be the most relevant rendered frames. When sending a user interface frame update to a client, a server may determine whether or not the new interface frame matches a cached frame using a similarity threshold comparison. If the new frame matches a cached frame within a determined threshold, the server sends instructions to the client to load or select the matching cached frame rather than send frame data for the new frame over the network the client. The new frame is then differenced against the selected current cache frame by the server and the encoded delta commands are sent in the same payload as the cache command. If the new frame does not match a cached frame, the server selects the least relevant cache frame based on some criteria. Based on the criteria, the selected cache frame is replaced with the new frame data and selected as the current active cache frame.
Remote Rendering System
• FIG. 1A is a block diagram of an embodiment of a system for remotely rendering a user interface. The system ofFIG. 1A includesserver110,network120 andclient130.Server110 includesapplication111 and user interface (UI)encoder114.Application111 includesUI framework112 and UI renderengine113. UI renderengine113 renders graphical data for a user interface, including a new frame, dirty rectangles representing portions having changed pixels between a current frame and new frame, and motion hints. In some embodiments, when rendering a new user interface frame, UI renderengine113 may move graphical objects within a current frame to render the new frame. As a result of moving the graphical objects, the UI render engine has knowledge of and may generate motion hints and determine dirty rectangles.
• UI Encoder114 includesnew frame115,current frame116, cachedframes117,motion encoder118 anddelta encoder119. As illustrated, several cached frames 0-N may be included in cachedframes117.Delta encoder119 determines and encodes differences or “deltas” between a new frame and current frame which may or may not be modified. The current frame may be modified by applying motion commands to the current frame. Delta encoder may encode RGB data, alpha data (or some other type of data format used to present graphical information) to generate a payload which is transmitted fromserver110 to theclient130.
• Application111 ofserver110 generate frames for a user interface viaUI framework112 and UI renderengine113. The frames, including dirty rectangles and motion hints, are sent to theUI encoder114 as a series ofnew frame115, processed by the cache logic withinUI Encoder114 andmotion encoder118, and then delta encoded againstcurrent frame116 before being sent to theClient130 as an encoded payload. Motion hints can be specified by the UI RenderEngine113 and passed to theUI Encoder114. The UI encoder will sort and verify these hints in themotion encoding logic118 to properly handle overlaps and verify that frame data hints result in useful motion commands. Operation ofapplication111,UI encoder114 and other server functionality is discussed in more detail below with respect toFIGS. 2B-10.
• In some embodiments, the present technology may be used to update only a portion of an entire user interface frame or separately and simultaneously update multiple portions of an entire user interface frame. Each user interface frame portion may include a sub-window, frames that don't consist of the entire user interface, or other portions of the user interface frame. The user interface frame portions may be visible or non-visible, have different sizes, and/or provide caching differently. When addressing one or more portions of a user interface frame, multiple instances ofUI encoder114 and UI decoder131 (as well as other modules, optionally) may be instantiated at a server and/or client. The multiple instantiations may each maintain payload streams between the server and the client. Though, the following discussion will address a single instantiation address an entire user interface frame, it is intended that the present technology can be applied to less than an entire user interface frame as well as simultaneously updating multiple portions of an user interface frame.
• Network120 may be implemented as the Internet or other WAN, a LAN, intranet, extranet, private network or other network or networks.
• Client130 may be a thin client used to remotely access and render interface graphics provided byserver110 overnetwork120.Client130 includesUI decoder131 anddisplay device135.UI decoder131 includespayload decode132,current frame133 and cache frames134.Payload decoder132 is a module or set of code and/or hardware that decodes a payload and applies commands, deltas and other payload data againstcurrent frame133 or one of cached frames134. Operation ofpayload decoder132 and other elements ofclient135 are discussed in more detail below with respect to FIGS.2A and14-19.
• FIGS. 1B-1E illustrate a progression of frame processing implemented by the current technology.FIG. 1B is an example of a current frame rendered in a user interface atclient130. The current frame inFIG. 1B includes three rectangles of pixel data which are associated with, from left to right, a skier icon, a rock climber icon and a face icon.FIG. 1C is an example of a new frame to be rendered in the user interface as a result of a user interface update rendered atUI Encoder114new frame115. The new frame ofFIG. 1C can be rendered in response to user input and will replace the current frame ofFIG. 1B. The new frame differs from the current frame in that the rock climber rectangle and face rectangle are in a different position, the skier rectangle is not present in the new frame, and a new horse rider rectangle is included.
• FIG. 1D illustrates how motion commands are applied to the rectangles in the current frame and the new frame. Motion commands may consist of a motion vector, shape information for the shape to more (such as a rectangle length and width), and an origin for the shape. A first motion vector indicates that the rock climber rectangle has moved from the center of the screen to the left side of the screen in a horizontal direction. A second motion vector indicates that the face rectangle has moved from the right side of the screen to the center and to a vertical position located below its previous position in the current frame. The system may also detect that the new position of the face rectangle overlaps with the original position of the rock climber rectangle. Thus, there is motion collision between the new pixels of the face rectangle and the old pixels of the rock climber rectangle. The motion collision process determines that the motion command for the rock climber rectangle must be applied before the motion command for the face rectangle so the original pixels comprising the rock climber rectangle are not overwritten by the movement of the face icon rectangle. Once motion commands are sorted and otherwise processed (discussed in more detail below), the motion commands are applied to the copy of the current frame stored at the server.
• After the motion commands have been applied to the current frame at the server, the difference in pixel values is determined between the modified current frame and the new frame.FIG. 1E illustrates the difference between the modified current frame ofFIG. 1D (after the motion commands have been applied) and the new frame ofFIG. 1C. The only difference is the horse riding rectangle on the right side of the frame inFIG. 1E. Since the positions of the rock climbing rectangle and the face rectangle should match or be within a threshold, there is no recognizable difference between these pixels as illustrated in the difference frame ofFIG. 1E. As a result, a payload for updating the current frame ofFIG. 1B to the new frame ofFIG. 1C would consist of encoded motion commands for the moving face rectangle and rock climbing rectangle and encoded pixel data for the difference comprising the horse riding rectangle.
• FIG. 1F shows an example of 3 current frames which a UI render engine could pass to theUI Encoder114.FIG. 1G shows 3 examples of the next frame showing changes in the user interface to be encoded and remoted to theclient130.
• FIG. 1H shows a graphical representation of the payload which would be generated from current frames inFIG. 1F and new frames inFIG. 1G using tradition dirty rectangle remoting techniques where all bitmap data within the dirty rectangles are remoted to theclient130.FIG. 1I shows a graphical representation of the payload which would be generated from current frames inFIG. 1F and new frames inFIG. 1G using a delta encoder, which shows only the pixels which were not similar being encoded and remoted toclient130.
Generation of Payload by Motion, Cache and Delta Encoding
• Server application111 andUI encoder114 may generate and transmit a payload toclient130 overnetwork120. In some embodiments, the payload may be generated and transmitted during a remote session betweenserver110 andclient130 and in response to input data received fromclient130 byserver110.FIG. 2A is a flowchart of an embodiment of a method for establishing a remote session. The method ofFIG. 2A provides details for establishing a remote session from a client point of view. Details regarding rendering, encoding and transmitting user interface graphics to the client in a payload after the remote session has been established is discussed in more detail below inFIG. 2B. Details regarding receiving and decoding user interface graphics byclient110 are discussed in more detail below inFIG. 12.
• First,client130 establishes a remote session withserver110 overnetwork120 atstep205. The user session is a remotesession allowing client130 to remotely render user interface graphics associated with an interface provided byserver110. Next, initial user interface graphics are received byclient130 fromserver110 atstep210. Providing the initial graphics may include providing an initial frame, one or more cached frames, or other graphical data, such as a set of initial backgrounds to be cached at the client. The user interface graphics are rendered atclient130 atstep215 andclient110 receives user input through the initial user interface (or detects some other event at the client) atstep220. In some embodiments,step220 may include detecting the occurrence of another event for which data is transmitted toserver110, such as the expiration of a timer. The discussion below will proceed with respect to receiving user input, but could be based on non-user initiated event as well. The received user input is translated to a command and transmitted toserver110 byclient130 atstep225.Server110 may receive the user command fromclient130 and process the command. Processing a command from a client to encode and transmit a payload is discussed in more detail below with respect toFIG. 2B.
• FIG. 2B is a flowchart of an embodiment of a method for encoding and transmitting a payload by a server. The method ofFIG. 2B may be performed in response to receiving a command, user input, or other data fromclient110. First,server110 receives a command fromclient130 and passes the command toapplication111 atstep230. Next,application111 determines that the received command changes the current frame graphics and UI renderengine113 generates new frame graphical data atstep235. In some embodiments, new frame graphical data may include any of a new frame, motion hints, and dirty rectangles representing changes between a current frame and the new frame. As discussed above with respect toFIG. 1A, when UI render engine renders a new frame, it may move objects within a current frame to create the new frame. By moving one or more objects, the UI render engine has knowledge of graphical content that moves between a current frame the new frame to replace the current frame. The UI render engine may then generate motion hints based on objects that it moves to create the new frame and dirty rectangles which comprise one or more bounding rectangles that enclose the source and destination of the objects that have moved. In some embodiments, different shapes may be used to indicate a “dirty” or changed portion of a frame, such as a circle, square, triangle, polygon, or any other shape. Generally speaking, rectangles will be referred to throughout this discussion for purposes of example only, and are not intended to limit the scope of the present technology. Generating new frame graphical data is discussed in more detail below with respect toFIG. 3.
• A determination is then made as to whether the graphical data includes dirty rectangles atstep240. If the graphical data generated atstep235 includes one or more dirty rectangles, method continues to step245. If the graphical data does not include one or more dirty rectangles, then no graphics in the current frame have been changed in the new frame, no payload needs to be sent toclient130 and the method ends atstep285.
• A determination is made as to whether an automatic caching process should be performed atstep245. In some embodiments, automatic caching can be performed if requirements are satisfied with respect to a full screen update, caching is enabled at the server and similarity of the new frame to the current frame. In some embodiments, automatic caching may be performed based on a subset of these requirements or other requirements. More detail for determining whether an automatic caching process should be performed is discussed in more detail below with respect toFIG. 5. If an automatic caching process should not be performed, the method ofFIG. 2B continues to step255. If the automatic caching process should be performed, then the method continues to step250.
• Caching logic is performed atstep250. Caching logic determines whether it would be more efficient to delta encode a new frame against a cached frame rather than the current frame. Caching logic may result in outputting a cache command into an encoded payload and switching to a new cache frame at the client.Server110 may maintain a copy of the cached frames maintained at the client and may switch to the corresponding new cache frame (maintained at the server) when cache commands are encoded. More detail for performing caching logic is discussed below with respect to the method ofFIG. 6. After performing caching logic, the method ofFIG. 2B continues to step265.
• In some embodiments, automatic caching and motion encoding are not performed on the same user interface update. Automatic caching involves determining that less data may be encoded into a payload be selecting an frame cached at the client (other than the current frame) and performing delta encoding between the selected cache frame and a new frame. Motion encoding involves determining one or more rectangles or other shapes of graphical data that have moved between a current frame and a new frame based on motion hints, reducing the motion hints and generating motion commands from the reduced set of motion hints, and applying the motion commands to the current frame. If delta encoding against a selected cached frame is determined to be more efficient than delta encoding against a current frame, then it is inherent that the selected cached frame and current frame are different. Therefore, applying the motion commands to the cached frame will not result in the same modified frame as applying the motion commands to the current frame (which is what the motion hints were generated from). Modifying the cached frame with motion commands generated for the current frame would likely require a large delta encoded payload to “fix” the modified cache frame into the new frame. In this embodiment, caching logic instep250 and motion encoding logic in260 are mutually exclusive in that only one set of logic operations will be performed on any given frame. However, there other embodiments exist in which these operations are not mutually exclusive.
• If it is determined that the automatic caching process should not be performed atstep245, a determination is made as to whether the new graphical data includes motion hints atstep255. The motion hints are required for motion encoding. If UI renderengine113 generated one or more motion hints atstep235, then motion encoding logic is performed atstep260. Motion encoding logic determines whether any motion commands should be generated from the motion hints and encodes any motion commands into the payload. In some embodiments, when considering whether to generate motion commands, the motion encoding logic may remove overlapping hints and clip, order and verify the motion hints. Generated motion commands may be output to a payload and/or applied to the current frame maintained at the server. Performing motion encoding logic is discussed in more detail with respect toFIG. 7. After performing motion encoding logic, the method ofFIG. 7 continues to step265.
• The current cache frame relevance value is adjusted atstep265. In some embodiments,UI encoder114 ofserver110 may track and adjust a relevance value for each frame cached atclient130. In some embodiments, the relevance value may be adjusted based on the use of the frame. For example, each time a particular frame is used in response to user input or some other event requiring an update to the user interface graphics, the relevance value for that frame may be incremented. The relevance value may be used to determine which cached frames to keep at the client, which cached frames can be overwritten when cache space is needed, and/or for other purposes. The relevance value for the current cache frame is adjusted, for example by incrementing the relevance value, atstep265.
• Delta encoder119 performs delta encoding of red-green-blue (RGB) data and adds the encoded RGB data to a payload atstep270. Delta encoding of RGB data may include determining the difference or “delta” between the new frame rendered by UI renderengine113 and either a modified current frame in which motion commands have been applied or a cache frame selected by the cache logic process. The delta encoding process may encode pixel differences in segments within a raster. The segments may be lossless and represent the exact difference in color value or be lossy to some degree based on a specified tolerance. For lossy segments, fragmentation between the segments is minimized by allowing short runs of non-delta values (pixels with the same data value) to be included in the raster segment. By including a short run of non-delta values in a segment containing delta values, creating a new raster segment is avoided along with the storage overhead in the output stream required by a new segment. Delta encoding RGB data is discussed in more detail below with respect toFIG. 7.
• In some embodiments, alpha data in a user interface frame may be encoded separately from RGB data. Because the encoded alpha data may not vary as much as the RGB data delta, separately encoding the alpha data from the RGB data can result in a smaller and more efficient payload. Alpha delta encoding and adding the encoded alpha data to the encoded payload is performed atstep275. Delta encoding the alpha data includes determining the difference between the alpha data in the new frame rendered by UI renderengine113 and the alpha data in either a modified current frame in which motion commands have been applied or a cache frame selected by the cache logic process. The delta encoding process may encode pixel differences in segments within a raster in similar manner to encoding of RGB data. The alpha encoded segments may be lossless and represent the exact difference in alpha value or be lossy to some degree based on a specified tolerance. Fragmentation between segments is minimized by allowing short runs of non-delta alpha values to be included in the raster segment to avoid creating a new raster segment which would require more storage overhead in the output stream than to simply include the short run of non-delta alpha values. In some embodiments, the alpha data may be run length encoded if the length of constant alpha values was long enough to store it more efficiently than as one or more segments. Run length encoding (RLE) of alpha values can be advantageous because alpha values are often constant over entire longer pixel runs, entire rasters, or even entire frames.
• After delta encoding is complete, the generated payload is transmitted toclient130 atstep280. One example format of a payload is illustrated inFIG. 11 and discussed in more detail below. In some embodiments, a single user interface graphical update may require several payloads, depending on the client cache and other factors. When sending multiple payloads, the last payload may be signified by a flag in the corresponding last payload. In some embodiments the last payload for a frame may also include a bounding shape to inform theUI Decoder131 what areas of the current frame needs to be updated on thedisplay device135 to minimize the operations needed to display the frame. Client logic for receiving and decoding a payload is discussed in more detail below with respect to the method ofFIG. 14.
• FIG. 3 is a flowchart of an embodiment of a method for generating new frame graphical data. In some embodiments, the method ofFIG. 3 provides more detail forstep235 of the method ofFIG. 2B. First, anew frame115 is rendered by UI renderengine113. The new frame may be rendered byUI engine113 in response to user input, an event, a command or some other data received fromclient130. While rendering the new frame, UI renderengine113 tracks portions of the new frame that change (step320). In some embodiments, bounding rectangle boxes (or other shapes) that contain changed objects are tracked
• Next, dirty rectangles are generated from the bounding boxes atstep330. The dirty rectangles contain one or more pixels that change between the current frame and the new frame. Motion hints are generated atstep340. A motion hint indicates detection of “motion” based on the position of an object in a current frame and a different position of the object in the new frame. A motion hint comprises a vector, an origin and data for the shape that moves, for example a length and width for a rectangle motion hint. Not all motion hints for objects detected to move during new frame rendering are translated into motion commands. For example, some motion hints collide with other motion hints and cannot be viewed in the current frame or new frame, additionally objects that rotate, scale or are transparent may not generate motion hints. Generating motion hints is discussed in more detail below with respect toFIG. 4.
• FIG. 4 is a flowchart of an embodiment of a method for generating motion hints. In some embodiments, the method ofFIG. 4 provides more detail forstep310 of the method ofFIG. 3 and is performed by UI renderengine113. First, renderengine113 makes a determination as to whether a particular object was detected to move during new frame rendering atstep410. An object has moved if the object was in a first position in a current frame and in a different location in the new frame. If a particular object did not move, then UI renderengine113 determines the object is not a valid motion hint atstep460.
• If an object has been detected to move, a determination is made as to whether the object is visible in the current frame atstep420. For example, an object may be visible in a current frame (or not visible) as a result of layering features implemented in the current frame. If an object is not visible in a current frame, the object is not a valid motion hint (step460). If the object is visible in the current frame, a determination is made as to whether the object is visible in the new frame atstep430. For example, an object may not be visible in a new frame if the object has been moved to a position outside the current frame or is behind an obstructing object or layer. If an object is not visible in the new frame, then the object is not a valid motion hint as indicated atstep460. If the object is visible in the new frame, a determination is made as to whether the object has been rotated or is transparent in the new frame atstep440. Objects that have been rotated or transparent are not processed as a motion hint. If the object has not been rotated or scaled, or is not transparent in the new frame, then the object is considered a valid motion hint atstep450.
• FIG. 5 is a flowchart of an embodiment of a method for determining if an auto caching process should be performed. In some embodiments, the method ofFIG. 5 provides more detail ofstep245 of the method ofFIG. 2B. First, a determination is made as to whether the new frame is a full screen update atstep510. The determination may involve whether the sum of the dirty rectangles is equal to the size of the current frame. In some embodiments, the determination may involve whether the sum of the dirty rectangles meets a percentage threshold of the current frame, such as eighty percent or ninety percent, or meets some other threshold indicating that a majority of the frame requires a graphical update. If the new frame is not a full screen update, a determination is made that the auto caching process should not be performed atstep550. Not performing the auto caching process results in performance of motion encoding logic atstep260.
• If the new frame is a full frame update, a determination is then made as to whether caching is enabled atstep520. Caching may be enabled ifclient130 andserver110 both have cache buffers enabled to perform caching of one or more frames or frame portions. If caching is not enabled, the auto caching process is not performed as determined atstep550. If caching is enabled, a determination is then made as to whether the new frame is similar to the current frame within a threshold atstep530. In some embodiments, the threshold is calculated as a percentage of similar pixels to dissimilar pixels, for example a frame may be deemed similar if 75% (or some other value) of the pixels are similar pixels. In some embodiments, a similar pixel may be defined to be identical color values or merely a similar color value within a color tolerance level. If the current frame and new frame are similar within the threshold, then no changes to the cache are required and the auto caching process is not performed perstep550. In this case, auto caching is not performed because the differences between the current frame and the new frame are small and relatively inexpensive to encode into the payload. If the current frame and new frame are not similar within the threshold, a determination is made that the auto caching process should be performed atstep540. As a result of this determination, cache logic is performed atstep250 of the method ofFIG. 2B.
• FIG. 6 is a flowchart of an embodiment of a method for performing cache logic. In some embodiments, the flowchart ofFIG. 6 provides more detail forstep250 of the method ofFIG. 2B. First, a determination is made as to whether the new frame is a constant color fill atstep610. In some embodiments, the UI render engine may determine whether the new frame has a constant color fill and may provide this information toUI encoder114. If the new frame is a constant color fill, then the cache command generated for a selected cache will include constant fill data as determined atstep625 and the method continues to step630. If the new frame is not a constant color fill, a determination is made as to whether the new frame is similar to an existing cache frame atstep615. If the new frame is similar to an existing cache frame, then the existing cache frame is selected atstep620 as the new cache frame and the method ofFIG. 6 continues to step650. The existing cache frame is selected because the delta encoded difference between the new frame and the existing cache frame was determined to be similar and will result in a smaller and more efficient payload to send to the client.
• If the new frame is not similar to an existing cache frame, a determination is made as to whether empty cache slots are available atclient130 atstep630. As illustrated inFIG. 1,server110 maintains a copy of the cached frames maintained atclient130. Therefore, the server has knowledge of the cached frames maintained atclient130 and whether any cache slots are empty atclient130. If there is an empty cache slot atclient130, the empty cache slot is used as the new cache frame atstep645. Using an empty cache frame may include encoding one or more cache commands to set the empty cache slot as the current frame slot. The method ofFIG. 6 then continues to step650.
• If there are no empty cache slots atclient130, the least relevant cache frame is selected as the new cache frame atstep635. The relevancy of a cache frame may be determined in several ways. In some embodiments, the relevancy may be determined by the last time a cached frame was used (for example, determined via timestamp) in the rendered user interface graphics. In some embodiments, the relevance may be determined by a relevancy value indicating the number of times a cached frame has been used relative to the other cache frames, or may be simply based on the how recently the cache has been referenced. Alternately, it could be determined by other means, such as a pre-determined ranking of pre-rendered cache content. Other mechanisms for determining cache frame relevancy can be used as well, and are considered within the scope of the present technology.
• After selecting the least relevant cache frame as the new cache frame, the relevance value of the current frame is reset atstep640. The relevance value is reset because new frame data will be encoded into the payload and placed in the cache slot associated with the lowest relevancy cache frame, thereby overwriting the previous cache frame in that slot. The new cache frame placed in the slot should not inherit the previous relevancy value, but should be associated with a reset relevancy value. The method ofFIG. 6 then continues to step650.
• A cache command for the selected frame is output into the encoded payload atstep650. The cache command way include constant color fill data if needed based on the determination made atstep610. The output cache command may include a command to set the current frame to a cached frame contained in a particular cache as identified by a cache identifier (cache ID number), and may include a color fill data. After outputting the cache command to the payload, the new cache frame is switched to the current cache frame atserver110 atstep655. Similar to step650, the new cache frame may be filled with a constant color if fill data was included in the payload cache command atstep650. After switching the new cache frame to the current cache frame at the server, cache logic is complete atstep660.
• If caching logic is not performed in the method ofFIG. 2B, motion encoding logic is performed.FIG. 7A is a flowchart of an embodiment of a method for performing motion encoding logic. In some embodiments, the method ofFIG. 7A provides more detail forstep260 of the method ofFIG. 2B.
• First, a determination is made as to whether the UI renderengine113 provided motion hints atstep710. Providing motion hints by UI renderengine113 was discussed above with respect to step340 of the method ofFIG. 3. If the UI render engine did not provide motion hints, motion encoding logic is complete atstep780 and no motion commands are generated.
• If UI renderengine113 provided one or more motion hints, then any of several steps can be performed to reduce the number of motion hints to make the motion encoding process more efficient. Motion hints that overlap are removed atstep715. In some embodiments, the overlapping motion hint having the smaller rectangle is removed from the motion hint list. Motion hint clipping is performed if needed atstep720. Motion hint clipping involves removing or “clipping” a portion of a rectangle associated with a motion hint because the portion is positioned off the frame in the destination. Conceptual steps of performing clipping of motion hints are illustrated inFIGS. 7B-7D. Motion hints in the list are ordered to avoid collisions atstep730. Additionally, in some embodiments, if more than one motion hint result in the same destination, all but one of those colliding motion hints are removed. In some embodiments, two or more motion hints having a destination rectangle that collides with a source rectangle of another motion hint are set to execute in an order that minimizes the overwriting of source rectangles before the motion command for source rectangle is applied.
• Motion verification is then performed using a verification threshold atstep740. The motion verification verifies that the rectangle associated with the motion hint would be most efficiently rendered by a motion command combined with delta encoding as opposed to delta encoding alone. In some embodiments, motion verification compares each pixel between a source rectangle and a destination rectangle and increments a pass counter or fail counter based on whether the pixels match (in some embodiments, within a comparison threshold). The verification is based on which of the pass counter or fail counter satisfy a corresponding threshold first, and motion commands are generated for motion hints that pass the verification process. Motion verification is discussed in more detail below with respect to the method ofFIG. 8A.
• Motion commands are output to a payload atstep760. Motion commands are generated for those motion hints that survive the motion hint reduction steps and satisfy a verification process. In some embodiments, a motion command may consist of a vector and rectangle data, including an origin, length and width. An example of the format of motion commands in a payload is illustrated bymotion payload1120 of the payload ofFIG. 11.
• After outputting the motion commands to the payload, the motion commands are applied to the current cache frame maintained atserver110 atstep770. The motion commands are applied to the current frame at the server in order to enable the delta encoder to correctly determine what needs to be encoded in the modified current frame in order to render the new frame. If the motion commands were not applied to the current cache frame atserver110, delta encoding the unmodified current frame against the new frame would result in much more encoded delta data included in the payload (which would affect processing of the thin client) and an unnecessary increase in network load. After applying the motion commands to the current cache frame, the motion encoding logic is complete atstep780.
• FIG. 7B is an example of object movement.FIG. 7B includes a current frame (on the left) and new frame (on the right). In the current frame, a source rectangle is shown in solid line and expected destination of the rectangle in a dotted line. In the new frame, the rectangle destination is shown after the current frame has been modified with a motion command. As illustrated, the rectangle moves to a position in the new frame such that a portion of the rectangle is no longer contained within the frame. The rectangle as a whole is not a good candidate for a motion vector as is because a portion of the rectangle is moved to a position outside of the frame.
• FIG. 7C illustrates a clipping process that can be performed on the rectangle.FIG. 7C illustrates an original rectangle on the left, the portion of the rectangle to be clipped in the middle, and resulting clipped rectangle on the right. In the middle ofFIG. 7C, the portion of the rectangle to remain in the motion hint after the rectangle is clipped is shaded. The shaded rectangle on the right inFIG. 7C represents the result of the clipping of the original rectangle.
• FIG. 7D illustrates the applying a motion command to the frame having the clipped rectangle. Previously, a portion of the moving rectangle was positioned outside the frame boundary in the new frame. InFIG. 7D, the moving rectangle has been modified such that only the portion which is contained within the new frame is processed with a motion hint. Thus, the portion of the object which was previously located outside the new frame has been clipped from the motion hint.
• FIG. 7E is an example of modified dirty rectangles.FIG. 7E includes a frame containing two overlapping dirty rectangles on the left and a frame of three adjacent dirty rectangles on the right. The two overlapping rectangles in the frame to the left partially overlap. Rather than encoding the overlapping portion twice, the two rectangles are broken up into three adjacent rectangles which do not overlap as shown in the frame to the right. In some embodiments, when reconstructing overlapping rectangles, the rectangles created to represent the overlapping rectangles are constructed to maximize a raster length for payload efficiency (thus, rectangles are chosen to maximize length rather than height). When the three adjacent rectangles are encoded, there will be no overlapping encoding and each rectangle can be represented by wide lengths of pixel segments described by origin and length.
• As discussed with respect to the method ofFIG. 7A, motion commands require that motion hints pass a verification process.FIG. 8A is a flowchart of an embodiment of a method for verifying motion using a threshold. In some embodiments, the method ofFIG. 8A provides more detail forstep740 of the method ofFIG. 7A. To verify motion associated with a motion hint, the method ofFIG. 7B determines whether it would be more efficient to encode the differences of the moving rectangle or apply a motion command to the moving rectangle and then encode the resulting differences. First, a first pixel of the source motion hint rectangle and destination motion hint rectangle is selected atstep805. The selected pixels are then compared atstep810. A determination is then made as to whether pixel color values between the compared pixels are within a threshold atstep815. In some embodiments, the threshold may be zero difference for a lossless new frame. In some embodiments, the threshold may be seventy percent, eighty percent, ninety percent of color matching, or some other tolerance level of color similarity. If the pixel color values are not within the threshold, a fail counter is incremented atstep850 and a determination is made as to whether the fail counter current value satisfies a fail threshold atstep855. Selecting a fail threshold value is discussed below. If the fail counter satisfies the fail threshold, the object hint has failed the motion verification process and a motion command is not generated for the motion hint atstep850. If the fail counter does not satisfy the fail threshold, the method continues to step835. In some embodiments, the determination of a valid move can be determined by calculating the total cost of the embedded delta payload without the motion applied and comparing to the total cost of the embedded payload with the motion applied, and deciding based on which results in a smaller payload.
• If the pixel color values are within a threshold atstep815, a pass counter is incremented atstep820 and a determination is made as to whether the current value of the pass counter satisfies a pass threshold atstep825. If the pass counter value satisfies the pass threshold, the object hint has passed the verification process and a motion command is generated for the motion hint atstep830. If the pass counter does not satisfy the pass threshold, the method continues to step835.
• In some embodiments, the pass threshold and fail threshold may be related in that they add up to one hundred percent. Thus, a pass threshold may have a value of sixty percent of rectangle pixels and the fail counter may have a value of forty percent of the rectangle pixels. In some embodiments, both the pass threshold and fail threshold may be set to fifty percent of the pixels. In any case, the system determines whether each counter has reached a pixel count which satisfies the threshold percentage of total pixels in the rectangle. In some embodiments, the pass threshold and fail threshold may be selected based on factors of network resources, type of user interface graphics which are being encoded, motion encoding and delta encoding resources, and other data.
• Atstep835, a determination is made as to whether more pixels exist in the motion hint rectangle (or other shape). If more pixels exist, the next pixel in the source motion hint rectangle and destination motion hint rectangle are selected and the method ofFIG. 8A continues to step810. If no further pixels exist in the motion hint, a pass or fail value is returned based on current counter values.
• In some embodiments, the destructive motion compensation process ofFIG. 8A determines whether it is more efficient to apply motion commands and delta commands to a current frame rather than just apply delta commands.FIGS. 8B-8D an example of the differences in between these two options with respect to an example frame.FIG. 8B is an example of a current frame and a new frame depicting a moving rectangle. The current frame on left inFIG. 8B includes a shaded background, a rectangle and a circle within the rectangle on the upper left portion of the frame. In new frame towards the right ofFIG. 8B, the location of the circle has not changed but the rectangle has moved from the upper left to the lower right position of the frame.
• FIG. 8C is an example of a progression of the current frame as motion command and delta encoding are applied. Of the three frames pictured inFIG. 8C, the left frame is the same as the current frame pictured inFIG. 8B. The center frame indicates the current frame after a motion command has been applied to the shape comprising the rectangle. Note that although only the rectangle moves to the new frame inFIG. 8B, the data comprising the rectangle in the current frame contains the circle as well, therefore creating an invalid state of the contents of the frame, which will be repaired by the delta encoder. Thus, the motion command moves a copy of the pixel data within the rectangle from the source position in the current frame to the destination position of the rectangle in the new frame. Next, the frame on the right indicates pixel data that the delta encoder determines needs to be encoded. The encoded data in the frame to the right is the encoding that the center frame inFIG. 8C requires to be changed into the new frame in the right side ofFIG. 8B. In this example, delta encoder replaces the background where the object was moved, and repairs the invalid state created by the motion of frame data.
• FIG. 8D is an example of a current frame wherein only delta encoding is applied to change the current frame ofFIG. 8B into the new frame ofFIG. 8B. As can be seen, the delta encoding requires that the destination location of the rectangle be encoded to match the background of the new frame (except for the location of the circle) and that the entire portion of the rectangle is encoded. The difference in encoding with and without encoding motion commands can be seen by comparing the encoded portions of the frame ofFIG. 8D and the frame in the right ofFIG. 8C. The frame in the right ofFIG. 8C requires the encoding the source position of the rectangle (except for the position of the circle) as inFIG. 8D, but requires much less encoding to complete the pixels in the vicinity of the rectangle destination (encoding the small circle rather than the entire rectangle).
• Some embodiments may verify motion by calculating the size of the payload produced both when a move is performed and when the move is not performed, and determines which option (delta encoding with or without the move operation) would produce the lowest delta encoded payload.
• FIG. 8E shows a current frame and a new frame and contains source rectangle S and destination rectangle D. The two rectangles represent the areas specified by a motion hint and indicate a movement in a UI element between the current frame and the new frame. In this example, an object in the UI framework was rendered at source rectangle S in the current frame and was then rendered at destination location D in the new frame.
• FIG. 8F shows the same current frame and the new frame depicted inFIG. 8E and contains destination rectangle D′ and source rectangle S′. The rectangle D′ presents the pixel data in the current frame which would be overwritten by the contents of rectangle S if a move is applied. The rectangle S′ represents the new pixel data in the new frame which will be encoded into rectangle S when encoding is applied.
• To calculate the cost of not performing the move operation for the motion hint and only using delta encoding, the following formula can be used: (S−S′)+(D′−D), where subtraction indicates an operation where each pixel is compared at identical pixel locations in the two areas of the current frame and new frame. The result of the operation would be number of pixels deemed non-similar. To compute the cost of performing the move prior to delta encoding, this formula can be used: (S−S′)−(S−D), which reflects applying the motion command by replacing D′ with S. To decide whether or not the move will result in the smallest possible payload, the following logic applies: Move=(((S−S′)+(D′−D))>((S−S′)+(S−D))). This reduces to: Move=((D′−D)>(S−D)).
• After performing motion encoding, data is delta encoded into the payload. The data to be delta encoded may include one or more of RGB data, alpha data, another format of graphical data, or some other data.FIG. 9 is a flow chart of an embodiment of a method for performing delta encoding for RGB data. In one embodiment, the method ofFIG. 9 provides more detail forstep270 of the method ofFIG. 2. First, a determination is made as to whetherdelta encoder119 has access to dirty rectangles atstep905. The dirty rectangles may be generated by UI renderengine113. If the delta encoder does not have access to any dirty rectangles, the method ofFIG. 9 is complete atstep960 because there is no data indicating any change between a current frame and a new frame.
• If the delta encoder does have access to dirty rectangles, the first dirty rectangle is selected atstep910 and raster points are calculated at the selected dirty rectangle for the new frame and current frame atstep915. Raster pointers are calculated to point to the first pixel of the first raster in dirty rect position in both the new frame and the current frame. Pixel pointers are then assigned to the same values as the raster pointers atstep920.
• A determination is made as to whether the pixels indicated by the pixel pointers are the same atstep925. The pixels are the same if they have the same RGB data values. In some embodiments, the determination may involve determining if the RGB values are within a certain threshold rather than whether the color values match exactly. If the pixels match (optionally, within a threshold) atstep925, the method continues to step935. If the pixels do not match, a difference has been detected and a raster segment is encoded atstep930. Encoding a raster segment may include determining the length of pixel pairs along which the pixel values differ from the current raster in the selected dirty rectangle. In some embodiments, a raster segment may include a string of pixels pairs that do not differ (or not meeting the difference threshold) to avoid fragmenting the delta encoded raster segments. Once the end of a raster segment is determined, the RGB data for the raster segment is copied into the payload and applied to the current frame maintained at the server. Encoding a raster segment is discussed in more detail below with respect to the method ofFIG. 10A with respect to RGB data and the method ofFIG. 10B with respect to alpha data.
• A determination is made atstep935 as to whether the end of the raster is detected atstep935. If the end of a raster is not detected, the pixel pointers are incremented to the next pixel in the raster atstep840 and the method continues to step925. If the end of the raster is detected, a determination is made as to whether more rasters exist in the current dirty rectangle atstep945. If more rasters exist in the current dirty rectangle, the raster pointers are incremented to the next raster atstep950 and the method continues to step920. Incrementing raster pointers will move the pointer to the next raster in the dirty rectangle. If there are no more rasters atstep945, a determination is made as to whether more dirty rectangles exist atstep955. Ifdelta encoder119 has access to more dirty rectangles to analyze,encoder119 increments to the next dirty rectangle atstep965 and the method returns to step915. If there are no further dirty rectangles to process, then RGB delta encoding is complete atstep960.
• Encoding a raster segment at step atstep930 in the method ofFIG. 9 may include detecting a segment of pixels along a raster and encoding the set of pixels in the payload. The pixels may include pixels that are different between a current frame and new frame and, in some embodiments, pixels that are the same.FIG. 10A illustrates a flowchart of an embodiment of a method for encoding raster segments for RGB data as instep930 ofFIG. 9.
• Upon detecting a pair of pixels that differ atstep925 ofFIG. 9, delta encoder scans forward along the current raster to determine the total length of the raster segment having dissimilar pixels atstep1010. The scan mere detects the difference between pixels in the raster and does not record the pixel values at this time. When the scan has completed,delta encoder119 determines whether the end of the length of the segment is the end of the current raster atstep1015. If the end of the segment corresponds to the end of the segment, the method continues to step1035 where the RGB raster segment is copied to a payload (discussed below). If the end of the segment is not the end of the raster, then a similar pixel has been detected. At this point,delta encoder119 scans ahead in the raster to determine the total count of pixels that are the same atstep1020. Once the end of the run of similar pixels is detected, a determination is made as to whether the end of the similar pixels corresponds to the end of the current raster atstep1025. If the end of the raster has been detected, the method continues to step1035.
• If the end of the raster has not been detected, then the run of similar pixels has ended with the detection of a pair of different pixels in the raster. At this point, a determination is made as to whether the number of pixels comprising the run of similar pixels is less than a threshold atstep1030. This determination decides whether the run of similar pixels is small enough such that the similar pixels may be included in the current raster segment. In some embodiments, the similar pixel length threshold may be selected based on factors of frame resolution, networking bandwidth, client memory size, and other factors. For example, the similar pixel length threshold instep1030 may be two pixels, three pixels, or some other number of pixels. If the number of similar pixels is less than the threshold, the length of the similar pixel run is added to the raster segment length atstep1050 and the process return to step1010. If the number of similar pixels is less than the threshold, the method continues to step1035. In inclusion of small runs of similar pixels in the raster segment avoids unnecessary fragmentation of the raster segment, which allows for more efficient storage of the delta data in the payload.
• The RGB data from the current raster segment is copied into the payload bydelta encoder119 atstep1035. The RGB data may include the raster, starting X position, the length of the raster segment, and data values for each pixel in the segment. An example ofRGB payload data1140 within apayload1100 is illustrated inFIG. 11 and discussed in more detail below. After copying the RGB raster segment data to the payload, the RGB raster segment is applied tocurrent frame116 maintained atserver110. By applying each raster segment to the copy of the current frame, the UI encoder may update the current frame to reflect the current frame one raster segment at a time. By the time UI encoder has processed every dirty rectangle, the current frame RGB data should be modified into the new frame. After updating the current frame with the raster segment data, the raster segment encoding is complete atstep1045 and the RGB delta encoding process returns to step935 of the method ofFIG. 9.
• FIG. 10bfollows the same basic structure of encoding alpha as used to encode RGB, the primary difference is that only the alpha values are compared and color data is ignored during this encoding, also of note is the alpha encoder may run length encode (RLE) the contents of the alpha raster segments to reduce the size of the alpha delta encoded payload as seen instep1085.
• Other data formats and color formats may be encoded in a similar manner as RGB data. In particular, segments of different data may be detected in a selected raster for any data format. The resulting segments may comprise the data that differs and in some embodiments a limited run of pixels in the raster (and adjacent to the changed pixels in the raster) that are the same. The raster segments may then be applied to the payload and a copy of a current frame maintained on the server.
• Some types of data may be encoded in other ways as well. The color data stored in the raster segment payloads can be compressed using other compression techniques or algorithms. For example, alpha data may be run length encoded (RLE) within the raster segment. RLE includes specifying a raster number, X location, length of pixels and a single alpha value. The alpha values for that run length will all be set to the particular alpha value. Alpha values often have the same data value or very little fluctuation over an entire raster. Thus, RLE can be more efficient that encoding a length of alpha values that have the same or similar values.
• FIG. 11 is an example of a format of a payload sent toclient130 by aserver110.Payload1100 includes aheader1110,motion payload1120,cache payload1130,RGB payload1140,alpha payload1150, and end offrame payload1160.Header1110 includes general information about the entire payload. For example, the header may include payload size, version and check sum information. The header may also indicate a number of payloads sent and/or where the present payload exists within those payloads. For example, a header may indicate that the current payload is a second of seven payloads sent or that it is the last payload of seven sent payloads.Header1110 may contain continuity information to allow the decoder to detect lost payloads.Header1110 may also include flags indicating whether data for the motion, cache, RGB and alpha portions of that payload are included. In some embodiments, a flag is set for each payload portion which contains data in the present payload. For example, if a payload header contained a motion payload flag and RGB payload flag set to true and cache payload flag and alpha payload flag set to false, then the motion and RGB portions of the payload would be expected to contain data and the cache and alpha payload portions would not be expected to contain data or to even be present in the payload
• Motion payload1120 may include motion vector data. In particular,motion payload1120 may indicate the number of commands contained in the motion payload and data for each motion command. Each motion command may indicate a rectangle (or some other shape) and a vector or measure of movement. The rectangles may indicate a location and shape of a rectangle, such as a coordinate point, width and height. The vector may indicate a direction of movement for that particular rectangle, a number of pixels the object has moved, a new location for the origin pixel, or some other indication of motion. For example, the motion vector may include data of “−10” pixels, “20 pixels down” or some other expression for indicating a distance and direction, a new origin coordinate of (20, 15), or some other data.
• Cache payload1130 may include commands for setting a particular frame cached at the client as a current frame. In some embodiments, a cache payload is exclusive of a motion payload; if a motion payload is included in a payload, a cache payload is not. Examples of cache commands included in a cache payload include commands to set cache ID number at a client and commands to set to a cache ID number and fill a particular value at a client. A command is set to a cache ID number and fill a particular background may be useful when a background has a relatively uniform color.
• RGB payload1140 provides data regarding pixel red-green-blue data. In particular, an RGB payload may indicate the number of rasters for which data is provided in the payload, the number of raster segments within each raster, a Y location (raster location) and number of segments within the Y location, an X location and length of each segment and a pixel data in RGB format for each pixel of a segment. The pixel data may indicate new pixel data to apply to the current frame or cached frame.
• Alpha payload1150 contains data in a format similar to the RGB payload format with the number of rasters in the payload, the raster number, the number of raster segments for the raster, an X location and length for each segment, and alpha pixel data for each pixel in each segment. The alpha payload may also optionally include an indication, for example a flag, as to whether the alpha payload is run length encoded.
• End ofFrame payload1160 may indicate the payload is the last payload needed to fully realize a frame update. When an End of frame flag is set, the payload may provide data regarding a shape or bounding rectangle to indicate toUI decoder131 the area of the frame which has changed and needs updating on the display device.
Client Receipt and Processing of Payload
• Thin client130 may receive the payload transmitted byserver110. Once received, the payload may be passed toUI decoder131 and parsed to retrieve encoded commands, frame data and other graphical update data. The commands are then executed in a particular order to render a new frame at the client. The new frame is rendered by updating a current frame or cached frame at the client with graphical updates contained in the payload (cache, motion and delta commands). The rendered frame is then presented through a display device atclient130.
• FIG. 12 is a flowchart of an embodiment of a method for performing decoder logic on a client. A payload is received byclient130 and passed toUI decoder132 on the client atstep1210. The payload header is then parsed byUI decoder131 atstep1215. As illustrated inFIG. 11,payload header1110 of apayload1100 may include size, version, checksum, sub-payload portion flags, and indication of whether the current payload is the last of several payloads, and optionally other data. After parsing the payload header, a determination is made as to whether the payload passes verification atstep1220. Payload verification may include determining whether the actual payload size matches an indicated payload size and confirming checksum data. Verifying a payload is discussed in more detail below with respect toFIG. 13. If a payload fails verification, then the client encoding process does not process the payload any further and does not provide updated graphical data in the form of a new frame. Rather, the method ends atstep1255.
• If a payload passes verification, any payload motion commands contained inmotion payload1120 are processed atstep1225. The motion commands may include a list of rectangle data (or other shape data) and a vector indicating where to move each rectangle. In some embodiments a payload may not contain any motion commands, such as if a payload containscache payload1130. Processing payload motion commands bypayload decoder132 is discussed in more detail below with respect to the method ofFIG. 14.
• Any payload cache commands contained incache payload1130 are processed atstep1230. Cache commands may include providing a cache ID for the payload decoder to select and optionally provide any fill data in the cached frame. Cache commands will not be included in a payload that includes motion commands, and vice versa. Processing payload cache commands bypayload decoder132 is discussed in more detail below with respect to the method ofFIG. 17.
• Payload RGB delta encode commands contained inRGB payload1140 are processed atstep1235. RGB delta encode commands may include a raster and X location to encode in an frame and color value data to copy to the frame. The encode RGB delta commands will be applied to a current frame. Processing payload RGB delta encode commands bypayload decoder132 is discussed in more detail below with respect to the method ofFIG. 18.
• Alpha payload commands inalpha payload1150, if any, are processed atstep1240. Alpha delta encode commands may include a raster, X location and alpha value data to copy to the frame. In some embodiments, RLE alpha values may be decoded fromalpha payload1150. Similar to RGB encode commands, alpha encode commands will be applied to the current frame. Processing payload alpha commands bypayload decoder132 is discussed in more detail below with respect to the method ofFIG. 19.
• After processing a payload, a determination is made as to whether or not to present the updated graphic to display device screen atstep1245. In one embodiment, an update from a current frame to a new frame may be specified over several payloads. The last payload of the several payloads may indicate that it is the latest payload. For example,header1110 ofpayload1100 may include a flag or other data indicating that is a “last payload.” In this embodiment, the UI decoder will determine to present the updated graphic to a display device screen if the most recently received payload included a flag which was set to indicate that the payload was the last payload. If the updated graphic data is not to be presented, the method ofFIG. 12 is complete. If the updated graphic data is to be presented, a bounding shape or rectangle containing graphical updates is parsed and the area from the current cache frame is presented to the display device atstep1250. The method then ends atstep1255.
• FIG. 13 is a flowchart of an embodiment of a method for verifying a payload. In some embodiments, the method ofFIG. 13 provides more detail forstep1220 of the method ofFIG. 12. First, a determination is made as to whether the actual payload size matches a payload size indicated inheader1110 of the payload. If the actual size and indicated size of the payload do not match, the payload fails verification atstep1360. If the actual size and indicated size of the payload match, a determination is made as to whether a checksum is contained in the header. If the checksum is not contained in the header, a checksum comparison is not performed and the payload is determined to pass the verification atstep1350. In some embodiments, a payload may be considered to fail verification if the payload does not contain a checksum. In other embodiments, the checksum may only be used as a testing mechanism
• If the checksum is contained in the header (or elsewhere in the payload), the checksum of the payload is calculated atstep1330. A determination is then made as to whether the calculated checksum matches the indicated checksum atstep1340. If the checksums match, the payload passes verification atstep1350. If the checksums do not match, the payload fails verification atstep1360.
• FIG. 14 is a flowchart of an embodiment of a method for processing payload motion commands. In some embodiments, the method ofFIG. 14 provides more detail forstep1225 of the method ofFIG. 12. First, a first motion command is selected in the payload bypayload decoder132 atstep1420. The motion commands are selected frommotion payload1120. The selected motion command is then parsed atstep1420. The parsing determines the rectangle and vector which comprise the motion command. The parsed motion command is then applied to the current frame atstep1430. Applying the motion command results in copying a rectangle (or other shape) from a first position in a current frame and applying the copy to a second position in a current frame, such as the resulting frame illustrated in the center frame ofFIG. 8C. After applying the selected motion command, a determination is made as to whether more motion commands exist in the motion payload atstep1430. If no additional motion commands exist in a payload, processing of payload motion commands is complete atstep1450. If additional motion commands exist in the payload, the next motion command is selected atstep1450 and the method returns to step1420 to parse the selected motion command.
• As discussed above, the present technology may accelerate remote user interface rendering through caching. Caching at the client may be implemented in an efficient manner by configuring use of a frame buffer as a virtual cache and using pointers to indicate which cached frame is a current frame.FIGS. 15A and 15B are examples of managing a frame cache with a frame buffer. As illustrated inFIG. 15A, a frame buffer contains a frame of “A” while a frame cache contains three frames of A, B and C. As a graphical update to the current frame is required, for example an update from frame of A to frame of B as illustrated inFIG. 15B, data comprising the B frame is copied into the frame buffer from the frame cache. Copying a frame from a cache into a frame buffer uses precious CPU processing resources and inefficiently requires some duplication of stored data at the client.
• In some embodiments of the present technology, frames may be cached using a virtual frame cache.FIGS. 16A and 16B are examples of configuring a frame buffer as a virtual frame cache. A frame pointer may be used to indicate which portion of a frame cache or virtual frame cache (formerly a frame buffer) contains the current frame to be displayed in the user interface. InFIG. 16A, the current frame pointer is set to the address of frame of A within the virtual frame cache. InFIG. 16B, the current frame pointer is set to the address of frame B within the frame cache. Utilizing a frame buffer as a virtual frame cache and a current frame pointer to identify the location of a cached current frame allows for quicker access to a larger size frame cache. This “zero copy cache” method also effectively provides one additional cache frame over the previous method using the same amount of memory storage, since there is no duplication of the current cache frame.
• FIG. 17 is a flowchart of an embodiment of a method for processing payload cache commands. In some embodiments, the method ofFIG. 17 provides more detail forstep1230 of the method ofFIG. 12. First, a cache command is parsed in a cache payload portion ofpayload1100 atstep1710. A cache ID is then selected as the current cache frame atstep1720. The selected cache ID corresponds to a cache ID contained in the parsed cache command. A determination is then made as to whether to fill the current cached frame with solid color as instructed by the cache command atstep1730. As discussed above,delta encoder119 may encode a cache command with fill data if the cache logic detects a constant color fill when processing the cache frame. If the cache command includes color fill data, the current cache frame is filled as indicated by the corresponding fill data atstep1740 and the method is complete atstep1750. If the cache command does not include color fill data, the current cache frame is not filled and the method is complete atstep1750.
• FIG. 18 is a flowchart of an embodiment of a method for processing payload RGB delta commands. In some embodiments, the method ofFIG. 18 provides more detail forstep1235 of the method ofFIG. 12. A raster count is first parsed atstep1810 and raster Y location is parsed atstep1820. The raster segment X location and sample count is then parsed atstep1830 and the RGB data encoded in the delta payload is copied to the current cache frame at the indicated X,Y position atstep1840. The alpha data for the updated pixels is preserved (unchanged) while the RGB data is copied into the current cache frame. After copying the RGB data, a determination is made as to whether any more raster segments exist to be parsed atstep1850. If more raster segments exist to be parsed, the method returns to step1850 where the next raster segment is selected and parsed. If no more raster segments exist to be parsed in the current raster, a determination is made as to whether any more rasters exist to be parsed atstep1860. If no more rasters exist to be parsed, processing of RGB delta commands in the payload is complete atstep1870. If more rasters exist to be parsed, the method returns to step1820 wherein the next raster is selected and the raster data is parsed.
• FIG. 19 is a flowchart of an embodiment of a method for processing payload alpha delta commands. In some embodiments, the method ofFIG. 19 provides more detail forstep1240 of the method ofFIG. 12. A raster count is first parsed atstep1910, the raster Y location is then parsed atstep1920, and a raster segment X location and sample count is parsed atstep1930. A determination is then made as to whether the alpha data is RLE encoded atstep1940. As discussed above, the alpha payload may have a flag or some other indication that the alpha data may be run length encoded rather than segment encoded to more efficiently encode alpha data having the same or similar values. If the alpha data is run length encoded, the RLE alpha data is decoded into the current cache at the determined X, Y location for the length indicated. While decoding the RLE alpha data, the RGB data values are preserved (not changed). If the alpha data is not run length encoded, the alpha data is encoded into the current cache in raster segments similar to how RGB data is encoded or copied into the current cache frame. After raster segment alpha data or RLE alpha data is decoded into the cache, a determination is made as to whether more raster segments (or RLE encodings) exist for the raster atstep1970. If more raster segments exist, the next segment is selected and the raster segment is parsed atstep1930. If more raster segments do not exist, a determination is made as to whether more rasters exist to be processed. If more rasters exist, the next raster is selected and the raster is parsed atstep1920. If more rasters do not exist, the method is complete atstep1990.
• FIG. 20 is a block diagram for an embodiment of a computing environment for implementing the present technology. In some embodiments, the computing environment ofFIG. 20 may be used to implementserver110 andclient130 of the system ofFIG. 1A.
• Computing environment2000 ofFIG. 20 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the technology herein. Neither should thecomputing environment2000 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in theexemplary operating environment2000.
• The technology described herein is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the technology herein include, but are not limited to, personal computers, server computers, hand-held or laptop devices, mobile phones or devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
• The technology herein may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth that perform particular tasks or implement particular abstract data types. The technology herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
• With reference toFIG. 20, an exemplary system for implementing the technology herein includes a general purpose computing device in the form of acomputer2010. Components ofcomputer2010 may include, but are not limited to, aprocessing unit2020, asystem memory2030, and a system bus2021 that couples various system components including the system memory to theprocessing unit2020. The system bus2021 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
• Computer2010 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed bycomputer2010 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed bycomputer2010. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
• Thesystem memory2030 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)2031 and random access memory (RAM)2032. A basic input/output system2033 (BIOS), containing the basic routines that help to transfer information between elements withincomputer2010, such as during start-up, is typically stored inROM2031.RAM2032 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on byprocessing unit2020. By way of example, and not limitation,FIG. 20 illustratesoperating system2034,application programs2035,other program modules2036, andprogram data2037.
• Thecomputer2010 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,FIG. 20 illustrates ahard disk drive2040 that reads from or writes to non-removable, nonvolatile magnetic media, amagnetic disk drive2051 that reads from or writes to a removable, nonvolatilemagnetic disk2052, and anoptical disk drive2055 that reads from or writes to a removable, nonvolatileoptical disk2056 such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. Thehard disk drive2041 is typically connected to the system bus2021 through a non-removable memory interface such asinterface2040, andmagnetic disk drive2051 andoptical disk drive2055 are typically connected to the system bus2021 by a removable memory interface, such asinterface2050.
• The drives and their associated computer storage media discussed above and illustrated inFIG. 20, provide storage of computer readable instructions, data structures, program modules and other data for thecomputer2010. InFIG. 20, for example,hard disk drive2041 is illustrated as storingoperating system2044,application programs2045,other program modules2046, andprogram data2047. Note that these components can either be the same as or different fromoperating system2034,application programs2035,other program modules2036, andprogram data2037.Operating system2044,application programs2045,other program modules2046, andprogram data2047 are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer200 through input devices such as akeyboard2062 andpointing device2061, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to theprocessing unit2020 through auser input interface2060 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). Amonitor2091 or other type of display device is also connected to the system bus2021 via an interface, such as avideo interface2090. In addition to the monitor, computers may also include other peripheral output devices such asspeakers2097 andprinter2096, which may be connected through anoutput peripheral interface2090.
• Thecomputer2010 may operate in a networked environment using logical connections to one or more remote computers, such as aremote computer2080. Theremote computer2080 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to thecomputer2010, although only amemory storage device2081 has been illustrated inFIG. 20. The logical connections depicted inFIG. 20 include a local area network (LAN)2071 and a wide area network (WAN)2073, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
• When used in a LAN networking environment, thecomputer2010 is connected to theLAN2071 through a network interface oradapter2070. When used in a WAN networking environment, thecomputer2010 typically includes amodem2072 or other means for establishing communications over theWAN2073, such as the Internet. Themodem2072, which may be internal or external, may be connected to the system bus2021 via theuser input interface2060, or other appropriate mechanism. In a networked environment, program modules depicted relative to thecomputer2010, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,FIG. 20 illustratesremote application programs2085 as residing onmemory device2081. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
• The foregoing detailed description of the technology herein has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The described embodiments were chosen in order to best explain the principles of the technology and its practical application to thereby enable others skilled in the art to best utilize the technology in various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the technology be defined by the claims appended hereto.

Claims (20)

What is claimed is:

1. A processor readable storage device having processor readable code embodied on said processor readable storage device, said processor readable code for programming a processor to:

receive a set of instructions by a client device from a server, the set of instructions including a cache instruction and difference data instructions;

set a first pointer to a cache slot at the client device in response to executing the cache instruction;

modify graphical frame data associated with the cache slot in response to executing the difference data instructions; and

present a frame associated with the modified graphical frame data through a display device by the client device.

2. The processor readable storage device ofclaim 1, wherein the cache instruction identifies the cache slot at the client device to which the first pointer is set.

3. The processor readable storage device ofclaim 1, wherein the instructions that cause the processor to set the first pointer to the cache slot cause the processor to either set the first pointer to an address of a frame within cache memory of the client device or to a frame buffer of the client device, the frame buffer stores graphical frame data for a single frame, the cache memory stores graphical frame data for a plurality of frames.

4. The processor readable storage device ofclaim 3, wherein only one copy of two or more graphical frames of data is stored in memory of the client device, the client memory including the cache memory and the frame buffer.

5. The processor readable storage device ofclaim 1, wherein the set of instructions include an instruction to fill a cached frame associated with the cache slot to which the first pointer is set with a color.

6. The processor readable storage device ofclaim 1, wherein the set of instructions includes difference data instructions in two different data formats used to present graphical information.

7. The processor readable storage device ofclaim 1, wherein the instructions that cause the processor to modify include instructions that cause the processor to:

apply updates to raster segments in the cache slot to which the first pointer is set with raster segment data contained in the difference data instructions.

8. A method comprising:

receiving a set of instructions by a client device from a server, the set of instructions including a cache instruction and difference data instructions;

setting a current frame pointer to a cache slot at the client device in response to executing the cache instruction;

modifying graphical frame data associated with the cache slot to which the current frame pointer is set in response to executing the difference data instructions; and

presenting a frame associated with the modified graphical frame data through a display device by the client device.

9. The method ofclaim 8, wherein the cache instruction identifies the cache slot at the client device to which the current frame pointer is set.

10. The method ofclaim 8, wherein the setting a current frame pointer to a cache slot at the client device in response to executing the cache instruction includes either:

setting the current frame pointer to an address of a frame in a cache memory on the client device that stores a plurality of frames of data; or

setting the current frame pointer to an address of a frame buffer on the client device that stores a single frame of data.

11. The method ofclaim 10, wherein only one copy of two or more graphical frames of data is stored in memory on the client device, the client memory including the cache memory and the frame buffer.

12. The method ofclaim 8, wherein the set of instructions include an instruction to fill a cached frame to which the current frame pointer is set with a color.

13. The method ofclaim 8, wherein the set of instructions includes difference data instructions in two different data formats used to present graphical information.

14. The method ofclaim 8, wherein said step of modifying includes:

applying updates to raster segments in the cache slot to which the current frame pointer is set with raster segment data contained in the difference data instructions.

15. The method ofclaim 8, wherein the set of instructions are received by the client device from the server as part of a cache payload, further comprising:

receiving a motion payload by the client device from the server, a payload includes either a cache payload or a motion payload but not both.

16. A system comprising:

a display device;

a processor readable storage device that stores a cache of frame data; and

a processor configured to:

receive a set of instructions over a network from a server, the set of instructions including a cache instruction and difference data instructions;

set a current frame pointer to a slot in the cache in response to executing the cache instruction, the current frame pointer identifies a current frame to be displayed in the display device;

modify graphical frame data associated with the cache slot in response to executing the difference data instructions; and

present a frame associated with the modified graphical frame data through the display device.

17. The system ofclaim 16, wherein the processor being configured to set the current frame pointer to the slot in the cache comprises the processor being configured to either set the current frame pointer to an address of a frame within cache memory in the processor readable storage device or to a frame buffer in the processor readable storage device.

18. The system ofclaim 16, wherein the set of instructions include an instruction to fill a cached frame associated with the cache slot to which the current frame pointer is set with a color.

19. The system ofclaim 16, wherein the set of instructions includes difference data instructions in two different data formats used to present graphical information.

20. The system ofclaim 16, wherein the processor being configured to modify includes the processor being configured to:

apply updates to raster segments in the cache slot to which the current frame pointer is set with raster segment data contained in the difference data instructions.

Images