US20180217932A1

Movatterモバイル変換

Info

Publication number: US20180217932A1
Application number: US15/422,691
Authority: US
Inventors: Tushar P. Ringe; Jamshed Jalal; Klas Magnus Bruce; Phanindra Kumar Mannava
Original assignee: ARM Ltd
Current assignee: ARM Ltd
Priority date: 2017-02-02
Filing date: 2017-02-02
Publication date: 2018-08-02
Anticipated expiration: 2037-02-02
Also published as: US10042766B1

Abstract

A home node of a data processing apparatus that includes a number of devices coupled via an interconnect system is configured to provide efficient transfer of data to a first device from a second device. The home node is configured dependent upon data bus widths of the first and second devices and the data bus width of the interconnect system. Data is transferred as a cache line serialized into a number of data beats. The home node may be configured to minimize the number of data transfers on the third data bus or to minimize latency in the transfer of the critical beat of the cache line.

Description

BACKGROUND

Data processing systems, such as a System-on-a-Chip (SoC) may contain multiple processor cores, multiple data caches and shared data resources. In a shared memory system for example, each of the processor cores may read and write to a single shared address space. Cache coherency is an issue in any system that contains one or more caches and more than one device sharing data in a single cached area. There are two potential problems with a system that contains caches. First, memory may be updated by another device after a cached device has taken a copy. At this point, the data within the cache is out-of-date or invalid and no longer contains the most up-to-date data. Second, systems that contain write-back caches must deal with the case where the device writes to the local cached copy at which point the memory no longer contains the most up-to-date data. A second device reading memory will see out-of-date data.

The data processing system may be arranged as a number of nodes coupled together via an interconnect system to form a network. One example of a protocol for maintaining cache coherency uses snoop requests. When a node of a network wishes to access data associated with a particular address, a snoop message is sent to other nodes that have a copy of requested data stored in a local cache.

However, the connected nodes may have different data bus widths. The presence of nodes having different data bus widths can result in timing and bus utilization inefficiencies.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings provide visual representations which will be used to more fully describe various representative embodiments and can be used by those skilled in the art to better understand the representative embodiments disclosed and their inherent advantages. In these drawings, like reference numerals identify corresponding elements.

FIG. 1 is a block diagram of a data processing apparatus, in accordance with various representative embodiments.

FIG. 2 is a diagrammatic representation of a sequence of data beats corresponding to a cache line in a data processing apparatus.

FIG. 3 is an example transaction flow diagram for a data transaction in a data processing apparatus.

FIGS. 4-6 are transaction flow diagrams for data transactions in a data processing apparatus, in accordance with embodiments of the disclosure.

FIG. 7 is a flow chart of a method of processing a data read request, in accordance with various representative embodiments.

FIG. 8 is a flow chart of a method of processing a data read response, in accordance with various representative embodiments.

DETAILED DESCRIPTION

While this invention is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detail specific embodiments, with the understanding that the present disclosure is to be considered as an example of the principles of the invention and not intended to limit the invention to the specific embodiments shown and described. In the description below, like reference numerals may be used to describe the same, similar or corresponding parts in the several views of the drawings.

In this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by “comprises . . . a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Reference throughout this document to “one embodiment,” “certain embodiments,” “an embodiment,” “implementation(s),” “aspect(s),” or similar terms means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of such phrases or in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments without limitation.

The term “or” as used herein is to be interpreted as an inclusive or meaning any one or any combination. Therefore, “A, B or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B and C.” An exception to this definition will occur only when a combination of elements, functions, steps or acts are in some way inherently mutually exclusive. Also, grammatical conjunctions are intended to express any and all disjunctive and conjunctive combinations of conjoined clauses, sentences, words, and the like, unless otherwise stated or clear from the context. Thus, the term “or” should generally be understood to mean “and/or” and so forth.

All documents mentioned herein are hereby incorporated by reference in their entirety. References to items in the singular should be understood to include items in the plural, and vice versa, unless explicitly stated otherwise or clear from the text.

Recitation of ranges of values herein are not intended to be limiting, referring instead individually to any and all values falling within the range, unless otherwise indicated, and each separate value within such a range is incorporated into the specification as if it were individually recited herein. The words “about,” “approximately,” “substantially,” or the like, when accompanying a numerical value, are to be construed as indicating a deviation as would be appreciated by one of ordinary skill in the art to operate satisfactorily for an intended purpose. Ranges of values and/or numeric values are provided herein as examples only, and do not constitute a limitation on the scope of the described embodiments. The use of any and all examples, or exemplary language (“e.g.,” “such as,” or the like) provided herein, is intended merely to better illuminate the embodiments and does not pose a limitation on the scope of the embodiments. No language in the specification should be construed as indicating any unclaimed element as essential to the practice of the embodiments.

For simplicity and clarity of illustration, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. Numerous details are set forth to provide an understanding of the embodiments described herein. The embodiments may be practiced without these details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the embodiments described. The description is not to be considered as limited to the scope of the embodiments described herein.

In the following description, it is understood that terms such as “first,” “second,” “top,” “bottom,” “up,” “down,” “above,” “below,” and the like, are words of convenience and are not to be construed as limiting terms. Also, the terms apparatus and device may be used interchangeably in this text.

As utilized herein, the terms “coupled”, “connected,” and/or similar terms are used generically. It should be understood that these terms are not intended as synonyms. Rather, “connected” is used generically to indicate that two or more components, for example, are in direct physical, including electrical, contact; while, “coupled” is used generically to mean that two or more components are potentially in direct physical, including electrical, contact; however, “coupled” is also used generically to also mean that two or more components are not necessarily in direct contact, but nonetheless are able to co-operate and/or interact. The term “coupled” is also understood generically to mean indirectly connected, for example, in an appropriate context.

The various embodiments and examples of the present disclosure as presented herein are understood to be illustrative of the present disclosure and not restrictive thereof and are non-limiting with respect to the scope of the present disclosure.

Further particular and preferred aspects of the present disclosure are set out in the accompanying independent and dependent claims. Features of the dependent claims may be combined with features of the independent claims as appropriate, and in combinations other than those explicitly set out in the claims.

The present disclosure is directed towards achieving an efficient trade-off between signal latency and data bus utilization in a data processing apparatus, such as a System-on-a-Chip (SoC).

A data processing apparatus may use multiple processor devices inside a chip to scale up performance and reduce latency. A program or task can be broken down into independent steps that can be executed in parallel by multiple devices. When the devices need to share data or control information, they communicate with one another through an on-chip interconnect.

Shared data is often stored off-chip in a main memory (such as DRAM) which is slow to access and consumes a lot of power. Efficiency is improved by using local L2 caches in each processor device that are accessible by other on-chip processor devices. This speeds up the inter-processor communication and reduces power consumption.

However, a mechanism is needed to ensure that any processor gets the latest data, rather than out-of-date or stale data. One approach is to use one or more home nodes in the interconnect to keep track of cache contents and manage memory access requests. A home node is a data handling module that can be considered to be within the interconnect or attached to it. Each home node manages requests for a region of the shared memory. A processing device sends data requests to the appropriate home node, and the home node manages how the request is processed. The processing device is referred to as a requestor node. The home node may serialize requests from all requestor nodes and execute them in serial order so as to maintain data coherency.

The memory space is divided between home nodes and is further divided into units called cache lines. For example, a cache line may contain 32, 64, or 128 bytes of data depending upon the design of the system. Coherency is managed at this level, so data is stored as cache lines in the local caches of processing devices and, when data is requested from a home node, a complete cache line is returned, containing the data associated with the requested address.

When a requestor node wants to access data in a cache line, it sends a request to the home node of the interconnect that is associated with that address. The home node maintains a record, in the form of a snoop filter, for example, of which nodes have a copy of the cache line in a local cache and sends a message, called a snoop request, to the identified node or nodes (the target nodes) requesting the data. Data not stored in local cache may be retrieved from the main memory via a memory controller, for example.

Data is transferred through the interconnect on data buses of a certain width. In addition, each node has a data bus of a certain width. These data bus widths may not be the same. In addition, the bus widths are usually not wide enough to transfer a complete cache line in a single parallel transfer. The cache line must then be split into multiple sections, called ‘beats’ and sent serially. For example, a 64 byte cache line (512 bits) could be sent as four 128-bit beats on a 128-bit data bus, or as two 256-bit beats on a 256-bit data bus. This serialization of the data transfer introduces a delay or latency into the system. Further, when the bus widths are not all the same, bus utilization depends upon how data is split and combined.

In accordance with embodiments of the disclosure, a home node of a data processing apparatus that includes a number of devices coupled via an interconnect is configured to provide efficient transfer of data to a first device from a second device. The home node is configured dependent upon data bus widths of the first and second devices and dependent upon the data bus width of the interconnect. Data is transferred as a cache line serialized into a number of data beats. The home node may be configured to minimize the number of data transfers on the third data bus or to minimize latency in the transfer of the critical beat of the cache line.

FIG. 1 is a block diagram of adata processing apparatus100, in accordance with various embodiments. Data processing systems, such as a System-on-a-Chip (SoC), may contain multiple processing devices, multiple data caches, and shared data resources. Thesystem100 may be implemented in a System-on-a-Chip (SoC) integrated circuit, for example. In the simplified example shown, thesystem100 is arranged as a network with a number of nodes connected together via an interconnect system. The nodes are functional blocks or devices, such as processors, I/O devices or memory controllers, for example. As shown, the nodes include

devices

102,104,106 and108. The devices are coupled via aninterconnect system110. Other devices, such as input/output devices and timers, have been omitted inFIG. 1 for the sake of clarity. A device such as102,104,106 or108, may be, for example, a processing core, a cluster of processing cores, a graphics post processor (GPP), a digital signal processor (DSP), a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC) device.

The devices may be coupled, viainterconnect system110 and one or more home nodes112 (HNs) to each other and to data resources such asmemory controller114 for accessing sharedmemory116.

Devices

102,104,106 and108 are able to generate data access requests and are referred herein as requestor nodes (RNs). When multiple nodes share a data or memory resource, a coherent protocol may be used, and devices may be referred to as fully coherent (e.g. RNF and HNF). Other devices may provide connections to another integrated circuit. Note that many elements of a SoC, such as clocks, have been omitted fromFIG. 1 for the sake of clarity.

Nodes that generate requests for data transactions, such as ‘load’ and ‘store’, for example, are referred to a requestor node and are end points for such transactions. Nodes that respond to the requests are referred to as target nodes. For example, inFIG. 1,

devices

102 and104 are requestor nodes and devices106 and108 are target nodes. When data requested by

requestor nodes

102 and104 is present at target node106 or108, it is passed back to the requestor node viainterconnect system110.

In a data transaction, the requestor node creates a message, comprising one or more packets, and injects it into the interconnect for transmission to its destination. A packet can contain a memory address, data from memory, or both, and may also hold control or routing bits. A control packet, such as a data read request, may contain only address information and control bits. In an on-chip network, a ‘packet’ is the meaningful unit of the upper-layer protocol, such as the cache-coherence protocol.

A packet may be broken into a number of flow-control digits or ‘flits’. A packet can contain a head flit, a body flit, and tail flit. The head denotes the beginning of a packet, containing routing and control information, the body contains the payload of the packet, and the tail denotes when the entire packet has been received. While individual packets may follow different routes in a network, the component flits of a packet are transmitted in sequence over a common route. Thus, a “flit” is the smallest unit of flow control maintained by the network. Generally, a packet consists of one or more number of flits, so if the packet size is larger than one flit, then the packet is split into multiple flits.

A physical channel may contain a data bus, which consists of lines dedicated to data transfer, together with additional lines dedicated to control data and metadata. In one embodiment, the data is divided into a number of chunks and the metadata includes ‘data-validity bits’ that indicate whether a corresponding data chunk contains valid data or not. As discussed below, the ability to transfer data beats that contain both valid and invalid chunks enables data transfer latency to be reduced. Typically, a ‘flit’ will contain a data beat in addition to other information that is transmitted in parallel. For example, the additional information may be validity bits that indicate which parts for data beat are valid and which are invalid. Herein, the action of sending a data beat and sending a flit are considered to be synonymous.

In a cache, data is typically stored as lines of data, for example 64 bytes of data, together with an address tag or line address. For an N-bit address and cache lines of 64 bytes, bits [N−1:6] of the address may be used as the cache line tag or address. The lower six bits of the address (bits [5:0]) denote the offset or position of the data within the cache line. Other length cache lines may be used.

In response to a data read request, a complete cache line is returned. Unless the data bus is the same width as a cache line, the data is sent as a number of data beats. For example, a 128-bit bus can sent 16 bytes at a time, so a 64 byte line is sent as four beats.Beat0 contains data bytes withindices 15→0 (hexadecimal indices 0x00 to 0x0f), beat1 contains data bytes with indices 31→16 (hexadecimal indices 0x10 to 0x10, beat2 contains data bytes with indices 47→32 (hexadecimal indices 0x20 to 0x2f), and beat3 contains data bytes with indices 63→48 (hexadecimal indices 0x30 to 0x3f). Thus, bits [5:4] of the requested data address indicate which beat contains the requested data bytes.

The time it takes for a flit or message to traverse the interconnect, from the time of injection to the time it is processed at its destination is referred to as latency. The total latency has three major components:

- Propagation latency: the time it takes for a bit to traverse the interconnect.
- Queuing latency: the time a packet waits in buffers/queues within the interconnect.
- Serialization latency: when a packet size is larger than the channel width, the packet is broken into multiple pieces or beats, resulting in additional time for a total packet traversal. For example, if a packet is 256 bits, and the channel is 64 bits-per-cycle wide, the serialization is 4 cycles.

In order to reduce serialization latency, nodes are configured to send the beat containing the requested data bytes first, when responding to a request. The remaining beats may then be sent in ‘wrap’ order to facilitate circular addressing.

Referring again toFIG. 1,Interconnect system110 may be used to connect devices having different bus widths by the inclusion ofdata combiner module118 anddata splitter module120. Multiple combiner and/or splitter modules may be used.Data combiner module118 anddata splitter module120 act as bridges, examples of which are described in co-owned patent application Ser. No. 14/965,237.

Data combiner module

118 can combine multiple incoming flits containing narrow beats (say 128-bit) into a single flit containing a wider beat (say 256-bit) provided that all narrow beats are contiguous in time, do not cross the wider beat boundary, and arrive in a given time window. If incoming (narrower) beats do not arrive in correct order as required by the data combiner, the outgoing (wider) data bus will carry partially-formed data beats thereby resulting in sub-optimal data bus utilization.

FIG. 2 is a diagrammatic representation of a sequence of beats corresponding to a cache line, for example. A sequence of narrower (128-bit) beats,202,204,206 and208, is shown in the upper sequence and a sequence of wider (256-bit) beats,210 and212, is shown in the lower sequence. In this example,narrower beats2 and3 (208 and206) can be combined together intowider beat212 andnarrower beats0 and1 (202 and204) can be combined together in wider bear210. However, beats1 and2 cannot be combined since they span the boundary between 256-bit beats210 and212.

narrower beats

202 and204 by a splitter module andwider beat212 may be split into

narrower beats

206 and208.

An interconnect system may contain Requestor Nodes (RNFs) with different data bus widths. For example, inFIG. 1,requestor node102 utilizes a 256-bit data bus and is denoted as ‘RNF-256’ whilerequestor node104 uses a 128-bit data bus and is denoted as ‘RNF-128’.Interconnect system110 utilizes a 256-bit bus. These data bus sizes are provided by way of example and other data bus sizes may be used. For example, a data bus width may be 32-bits, 64-bits, 128-bits or 256-bits. In addition, widths that are not powers of two may be used. Although the example shown inFIG. 1 uses only two different bus widths, the interconnect may couple between more than two different bus widths. Requestor node108 sends data to the interconnect viadata combiner module118, andrequestor node104 receives data from the interconnect viadata splitter module120.

Cache coherency is an issue in any system that contains one or more caches and more than one device sharing data in a single data resource. There are two potential problems with a system that contains caches. Firstly, memory may be updated by another device after a cached device has taken a copy. At this point, the data within the cache is out-of-date or invalid and no longer contains the most up-to-date data. Secondly, systems that contain write-back caches must deal with the case where the device updates the local cached copy, at which point the memory no longer contains the most up-to-date data. A second device reading memory will see out-of-date (stale) data. Cache coherency may be maintained through the exchange of ‘snoop’ messages between the processing devices, such as

nodes

102,104,106 and108, for example.

To maintain coherence, each processing device includes a snoop control unit that issues and receives coherence requests and responses, snoop messages, via theinterconnect system110 from other devices.

Snoop requests and responses are managed byhome node112.Home node112 receives a data read request on

interconnect bus

122 or124, generates corresponding snoop messages on snoop

buses

126 and128, receives data oninterconnect buses130 and/or132, and returns the data to the requestor node onbuses134 and/or136. Data not found in any local caches of the target node may be retrieved from other locations, such as a memory viamemory controller114.

In some embodiments, snoopfilter138 is used to reduce the number of snoop messages by tracking which local caches have copies of data and filtering out snoop messages to other local caches.

In the example system shown inFIG. 1,nodes102 and106 each have a 256-bit data bus (140 and142, respectively), whilenodes104 and108 each have a 128-bit data bus (144 and146, respectively) and theinterconnect system110 has 256-bit data buses (130,132,134 and136). Other combinations of bus sizes may be used without departing from the present disclosure.

In the example discussed below with reference toFIG. 3,interconnect system110 has 256-bit bus widths, the requestor bus width is 256 bits, and the target bus width is 128 bits. This corresponds, for example, to a coherent read transaction initiated byrequestor102 and targeted to node108 inFIG. 1.

FIG. 3 is a flow diagram of an example transaction in a system that has not been optimized. In the transaction, a read request302 (denoted as RD_REQ [01]) is sent at time T1 from requestor node RNF-256 to Home Node HNF-256 in the interconnect system. The request includes a data address (ADDR) with some parts of address (ADDR[5:4], for example, as discussed above) denoting the beat (the serialized portion of the cache line) that will contain the requested data. This beat is referred to as the ‘critical’ beat. In this example, ADDR[5]=0 and ADDR[4]=1, which is denoted herein as ADDR[5:4]=2′b01. Thus, inFIG. 3, RD_REQ [01] denotes a request for data inbeat1.

If the home node determines that node RNF-128 has a copy of the requested data, the ADDR is passed as a snoop request (SNP_REQ [01])304 towards snoop target RNF-128 without any changes. The response data (SNP_RSP_DATA) is sent as four beats (306,308,310 and312) to the data combiner. A snoop target sends the response data in wrap order, starting with the critical beat first. In this example, the critical beat information is in beat1 (306), so this beat is sent first followed by the other beats (308,310,312) in wrap or cyclic order. If the critical beat happens to be misaligned to wider data bus (e.g. 256 bits) width, the data combiner module will not be able to combine multiple narrow beats into a single wide beat efficiently, thereby resulting in sub-optimal data bus utilization. In the example shown, the data combiner receives beat1 (306) and waits to see if it can be combined with the next beat. The next beat is beat2 (308) which cannot be combined, so beat1 is sent infirst flit314.First flit314 thus contains a single narrower beat. Beat3 (310) arrives next and may be combined with beat2 (308). Thus,second flit316 contains two narrower beats. Beat0 (312) arrives last. Since no additional beats are received within a designated time window,third flit318 is sent containing the single remaining beat.

The home node (HNF-256) performs the final combination of data and it will wait for all sub-beats to arrive before combining them to form the critical beat. As a result, the critical beat (beat1) is delivered to requestor inflit320 at time T2, which is later than the delivery of the non-critical beats inflit322. The transaction latency is the time from T1 until T2.

There are two issues with above transaction flow:

a. Critical beat (beat1) arrives at the requestor node later than non-critical beats, even though the snoop target delivered the critical beat first (306), and b. Three 256-bit data beats (314,316 and318) are used to transport four narrower beats (of 128 bits each) between the Data Combiner module and the home node HNF-256, whereas two data beats would be sufficient.

In accordance with a first embodiment of the disclosure, a HNF in an interconnect system is configured to detect the data bus widths of a requestor, a snoop target, and the interconnect system and, based on various data bus widths combination, determine whether and how to align the incoming requested address before sending the address towards snoop target.

FIG. 4 shows a transaction flow diagram of a data processing apparatus in accordance with an embodiment of the disclosure. In this embodiment, a read request402 (denoted as RD_REQ [01]) is sent at time T1 from requestor node RNF-256 to Home Node HNF-256 in the interconnect system. As before, the request includes an data address (ADDR) with some parts of address (ADDR[5:4], for example) denoting critical beat information. In this example, ADDR[5]=0 and ADDR[4]=1, so the requested data is located inbeat1.

The home node HNF-256 determines that snoop target bus (128-bit) is narrower than the Requestor/Interconnect data-bus widths (256-bit), so the home nodes HNF-256 aligns the snoop address to a 256-bit boundary and sends snoop request404 (SNP_REQ [00]). Thus, the snoop address is the requested address except that ADDR[4] is set to zero to align the address to a 256-bit boundary. Thus the snoop request indicates a request for data in the same cache line, but located inbeat0, rather thanbeat1. The snoop target RNF-128, thinkingbeat0 is the critical beat, responds with data beats406,408,410 and412, in contiguous and naturally aligned order. This enables the data combiner to efficiently combine narrow beats into fully-

populated flits

414 and416 when transferring data towards the home node HNF-256. The actual critical beat data (beat1) is first to reach requestor node inflit418. The remaining data arrives inflit420. This scheme minimizes the critical beat latency while maximizing data-bus efficiency, since only two flits (414 and416) are sent from the home node HNF-256 to the requestor RNF-256.

Snoop address alignment, as described above, solves two issues:

1) The critical beat (beat1) arrives at the requestor first, while non-critical beats arrive later (beat2/beat3). Therefore, the critical beat does not get exposed to any non-critical beat latency.

2) Between the data combiner module and HNF-256, only two 256-bits data beats (414 and416) are used to transport four beats of 128 bits each, thereby optimizing data bus utilization.

However, there are other scenarios where alignment of a snoop address will result in the best utilization of the data bus, but it may not be ideal for minimizing critical beat latency.

FIG. 5 shows a transaction flow diagram of a data processing apparatus in accordance with an embodiment of the disclosure. As indicated inFIG. 5, requestor RNF-128 on the left of the diagram and Snoop Target RNF-128 on the right of the diagram both have data-bus widths of 128 bits, while the interconnect system and the home node has a data-bus width of 256 bits. Readrequest502 for data inbeat1 is received by the home node HNF-256. The home node HNF-256, based on various data widths, is configured to align the address in snooprequest504 to 256 bits, as indicated by SNP_REQ [00] so as to achieve the best utilization of the data bus. Data bus utilization is optimized since the four beats of 128 bit data (506,508,510,512) are transported using only two 256-bit data beats (514 and516, then 518 and 520) in the interconnect system. The 256-bit beats are separated in the data splitter into split four 128-bit beats (522,524,526,528) in natural order. However, latency is not minimized since the critical beat524 (beat1) arrives at requestor later than beat0 (522).

In the transaction shown inFIG. 5, data bus utilization is optimized but critical beat latency is not optimized. If it is determined that critical beat latency is more important in this system, the home node HNF-256 can choose to not align the snoop address when sending a request to snoop target. Additionally, when the critical beat arrives at the home node, the home node may be configured to send partially formed critical beats directly to requestor, without waiting on wider beat being fully populated. This will result in the best critical beat latency, as shown inFIG. 6.

FIG. 6 shows a transaction flow diagram of a data processing apparatus in accordance with an embodiment of the disclosure. As indicated inFIG. 6, requestor RNF-128 on the left of the diagram and Snoop Target RNF-128 on the right of the diagram both have a data-bus width of 128 bits, while the interconnect system has a data-bus width of 256 bits. Read request602 (RD_REQ [01]) for data inbeat1 is received by the Home node HNF-257. The home node HNF-256 is configured to minimize the critical beat latency, so the address in snooprequest604 is sent unchanged, as indicated by the notation SNP_REQ [01]. The snoop target RNF-128 responds with four beats of 128 bit data (606,608,610,612), which are sent in wrap order with the critical beat first. The data combiner sends partially filled flit614 when beat608 arrives and cannot be combined.

Beats

608 and610 are combined intoflit616 and beat612 is sent inflit618 when no further flits are received by the data combiner within a designed time window.

HNF-256 is configured for minimum latency, so as soon as flit614 is received containingcritical beat1, flit620 is generated, even though the data beat is not completely filled. Signaling bits, such as data validity bits, are used to indicate to a recipient that the data beat is partially filled and which part of the data should be ignored.Flit622 is generated whenflit616 is received and flit624 is generated whenflit618 is received.

As before, the 256-bit beats are separated in the data splitter into split four 128-bit beats (626,628,630 and632) in wrap order with the critical beat being sent first. In this way latency is minimized, since the critical beat626 (beat1) arrives at requestor first. However, bus utilization is not optimized since three flits (614,616 and618 or620,622 and624) are used for transport in the interconnect.

The approaches described above enable a user or designer to configure an interconnect system for optimal latency or for optimal bus utilization. In some cases these may be achieved together. In one embodiment, the configuration may be performed dynamically using software controlled configuration registers, for example.

FIG. 8 is aflow chart800 of a method for processing a data read response in accordance with various representative embodiments. Followingstart block802, snoop response data is received at a home node of an interconnect system atblock804. The response comprises a sequence of one or more beats, where each beat is a portion of a cache line containing data requested by a requestor node. Atblock806, the received data is merged with any existing data in a cache line buffer in the home node. The system may be optimized for minimum latency or optimal bus utilization. If the system is optimized for minimum latency, as depicted by the positive branch fromdecision block808, a check is made atdecision block810 to determine if the full critical word (the word containing the critical data) is available in the cache line buffer. If the full critical word is available, as depicted by the positive branch fromdecision block810, the critical word data is forwarded to the requestor node, atblock812, and the process terminates atblock814—even if the outgoing beat is not completely filled. Unused data chunks may be marked as ‘invalid’ using associated validity-bits in the metadata. This eliminates the latency that would be introduced if the home node waited for an outgoing beat to be completely filled before forwarding it to the requestor node.

If the full critical word is not available in the cache line buffer in the home node, as depicted by the negative branch fromdecision block810, or if the system is not optimized for minimum latency, as depicted by the negative branch fromdecision block808, flow continues todecision block816. If the cache line buffer has sufficient data available to fully populate the interconnect bus width, as depicted by the positive branch fromdecision block816, the buffered data is forwarded to the requestor node atblock818. If the cache line buffer does not contain sufficient data to fill an outgoing beat, as depicted by the negative branch fromdecision block816, no action is taken until additional snoop data is received and the method ends attermination block814.

Thus, in accordance with various embodiments of the disclosure, a home node (HNF) is configured to respond to the data request dependent upon the following configuration parameters:

- 1) The data bus width of requestor node (REQ_BW)
- 2) The data bus width of snoop target(s) (SNP_BW)
- 3) The data bus width of interconnect system (ICN_BW), and
- 4) The relative importance of critical beat latency (CB_Lat) versus data bus utilization (DataBus_Util)

Based upon the above information, the home node determines if, and how, to align the address in the snoop request and whether to send partial critical data beat to requestor. The actions of the home node are summarized in Table 1 below.

	TABLE 1

	Configuration

CB_Lat more

Actions

Config.	REQ_BW <	SNP_BW <	important than	Align Snoop	HNF sends partial
Number	ICN_BW?	ICN_BW?	DataBus_Util?	Address?	critical beat first?

1	0	0	0 or 1	No	No
2	0	1	0 or 1	Yes	No
3	1	0	0	No	No
4	1	0	1	No	Yes
5	1	1	0	Yes	No
6	1	1	1	No	Yes

One benefit of the approach discussed above is improved data bus utilization. Table 2 summarizes the improvement in an example data processing apparatus having

- 1. Requestor bus width (REQ_BW)=256 bits.
- 2. Snoop target bus width (SNP_BW)=128 bits.
- 3. Interconnect bus width (ICN_BW)=256 bits.

TABLE 2

	Number of data
Critical beat	beats between
containing	Data Combiner	Number of data beats
requested	and HNF if	between Data
data	Snoop Address	Combiner and HNF if
(Addr[5:4])	is not aligned	Snoop Addr. is aligned	Note

00	2	2
01	3	2	Better combining
10	2	2
11	3	2	Better combining
Total	10	8

Assuming equal distribution of address offsets within cache lines, there is 20% savings on Data Bus utilization. That is, 20% less snoop response data traffic on the interconnect data bus between the Data Combiner and home node (HNF).

Those skilled in the art will recognize that the present invention may be implemented using a programmed processor, reconfigurable hardware components, dedicated hardware components or combinations thereof. Similarly, general purpose computers, microprocessor based computers, micro-controllers, optical computers, analog computers, dedicated processors and/or dedicated hard wired logic may be used to construct alternative equivalent embodiments of the present invention.

Dedicated or reconfigurable hardware components may be described by instructions of a Hardware Description Language. These instructions may be stored on non-transient computer readable medium such as Electrically Erasable Programmable Read Only Memory (EEPROM); non-volatile memory (NVM); mass storage such as a hard disc drive, floppy disc drive, optical disc drive; optical storage elements, magnetic storage elements, magneto-optical storage elements, flash memory, core memory and/or other equivalent storage technologies without departing from the present invention. Such alternative storage devices should be considered equivalents.

The various representative embodiments, which have been described in detail herein, have been presented by way of example and not by way of limitation. It will be understood by those skilled in the art that various changes may be made in the form and details of the described embodiments resulting in equivalent embodiments that remain within the scope of the appended claims.