WO2025101823A1 - A die-to-die interface coupled between a virtual phy and virtual mac - Google Patents

Abstract

A virtualised PHY (V-PHY) Interface for the PCI Express protocol is provided to enable a first PIPE-compliant component located on a first circuit die to communicate with a second PIPE-compliant component located on a second circuit die, where the first and second circuit dies are communicatively coupled to one another via a die-to-die (D2D) interface. The V-PHY interface can enable this communication transparently so that neither the first component nor second component needs to have any knowledge of the D2D interface or indeed that they are communicating across dies.

Description

A DIE-TO-DIE INTERFACE COUPLED BETWEEN A VIRTUAL PHY AND VIRTUAL MAC

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of US Provisional Patent Application 63/597,573, filed November 9, 2023, naming Peter Korger, Alexander Koch, and Victor Perrin, entitled “A Die-to-Die Interface Coupled Between a Virtual PHY and Virtual MAC” which is incorporated herein by reference in its entirety for all purposes.

REFERENCES

[0002] The following references are herein incorporated by reference in their entirety for all purposes:

[0003] Intel Corporation, ‘PHY Interface For the PCI Express, SATA, USB 3.1, DisplayPort, and Converged IO Architectures’, Version 5.1, 2018, hereinafter identified as [PIPE Specification], accessible at www[dot]intel[dot]com/content/www/us/en/io/pci-express/phy -interface-pci-express- sata-usb30-architectures-3 - 1. html .

[0004] U.S. Patent Application No. 13/030,027, filed February 17, 2011, naming Harm Cronie, Amin Shokrollahi and Armin Tajalli, entitled “Methods and Systems for Noise Resilient, Pin-Efficient and Low Power Communications with Sparse Signaling Codes”, hereinafter identified as [Cronie],

[0005] U.S. Patent Application No. 14/612,241, filed February 2, 2015, naming Ali Hormati, Amin Shokrollahi and Roger Ulrich, entitled “Method for Code Evaluation Using ISI Ratio”, hereinafter identified as [Hormati],

[0006] PCIe SIG, ‘PCI Express Base Specification Revision 5.0, Version 1.0’, May 28, 2019, hereinafter identified as [PCIe Specification],

[0007] U.S. Patent Application No. 17/931,448, filed September 12, 2022, naming Brian Holden, entitled “Pre-Scaler for Orthogonal Differential Vector Signalling”, hereinafter identified as [Holden], BACKGROUND

[0008] In integrated circuits there is often a need to send signals off-die, including, e.g. from a first die to a second die. This can allow a component located on the first die to communicate with a component located on the second die, or a component coupled to the second die.

[0009] In order to facilitate this die-to-die (D2D) communication, a D2D interface is provided. The D2D interface has a component on each die and one or more wires coupled between these components, to enable signals to be exchanged between the dies. [0010] The D2D interface has a D2D protocol associated with it to allow each of the D2D interface components to communicate with the other. The D2D protocol is not necessarily the same as a protocol used by components on the first die and second die. This is because the D2D protocol tends to be optimised in some manner for off-die communication, whereas components on the die tend to communicate using protocols optimised for interoperability.

[0011] It is desirable to provide a D2D communication technique that is capable of enabling off-die communication in a manner that presents little or no disruption to components communicating between dies.

BRIEF DESCRIPTION

[0012] A virtualised PHY (V-PHY) Interface for the PCI Express protocol is provided to enable a first PIPE-compliant component located on a first circuit die to communicate with a second PIPE-compliant component located on a second circuit die, where the first and second circuit dies are communicatively coupled to one another via a die-to- die (D2D) interface. The V-PHY interface can enable this communication transparently so that neither the first component nor second component needs to have any knowledge of the D2D interface or indeed that they are communicating across dies.

[0013] More specifically, a D2D communication technique is provided that includes a virtual physical layer (V-PHY) interface circuit located on a first circuit die and a virtual Media Access Control (V-MAC) interface circuit located on a second circuit die. The V-PHY interface circuit is coupled to a first D2D interface circuit on the first circuit die and to a Media Access Control (MAC) circuit also on the first circuit die. The V-MAC interface circuit is coupled to a second D2D interface circuit on the second circuit die and to a physical layer (PHY) circuit also located on the second circuit die. This arrangement enables the MAC circuit on the first circuit die to communicate with the PHY circuit on the second circuit die. The MAC circuit can be part of, or otherwise associated with, a first component and the PHY circuit can be part of, or otherwise associated with, a second component. Thus, communication between the first component on the first circuit die and the second component on the second circuit die is made possible.

[0014] In the case of communication in the transmit direction, i.e. from the MAC of the first circuit die to the PHY of the second circuit die, the V-PHY interface circuit is involved in converting data and control information from a first component on the first circuit die from a first protocol into a second, different protocol that is used by the D2D interface circuit. The V-MAC interface circuit is involved in the reverse conversion, i.e. conversion from the second protocol back to the first protocol, before sending on the data and control information to a second component on the second circuit die. The reverse is true in the case of communication in the opposite direction from the PHY to the MAC, termed the receive direction herein. This arrangement effectively renders the D2D interface circuits and multi-wire D2D communication link transparent from the perspective of the MAC and PHY of first and second components, respectively.

[0015] The first protocol may be the PHY Interface for Peripheral Component Interconnect Express (PIPE) protocol as described in [PIPE Specification], Herein, version 5.1 of the PIPE protocol is referenced by way of explanation only, and this is not limiting as other versions of PIPE protocol, including future versions that are not published at the time of filing of this specification, can also be used. The second protocol may be the Glasswing protocol provided by the Applicant and described in detail in [Cronie] and [Hormati].

[0016] In order to ensure compliance with timing critical aspects of the first protocol, the V-PHY interface circuit is configured to locally synchronize data and control information associated with the first protocol for interoperation with the MAC circuit. Similarly, the V-MAC interface circuit is configured to locally synchronize the data and the control information for interoperation with the PHY circuit.

[0017] In a first aspect an apparatus is provided, the apparatus comprising: a serialiser/deserialiser (SERDES) physical layer (PHY) circuit on a first circuit die, the PHY circuit configured to send and receive serial data; a die-to-die (D2D) interface circuit on the first circuit die, configured to send and receive time-division multiplexed data frames and control frames over a multi-wire D2D communication link, and further configured to exchange data and control information associated with the data frames and the control frames; and a virtual media access control (V-MAC) interface circuit on the first circuit die, the V-MAC interface circuit connected to the D2D interface circuit and connected to the PHY circuit, the V-MAC interface circuit configured to locally synchronize the data and the control information for interoperation with the PHY circuit.

[0018] In a second aspect an apparatus is provided, the apparatus comprising: a media access control (MAC) circuit on a first circuit die, the MAC circuit configured to send and receive data; a die-to-die (D2D) interface circuit on the first circuit die, configured to send and receive time-division multiplexed data frames and control frames over a multi-wire D2D communication link, and further configured to exchange data and control information associated with the data frames and the control frames; and a virtual serialiser/deserialiser (serdes) physical layer (V-PHY) interface circuit on the first circuit die, the V-PHY interface circuit connected to the D2D interface circuit and connected to the MAC circuit, the V-PHY interface circuit configured to locally synchronize the data and the control information for interoperation with the MAC circuit.

[0019] In a third aspect an apparatus is provided, the apparatus comprising: a first circuit die having: a first logic circuit having a plurality of media access control (MAC) circuits that are each configured to implement at least a portion of a PHY Interface for Peripheral Component Interconnect Express (PIPE) standard to interoperate with a physical layer (PHY) interface; a first die-to-die (D2D) interface circuit configured to communicate with a second circuit die; and, a plurality of virtual PIPE PHY interface circuits connected between respective ones of the plurality of MAC circuits and the first D2D interface, each of the plurality of virtual PIPE PHY interface circuits configured to emulate a PIPE PHY circuit by conveying PIPE signals between respective ones of the MAC circuits and the D2D interface circuit; a second circuit die having: a plurality of PHY circuits that are each configured to implement at least a portion of a PIPE standard; a second D2D interface circuit configured to communicate with the first circuit die; and a plurality of virtual PIPE MAC interface circuits connected between respective ones of the plurality of PHY circuits and the second D2D interface, each of the plurality of virtual PIPE MAC interface circuits configured to emulate a PIPE MAC circuit by conveying PIPE signals between respective ones of the PHY circuits and the D2D interface circuit and configured to locally generate one or more control signals for controlling respective ones of the plurality of PHY circuits.

[0020] In a fourth aspect a method is provided, the method comprising: sending and receiving serial data using a serialiser/deserialiser (serdes) physical layer (PHY) circuit on a first circuit die; sending and receiving, by a die-to-die (D2D) interface circuit on the first circuit die, time-division multiplexed data frames and control frames over a multi-wire D2D communication link that is connected to the D2D interface circuit, the sending and receiving including exchanging data and control information associated with the data frames and the control frames; and locally synchronizing the data and the control information for interoperation with the PHY circuit using a virtual media access control (V-MAC) interface circuit on the first circuit die, the V-MAC interface circuit connected to the D2D interface circuit and connected to the PHY circuit.

[0021] In a fifth aspect a method is provided, the method comprising: sending and receiving data by a media access control (MAC) circuit on a first circuit die; sending and receiving, by a die-to-die (D2D) interface circuit on the first circuit die, timedivision multiplexed data frames and control frames over a multi-wire D2D communication link that is connected to the D2D interface circuit, the sending and receiving including exchanging data and control information associated with the data frames and the control frames; and locally synchronizing the data and the control information for interoperation with the MAC circuit using a virtual serialiser/deserialiser (serdes) physical layer (V-PHY) interface circuit on the first circuit die, the V-PHY interface circuit connected to the D2D interface circuit and connected to the MAC circuit.

[0022] In a sixth aspect a method is provided, the method comprising: transmitting, by one of a plurality of media access control (MAC) circuits located on a first circuit die, a PHY Interface for Peripheral Component Interconnect Express (PIPE) compliant signal; receiving, by one of a plurality of virtual PIPE PHY interface circuits located on the first circuit die and connected between respective ones of the plurality of MAC circuits and a first die-to-die (D2D) interface circuit located on the first circuit die, the PIPE compliant signal; generating, by the one of a plurality of virtual PIPE PHY interface circuits, a die-to-die (D2D) signal based on the PIPE compliant signal; transmitting, by the one of the plurality of virtual PIPE PHY interface circuits, the D2D signal to the first D2D interface circuit; transmitting, by the first D2D interface circuit, the D2D signal to a second D2D interface circuit located on a second circuit die; transmitting, by the second D2D interface circuit, the D2D signal to one of a plurality of virtual PIPE MAC interface circuits located on the second circuit die and coupled between the second D2D interface circuit and respective ones of a plurality of PHY circuits located on the second circuit die; generating, by the one of a plurality of virtual PIPE MAC interface circuits, a second PIPE compliant signal based on the D2D signal; transmitting, by the one of the plurality of virtual PIPE MAC interface circuits, the second PIPE compliant signal to one of the plurality of PHY circuits; locally generating, by the one of the plurality of virtual PIPE MAC interface circuits, one or more control signals for controlling the one of the plurality of PHY circuits; and transmitting, by the one of the plurality of virtual PIPE MAC interface circuits, the one or more control signals to the one of the plurality of PHY circuits.

BRIEF DESCRIPTION OF FIGURES

[0023] FIG. 1 is a schematic diagram providing an overview of an apparatus that enables D2D communication, according to an embodiment.

[0024] FIG. 2 is a schematic diagram of a D2D communication circuit employing an Orthogonal Differential Vector Signaling (ODVS) protocol, that may form part of the D2D interface of FIG. 1.

[0025] FIG. 3 is a schematic diagram of the first circuit die of FIG. 1 showing components operational in a transmit direction, including some additional details of certain elements.

[0026] FIG. 4 is a schematic diagram of the second circuit die of FIG. 1 showing components operational in a transmit direction, including some additional details of certain elements.

[0027] FIG. 5 is a schematic diagram of the second circuit die of FIG. 1 showing components operational in a receive direction, including some additional details of certain elements.

[0028] FIG. 6 is a schematic diagram of the first circuit die of FIG. 1 showing components operational in a receive direction, including some additional details of certain elements.

[0029] FIG. 7 is a flowchart of a method of adjusting the rate of a locally generated clock signal to match a new rate of another clock signal generated on a different circuit die, according to an embodiment. [0030] FIG. 8 provides a graphical representation of a data frame suitable for use with the D2D interface of FIG. 1, according to an embodiment.

[0031] FIG. 9 provides a graphical representation of a control frame suitable for use with the D2D interface of FIG. 1, according to an embodiment.

[0032] FIG. 10 provides a graphical representation of a second control frame suitable for use with the D2D interface of FIG. 1, according to an embodiment.

[0033] FIG. 11 provides a graphical representation of a third control frame suitable for use with the D2D interface of FIG. 1, according to an embodiment.

[0034] FIG. 12 provides a graphical representation of a D2D control frame suitable for use with the D2D interface of FIG. 1, according to an embodiment.

[0035] FIG. 13 is a schematic diagram of an apparatus having multiple circuit dies, according to an embodiment.

[0036] FIG. 14 is a schematic diagram of an apparatus having four circuit dies coupled in a ring topology, according to an embodiment.

[0037] FIG. 15A is a schematic diagram showing routing of data frames and control frames in a downstream direction, in the four circuit die apparatus of FIG. 14.

[0038] FIG. 15B is a schematic diagram showing routing of data frames and control frames in an upstream direction, in the four circuit die apparatus of FIG. 14.

[0039] FIG. 16 is a flowchart showing a method of sending data and control information over a D2D interface from the perspective of a follower circuit die, according to an embodiment.

[0040] FIG. 17 is a flowchart showing a method of sending data and control information over a D2D interface from the perspective of a leader circuit die, according to an embodiment.

[0041] FIG. 18 is a flowchart showing a method of transmitting data and control information over a D2D interface, according to an embodiment.

DETAILED DESCRIPTION

[0042] FIG. 1 is a schematic diagram of an apparatus 100 according to an embodiment. Apparatus 100 can be used to exchange data and control information between circuit dies. Apparatus 100 includes a first circuit die 105 and a second circuit die 110. Located on first circuit die 105 is a first logic circuit 115, a plurality of MAC circuits 120a-120n, a plurality of virtual PHY (V-PHY) interface circuits 120a-120n and a first die-to-die (D2D) interface circuit 130. [0043] Located on second circuit die 110 is a second D2D interface circuit 135, a plurality of virtual MAC (V-MAC) interface circuits 135a-135n, and a plurality of PHY circuits 140a-140n. In this embodiment the PHY circuits 140a-140n are serializer/deserializer (SERDES)-capable PHY circuits that send and receive a serial data stream off-die, e.g. over a wire or other such channel. This is not essential as other embodiments in which PHY circuits 140a-140n do not incorporate SERDES functionality are possible. Each of the components on each circuit die is described in more detail below.

[0044] Although the various components are shown as distinct in FIG. 1, this is purely for clarity of illustration and does not imply limitation. Arrangements in which one or more of the elements are combined such that a single circuit component or logic block performs all of the functions of the combined elements are also possible. For example, first logic circuit 115 can incorporate MAC circuits 120a-120n and/or D2D interface circuit 130 can incorporate V-PHY interface circuits 125a-125n and/or D2D interface circuit 135 can incorporate V-MAC interface circuits 140a-140n. Other modifications on this theme are also possible.

[0045] The first logic circuit 115 can be any logic circuit having any function. One example is a switch core circuit providing switching functionality. The switch core circuit may comply with the PCIe protocol and PCIe Specification for switching PCIe packet data such as Transport Layer Packet (TLP) data. Another example is a radio circuit, e.g. a 5G radio circuit, providing connectivity to a radio network. Other types of logic circuit will be apparent to a person skilled in the art having the benefit of this disclosure.

[0046] The MAC circuits 120a-120n are each configured to provide an interface between the transport layer and the data link layer (DLL), using the Open Systems Interconnection (OSI) model to define layers. The physical layer, or PHY is implemented by the PHY circuits 145a-145n on the second circuit die 110 and emulated by the V-PHY interface circuits 125a-125n on the first circuit die 105. Similarly, the V-MAC interface circuits 140a-140n on the second circuit die 110 emulate the MAC circuits 120a-120n on the second circuit die 110.

[0047] Emulation is understood to mean that the V-MAC interface circuits and V-PHY interface circuits appear to other components as a MAC circuit and PHY circuit, respectively. This is equivalently referred to as virtualisation of the MAC / PHY function, leading to the ‘virtual’ descriptor used herein for the V-MAC interface circuits and V-PHY interface circuits. It is not necessary for a V-MAC interface circuit or V- PHY interface circuit to implement all of the functions of a MAC circuit or PHY circuit, respectively, as some functions may be adjusted, modified, added, or even removed entirely without adverse effects. Examples of this are set out in detail in this specification.

[0048] In the illustrated embodiment there are the same number of MAC circuits, V- PHY interface circuits, V-MAC interface circuits and PHY circuits. In such a configuration, each MAC circuit is coupled to a respective V-PHY interface circuit on the first circuit die and each V-MAC interface circuit is coupled to a respective PHY circuit on the second circuit die. Additionally, each V-MAC interface circuit is associated with a respective one of the MAC circuits and each V-PHY interface circuit is associated with a respective one of the PHY circuits.

[0049] This can be conceptualised as each V-MAC interface circuit emulating a specific MAC circuit and each V-PHY interface circuit emulating a specific PHY circuit, with the MAC circuit and PHY circuit being in communication with one another. However, this ‘one-to-one’ relationship is not strictly required and configurations in which the numbers of these interface circuits are not the same are also possible. Generally, there can be any number of MAC circuits, any number of V-PHY interface circuits, any number of V-MAC interface circuits and any number of PHY circuits. Note also that these circuits may be implemented as a single circuit providing separate logical functions, such as a single V-MAC having a plurality of MAC outputs to serve a respective plurality of PHY circuits.

[0050] D2D interface circuit 130 communicates with D2D interface circuit 135 using a multi-wire D2D communication link 150. D2D interface circuit 130 is configured to send data frames and control frames to D2D interface circuit 135. The data frames contain data, and the control frames contain control information. The data frames and control frames are time-division multiplexed, meaning that the data frames and control frames are sent over the same multi -wire D2D communication link 150. Stated alternately, it is not the case that there is a D2D communication link for data and a second, separate D2D communication link for control information.

[0051] In the illustrated embodiment D2D interface circuit 135 also communicates with D2D interface circuit 130 in the receive direction over the multi -wire D2D communication link 150. This is referred to as a bi-directional communication link. A bi-directional link is suitable for use in a two-die arrangement like that shown in FIG. 1. Embodiments in which more than two dies are present are also possible, and in such embodiments each D2D interface circuit can receive data frames and control frames from one die and transmit data frames and control frames to a second, different die. In such configurations the multi -wire D2D communication link 150 comprises a receive link with a first circuit die and a transmit link with a second circuit die. The first and second circuit dies can be part of a set of circuit dies coupled together in a ring topology, for example. In a further embodiment, one or more intermediate dies may have full duplex bidirectional links to adjacent dies.

[0052] D2D interface circuit 130 and D2D interface circuit 135 communicate using a D2D protocol, for example a vector signalling protocol as used in an Orthogonal Differential Vector Signaling (ODVS) D2D link (see FIG. 2 for details). The D2D protocol is adapted for die-to-die signalling and is different from the PHY/MAC protocol used by MAC circuits 120a-120n to communicate with PHY circuits 145a- 145n, for example the PHY Interface for PCIe (PIPE) protocol. V-PHY interface circuits 125a-125n and D2D interface circuit 130 collectively function to convert between the D2D protocol and PHY/MAC protocol in one direction and to convert between the PHY/MAC protocol and the D2D protocol in the opposite direction. Similarly, V-MAC interface circuits 140a-140n and D2D interface circuit 135 collectively function to convert between the D2D protocol and PHY/MAC protocol in one direction and to convert between the PHY/MAC protocol and the D2D protocol in the opposite direction. In the illustrated embodiment, the PHY/MAC protocol is PIPE (see [PIPE Specification]) and the D2D protocol is ODVS, but this is not limiting as other protocols can be used instead.

[0053] It will be appreciated that the diagram of FIG. 1 does not provide an exhaustive list of components, as in a practical implementation other components that are not shown may be present on either circuit die, or both.

[0054] FIG. 2 illustrates an embodiment of a communication circuit employing an ODVS D2D protocol suitable for use in the configuration of FIG. 1. Curly braces 130, 135 and 150 indicate the correspondence between subset of the components shown in FIG. 2 and D2D interface circuit 130, D2D interface circuit 135 and multi -wire D2D communication link 150 of FIG. 1. It will be appreciated that FIG. 2 does not provide an exhaustive set of components for D2D interface circuit 130, D2D interface circuit 135 and multi -wire D2D communication link 150 such that each of D2D interface circuit 130, D2D interface circuit 135 and multi-wire D2D communication link 150 can additionally include components not shown in FIG. 2.

[0055] The D2D protocol of FIG. 2 is an ODVS protocol; that is, it uses a vector signaling code. A D2D transmitter 210 receives source data, herein illustrated as So, Si, S2, S3, S4. The source data can be generated by D2D interface circuit 130 based on data and control information received from V-PHY interface circuits 125a-125n. The source data takes the form of a data frame in the case of data or a control frame in the case of control information. As data frames and control frames are each typically larger than five bits (e.g. 150 bits), the data word size of this particular implementation of the communication circuit of FIG. 2, a part of a data frame or control frame enters as a source data word 200 into encoder 212. The size of the source data word may vary and depends on the parameters of the vector signaling code.

[0056] The encoder 212 generates a codeword of the vector signaling code based on the portion of the data frame or control frame that the encoder is currently processing. In operation, the codeword produced by encoder 212 is used to control PMOS and NMOS transistors within driver 218, generating two, three, four, or more distinct voltages or currents on each of the N communication wires 225 of communications channel 220, to represent the N symbols of the codeword. The N communication wires 225 serve as at least part of the multi -wire D2D communication link 150 of FIG. 1. As noted above, the source data forms part of a data frame or a control frame generated by D2D interface circuit 130 for transmission to D2D interface circuit 135.

[0057] In the embodiment of FIG. 2, the size of the source data word is shown as five bits and the codeword size is six symbols. Thus, communications channel 220 is shown as being comprised of six signal wires 225, each transporting one codeword symbol. One familiar with the encoding arts may also describe this code as having a block length of six (i.e. producing an output word of six symbols) and a code size of 32 (i.e. having 32 distinct codewords, sufficient to encode 5 binary bits of data.). This is purely an illustrative example and should not be construed as limiting, as other sizes of source data words and codeword sizes can be used instead. One such example is a Hadamard- based encoding / decoding matrix which uses e.g. source data words of three bits and a communication channel of four wires. In this case, the encoding / decoding matrix is based on a Hadamard matrix.

[0058] While illustrated as one set of links in FIG. 2, this is not limiting as communication channel 220 can comprise multiple parallel sets of links. Each set of links can comprise any number of signal wires. For example, in an embodiment communication channel 220 comprises four sets of links, each link comprising six signal wires for a total of 24 signal wires. These values are purely illustrative and are not limiting as other link sizes can alternatively be used.

[0059] The ODVS protocol used by communication channel 220 can be a so-called ‘ensemble NRZ’ or ENRZ protocol that makes use of N wires to transmit n-1 bits. In an example, communication channel 220 can be comprised of four signal wires and transmit three bits per unit interval. Each bit is transported by a so-called ‘subchannel’ that is a superposition of signals transmitted over multiple signal wires. In the case of ENRZ signals relating to a given bit are transmitted over all of the wires, i.e. each subchannel includes a signal component from every wire. In the four signal wires transporting three bits example, this means there are three subchannels (one for each bit) with each subchannel being formed from a superposition of signals from all of the four wires. This can be termed ‘three bits over four wires’. In another case, the same principles can be used to transport seven bits per unit interval over eight wires. ENRZ encoding and decoding is based on the Hadamard matrix. Examples of ENRZ codes are set out in [Holden], Alternatively, ODVS codes can be used in which at least one of the bits is carried by a proper subset of the wires, i.e. every wire does not carry signals relating to every bit. Examples of these ‘Chord Non-Return to Zero’ (CNRZ) codes are provided in [Holden], Unlike ENRZ codes, CNRZ encoding matrices are not based on the Hadamard matrix. One example is a code called ‘CNRZ-7’ in which seven bits are transmitted over eight wires. In this case, the subchannels are grouped such that there are four pair subchannels that are carried by a respective two of the eight wires, two tetrad subchannels that are carried by a respective four of the eight wires, and one octad subchannel that is carried by all eight wires.

[0060] Within D2D receiver 230, detector 232 reads the voltages or currents on wires 225, possibly including amplification, frequency compensation, and common mode signal cancellation. In the present example, the received results 240, herein shown as Ro, Ri, R2, R3, R4, are provided directly by detector 232, without need of optional decoder 238. The received results 240 form part of a data frame or a control frame received by D2D interface circuit 135 from D2D interface circuit 130.

[0061] As will be readily apparent, different codes may be associated with different block sizes and different codeword sizes; for descriptive convenience and without implying limitation, the example of FIG. 2 illustrates a system using an ODVS code capable of encoding five binary bit values for transmission over six wires, a so-called 5b6w code.

[0062] Depending on which vector signaling code is used, there may be no decoder, or no encoder, or neither a decoder nor an encoder. For example, for the 8b8w code disclosed in [Cronie], both encoder 212 and decoder 238 exist. On the other hand, for the 5b6w code of the present embodiment, an explicit decoder is unnecessary, as the system may be configured such that detector 232 generates the received results 240 directly.

[0063] It will be appreciated that the communications circuit shown in FIG. 2 enables communication in one direction, from D2D interface circuit 130 to D2D interface circuit 135. This is also referred to herein as the transmit direction, or downstream direction. To enable two-way communication, a second instance of the communications circuit of FIG. 2 can be provided. In this second instance, the transmit and receive components swap locations, so that D2D interface circuit 130 includes a D2D receiver 230 and D2D interface circuit 135 includes a D2D transmitter 210. This provides bi-directional communication between first circuit die 105 and second circuit die 110. Each instance of the communications circuit if FIG. 2 would have its own set of wires between circuits, i.e. in the illustrated embodiment six wires per instance of the communications circuit of FIG. 2 are present (excluding any wires carrying clocking information).

[0064] In arrangements having more than two circuit dies, each circuit die can be connected to at least two other circuit dies. In such cases each circuit die has a D2D transmitter 210 connected to one circuit die and a D2D receiver 230 connected to another circuit die. This allows each circuit die to send data frames and control frames to one other circuit die and to receive data frames and control frames from a different circuit die. This type of configuration creates a set of circuit dies coupled in a unidirectional ring topology. Other topologies, e.g. a bidirectional ring topology or point to point topology can alternatively be established by adjusting the configuration of D2D transmitter 210 and D2D receiver 230 such that each performs both encoding and decoding, i.e. they both become D2D transceivers. More information on such configurations is provided later in this specification.

[0065] FIG. 3 is a schematic diagram of first circuit die 105 including some additional details of V-PHY interface circuit 125a and D2D interface circuit 130. First circuit die 105 is shown in the transmit direction in FIG. 3, i.e. data and control information is being sent from MAC circuit 120a to PHY circuit 145a.

[0066] While only one V-PHY interface circuit 125a is shown in FIG. 3, it will be appreciated that the other V-PHY interface circuits 125b-125n are of identical construction. The components of V-PHY interface circuit 125a are shown in FIG. 3 beneath the curly brace labelled ‘ 125a’, this being the conceptual boundary of the V- PHY function. Similarly, the components of D2D interface circuit 130 are shown beneath the curly brace labelled ‘ 130’. As noted earlier, these distinctions are arbitrary and made for the sake of clarity of explanation, such that any part of the V-PHY interface circuit and D2D interface circuit 130 can be incorporated into the other entity without departing from the scope of this disclosure.

[0067] In FIG. 3 clock signals are shown using dashed arrows to distinguish from data, control information, data frames and control frames that are all shown using solid arrows. This convention is used throughout the drawings of this specification.

[0068] As an overview, first circuit die 105 comprises MAC circuit 120a that is configured to send and receive data. Also located on first circuit die 105 is D2D interface circuit 130 which is configured to send and receive time-division multiplexed data frames and control frames over multi -wire D2D communication link 150, and further configured to exchange data and control information associated with the data frames and the control frames. V-PHY interface circuit 125a is also located on first circuit die 105. V-PHY interface circuit 125a is connected to the D2D interface circuit 130 and connected to the MAC circuit 120a. V-PHY interface circuit 125a is configured to locally synchronize the data and the control information for interoperation with the MAC circuit 120a. Additional information on the local synchronization is provided below.

[0069] In FIG. 3 only one MAC circuit 120a is shown, for clarity of explanation. The other MAC circuits 120b-120n are of identical construction. In FIG. 3 MAC circuit 120a is shown as part of first logic circuit 115, but this is not limiting as MAC circuit 120a can alternatively be separate as shown in FIG. 1. First logic circuit 115 can be a switch core, for example.

[0070] In FIG. 3 MAC circuit 120a uses the PIPE protocol to communicate with PHY circuit 145a (see FIGS. 1 and 4) and D2D interface circuit 130 uses an ODVS protocol to communicate with D2D interface circuit 135 (see FIGS. 1 and 4). V-PHY interface circuit 125a and D2D interface circuit 130 thus collectively function as a PIPE to ODVS converter. It will be appreciated that MAC circuit 120a does not need to have any knowledge of the ODVS protocol, nor does it need to know that PHY circuit 145a is located on a different circuit die. From the perspective of MAC circuit 120a, it is communicating directly with PHY circuit 145a using an agreed-upon version of the PIPE protocol.

[0071] V-PHY interface circuit 125a includes a control buffer 300. V-PHY interface circuit 125a writes control information into control buffer 300. The control information is read out from control buffer 300 by D2D interface circuit 130, specifically protocol converter 305. The function of protocol converter 305 is described later.

[0072] In the case of the PIPE protocol, the control information that is written into control buffer 300 is PIPE command words generated by MAC circuit 120a on the PIPE Message Bus interface. See section 6.1.4 of [PIPE Specification] for further details regarding the Message Bus interface.

[0073] PIPE messages are 8 bits in length but the PIPE command words they relate to can be 8, 16 or 24 bits. For this reason, a single PIPE command word can require up to three PIPE messages to convey it fully. Each PIPE message can be written into and read out of control buffer 300 separately, necessitating an 8-bit wide control buffer 300. [0074] Alternatively, as illustrated in FIG. 3, 8-bit to 24-bit converter logic 310 can be provided between MAC circuit 120a and control buffer 300. Converter logic 310 functions to collect together up to three PIPE messages relating to the same PIPE command word and to write the resulting 24 bits into control buffer 300, which is consequently 24 bits wide in this case. In cases where the PIPE command word is 8 or 16 bits, 16 or 8 bits of padding, respectively, are added by converter logic 310 to ensure that 24 bits of data are available for writing into control buffer 300. Converter logic 310 can use the first bit of the first PIPE message to determine the length of the PIPE command word that the first PIPE message relates to, and hence determine whether padding will be needed and, if yes, how many bits of padding are required.

[0075] Converter logic 310 can result in more efficient usage of the available bandwidth of multi -wire D2D communication link 150 because a complete PIPE command word can be sent in a single control frame over link 150, rather than the two or three control frames that would otherwise be necessary for 16 and 24-bit PIPE command words, respectively.

[0076] It will be appreciated that for other protocols, or for versions other than version 5.2 of the PIPE protocol, the configuration of the control buffer 300 can be adapted as necessary to accommodate the specifics of those other protocols. Control buffer 300 can also be configured to support multiple different protocols and/or multiple different versions of the same protocol.

[0077] In the illustrated embodiment control buffer 300 is configured as a clock domain crossing (CDC) ‘first in first out’ (FIFO) buffer. The CDC itself is represented via a dashed line in FIG. 3 which conceptually separates the two distinct clock domains that control buffer 300 operates within. This nomenclature is used throughout this specification.

[0078] The first clock domain, shown on the left-hand side of control buffer 300 in FIG. 3, is the PHY Clock (PCLK) domain. As shown, a PCLK clock signal is generated by MAC circuit 120a and is used to write control information into control buffer 300. Further information about PCLK can be found on page 79 of [PIPE Specification], [0079] The second clock domain, shown on the right-hand side of control buffer 300 in FIG. 3, is the D2D clock domain. As shown, a D2D clock signal (‘D2D CLK’) is generated by D2D interface circuit 130, and specifically in this case D2D transmitter 210. The scope of this disclosure is however not limited to D2D transmitter 210 generating D2D CLK as other components that are part of D2D interface circuit 130 can generate D2D CLK, or as a further alternative D2D CLK can be generated by a separate circuit that is not part of D2D interface circuit 130. Control information is read out of control buffer 300 according to D2D CLK such that control buffer 300 facilitates a clock domain crossing from the PCLK domain to the D2D CLK domain.

[0080] The frequency of PCLK, also referred to interchangeably herein as the rate of PCLK, can be variable and can be specified by control information. For example, MAC circuit 120a can provide a PCLK rate signal that specifies the rate of PCLK. The PCLK rate signal can be a multi -bit signal. An exemplary set of possible rates for PCLK are: 62.5 MHz, 125 MHz, 250 MHz, 500 MHz, 1000 MHz, 2000 MHz and 4000 MHz. These possible values are as set out on page 53 of [PIPE Specification] under the heading ‘PCI Express Mode’ . These values are purely illustrative - rates for PCLK other than those listed above are also possible.

[0081] The frequency (equivalently, rate) of D2D CLK may be the same as, or different to, the current PCLK rate. Exemplary rates for D2D CLK are 500 MHz, 781.25 MHz and 1000 MHz. These values are purely illustrative - rates for D2D CLK other than those listed above are also possible. The rate of D2D CLK can be fixed or it can vary. Variations to the rate of D2D CLK may occur based on with variations to the rate of PCLK.

[0082] The rate of D2D CLK is preferably set so that the rate that data is read out of control buffer 300 is greater than or equal to the rate at which data is written into control buffer 300, to avoid the possibility of overflow of control buffer 300. An appropriate value for the rate of D2D CLK will depend at least on the current rate of PCLK as well as the width of control buffer 300. The D2D CLK rate can be set based on a ‘worst case’ scenario, i.e. the scenario in which PCLK rate is equal to a maximum value (MAX PCLK).

[0083] Given the set of PCLK rates above, a ‘worst case’ scenario is PCLK rate = 4000 MHz, as this is the maximum PCLK rate at which control buffer 300 is written. In the illustrated embodiment, 24 bits of control information is written into control buffer 300 from converter logic 310 every fourth PCLK cycle, as it takes three PCLK cycles for converter logic 310 to construct the 24 bits of control information based on the message(s) received from MAC interface circuit 120a. This gives a maximum data write rate of 4000 x 0.25 x 24 = 24Gbps. In the case of the embodiment of FIG. 3, 24 bits of data is read out of control buffer 300 every D2D CLK cycle, meaning that to match the 24Gbps maximum write rate the rate of D2D CLK should be at least 1000 MHz. This calculation is purely illustrative and these values should not be construed as limiting.

[0084] In the embodiment of FIG. 3, V-PHY interface circuit 125a also includes another path for control information to be conveyed from MAC circuit 120a to D2D interface circuit 130. This other path is control wires 315. Control wires 315 may carry at least some of the control signals that are used by a PIPE interface other than the PIPE commands sent via MAC circuit 120a via the PIPE message bus. Specifically, control wires 315 carry at least some PIPE control signals that are sent via MAC circuit 120a via the PIPE Command Interface and also the PIPE Status Interface in the MAC to PHY direction. Details of these signals can be found in sections 6.1.2 and 6.1.3 of [PIPE Specification], The number of control wires 315 will vary according to the specific PIPE version and protocols supported, and has therefore been shown as ‘M’ wires in FIG. 3 to take account of this variation, M being a positive integer.

[0085] The specific set of signals carried by control wires 315 can vary according to PIPE operating mode, e.g. Original PIPE mode compared with SERDES PIPE mode. It should be appreciated that control wires 315 do not necessarily carry a complete set of control signals as defined in a PIPE specification or other such protocol document as some control information is generated locally and hence it is not necessary to send these signals or symbols over the multi -wire D2D communication link 150.

[0086] As shown in FIG. 3, control wires 315 are arranged to transmit PIPE control signals from MAC circuit 120a to D2D interface circuit 130, and specifically protocol converter 305. The function of protocol converter 305 is described later.

[0087] V-PHY interface circuit 125a also includes a data buffer 320. Data buffer 320 is coupled to MAC circuit 120a and receives data from MAC circuit 120a. Here, ‘data’ refers to any signals that are not control information as discussed above. In the case of MAC circuit 120a implementing a PIPE protocol, the data is PIPE data sent over the PIPE Data Interface as described in section 6.1.1 of [PIPE Specification],

[0088] MAC circuit 120a is coupled to data buffer 320 via N wires, N being a positive integer. The value of N will depend on the specifics of the protocol implemented by MAC circuit 120a. In the case of Original PIPE, N = 8, 16 or 32. In the case of SERDES PIPE, N = 10, 20 or 40. These values are purely exemplary and other values of N can be implemented as necessitated by the protocol used by MAC circuit 120a. It will be appreciated that in cases where a set of protocols are supported by MAC circuit 120a, N can be selected to provide a sufficient number of wires for whichever protocol of the set of protocols supported by MAC circuit 120a requires the greatest number of wires.

[0089] In the illustrated embodiment data buffer 320 is configured as a CDC FIFO buffer. The CDC itself is shown via a dashed line in FIG. 3 which conceptually separates the two distinct clock domains that data buffer 320 operates within.

[0090] The first clock domain, shown on the left-hand side of data buffer 320 in FIG. 3, is the PHY Clock (PCLK) domain. As shown, the PCLK clock signal discussed above that is generated by MAC circuit 120a is used to write in data into data buffer 320.

[0091] The second clock domain, shown on the right-hand side of data buffer 320 in FIG. 3, is the D2D clock domain discussed above. Data is read out of data buffer 320 according to the D2D clock. Reference is made to the discussion above relating to the clocking of control buffer 300, as data buffer 320 is clocked in the same manner.

[0092] In the embodiment of FIG. 3, data is read out of data buffer 320 via 2N wires. This is not essential, but is useful in preventing buffer overflow as it doubles the data read out rate relative to the data write rate. As with control buffer 300, it is desirable to select the number of wires on the ‘read out’ side of data buffer 320 and the rate of D2D CLK such that the data read out rate is greater than or equal to the data write in rate to prevent overflow of data buffer 320. Nevertheless, it is not essential that data is read out of data buffer via 2N wires, and other numbers of wires can be used instead.

[0093] As touched upon earlier, D2D interface circuit 130 includes protocol converter 305 that functions to convert between a first protocol used by MAC circuit 120a and a second protocol used by the D2D interface circuit. The first protocol in the case of FIG. 3 is PIPE and the second protocol is ODVS. This is however not limiting as the principles taught herein can be applied to conversion between other protocols.

[0094] Details of the first protocol are not provided herein as such details will be available from whichever source is authoritative in respect of the first protocol, e.g. [PIPE Specification] in the case of PIPE v. 5.1.

[0095] Details of the second protocol are provided herein in the context of the first protocol being PIPE, in order to assist in the reader’s understanding. However, it is understood that the first protocol does not need to be PIPE as the principles established herein can apply to other first protocols that have separate transmission of data and control information, e.g. Universal Chiplet Interconnect Express (UCIe).

[0096] D2D interface circuit 130 operates as follows. Protocol converter 305 receives control information from control buffer 300 and also from control wires 315. Protocol converter 305 also receives data from data buffer 320. The control information is packaged into control frames by protocol converter 305 and transmitted by D2D transmitter 210 over multi-wire D2D communication link 150 using an ODVS code to another circuit die. Similarly, the data is packaged into data frames by protocol converter 305 and transmitted by D2D transmitter 210 over multi -wire D2D communication link 150 using the ODVS code to the other circuit die. It will be apparent that the designation of ‘control’ or ‘data’ is in accordance with the first protocol, PIPE in this embodiment.

[0097] The control frames and data frames are time-division multiplexed. That is, the control frames and data frames are interleaved and transmitted over the same multiwire D2D communication link 150. This is in contrast to the PIPE protocol and other similar protocols where data and control information are transmitted over separate sets of wires that are distinct from one another.

[0098] In the embodiment of FIG. 3 the multi -wire D2D communication link 150 has six wires per D2D transmitter and uses a ‘5 bits over 6 wires per clock cycle’ (‘5b6w’) ODVS code termed a ‘Glasswing’ code. Such a code is described in more detail in [Hormati],

[0099] D2D transmitter 210 generates a forwarded clock signal FWD CLK that is also transmitted over multi-wire D2D communication link 150, hence the name ‘forwarded’ . FWD CLK can be sent over dedicated clock wires that are separate from the wires that carry data frames and control frames, e.g. a differential pair of wires can carry FWD CLK. This is not limiting as other techniques for transmitting FWD CLK can be used instead. A corresponding D2D receiver (discussed later) on another die is able to use FWD CLK to recover the data frames and control frames sent over multi-wire D2D communication link 150. FWD CLK has a rate that is typically significantly higher than the rate of D2D CLK, e.g. 6.25 GHz. This is because protocol converter 305 sends 150 bits (one frame) to D2D transmitter 210 per D2D CLK cycle, whereas D2D transmitter 210 transmits only 5 bits per FWD CLK cycle. These values are exemplary and should not be construed as limiting, as other rates for FWD CLK and/or other sizes for each frame are possible. Note that in this embodiment the D2D CLK is a quarter rate clock, meaning that the D2D transmitter and D2D receiver actually operate at 25GHz.

[00100] Although just one D2D transmitter 210 is shown in FIG. 3, this is not a limitation as multiple D2D transmitters 210 can be present, each coupled to a distinct set of wires that collectively form the multi -wire D2D communication link 150. As an example of this, four D2D transmitters that are each like D2D transmitter 210 can be present, each transmitter coupled to a respective set of six wires (see FIG. 14). This arrangement thus has 24 wires total (excluding wires carrying FWD CLK) in the multiwire D2D communication link 150, grouped into four sets of 6 wires coupled to a respective one of the four transmitters, each set of 6 wires carrying 5 bits per clock cycle using the 5b6w ODVS code discussed earlier. In the case where FWD CLK has a rate of 6.25GHz, as it is a quarter rate clock the D2D transmitter operates at 25GHz. This provides a total capacity of 25GHz x 5 bits x 4 transmitters = 500Gpbs for multiwire D2D communication link 150. Other configurations having more or fewer transmitters and wires, and/or using a different ODVS code, are also possible. As mentioned above, one or more additional wires can be present for carrying FWD CLK - this/these wire(s) are referred to as ‘dedicated clock’ wire(s) herein.

[00101] It will be appreciated that if a protocol other than PIPE is being virtualised, the configuration of V-PHY interface circuit 125a can be adjusted as necessary to take account of the specifics of this other protocol. For example, the size of control buffer 300 and/or data buffer 320, and/or the number of wires coupled to each buffer and/or clocking scheme can be adjusted as needed.

[00102] Referring now to FIG. 4, a detailed view of second circuit die 110 is provided. This shows second circuit die 110 in the transmit direction, i.e. data and control information is being sent from MAC circuit 120a to PHY circuit 145a. As in FIG. 3, curly braces 135, 140a and 145a have been used to show those components that are part of D2D interface 135, V-MAC interface circuit 140a and PHY circuit 145a, respectively. These boundaries are not to be construed as limiting, as it is possible to incorporate the function of one part of one of these elements into another of these elements.

[00103] Second circuit die 110 receives ODVS signals via multi-wire D2D communication link 150, e.g. as transmitted by D2D transmitter 210 from first circuit die 105. The ODVS signals are received by D2D receiver 230 that is part of D2D interface circuit 135. D2D receiver 230 operates as described above in connection with FIG. 2 to translate ODVS signals received via multi-wire D2D communication link 150 into data frames and control frames that are each 150 bits in size in this embodiment. This is essentially the inverse process as described in relation to D2D transmitter 210. D2D receiver 230 generates a 25GHz clock based on FWD CLK that is also received via the multi -wire D2D communication link 150, e.g. via a dedicated clock wire or a differential pair of dedicated clock wires. Multiple D2D receivers like D2D receiver 230 can be present, e.g. four D2D receivers, each with their own set of wires.

[00104] Second circuit die 110 has located on it serialiser/deserialiser (SERDES) physical layer (PHY) circuit 145a. PHY circuit 145a is configured to send and receive serial data. The serial data can be sent off-die.

[00105] Second circuit die 110 also has located on it D2D interface circuit 135 that is configured to send and receive time-division multiplexed data frames and control frames over multi-wire D2D communication link 150. Time-division multiplexing is discussed earlier in this specification and has the same meaning here as set out above.

[00106] D2D interface circuit 135 is also configured to exchange data and control information associated with the data frames and the control frames, respectively. The data can be extracted by D2D interface circuit 135 from data frames and the control information can be extracted by D2D interface circuit 135 from the control frames. The data and control information are exchanged with other components located on second circuit die 110, as discussed in more detail below.

[00107] Second circuit die 110 also has V-MAC interface circuit 140a located on it. V- MAC interface circuit 140a is connected to D2D interface circuit 135 and connected to PHY circuit 145a. V-MAC interface circuit 140a is configured to locally synchronize the data and the control information for interoperation with PHY circuit 145a. Local synchronization refers to the temporal alignment of data and control information; that is, alignment in time such that PHY circuit 145a receives data and control information in an order that it expects to receive data and control information as defined by the protocol that PHY circuit 145a is using (PIPE in this embodiment). Some specific examples of local synchronization are provided later in this specification. The term ‘local’ is used because the synchronization occurs locally to PHY circuit 145a, e.g. on the circuit die 110 that PHY circuit 145a is located on. This enables interoperation between the V-MAC interface circuit 140a and PHY circuit 145a because V-MAC interface circuit 140a is able to provide data and control information to PHY circuit 145a in an order that PHY circuit 145a expects to receive it, such that the requirements of the protocol that PHY circuit 145a uses (PIPE in this embodiment) are respected. This allows V-MAC interface circuit 140a, as well as D2D interface circuits 130, 135, multi-wire D2D communication link 150 and V-PHY interface circuit 125a to be entirely transparent to PHY circuit 145a, such that PHY circuit 145a operates as though it were directly coupled to MAC circuit 120a.

[00108] V-MAC interface circuit 140a comprises a receive data buffer 405. This functions similarly to data buffer 320 but operating essentially in an inverse manner. Specifically, data is written into receive data buffer 405 by protocol converter 400 at a rate based on the rate of D2D CLK as generated by D2D interface circuit 135, specifically D2D receiver 230 in this embodiment, which in this case is 25GHz. The data written into receive data buffer 405 is extracted from data frames received by D2D receiver 230 via multi-wire D2D communication link 150.

[00109] Data is read out of receive data buffer 405 by PHY circuit 145a at a rate set by a locally generated version of PCLK, referred to herein as L PCLK. L PCLK is generated by V-MAC interface circuit 140a and provided to PHY circuit 145a. This emulates a SERDES PIPE operation mode under which a MAC circuit is required to provide PCLK to a PHY circuit. [00110] Note however that L PCLK is not the same PCLK signal as generated by MAC circuit 120a. This is a departure from operation according to the PIPE specification, specifically SERDES PIPE mode, in which PCLK is generated by a MAC circuit and provided to a PHY circuit.

[00111] L PCLK can be generated in the following manner, which is provided as one way of generating L PCLK and thus is not limiting. V-MAC interface circuit 140a includes an Integrated Clock Gating (ICG) cell 410 that is coupled to a system clock 450. System clock 450 provides a system clock signal SysCLK as an output. System clock 450 can be located on the same circuit die as V-MAC interface circuit 140a, i.e. second circuit die 110 in this embodiment. System clock 450 can be coupled to a reference clock (not shown in FIG. 4) that is common to all circuit dies in a given apparatus or system, meaning that each system clock generates a version of SysCLK with the same frequency but a different phase, i.e. the various SysCLK signals are mesochronous.

[00112] As shown in FIG. 4, the SysCLK output of system clock 450 is coupled to an input of ICG cell 410. ICG cell 410 operates to gate a clock signal provided as an input, meaning that the output of ICG cell 410 is a gated version of SysCLK. This output is L PCLK, i.e. L PCLK is a gated version of SysCLK.

[00113] V-MAC interface circuit 140a is configured to set a rate of the L PCLK signal based on PCLK rate control information that is part of the control information. This control information can be received via one or more control frames that have been sent over multi -wire D2D communication link 150.

[00114] The specific nature of the control information used to set the rate of the L PCLK signal will depend on the first protocol that is used by MAC circuit 120a and PHY circuit 145a. In the case where the first protocol is a PIPE protocol, the control information is PCLK rate control information ‘PCLK Rate[4:0]’ as discussed on page 53 of [PIPE Specification], This PCLK rate control information is provided by V-MAC interface circuit 140a to a second input of ICG cell 410 and is used by ICG cell 410 to control the gating of SysCLK and thus to control the rate of L PCLK.

[00115] System clock 450 can be configured to set a frequency of the system clock signal equal to a maximum allowed value of a maximum PCLK frequency (‘Max PCLK’) value that is part of the control information. Max PCLK is described on page 79 of [PIPE Specification], The rationale here is that by selecting the maximum allowed value that Max PCLK can validly take according to the first protocol (PIPE in this embodiment), it is always possible to obtain any of the necessary lower frequencies that may be required during operation. In the case of PIPE v. 5.2 implementing PCIe, the maximum allowed value of Max PCLK is 4 GHz when at a 32 gigatransfers per second (PCIe generation 5.0) mode of operation. Thus, SysCLK can be set to 4 GHz in this case, providing support for all possible PCLK rates that can be validly requested according to the relevant version of the PIPE protocol. These values are not limiting as it will be appreciated that in practice an appropriate frequency for SysCLK will depend on implementation details.

[00116] The maximum allowed value of a maximum PCLK frequency can be part of the control information. See Max PCLK described on page 79 of [PIPE Specification], for example. Changes in the value of Max PCLK or equivalent can be detected by V- MAC interface circuit 140a and/or V-PHY interface circuit 125a and communicated to system clock 450 (or an equivalent located on first circuit die 105, in the case of V- PHY interface circuit 125a performing the detecting) to instruct the system clock to change the frequency of SysCLK based on the new value of Max PCLK.

[00117] The system clock can alternatively be configured to set a frequency of the system clock signal greater than a maximum allowed value of a maximum PCLK frequency value that is part of the control information. This allows all of the frequencies that may be required for L PCLK during operation to be generated by an ICG cell. Additionally, the system clock can be configured to output multiple system clock signals each having a common frequency and a different phase. V-MAC interface circuit 140a (or V-PHY interface circuit 125a in the case of Original PIPE operation) is configured to select one of the multiple system clock signals to provide as an input to the ICG cell. As each of the multiple system clock signals has a different phase, selecting a specific one of the multiple system clock signals allows a relative delay between different lanes of a link to be introduced. Each lane has a respective V-MAC interface circuit and V-PHY interface circuit, the relevant one of which can select a particular system clock phase so as to enable a deliberately introduced delay between lanes. This delay can be introduced to reduce, or even entirely eliminate, a skew between the lanes, skew referring to a difference in arrival time of signals on one lane relative to another lane. This is because in a situation where signals arrive on a first lane earlier than on a second lane, a V-PHY interface circuit or V-MAC interface circuit of the first lane can select a system clock signal with a greater phase than the second lane to introduce a delay in the first lane that compensates at least partially for the early arriving of signals on the first lane.

[00118] It is noted that in the embodiment of FIG. 4 V-MAC interface circuit 140a is emulating a SERDES PIPE architecture in which a MAC circuit is required to provide a PCLK signal to a PHY circuit. In an Original PIPE architecture the PCLK signal is instead generated by the PHY circuit and provided to the MAC circuit. In this case the embodiment of FIG. 4 can be modified such that ICG cell 410 is turned off or removed entirely if emulation of a SERDES PIPE architecture is not needed. The system clock and ICG cell configuration shown in FIG. 4 can be provided on first circuit die 105 to operate in the same manner as described above, with the difference that the output of the ICG cell located on the first circuit die 105 is provided to MAC circuit 120a. As the PCLK Rate[4:0] control information is generated by MAC circuit 120a in both Original PIPE and SERDES PIPE modes, the control information to set the rate of L PCLK in the case of Original PIPE is obtained directly as an output of MAC circuit 120a rather than extracting it from one or more control frames received via the multiwire D2D communication link 150. Thus, an input of the ICG cell on the first circuit die 105 is coupled directly to the output wires of MAC circuit 120a on which PCLK Rate [4:0] signal is sent.

[00119] If simultaneous support for both Original PIPE and SERDES PIPE is desired, then both the ICG cell configuration shown in FIG. 4 and the ICG cell configuration discussed in the immediately preceding paragraph can be provided, with ICG cell 410 being inactive in Original PIPE mode and the ICG cell located on first circuit die 105 being inactive in SERDES PIPE mode.

[00120] As touched upon earlier, second circuit die 110 has D2D receiver 230 located on it. D2D receiver 230 locally generates a D2D clock called D2D CLK. This can be the same frequency as the D2D clock generated by D2D transmitter 210, differing only in a phase between the two D2D clocks. This is however not essential as the frequency of the D2D CLK generated by D2D receiver 230 can be different to the D2D CLK generated by D2D transmitter 210. It is also possible for a different component, e.g. protocol converter 400, to generate D2D CLK instead of D2D receiver 230.

[00121] Protocol converter 400 receives control frames and data frames from D2D receiver 230. The control frames and data frames are each 150 bits in this embodiment, but this is purely illustrative and frames of other sizes are possible. Moreover, the size of a data frame does not need to match the size of control frame. More detail is provided on data frames and control frames later in this specification in connection with FIGs. 8 to 12.

[00122] Protocol converter 400 performs the inverse operation to protocol converter 305. That is, protocol converter 400 converts from the second protocol to the first protocol. In this embodiment, protocol converter 400 converts from frames of an ODVS code to PIPE control information and PIPE data. Other protocol conversions are possible. As shown in FIG. 4, protocol converter 400 is also part of D2D interface circuit 135.

[00123] Protocol converter 400 is configured to extract data from one or more received data frames and to extract control information from one or more received control frames. To enable this, protocol converter 400 is able to distinguish between data frames and control frames. As discussed in more detail later, a ‘frame type’ header can be present in each frame, such that protocol converter 400 can read the frame type header and determine whether the frame is a control frame containing control information or a data frame containing data. The frame type header can be a two-bit header, for example, although this is not limiting and frame type headers of other sizes can alternatively be used.

[00124] D2D interface circuit 135 can include a router circuit (not shown) that is configured to detect routing data in inbound data frames and inbound control frames. ‘Inbound’ here refers to data frames and control frames that are received by D2D receiver 230 from other circuit dies via multi-wire D2D communication link 150. This can be termed a ‘receive link’.

[00125] The router circuit can be configured to responsively retransmit the inbound data frames and inbound control frames on the transmit link. The retransmission can occur over a transmit link of the multi -wire D2D communication link 150, where the transmit link is coupled to a different circuit die to the receive link.

[00126] The router circuit can be configured to responsively retransmit any inbound data frames and inbound control frames having routing data specifying a circuit die other than the circuit die that the router circuit is located on. The retransmitted data frames and control frames are retransmitted as outgoing data frames and outgoing control frames, respectively.

[00127] The routing data can be contained within a frame. Additional details are provided later in this specification, but it is sufficient to note here that the routing data can comprise one or more fields in a frame (data or control, both can use the same routing data) that identify a circuit die in some manner. For example, each circuit die in a given arrangement could be assigned a number, e.g. 0, 1, 2, 3, etc. and each frame can include a ‘Die ID’ field that identifies the circuit die that the payload of the frame needs to be delivered to. More details on this are provided later.

[00128] The router circuit can be configured to know which tile it is located on, i.e. to store or otherwise have access to the number assigned to the tile that the router circuit is located on. The router circuit can be configured to compare the number stored in the ID field of incoming data frames and incoming control frames and to retransmit any frames having a value stored in the ID field that does not match the number assigned to the tile that the router circuit is located on. This routing technique has a relatively low latency because frames not addressed to a component on the circuit die that has received them are retransmitted shortly after receipt. The bulk of the latency for a frame being transmitted from one die to another is that associated with the multi-wire D2D communication link 150 itself. In the case of a six wire D2D communication link using a 5b6w ODVS code, latency can be of the order of 7ns per D2D link. This compares favourably to typical latencies for switch cores, which are typically at least 20 ns and often significantly greater. Additionally, a switch core is not required on all but one die in a set of dies, reducing design and manufacturing cost as well as freeing up die area for use with other components or alternatively enabling the overall area of the die to be reduced.

[00129] V-MAC interface circuit 140a also includes a control buffer 415. This functions similarly to control buffer 300, with reference being made to the description of control buffer 300 earlier in this specification. Control buffer 415 is configured to store control information received in control frames via multi -wire D2D interface 150 and extracted by protocol converter 400. In this embodiment protocol converter 400 is configured to identify control information that is in the form of PIPE command words that were originally sent by MAC 120a on the PIPE Message Bus interface. See section 6.1.4 of [PIPE Specification] for further details regarding the Message Bus interface. [00130] In this embodiment the control information is written into control buffer 415 in 24-bit chunks, these corresponding to the maximum size of a single PIPE command word. As noted earlier in this specification, 8-bit and 16-bit PIPE command words can be padded by 8-bit to 24-bit converter 310 to make a 24-bit chunk of control information. This is optional, as 8-bit to 24-bit converter 310 can be omitted and instead control information can be written into control buffer 415 in 8-bit chunks. [00131] Control information is written into control buffer 415 according to D2D CLK generated by D2D interface 135, specifically D2D receiver 230 in this embodiment.

[00132] In this embodiment, V-MAC interface circuit 140a further includes a 24-bit to 8-bit converter 420 that is configured to convert 24-bit chunks of control information that converter 420 reads out of control buffer 415 into 8, 16 or 24-bit PIPE command words.

[00133] 24-bit to 8-bit converter 420 determines whether a given 24-bit chunk of control information contains an 8-bit, 16-bit or 24-bit PIPE command word based on the first bit of each 24-bit chunk, which indicates the length of the PIPE command word. In the case of 8-bit or 16-bit PIPE command words, 24-bit to 8-bit converter 420 discards the 16 bits or 8 bits of padding, respectively. The resulting 8-bit chunks of control information are subsequently transmitted by 24-bit to 8-bit converter 420 to PHY circuit 145a, specifically to the PIPE Message Bus interface of PHY circuit 145a. Thus, from the perspective of PHY circuit 145a, PIPE command words appear to have been received directly from MAC circuit 120a. V-PHY interface circuit 125a, D2D interfaces 130, 135 and multi-wire communication link 150 are all transparent to PHY circuit 145a.

[00134] 24-bit to 8-bit converter 420 is configured to read 24 bits of control information out of control buffer 415 each L PCLK cycle where control buffer 415 is not empty. Control buffer 415 thus functions as a CDC buffer to allow crossing from the D2D CLK domain and the L PCLK domain.

[00135] In an embodiment where control buffer 415 stores 8-bit chunks of control information, 24-bit to 8-bit converter 420 is not required and can be omitted. In this embodiment, PHY circuit 145a reads 8 bits of control information directly out of control buffer 415 each L PCLK cycle where control buffer 415 is not empty.

[00136] In the embodiment of FIG. 4, V-MAC interface circuit 140a also includes another path for control information to be conveyed from D2D interface circuit 135 to PHY 145a. This path is control wires 425. Control wires 425 carry all control signals that are required by a PIPE interface other than the PIPE commands that are sent by MAC circuit 120a via the PIPE message bus. Specifically, control wires 425 carry PIPE control signals that are sent by MAC circuit 120a via the PIPE command interface and also the PIPE status interface. Details of these signals can be found in sections 6.1.2 and 6.1.3 of [PIPE Specification], The number of control wires 425 will vary according to the specific PIPE version and protocols supported, and has therefore been shown as ‘X’ wires in FIG. 4 to take account of this variation, X being a positive integer. X may be equal to M (see FIG. 3), i.e. the same number of control wires are present in both V- PHY interface circuit 125a and V-MAC interface circuit 140a.

[00137] Control frames that contain control information that is for D2D interface circuit 135 to action are also possible. D2D interface circuit 135 can be configured to identify a received control frame containing an instruction to generate one or more symbols for insertion into receive data buffer 405 and to responsively generate and insert said one or more symbols into receive data buffer 405. The control frame can specify the type of symbol to insert and the number of times the identified symbol should be inserted.

[00138] This type of control frame, referred to hereinafter as a ‘D2D control frame’, can be identified by protocol converter 400 using a field within the D2D control frame that is set to a value to indicate that it is a D2D control frame containing an instruction for D2D interface circuit 135 to act on. This field can be the frame type header discussed earlier. In the case of a two-bit frame type header, a set of values can be defined as directly below:

[00139] These values are purely exemplary and should not be construed as limiting. Other frame type header values and/or schemes are possible.

[00140] D2D control frames can be created by D2D interface 130 when it reads data out of data buffer 320 that matches a symbol of a set of symbols that can be represented by D2D control frames. In this case D2D interface 130, e.g. protocol converter 305, is configured to identify the symbols in the data read from data buffer 320 and to discard these symbols. In their place, D2D interface 130 is configured to create a D2D control frame corresponding to the discarded symbols for transmission over the multi-wire D2D communication link 150. Here, ‘discard’ means that the discarded symbol(s) is/are not sent over multi -wire D2D communication link 150. The D2D control frame identifies the discarded symbol(s) and also the number of discarded symbols so that D2D interface 135 can determine which symbol to insert into data buffer 405 and how many times to insert this symbol.

[00141] D2D interface circuit 135 can have access to a storage component (not shown) such as a register, memory etc. that stores each symbol that D2D interface circuit 135 may need to insert into receive data buffer 405. D2D interface circuit 135 can be configured to read out from the storage component whichever symbol has been specified by the relevant D2D control frame and insert that symbol into receive data buffer 405 the number of times specified by the D2D control frame.

[00142] An example of a symbol that can be inserted into receive data buffer 405 is a skip ordered-set (SKP) as defined in the PCIe protocol. PCIe requires SKP symbols to be inserted at certain points in a PCIe-compliant data stream. The format of a SKP symbol depends on the PCIe generation, as follows:

[00143] In the table above, ‘x’ is a placeholder for bits whose value are not specified by the PCIe protocol and thus can take any value, e.g. for transmission of vendor- defined messages. Square brackets are used to indicate that the value enclosed can appear the number of times stated in subscript adjacent the closing bracket.

[00144] Transmitting these symbols over the multi -wire D2D communication link 150 consumes bandwidth of this link. Instead, one D2D control frame can be sent over multi -wire D2D communication link 150 in place of multiple data frames each containing SKP symbols, reducing the bandwidth used. This is however not essential, as it is possible to send symbols like a SKP symbol over multi-wire D2D communication link 150 in data frames instead.

[00145] In the embodiment of FIG. 4, V-MAC interface circuit 140a also includes an electrical idle symbol detection circuit 430 configured to monitor the data extracted from received data frames for the presence of an electrical idle symbol (electrical idle ordered set, EIOS) and an exit electrical idle symbol (electrical idle exit ordered set, EIEOS). The EIOS and EIEOS symbols are defined in the PCIe specification and take various forms depending on the PCIe generation in use.

[00146] The electrical idle symbol and the exit electrical idle symbol respectively cause a transmitter of PHY circuit 145a to turn off and on. The transmitter can be within the SERDES portion of the PHY circuit 145a, and is responsible for transmitting serial data out from second circuit die 110 to some other entity that is not shown in the figures.

[00147] As shown in FIG. 4, electrical idle symbol detection circuit 430 is configured to monitor the data sent from protocol converter 400 to receive data buffer 405 in order to detect EIOS and EIEOS.

[00148] Electrical idle symbol detection circuit 430 is further configured to, responsive to detection of an electrical idle symbol in the data, set at least one control bit in receive data buffer 405 to a first value to indicate detection of the electrical idle symbol. The at least one control bit is part of a data word in receive data buffer 405 that also includes the electrical idle symbol. The first value could be, for example, a 2-bit value such as ‘ 10’ for data words in receive data buffer 405 that contain an EIOS. Similarly, a second value can be set for the at least one control bit responsive to detection of an exit electrical idle symbol in the data. The second value could be, for example, a 2-bit value such as ‘OF for data words in the receive data buffer 405 that contain an EIEOS. The at least one control bit could be set to a third 2-bit value, e.g. ‘00’, for data words in the receive data buffer 405 that contain neither an EIOS nor an EIEOS. These values are not limiting as other values and schemes can be used instead. These values are referred to herein as ‘EIOS found’ value and ‘EIEOS found’ value, respectively.

[00149] The EIOS found value, or EIEOS found value, is written to the at least one control bit in receive data buffer 405 by electrical idle symbol detection circuit 430 when an EIOS or EIEOS is detected, respectively. In the embodiment of FIG. 4, electrical idle symbol detection circuit 430 triggers an electrical idle register 435 on detection of an EIOS or EIEOS. The value held in electrical idle register 435 is based on the signal level of the most recently extracted value for a transmitter electrical idle (‘TxElecIdle’) signal as defined on page 43 of [PIPE specification]. Specifically, an asserted value of TxElecIdle causes the electrical idle register 435 to hold the EIOS found value, and a deasserted value of TxElecIdle causes the electrical idle register 435 to hold the EIEOS found value. [00150] The TxElecIdle signal value is extracted from a control frame by protocol converter 400. Electrical idle symbol detection circuit 430 is configured to trigger electrical idle register 435 to write the value it currently holds to receive data buffer 405 when an EIOS or EIEOS is detected. This arrangement enables the EIOS found value or EIEOS found value to be written to the same data word as the EIOS or EIEOS itself in receive data buffer 405.

[00151] The arrangement of FIG. 4 relating to detection and writing of EIOS and EIEOS symbols into receive data buffer 405 is not to be understood as limiting. Alternative arrangements that enable the EIOS found value and EIEOS found value to be written into receive data buffer 405 in the same data word as the EIOS or EIEOS itself are also within the scope of this disclosure.

[00152] V-MAC interface circuit 140a is configured to read out a data word from receive data buffer 405 including the at least one control bit and, based on a value of the at least one control bit, generate a corresponding transmitter control signal and provide the transmitter control signal to PHY circuit 145a.

[00153] In the case where the value of the at least one control bit is the EIOS found value, V-MAC interface circuit 140a generates a transmitter electrical idle (‘TxElecIdle’) signal as defined on page 43 of [PIPE specification]. This signal is driven at the ‘asserted’ level and transmitted to PHY 145a via a subset of control wires 425 specifically designated to carry the TxElecIdle signal(s). (In some embodiments TxElecIdle is a single signal, and in others it is multiple signals - see [PIPE specification], page 43). This causes PHY 145a to turn off a transmitter (not shown) of PHY 145a, e.g. that is part of the PHY SERDES.

[00154] In the case where the value of the at least one control bit is the EIEOS found value, V-MAC interface circuit 140a generates the TxElecIdle signal as defined on page 43 of [PIPE specification]. This signal is driven at the ‘deasserted’ level and transmitted to PHY 145a via the subset of control wires 425. This causes PHY 145a to turn on its transmitter.

[00155] The inclusion of the EIOS found / EIEOS found value in the same data word as the symbol itself is an example of local synchronization of data and control information by V-MAC interface circuit 140a. This is because the data (EIOS / EIEOS) and control information (TxElecIdle signal level) are presented to PHY circuit 145a by V-MAC interface circuit 140a in a specific time-ordered manner. Specifically, PHY circuit 145a receives a change in the TxElecIdle signal level (asserted to deasserted, or vice versa) at the same time as, or shortly after, the EIOS or EIEOS symbols are read out of receive data buffer 405 by PHY circuit 145a. This ensures that the timing requirements of the first protocol, in this case PIPE, are respected without placing any additional requirements on either MAC circuit 120a or PHY circuit 145a.

[00156] The configuration of electrical idle symbol detection circuit 430 shown in FIG. 4 is not the only way of locally synchronizing the EIOS / EIEOS symbols and TxElecIdle signal levels.

[00157] In another non-illustrated embodiment, electrical idle symbol detection circuit 430 is configured to monitor data being read out of receive data buffer 405. Responsive to an electrical idle symbol being read out of the data buffer, electrical idle symbol detection circuit 430 transmits a corresponding electrical idle control signal to PHY circuit 145a. The electrical idle control signal can be the TxElecIdle signal set to the relevant level depending on whether the electrical idle symbol read out is an EIOS or EIEOS. By triggering the transmission of the electrical idle control signal from the detection of the electrical idle symbol in the data read out of receive data buffer 405, synchronization is achieved because PHY circuit 145a will receive the electrical idle control signal concurrently with, or shortly after, the electrical idle symbol.

[00158] In a modification to this non-illustrated embodiment, electrical idle symbol detection circuit 430 can additionally be configured to monitor the control information for the presence of an electrical idle control signal and an exit electrical idle control signal. This could be, for example, monitoring control frames received by protocol converter 400 to identify control frames containing an electrical idle control signal or an exit electrical idle control signal, e.g. an asserted or deasserted level for TxElecIdle, respectively. Transmission of the electrical idle symbol by electrical idle symbol detection circuit 430 to PHY circuit 145a can be delayed until electrical idle symbol detection circuit 430 has detected an electrical idle control signal or an exit electrical idle control signal in a control frame, and also detected the corresponding electrical idle symbol being read out of receive data buffer 405. In addition to synchronizing the electrical idle control signal and electrical idle symbols, this modification also ensures that the state of the electrical idle control signal is the same at both MAC circuit 120a and PHY circuit 145a.

[00159] Referring now to FIG. 5, circuit die 110 is shown again but in FIG. 5 the receive signal path is shown, i.e. PHY circuit 145a sending control information and data to MAC circuit 120a. It will thus be appreciated that in practice an embodiment of circuit die 110 includes both the forward signal path (MAC circuit 120a to PHY circuit 145a) shown in FIG. 4 and the reverse signal path shown in FIG. 5.

[00160] There are some components shown in FIG. 5 that are also shown in FIG. 4. To make this clear, these components retain in FIG. 5 the reference numeral assigned to them in FIG. 4. Other components in FIG. 5 provide similar functionality to respective components in FIG. 3 - to indicate this, the same reference numeral suffix has been assigned to these components in FIG. 5 as FIG. 3.

[00161] V-MAC interface circuit 140a includes a control buffer 500. Control information generated by PHY 145a is written into control buffer 500. Control buffer 500 is thus distinguished from control buffer 415 in that control buffer 415 holds control information generated by MAC 120a, i.e. control information flowing in the opposite direction to the control information stored in control buffer 500.

[00162] In the illustrated embodiment control information is written into control buffer 500 based on a receiver clock (RxCLK) generated by PHY 145a. See Table 6-11 of [PIPE Specification] for more information on the RxCLK signal. This indicates that PHY 145a is operating in SERDES PIPE mode. In the case where PHY 145a is operating in Original PIPE mode, RxCLK is replaced everywhere it is present in FIG. 5 by L PCLK as provided by ICG cell 410.

[00163] Functionally, control buffer 500 operates in the same manner as control buffer 300 and thus reference is made here to the earlier description of control buffer 300. It is noted that, due to the reversal in direction of traffic in FIG. 5 relative to FIG. 3, control information is written into control buffer 500 by PHY 145a according to RxCLK (or L PCLK in original PIPE mode) and read out of control buffer 500 by protocol converter 400 according to D2D CLK.

[00164] The embodiment of FIG. 5 includes an 8-bit to 24-bit converter 510 coupled to a PIPE Message Bus interface of PHY circuit 145a and also coupled to control buffer 500 on the write side. 8-bit to 24-bit converter 510 functions in the same manner as 8- bit to 24-bit converter 310. Reference is thus made to the earlier description of 8-bit to 24-bit converter 310. It is possible to omit 8-bit to 24-bit converter 510, in which case control buffer 500 is coupled directly to the PIPE Message Bus interface of PHY circuit 145a.

[00165] Transmit data buffer 520 functions in the same manner as data buffer 320 of FIG. 3. Briefly, PHY 145a writes data (e.g. data formatted according to a PIPE protocol) into transmit data buffer 520 according to RxCLK (L PCLK in the case of Original PIPE). Protocol converter reads data out of transmit data buffer 520 according to D2D CLK.

[00166] Control wires 515 are coupled between PHY circuit 145a and protocol converter 400. Control wires 515 are similar to control wires 315 in that they carry all control information that is not sent over the PIPE Message Bus. However, the control information sent from PHY circuit 145a to MAC circuit 120a is not the same as the control information sent in the opposite direction (FIG. 3), hence Y control wires are present, Y being a positive integer that may or may not be equal to M of FIG. 3.

[00167] In the case where the first protocol is a PIPE protocol, control wires 515 carry at least some of the signals specified in sections 6.1.2 and 6.1.3 of [PIPE Specification] that are sent from a PHY to a MAC. These are control signals sent from a PHY to a MAC over the PIPE Command Interface and PIPE Status Interface, respectively. This is not an exhaustive list of signals that can be carried by control wires 515, as control wires 515 can carry any control information going in the PHY to MAC direction that is not sent over the PIPE Message Bus.

[00168] The specific set of signals carried by control wires 515 can vary according to PIPE operating mode, e.g. Original PIPE compared with SERDES PIPE. It should be appreciated that control wires 515 do not necessarily carry a complete set of control signals as defined in a PIPE specification or other such protocol document as some control information is generated locally and hence it is not necessary to send this control information over the multi -wire D2D communication link 150.

[00169] Protocol converter 400 is configured to operate in the same manner as protocol converter 305. Specifically, protocol converter 400 receives control information and data from V-MAC interface circuit 140a, generates control frames and data frames respectively containing the control information and data, and sends the control frames and data frames to D2D transmitter 150 for transmission over multi -wire D2D communication link 150. Multi -wire communication link 510 can be coupled to circuit die 105 in the two-die bidirectional link case, or another circuit die in an arrangement having more than two dies (e.g. three or more dies coupled in a ring topology).

[00170] D2D transmitter 510 operates in the same manner as D2D transmitter 210 and thus reference is made here to the earlier discussion of D2D transmitter 210.

[00171] D2D CLK as shown in FIG. 5 is generated by D2D interface 135 as discussed above, with reference being made here to that discussion. FIG. 5 illustrates a SERDES PIPE operating mode, meaning that data and control information is written into transmit data buffer 520 and control buffer 500, respectively, according to RxCLK as generated by PHY circuit 145a. In the case where the apparatus of FIG. 5 is operating in an Original PIPE mode, RxCLK is replaced with L PCLK as discussed above. L PCLK is provided by ICG cell 410.

[00172] D2D interface circuit 135 (e.g. protocol converter 400) is configured to read data out of transmit data buffer 520 and to insert the data that has been read out into one or more transmit data frames. The one or more transmit data frames can then be transmitted by D2D transmitter 510 over multi -wire D2D communication link 150 to another circuit die, e.g. circuit die 105 or another circuit die that is part of a ring topology.

[00173] As shown in the upper right of FIG. 5, V-MAC interface circuit 140a can be configured to monitor status information provided by PHY circuit 145a. The status information can include a PCLK change ok signal (‘PCLKChangeOk’) signal indicating that PHY circuit 145a is ready to change a rate of the PCLK (and L PCLK) signal. More information is provided about the PCLKChangeOk signal in [PIPE Specification] on page 64.

[00174] The status information can additionally or alternatively include a receiver electrical idle signal (‘RxElecIdle’). This indicates that a receiver of PHY circuit 145a (e.g. that is part of the SERDES of PHY circuit 145a) has detected one or more electrical idle symbols in data that the receiver has received. More information is provided about the RxElecIdle signal in [PIPE Specification] on page 61.

[00175] The status information can be used as part of any process that requires knowledge of the current state of PHY circuit 145a. One example of such a process is changing the rate of PCLK. This is discussed in more detail below in connection with FIG. 7.

[00176] Another instance where knowledge of the current state of PHY circuit 145a is necessary is detection of receiver electrical idle signals. To facilitate this, V-MAC interface circuit 140a can be configured to monitor status information provided by PHY circuit 145a to detect a receiver electrical idle signal (RxElecIdle). Responsive to detection of the receiver electrical idle signal, V-MAC interface circuit 140a can transmit a standby command to PHY circuit 145a. The standby command causes the PHY circuit 145a to turn off the receiver (e.g. located with the SERDES). Here, ‘turn off includes switching the receiver to a low power mode, standby mode or other such ‘non-operational’ mode. The standby command can be the PIPE RxStandby command described on page 55 of [PIPE Specification], for example.

[00177] V-MAC interface circuit 140a can be configured to send the standby command once it has detected both the receiver electrical idle signal and also detected at least one electrical idle symbol (EIOS) in data provided by the PHY circuit 145a. To achieve this, V-MAC interface circuit 140a can be configured to monitor data provided by the PHY circuit 145a. This could be achieved by monitoring data being written into transmit data buffer 520 for electrical idle symbols (EIOS). The monitoring can be performed by logic similar to electrical idle symbol detection circuit 430.

[00178] It should be noted that the receiver electrical idle signal is not consumed by V- MAC interface circuit 140a despite V-MAC interface circuit 140a acting on this signal. Instead, the receiver electrical idle signal is received by D2D interface circuit 135. D2D interface circuit 135 (e.g. protocol converter 400) generates a corresponding receiver electrical idle control frame and transmits the receiver electrical idle control frame over multi-wire D2D communication link 150. This means that MAC 120a receives the receiver electrical idle signal.

[00179] This configuration advantageously ensures that PHY circuit 145a receives an instruction (from V-MAC interface circuit 140a) to turn off its receiver in a timely manner, i.e. without waiting for the additional latency that is introduced whilst information is exchanged with MAC 120a via multi-wire D2D communication link 150. This can reduce the chance of clock data recovery (CDR) circuitry that is part of PHY circuit 145a being active when no data is being received, which is desirable as a CDR circuit can derail or otherwise enter a non-operational state if operated whilst invalid data is being received. This is achieved in a manner that is transparent to MAC circuit 120a and PHY circuit 145a but which also maintains the correct synchronization between these components.

[00180] Referring now to FIG. 6, circuit die 105 is shown again but in FIG. 6 the receive signal path is shown, i.e. command information and data sent by PHY circuit 145a to MAC circuit 120a. It will thus be appreciated that in practice an embodiment of circuit die 105 includes both the forward signal path (MAC circuit 120a to PHY circuit 145a) shown in FIG. 3 and the reverse signal path shown in FIG. 6.

[00181] There are some components shown in FIG. 6 that are also shown in FIG. 3. To make this clear, these components retain in FIG. 6 the reference numeral assigned to them in FIG. 3. Other components in FIG. 6 provide similar functionality to respective components in FIG. 4 - to indicate this, the same reference numeral suffix has been assigned to these components in FIG. 6 as FIG. 4.

[00182] As shown in FIG. 6, the receive portion of D2D interface 130 includes D2D receiver 630. This operates in the same manner as D2D receiver 230 to receive data frames and control frames transmitted from another circuit die (e.g. circuit die 110) via multi -wire D2D communication link 150.

[00183] Received data frames and control frames are unpacked by protocol converter 305 as discussed above in respect of protocol converter 400. The extracted data and control information resulting from the unpacking is in the correct format for the first protocol, e.g. PIPE.

[00184] The extracted data is written into receive data buffer 605 by protocol converter 305 in the same manner as data is written into receive data buffer 405. D2D interface circuit 130 (e.g. D2D receiver 630) can be configured to generate D2D CLK and to write the extracted data into the receive data buffer 605 based on D2D CLK.

[00185] Data is read out of receive data buffer 605 by MAC circuit 120a. Attention is drawn to the fact that data is read out of receive data buffer 605 not according to cycles of PCLK as generated by MAC circuit 120a, but instead in a manner such that MAC circuit 120a only reads out data when receive data buffer 605 is not empty (i.e. contains unread valid data). This prevents underrun of receive data buffer 605 that may be caused by late arrival of control information from PHY circuit 145a (e.g. a RxElecIdle signal) owing to latency associated with the multi -wire D2D communication link 150. [00186] One way in which MAC circuit 120a can be configured to read out data only when receive data buffer 605 contains unread valid data is shown in FIG. 6. As shown, an ICG cell 650 is provided as part of V-PHY interface circuit 125a. One input of ICG cell 650 receives the D2D CLK signal and another input of ICG cell 650 receives a fill level signal from receive data buffer 605. The fill level signal indicates whether receive data buffer 605 currently contains unread valid data. For example, when unread valid data is present in receive data buffer 605, the fill level signal could be set to a HIGH level, and when there is no unread valid data in receive data buffer 605 the fill level signal could be set to a LOW level. This is not limiting as other types of fill level signal are also possible.

[00187] ICG cell 650 provides a gated D2D interface clock signal (‘G D2D CLK’) at an output. G D2D CLK is gated based on the fill level signal from receive data buffer 605, meaning that the G D2D CLK signal is ‘active’ (e.g. contains cycles rather than remaining at a constant level) only when there is unread valid data in receive data buffer 605. MAC circuit 120a can be configured to read out data from receive data buffer 605 based on G D2D CLK. This can involve MAC circuit 120a reading data out from receive data buffer 605 only when a transition is detected in G D2D CLK (e.g. on each rising edge), meaning that MAC circuit 120a only reads data from receive data buffer 605 when the fill level signal is set to a level that causes transitions in the level of G D2D CLK. Thus, MAC circuit 120a reading from receive data buffer 605 when it does not contain any unread valid data is prevented.

[00188] V-PHY interface circuit 125a also includes control buffer 615 and optionally also 24-bit to 8-bit converter 620. These components operates in a similar manner to control buffer 415 and 24-bit to 8-bit converter 420, respectively, with reference being made here to the discussion of control buffer 415 and converter 420 above.

[00189] V-PHY interface circuit 125a includes a set of control wires 625. Control wires 625 carry all control signals that are required by a PIPE interface other than the PIPE commands that are sent by PHY circuit 145a via the PIPE message bus. Specifically, control wires 625 carry PIPE control signals that are sent by PHY circuit 145a via the PIPE command interface and also the PIPE status interface. Details of these signals can be found in sections 6.1.2 and 6.1.3 of [PIPE Specification], The number of control wires 625 will vary according to the specific PIPE version and protocols supported, and has therefore been shown as ‘Z’ wires in FIG. 6 to take account of this variation, Z being a positive integer. Z may be equal to Y (see FIG. 5), i.e. the same number of control wires can be present in both V-PHY interface circuit 125a and V-MAC interface circuit 140a. This is not always the case, however, as some signals are generated locally by V-PHY interface circuit 125a or V-MAC interface circuit 140a, obviating the need for a dedicated control wire.

[00190] V-PHY interface circuit 620 also includes receiver electrical idle gating logic. In this embodiment receiver electrical idle gating logic comprises an AND gate 655 having a first input configured to receive a receiver electrical idle status signal from D2D interface 130. In this embodiment, the first input of AND gate 655 is coupled to control wires 625, specifically to a subset of control wires 625 that carry a receiver electrical idle signal. In the case of PIPE, the receiver electrical idle signal is RxElecIdle (see [PIPE Specification], page 61) and is carried by one wire. Thus, the first input of AND gate 655 is coupled to the wire of control wires 625 that carries the RxElecIdle signal. [00191] A second input of AND gate 655 is coupled to receive data buffer 605. The second input receives the fill level signal discussed above from receive data buffer 605. Depending on the fill level signal configuration, the second input may be an inverting input as the objective is to have the AND gate 655 output HIGH when the fill level signal indicates that there is no unread valid data in receive data buffer 605.

[00192] The receiver electrical idle gating logic has a first output that is coupled to MAC 120a to provide a logically gated version of the receiver electrical idle status signal to MAC 120a. This is shown in FIG. 6 as an output of AND gate 655 providing signal G RxElecIdle (gated RxElecIdle) to MAC circuit 120a. G RxElecIdle is provided to MAC circuit 120a on the wire(s) that it expects to receive RxElecIdle, such that MAC circuit 120a treats G RxElecIdle as if it were RxElecIdle. This is an example of local synchronization of data and control information because it ensures that MAC circuit 120a receives the electrical idle status signal (control information) once this signal has been asserted by PHY 145a and only after all valid data has been read out of receive data buffer 605.

[00193] At various points earlier in the specification reference has been made to changing of the rate of PCLK. FIG. 7 sets out a flowchart providing details on how the rate of locally generated signal L PCLK is changed to match a new rate of PCLK.

[00194] While elements of FIG. 7 have been drawn in a specific order, this does not imply limitation to the execution of these elements only in the order shown. The scope of this disclosure extends also to execution of these elements in a different order to that shown. One or more elements may also be executed in parallel rather than sequentially. One or more elements may be omitted, depending on the specifics of the implementation at hand.

[00195] In element 700, V-MAC interface circuit 140a detects a PCLK rate signal in the control information, the PCLK rate signal indicating the target frequency. The PCLK rate signal can be PCLK Rate[4:0] as described on page 53 of [PIPE Specification], The target frequency is that of the PCLK signal, as it specifies a new PCLK rate that MAC circuit 120a intends to adjust PCLK to in the future.

[00196] The PCLK rate signal can be sent by MAC circuit 120a via control wires 315 (FIG. 3). V-MAC interface circuit 140a can detect the reconstruction of this PCLK rate signal that is provided via control wires 425 (FIG. 4) after the information that this signal represents has traversed the multi -wire D2D communication link 150 via one or more control frames. V-MAC interface circuit 145a can include signal level monitoring logic (not shown) that is coupled to the control wire(s) that carry the PCLK rate signal, in order to facilitate detection of the PCLK rate signal.

[00197] V-MAC interface circuit 140a can store the target frequency that PCLK rate specifies in some manner, e.g. in a register (not shown) that can be part of ICG cell 410, for example. This enables V-MAC interface circuit 140a to retrieve this target frequency at a later time when adjusting a rate of L PCLK to the target frequency, as discussed in more detail below.

[00198] In element 705, V-MAC interface circuit 145a monitors status information provided by the PHY circuit 145a to detect a PCLK change ok signal indicating that the PHY circuit 145a is ready to change a rate of the PCLK signal. The PCLK change ok signal can be the PclkChangeOk signal described on page 64 of [PIPE Specification], for example. The status information can be part of the control information discussed above and can be sent by PHY circuit 145a on one or more of control wires 515 (FIG. 5). V-MAC interface circuit 120a can include signal level monitoring logic (not shown) that is coupled to the control wire(s) that carry the PCLK change ok signal, in order to facilitate detection of the PCLK change ok signal.

[00199] In element 710, V-MAC interface circuit 145a changes a frequency of the locally generated PCLK signal (L PCLK) to the target frequency. This action can be performed responsive to the detection by V-MAC interface circuit 145a of the PCLK change ok signal. V-MAC interface circuit 145a can change the rate of L PCLK by updating a configuration of ICG cell 410 (FIG. 4) that is part of the V-MAC interface circuit 145a. Updating the configuration of ICG cell 410 can include adjusting a gating setting of ICG cell 410 such that the gating of the clock signal that ICG cell 410 accepts as an input (SysCLK) is changed. Adjusting the gating setting can involve changing a gating frequency of ICG cell 140, where the gating frequency is the frequency at which the input clock signal (SysCLK) is provided at an output of ICG cell 410. The end result is that the frequency of L PCLK is adjusted by adjusting the gating setting.

[00200] In element 715, V-MAC interface circuit 140a transmits an acknowledgement signal to PHY circuit 145a once the frequency of the locally generated PCLK signal L PCLK has been changed to the target frequency. This acknowledgement signal can be PclkChangeAck as described on page 57 of [PIPE Specification], The acknowledgement signal can be transmitted to PHY circuit 145a by V-MAC interface circuit 140a via one or more of control wires 425 (FIG. 4). In the case where the acknowledgement signal is PclkChangeAck, V-MAC interface circuit 140a can transmit PclkChangeAck via one of control wires 425 that is coupled to a PIPE Status Interface of PHY circuit 145a.

[00201] The acknowledgement signal can be generated locally by V-MAC interface circuit 140a without the involvement of MAC circuit 120a. This ensures that PHY circuit 145a is promptly informed of the completion of the change of rate of L PCLK, as the latency associated with the multi -wire D2D communication link 150 does not affect the locally generated acknowledgement signal.

[00202] Advantageously, V-MAC interface circuit 145a acting responsive to the local detection of the PCLK change ok signal ensures that the L PCLK rate adjustment is performed promptly. Specifically, the rate of L PCLK is adjusted and L PCLK is stable before MAC circuit 120a completes the same process but in respect of PCLK. This is significant because it ensures that L PCLK is stable at the new rate before an acknowledgement signal that is generated by MAC circuit 120a can traverse the die-to- die link and be received by PHY circuit 145a. This avoids an error-introducing scenario where PHY circuit 145a understands L PCLK to be stable based on a local reconstruction of an acknowledgement signal from MAC circuit 120a whereas in fact V-MAC interface circuit 140a has not yet stabilised L PCLK at the new rate.

[00203] Element 715 can be omitted in the case where the first protocol is a PIPE protocol and MAC circuit 120a is communicating with PHY circuit 145a in Original PIPE mode. In this case, no acknowledgment signal need be sent by V-MAC interface circuit 140a.

[00204] In element 720, D2D interface circuit 135 receives the PCLK change ok signal and generates a corresponding PCLK change ok control frame. D2D interface circuit 135 also transmits the PCLK change ok control frame over the multi -wire D2D communication link 150. The PCLK change ok control frame can be addressed to the die on which MAC circuit 120a is located (tile 105 in the figures). Sending the PCKL change ok information to MAC circuit 120a ensures that the state of MAC circuit 120a is kept in sync with the state of PHY circuit 145a even though PCLK as generated by MAC circuit 120a is not actually used by PHY circuit 145a.

[00205] As noted above, if operating in SERDES PIPE mode MAC circuit 120a will generate an acknowledgement signal in respect of successful adjustment of the rate of PCLK. As PHY circuit 145a is not actually receiving PCLK, this acknowledgment signal is both duplicative and unnecessary. Protocol converter 305 or protocol converter 400 can be configured to drop the acknowledgement signal from MAC circuit 120a, e.g. by not transmitting it via multi-wire D2D communication link 150 in the case of protocol converter 305 or by not transmitting it to V-MAC interface circuit 145a in the case of protocol converter 400. Dropping the acknowledgement signal can include not coupling any of control wires 315 to the PclkChange Ack output of the PIPE Status Interface of MAC circuit 120a, for example, such that this signal is not sent to protocol converter 305. Additionally / alternatively, the one of control wires 425 that is coupled to the PclkChangeAck input of the PIPE Status Interface of PHY circuit 145a can be coupled only to logic of V-MAC interface circuit 140a that generates PclkChangeAck such that there is no signal path that enables the PclkChangeAck signal generated by MAC circuit 120a to reach PHY circuit 145a.

[00206] The PCLK rate change handshake procedure specified by the PIPE protocol ([PIPE Specification], section 8.6) also specifies a PhyStatus signal sent from PHY circuit to MAC circuit to complete the rate change procedure. For completeness, it is mentioned here that the PhyStatus signal can be sent from PHY circuit 145a to MAC circuit 120a in one or more control frames over multi -wire D2D communication link 150 in the manner described herein. The deassertion of PclkChangeOk on the trailing edge of PhyStatus also required by [PIPE Specification] can also be communicated to MAC circuit 120a in one or more control frames over multi -wire D2D communication link 150 in the manner described herein.

[00207] FIG. 8 shows a graphical representation of a data frame 800 according to an embodiment. Data frames carry data, this being defined as any information destined for a receive data buffer like receive data buffer 405, 605. This includes, but is not limited to, signals sent by MAC circuit 120a via a PIPE data interface.

[00208] The blocks shown in FIG. 8 each respectively represent a different field of the data frame. The term ‘field’ is intended to refer to a specific proper subset of the bits of the data frame, e.g. field 1 = bits 3:0. This nomenclature is used throughout this specification. The width of each block as drawn is approximately representative of the size of the associated field in bits, but this is not adhered to strictly.

[00209] Data frame 800 is suitable for use in the context of MAC circuit 120a communicating with PHY circuit 145a using PCIe over PIPE. This should not be construed as limiting as it is possible to adapt the frames disclosed herein to other protocols.

[00210] Data frame 800 is 150 bits in total. The various fields are shown in the table directly below.

[00211] Frame type field 805 indicates whether the frame is a control frame or data frame. Subdivisions within this are possible, e.g. the D2D control frame subtype discussed earlier. For data frames, frame type field 805 can have a binary value 10, for example.

[00212] Payload field 810 contains data. It is not necessary to fill the entire payload field before transmitting a data frame over the multi-wire D2D communication link 150, so in some cases one or more bits of the payload field are unused.

[00213] It is possible to divide the payload field 810 up into subfields. This is useful in circumstances where a single frame contains data that is destined for multiple different components. An example of such circumstances is different PHY circuits in the case of multiple lanes in a single PCIe link. In the case of a PCIe link having n lanes, payload field 810 can be divided up into n subfields each 128 / n bits in size (assuming a full payload field), each subfield corresponding to a given lane of the link. The Link ID field 815 can be used to specify the lowest numbered PCIe lane in the link that the data is destined for, with it being understood that the first subfield of payload field 810, i.e. bits ((128 / n) + 1):2 of the payload field, contains data for the lowest numbered PCIe lane in the link. The second subfield of payload field 810, i.e. bits (2*(128 / n) + 1): ((128 / n) + 2) of the payload field, contains data for the next lowest numbered PCIe lane in the link, and so on. The value of n is set for each link during configuration, e.g. in firmware, and can thus be looked up by D2D interface 130 and D2D interface 135.

[00214] Splitting the payload field 810 into multiple subfields as described above can advantageously reduce the average latency in communication between MAC circuits 120a ... 120n and PHY circuits 145a ... 145n. This is because subfields allow each PHY circuit in a link to start reading out of its corresponding receive data buffer after a sufficient depth has been achieved, e.g. data from one frame has been sent over the multi -wire D2D communication link 150. This is because in this case every receive data buffer in the link has some data in it. In the case where subfields are not used, the receive data buffer that is last to be filled must wait for, at a minimum, n data frames to be sent over the multi -wire D2D communication link 150, n being the number of lanes in the link.

[00215] It is possible that not all lanes of a link are in use, e.g. where an endpoint device negotiates for fewer lanes than a given link supports. In such a case, each subfield of payload field 810 in a data frame that is associated with an unused lane can contain electrical idle data (e.g. EIOS).

[00216] The die ID field 820 indicates which die the frame is destined for. Dies can be assigned a numerical value in ascending order, e.g. die 105 is assigned the binary value 0000 and die 110 is assigned the binary value 0001. The 4-bit die ID field allows for up to 2⁴ = 16 dies coupled together. Die ID field 820 can be increased in size if necessary to allow additional dies to be added. The router circuit of D2D interface 135 discussed above can use the die ID field to determine whether to process a given frame or to forward the frame on to the next die in the chain.

[00217] It is possible to reserve a die ID value for broadcast frame capabilities. In this case, any frame having the broadcast die ID is both consumed (processed) and forwarded on by a given D2D interface circuit. This has utility in a scenario where it is desired to transmit certain data from a leader die on which MAC 120a is located (e.g. die 105) to all follower dies that the leader die is connected to, each follower die being like circuit die 110.

[00218] The payload size field 825 specifies the number of bytes of the payload field 810 that contain valid data. For example, a payload size field value of 0001 indicates that one byte of payload field is valid, i.e. bits 9:2 of frame 800 are valid and the rest should be ignored. This can be relevant where MAC circuit 120a or PHY circuit 145a has less than 128 bits of data left to send over multi-wire D2D communication link 150, as in such a case payload field 810 is partially filled.

[00219] The D2D protocol field 830 contains information relating to the D2D protocol itself. This can include a frame priority (e.g. high, medium, low) and/or a second frame type that indicates that the frame contains information that is to be consumed by the D2D interface circuit on the receive side of the multi-wire D2D communication link 150. Some bits of D2D protocol field 830 may be reserved for future use. D2D protocol field 830 can vary according to the specific implementation of the D2D interface.

[00220] It will be appreciated that FIG. 8 shows just one way of constructing a data frame. Many variations are possible, e.g. data frames of a different total size can be used and/or the field ordering can be changed relative to that shown in FIG. 8 and/or some fields may be removed and/or changed in size relative to that shown in FIG. 8. Additionally or alternatively, fields not shown in FIG. 8 can be added to data frame 800. The scope of this disclosure is thus not limited to the particular data frame shown in FIG. 8, which is provided as an illustrative embodiment without limiting the scope of this disclosure.

[00221] FIG. 9 shows a graphical representation of a control frame 900 according to an embodiment. Control frame 900 carries control information, in this case control information that is sent via a PIPE Message Bus interface by either MAC circuit 120a or PHY circuit 145a.

[00222] Control frame 900 is suitable for use in the context of MAC circuit 120a communicating with PHY circuit 145a using PCIe over PIPE. This should not be construed as limiting as it is possible to adapt the frames disclosed herein to other protocols.

[00223] Control frame 900 is 150 bits in total. The various fields are shown in the table directly below. Payload field in this case is subdivided into three subfields, shown in italic text.

[00224] The frame type field 905, die ID field 920, payload size field 925 and D2D protocol field 930 are all the same as their counterparts in frame 800 of FIG. 8. Reference is thus made to the description above in respect of those fields.

[00225] Payload field 910 contains control information. In the case of frame 900, the control information is a PIPE command word. The PIPE command word is stored in the PIPE command word subfield 910a. As noted earlier, the maximum size of a PIPE command word is 24 bits, hence the PIPE command word subfield 910a is 24 bits to accommodate the largest PIPE command words. In the case of 8-bit or 16-bit PIPE command words, the 16 bits or 8 bits of the PIPE command word subfield 910a are empty, respectively.

[00226] Payload field 905 also includes a PHY ID subfield 910b. PHY ID subfield 910b includes one bit for each PHY circuit that is present on circuit die 110. PHY ID subfield 910b in this embodiment is 8 bits, corresponding to the fact that there are 8 PHY circuits like PHY circuit 145a on circuit die 105. This is not limiting in scope, however, as a different number of PHY circuits can instead be provided on circuit die 105.

[00227] In the general case, each bit of the PHY ID subfield 910b is respectively mapped to one of the PHY circuits on the circuit die having a die identifier corresponding to the value in Die ID field 920. In the case of 8 PHY circuits, the mapping can be as follows:

This mapping is purely exemplary and is not limiting as other mapping schemes can be used instead. [00228] Each bit of PHY ID subfield 910b is set to a first value, e.g. ‘O’, in the case where the PHY circuit associated with the bit does not need to receive the PIPE Command word stored in the PIPE command word subfield 910a. Each bit of PHY ID subfield 910b is set to a second value, e.g. ‘ 1’, in the case where the PHY circuit associated with the bit does need to receive the PIPE Command word stored in the PIPE command word subfield 910a. Thus, if PHY circuits 0 and 1 need to receive the PIPE command word stored in PIPE command word subfield 910a and PHY circuits 2 through 7 do not, PHY ID subfield 910b would have the binary value 00000011, on the understanding that the rightmost bit is bit 26, the next rightmost bit is bit 27, and so on. This scheme is purely exemplary and is not limiting as other schemes can be used instead.

[00229] The PHY ID subfield 910b enables a single PIPE command word to be distributed to any number of PHY circuits on a circuit die whilst consuming only one frame’s worth (150 bits in this embodiment) of bandwidth of the multi-wire D2D communication link 150. It is often the case that a given PIPE command word needs to be sent to all PHY circuits in a link and use of the PHY ID subfield 910b makes this possible with relatively little consumption of D2D communication link bandwidth.

[00230] On the receive side of multi -wire D2D communication link 150, D2D interface 135 (e.g. protocol converter 400) is configured to insert a copy of the PIPE command word stored in PIPE command word subfield 910a into every control buffer that is coupled to a PHY circuit that is indicated as a recipient of the PIPE command word by the PHY ID subfield 910b. D2D interface 135 therefore effectively replicates the PIPE command word as many times as necessary to ensure that all of the PHY circuits that should receive the PIPE command word do indeed receive it.

[00231] As the combination of the PIPE command word 910a and PHY ID 910b fields are 32 bits in size, the payload size field 925 would hold a value of 32 in this case.

[00232] In the embodiment of FIG. 9 bits 129:34 are unused and labelled as ‘unused subfield 910c’. Modifications to frame 900 that make use of unused subfield 910c are possible and within the scope of this disclosure. For example, as the PIPE command word and PHY ID subfields collectively occupy 32 bits, this pair of subfields could be replicated once, twice or three times more within unused subfield 910c. Thus, up to four PIPE command words could be transmitted in a single frame, each with an associated PHY ID field to direct the PIPE command word to the correct set of PHY circuits. The value stored in payload size field 925 would be adjusted as necessary to take account of the additional copy/copies of the PIPE command word and PHY ID subfields.

[00233] Optionally, frame 900 can include extended PHY ID field 915. This field has utility in the case where there are more PHY circuits on a given die than is possible to map to using PHY ID subfield 910b. In this embodiment, extended PHY ID field 915 would thus have utility in the case where there are more than 8 PHY circuits on circuit die 110. Extended PHY ID field 915 can store a value that indicates an offset at which to start the mapping between bits of PHY ID subfield 910b and PHY circuits. For example, if extended PHY ID field 915 stores a value of 1, the first bit of PHY ID subfield 910b maps to PHY circuit 8, the second bit of PHY ID subfield 910b maps to PHY circuit 9, and so on. In the case where extended PHY ID field 915 is 4 bits, up to 136 PHY circuits can be addressed per circuit die. A value of 0 in extended PHY ID field 915 can indicate that no offset is to be applied and thus the mapping discussed in paragraph 0188 above applies.

[00234] FIG. 10 shows a graphical representation of a control frame 1000 according to an embodiment. Control frame 1000 is a variation of control frame 900 and is capable of broadcasting a given PIPE command word to PHYs on multiple dies. Control frame 1000 thus has particular utility in embodiments having a leader circuit die like circuit die 105 with first logic circuit 115 (e.g. a switch core) communicating with PHY circuits on two or more follower circuit dies that are each like circuit die 110.

[00235] The fields of frame 1000 are the same as their respective counterparts of frame 900, other than as follows.

[00236] The main difference between frames 900 and 1000 is that frame 1000 occupies at least some the unused subfield 910c of frame 900. Specifically, this subfield is replaced with up to 13 PHY ID subfields 1010b - lOlOn, each of 8 bits. 13 subfield are possible in the illustrated embodiment because 104 bits are available. This is not limiting however, as other frame sizes with fewer or more available bits for use as PHY ID subfields are possible.

[00237] Each PHY ID subfield 1010b - lOlOn is the same as PHY ID subfield 910b in that it includes one bit for each PHY circuit that is present on a given circuit die. However, in this embodiment each PHY ID subfield is mapped to a different circuit die, rather than being associated with the circuit die identified in Die ID field 1020. One mapping scheme that is usable is to map PHY ID subfield 1010b to the PHY circuits on a first follower circuit die, and map PHY ID subfield 1010c to the PHY circuits on the second follower circuit die, and so on. Follower circuit dies can be statically enumerated during manufacture or dynamically enumerated in a start-up or configuration mode, for example.

[00238] By way of illustrative example, assuming a four circuit die configuration, with one leader circuit die with Die ID 0000 and three follower circuit dies with respective Die IDs as shown below, a possible mapping is as follows. These values are provided to assist in the understanding of this disclosure and are not limiting.

[00239] In this example, the PIPE command word contained in PIPE command word subfield 1010a is transmitted to PHY circuits 0 and 1 on follower circuit die 1, PHY circuits 6 and 7 on follower circuit die 2 and PHY circuits 0, 1, 2 and 3 on follower circuit die 3. As there are no more circuit dies in this example configuration, PHY ID subfields lOlOe - lOlOn are unused. Advantageously, a PIPE command word can thus be transmitted to multiple PHY circuits on multiple circuit dies using just one control frame. This can reduce the bandwidth of multi -wire D2D interface 150 that is consumed by control frames.

[00240] To indicate that frame 1000 is a broadcast control frame, Die ID field 1020 can be set to the identifier corresponding to a leader circuit die (typically 0000, but other values can be used). Each D2D interface on the respective follower circuit dies can be configured to treat a control frame with a Die ID value equal to that of the leader circuit die as a broadcast control frame, such that each router circuit of each D2D interface both processes the control frame and forwards it onward to the next circuit die.

[00241] FIG. 11 shows a graphical representation of a control frame 1100 according to an embodiment. Control frame 1100 carries control information that is not sent via a PIPE Message Bus interface by either MAC circuit 120a or PHY circuit 145a.

[00242] Control frame 1100 is suitable for use in the context of MAC circuit 120a communicating with PHY circuit 145a using PCIe over PIPE. This should not be construed as limiting as it is possible to adapt the frames disclosed herein to other protocols.

[00243] Control frame 1100 is 150 bits in total. The various fields are shown in the table directly below. Payload field in this case is subdivided into four subfields, shown in italic text.

[00244] The frame type field 1105, Die ID field 1120, payload size field 1125 and D2D protocol field 1130 are all the same as their counterparts in frame 800 of FIG. 8. Reference is thus made to the description above in respect of those fields.

[00245] In the embodiment of FIG. 11, payload field 1110 is split into four chunks of 32 bits each. These are referred to above as ‘ctrl’ (control) chunks 0 to 3 1110a - 11 lOd. Each control chunk contains control information that is destined for a particular PHY circuit on the circuit die that is specified in the Die ID field 1120. The PHY ID 1115 field specifies the lowest numbered PHY on the circuit die that the control information is destined for. The first 32 bits of payload 1110, i.e. control chunk 0, are destined for the PHY circuit that corresponds to the value in PHY ID 1115, the next 32 bits of payload 1110, i.e. control chunk 1, are destined for the PHY circuit that corresponds to the value in PHY ID 1115 positively incremented by one, and so on.

[00246] Up to four PHY circuits can be provided with control information per frame 1100. In the case of a circuit die with more than four PHY circuits, multiple control frames like frame 1100 can be sent, with the value of PHY ID 1115 being set some frames to specify the higher-numbered PHY circuits. In the case of a circuit die with fewer than four PHY circuits, one or more of control chunks 1110b, 1110c and 11 lOd can be empty. In this case, the value of the payload size field 1125 is set to reflect this. [00247] Each control chunk 1110a to 11 lOd is itself mapped to a set of control signals for the first protocol, i.e. PIPE in this embodiment. That is, the first bit of every control chunk is mapped to PIPE signal A, the second bit is mapped to PIPE signal B, etc., where A and B are any signals specified in Section 6.1.2 or 6.1.3 of [PIPE Specification], Multi-bit signals can be mapped to a corresponding number of bits of the control chunk. The details of this mapping will depend on the specifics at hand and as such the scope of this disclosure extends to all possible mappings. Each of D2D interface 130 and D2D interface 135 is aware of the mapping, e.g. it is set in firmware or dynamically agreed between the interfaces during start-up or during a configuration mode, such that each D2D interface 130, 135 can construct and decode frames like frame 1000.

[00248] Typically there will be fewer than 32 PIPE signals and so some bits of each control chunk are unused. It will be appreciated that any PIPE signal that is locally generated, e.g. PclkChangeAck, does not need to have a corresponding bit mapped to it in the control chunk as locally generated signals do not need to be sent across the multi -wire D2D communication link 150.

[00249] FIG. 12 shows a graphical representation of a D2D control frame 1200 according to an embodiment. D2D control frame 1200 can send an instruction for D2D interface circuit 135 to act on. This action can be, for example, inserting a specified number of symbols into receive data buffer 405.

[00250] D2D control frame 1200 is suitable for use in the context of MAC circuit 120a communicating with PHY circuit 145a using PCIe over PIPE. This should not be construed as limiting as it is possible to adapt the frames disclosed herein to other protocols.

[00251] D2D control frame 1200 is 150 bits in total. The various fields are shown in the table directly below.

[00252] The frame type field 1205, PHY ID field 1215, Die ID field 1220, payload size 1255 and D2D protocol field 1230 are the same as their respective counterparts in FIG. 11. Reference is thus made to the earlier description in respect of those fields.

[00253] The instruction field 1210 stores the instruction(s) for D2D interface 135. The instructions can be executed by e.g. protocol converter 400. The instructions can be, for example, to insert one or more symbols into receive data buffer 405. The symbol(s) inserted can be the SKP symbol discussed earlier in this specification. As the number of instructions is variable, payload size field 1225 contains a value that specifies the number of bits of the instruction field 1210 contain valid instructions.

[00254] Instruction field 1210 can be split into 8-bit chunks, each chunk containing an instruction relating to a particular PHY circuit. Each chunk can be split into two portions - a 3 -bit PHY identifier portion and a 5 -bit symbol number portion. The PHY identifier portion identifies a specific PHY circuit located on the circuit die specified by Die ID. The symbol number portion specifies a number of symbols, e.g. SKP symbols, that are to be written into a receive data buffer that is associated with the PHY circuit corresponding to the PHY circuit identified by the PHY identifier portion. The value 00010000 thus indicates that 16 SKP symbols should be written into the receive data buffer associated with PHY 0, and the value 10000100 indicates that 4 SKP symbols should be written into the receive data buffer associated with PHY 4. Generally, the value aaabbbbb indicates that bbbbb SKP symbols are to be written into the receive data buffer associated with PHY aaa. This is just one possible format for encoding instructions for D2D interface 135 and is thus not limiting, as other formats can be used instead.

[00255] FIG. 13 shows in schematic form an apparatus 1300 according to an embodiment. In FIG. 13, components that have been described above have been given the same reference sign as assigned above to make clear the relationship between the earlier figures and FIG. 13.

[00256] Apparatus 1300 includes circuit die 105 and circuit die 110 coupled together by multi -wire D2D interface 150. Circuit die 105 is a leader die as it includes first logic circuit 115, which in this embodiment is a switch core. Circuit die 110 is a follower tile. Apparatus 1300 is part of a four-die ring switch topology meaning that there are two additional instances of circuit die 110 that are not shown in FIG. 13 (see FIGs. 15 A and 15B).

[00257] Switch core 115 is coupled to a root complex 1305 via an uplink that in this embodiment comprises two PHY circuits 1310a, 1310b. Each PHY circuit 1310a, 1310b is coupled to a respective MAC circuit 1325a, 1325b that is part of the switch core 115 in this embodiment. The PCIe protocol is used by root complex 1305 to communicate with various endpoints via switch core 115.

[00258] Switch core 115 is also coupled to one or more local endpoints (‘EP’) 1330a - 1330n via local PHY circuits 1315a - 1315f. Each local PHY circuit has a corresponding MAC circuit 1320a - 1320f that is shown as part of switch core 115 in this embodiment. ‘Local’ here indicates that the local endpoints are coupled to the switch core 115 via MAC and PHY circuits that are also located on circuit die 105. That is, multi -wire D2D communication link 150 is not needed when switch core 115 communicates with local endpoints 1330a - 1330n.

[00259] In this embodiment circuit die 105 has eight local PHY circuits and 8 corresponding local MAC circuits, each local PHY circuit coupled to a respective local MAC circuit. Two of the local PHY and MAC circuits (1310a, 1310b & 1325a, 1325b) form an uplink connection to root complex 1305. The remaining six local PHY and MAC circuits are coupled to one or more endpoints 1330a - 133 On and thus form a downlink.

[00260] The number of endpoints is variable because each PHY represents a single lane of PCIe traffic, meaning that up to six lanes are supported by circuit die 105. As the PCIe protocol at the time of writing supports link widths of 1, 2, 4, 8 and 16 lanes, the following endpoint combinations are possible: six single lane endpoints, one 4-lane and one 2-lane endpoint, and three 2-lane endpoints. It is also possible to have some lanes unused, e.g. a single 4-lane endpoint with two lanes unused, but this is inefficient and thus avoided if possible. Nevertheless, any possible combination of endpoints, including combinations that are not currently permitted by the PCIe protocol but which become possible in the future, are within the scope of this disclosure.

[00261] Switch core 115 also includes 24 MAC circuits 1335a - 1335x that are each configured to communicate with a respective PHY circuit that is off-die. That is, each of MAC circuits 1335a - 1335x are configured to communicate with a respective PHY circuit via multi -wire communication link 150. Notably, from the perspective of switch core 115, there is no difference between an on-die and off-die PHY circuit - both appear the same to switch core 115.

[00262] Each MAC circuit 1335a - 1335x is coupled to a respective V-PHY interface circuit 125a - 125x. Each V-PHY interface circuit 125a - 125x is the same as, or similar to, the V-PHY interface circuit 125a described above. Each V-PHY interface circuit 125a - 125x is also coupled to protocol converter 305, the operation of which is described above. Note that although protocol converter 305 is shown in FIG. 13 as a single component, this is not limiting as the function of protocol converter 305 can instead be performed by multiple separate components.

[00263] Protocol converter 305 is coupled to multi-wire D2D interface 150 to enable communication with other circuit dies, e.g. circuit die 110. In this embodiment multi- wire D2D interface 150 comprises, in each direction, 24 data wires and a differential pair of clock wires (shown with dashed lines) to carry FWD CLK, i.e. 26 wires in the transmit direction and 26 wires in the receive direction to give 52 wires in total. A 5b6w ODVS code is used, requiring four transmitters 210a - 210d, 510a - 510d and four receivers 630a - 63 Od, 230a - 23 Od per circuit die, with each transmitter / receiver pair being coupled to a different set of 6 wires of the 48 data-carrying wires. This arrangement can provide a die-to-die bandwidth of 500 Gbps in each direction when FWD CLK has a rate of 6.25GHz. All of the values discussed in this paragraph are purely exemplary and are not limiting on the scope of this disclosure.

[00264] Circuit die 110 is essentially as described and shown in FIGs. 4 and 5, and reference is thus made to this disclosure. Circuit die 110 has eight PHY circuits 145a

- 145h that are coupled to one or more endpoints 1340a - 1340n. These numbers are purely exemplary and are not limiting in scope.

[00265] As the PCIe protocol at the time of writing supports link widths of 1, 2, 4, 8 and 16 lanes, the following endpoint combinations are possible for circuit die 110: eight single lane endpoints, one 4-lane and two 2-lane endpoint, four 2-lane endpoints, two 4-lane endpoints, and one 8-lane endpoint. It is also possible to have some lanes unused, e.g. a single 4-lane endpoint and one single-lane endpoint, with 3 lanes unused, but this is inefficient and thus avoided if possible. Nevertheless, any possible combination of endpoints, including combinations that are not currently permitted by the PCIe protocol but which become possible in the future, are within the scope of this disclosure.

[00266] Each PHY circuit 145a - 145h is coupled to a respective V-MAC interface circuit 140a - 140h, of the type described above. Each V-MAC interface circuit 140a

- 140h is also coupled to protocol converter 400, thus enabling communication with other circuit dies via multi -wire D2D communication link 150.

[00267] Two additional follower circuit dies that are like follower circuit die 110 and which are not shown in FIG. 13 are also present in this embodiment, to make a total of four circuit dies. The circuit dies are coupled in a ring topology as is shown in FIG. 14. Specifically, FIG. 14 shows a leader circuit die 105 and three follower circuit dies each like circuit die 110 coupled in a ring topology. Note that some components have been omitted in FIG. 14 relative to earlier figures for increased clarity. The arrangement of FIG. 14 can be part of a multi-chip module (MCM), e.g. a MCM operating as a highspeed multi-die switch. [00268] As can be seen in FIG. 14, leader circuit die 105 has a transmit D2D link 150a to first follower circuit die 110a and a receive D2D link 150d from third follower circuit die 110c. First follower circuit die 110a has a transmit D2D link 150b to second follower circuit die 110b and a receive D2D link 150a from leader circuit die 105. Second follower circuit die 110b has a transmit D2D link 150c to third follower circuit die 110c and a receive D2D link 150b from first follower circuit die 110a. Third follower circuit die 110c has a transmit D2D link 150d to leader circuit die 105 and a receive D2D link 150c from second follower circuit die 110b. This creates a unidirectional ring topology in which control frames and data frames can be transmitted in a clockwise direction around the ring as it is shown in FIG. 14.

[00269] Each follower circuit die HOa-c has eight PHY circuits located on it. Specifically, first follower circuit die 110a has PHY circuits 145a-h located on it, second follower circuit die 110b has PHY circuits 145i-p located on it, and third follower circuit die 110c has PHY circuits 145q-x located on it. Each PHY circuit corresponds to a lane of a PCIe link.

[00270] It will be appreciated that apparatus 1300 can be modified to a two-die embodiment by omitting second and third follower circuit dies 110b, 110c, and coupling D2D transmitters 510a - 510d to D2D receivers 630a - 63 Od, such that follower circuit die 110 is coupled to leader circuit die 105 in both the transmit and receive directions. Equally, a three-die embodiment is possible by omitting third follower circuit die 110c. Alternatively, additional follower circuit dies like circuit die 110 can be added to that shown in FIG. 14 to create arrangements with four or more follower tiles. Further modification can be made to any of these embodiments, or the embodiment of FIG. 14, by adjusting the D2D transmitter and receiver configuration such that each group of four instead contains two transmitters and two receivers. This allows for a bi-directional ring structure in which each circuit die is directly coupled to its neighbouring dies in both the transmit and receive direction. This bidirectional ring has lower latency than the unidirectional ring because at most a data or control frame will need to make two D2D ‘hops’ before getting to its destination die, whereas in the unidirectional ring three hops can be required in the worst case. Additional clock wires can be provided to allow for two independent FWC CLK signals to be sent, one associated with each direction of the ring.

[00271] A point-to-point arrangement is also contemplated in which each circuit die is coupled to every other circuit die via a multi-wire D2D communication link. In the case of the embodiment of FIG. 14, up to four follower circuit dies can be present for a total of five circuit dies including the leader circuit die, with each circuit die coupled directly to every other circuit die by a respective one of the four D2D transmitter / receiver pairs present on each circuit die. Additional clock wires can be provided to allow for up to four independent FWC CLK signals to be sent, one associated with each point-to-point link. This embodiment reduces latency relative to both the unidirectional ring and bidirectional ring topology. More D2D transmitter / receiver pairs can be provided to increase the circuit die count beyond five if desired. It is also possible to have some D2D links with greater bandwidth than others, e.g. in a modification of the embodiment of FIG. 14 with one leader die and three follower dies, each circuit die has one ‘spare’ D2D transmitter / receiver pair that can be used as a second link to one particular circuit die to double the bandwidth between that pair of circuit dies. This could be particularly useful in a configuration where a given endpoint has greater bandwidth requirements than others, with the double bandwidth D2D link being provided between e.g. the leader circuit die and the circuit die coupled to the high bandwidth-requiring endpoint.

[00272] The apparatus of FIG. 14 can thus be provided in various forms having a different total number of circuit dies. A two-die embodiment provides an apparatus comprising a first circuit die 105 and a second circuit die 110. The first circuit die comprises a first logic circuit 115, e.g. a switch core, having a plurality of MAC circuits 1335a-h that are each configured to implement at least a portion of a PIPE standard to interoperate with a PHY interface. That is, each of the plurality of MAC circuits is capable of communicating with a generic PHY interface using the PIPE protocol. These MAC circuits communicate with non-local PHY circuits, i.e. PHY circuits that are remote as they are on another circuit die (in this embodiment, circuit die 110). The first circuit die 105 can also include additional MAC circuits 1320a-f, 1325a, 1325b that communicate with local PHY circuits 1315a-1315f, 1310a, 1310b.

[00273] The apparatus also includes a first D2D interface circuit 130 configured to communicate with a second circuit die 110. The apparatus further includes a plurality of virtual PIPE PHY interface circuits 125a-h connected between respective ones of the plurality of MAC circuits 1335a-h and the first D2D interface 130, each of the plurality of virtual PIPE PHY interface circuits 125a-h configured to emulate a PIPE PHY circuit by conveying PIPE signals between respective ones of the MAC circuits 1335a-h and the D2D interface circuit 130. [00274] The apparatus further includes a second circuit die 110 having a plurality of PHY circuits 145a-h that are each configured to implement at least a portion of a PIPE standard. The second circuit die 110 also includes a second D2D interface circuit 135 configured to communicate with the first circuit die 105. The second circuit die 110 further includes a plurality of virtual PIPE MAC interface circuits 140a-h connected between respective ones of the plurality of PHY circuits 145a-h and the second D2D interface 135, each of the plurality of virtual PIPE MAC interface circuits 140a-h configured to emulate a PIPE MAC circuit by conveying PIPE signals between respective ones of the PHY circuits 145a-h and the D2D interface circuit 135 and configured to locally generate one or more control signals for controlling respective ones of the plurality of PHY circuits 145a-h.

[00275] Each of the plurality of virtual PIPE MAC interface circuits 140a-h can be configured to locally generate one or more control signals for controlling respective ones of the plurality of PHY circuits by being configured to generate a receiver standby control signal (e.g. the RxStandby signal discussed above).

[00276] The receiver standby signal can be generated responsive to detection of a receiver electrical idle signal (e.g. the RxElecIdle signal discussed above) provided by a control channel of the respective PHY circuit of the plurality of PHY circuits 145a-h. [00277] The receiver standby signal can be generated responsive to the detection of the receiver electrical idle signal and responsive to detection of one or more transmitter electrical idle symbols (e.g. EIOS symbols) provided by a data channel of the respective PHY circuit of the plurality of PHY circuits. That is, the receiver standby signal can be generated only once both a receiver electrical idle signal and receiver electrical idle symbols have been detected (e.g. RxElecIdle signal as a control signal and EIOS in data received from the receiver.

[00278] The first D2D interface circuit 130 can comprise a first D2D transmitter 210a and a first D2D receiver 630a, and the second D2D interface circuit 135 can comprise a second D2D transmitter 510a and a second D2D receiver 230a. Additional transmitters and/or receivers can also be present on each circuit die, e.g. four transmitter 210a-d, 510a-d and four receivers 630a-d, 230a-d. The number of transmitter and receivers can be selected based on the bandwidth required for the multi-wire D2D communication link 150. This disclosure is thus not limited to any particular number of transmitters and receivers. [00279] The first D2D transmitter 210a can be communicatively coupled to the second D2D receiver 230a and the first D2D receiver 630a can be communicatively coupled to the second D2D transmitter 210a. A corresponding relationship can be established for any additional transmitter/receiver pairs present on the circuit dies 105, 110. This configuration enables a bi-directional D2D link between two circuit dies, leader circuit die 105 and follower circuit die 110.

[00280] As touched on above, embodiments with more than two circuit dies are also contemplated. In a three-die embodiment, the apparatus described above further comprises a third circuit die 110b having a plurality of PHY circuits that are each configured to implement at least a portion of a PIPE standard, a third D2D interface circuit configured to communicate with the second circuit die 110; and a plurality of virtual PIPE MAC interface circuits connected between respective ones of the plurality of PHY circuits and the third D2D interface, each of the plurality of virtual PIPE MAC interface circuits configured to emulate a PIPE MAC circuit by conveying PIPE signals between respective ones of the PHY circuits and the third D2D interface circuit and configured to locally generate one or more control signals for controlling respective ones of the plurality of PHY circuits.

[00281] In this three-die embodiment, the third D2D interface circuit comprises a third D2D transmitter and a third D2D receiver, wherein the first D2D transmitter is communicatively coupled to the second D2D receiver and the second D2D transmitter is communicatively coupled to the third D2D receiver. A communication path is thus established between the three circuit dies; leader circuit die 105 is coupled to the first follower circuit die 110a in the transmit direction, and the first follower circuit die 110a is coupled to the second follower circuit die 110b in the transmit direction. The loop can be closed by coupling the third D2D transmitter on the second follower circuit die 110b to the first D2D receiver on the leader circuit die 105, to create a unidirectional ring topology.

[00282] The three-die embodiment can be extended to a four-die embodiment as follows. The apparatus can further comprise a fourth circuit die 110c having a plurality of PHY circuits that are each configured to implement at least a portion of a PIPE standard; a fourth D2D interface circuit configured to communicate with the third circuit die; and, a plurality of virtual PIPE MAC interface circuits connected between respective ones of the plurality of PHY circuits and the fourth D2D interface, each of the plurality of virtual PIPE MAC circuits configured to emulate a PIPE MAC circuit by conveying PIPE signals between respective ones of the PHY circuits and the third D2D interface circuit and configured to locally generate one or more control signals for controlling respective ones of the plurality of PHY circuits

[00283] In the four-die embodiment, the coupling between dies is as shown in FIG. 14. The fourth D2D interface circuit comprises a fourth D2D transmitter and a fourth D2D receiver, and wherein the third D2D transmitter is communicatively coupled to the fourth D2D receiver and the fourth D2D transmitter is communicatively coupled to the first D2D receiver. This creates a four-die unidirectional ring structure.

[00284] Irrespective of the total number of circuit dies, each D2D interface circuit can be configured to receive a clock signal based on a reference clock signal provided by a common reference clock. The common reference clock can be located on one of the circuit dies or it can be located on another die that is not one of the circuit dies. The use of a common reference clock creates a mesochronous clock network in which each D2D interface circuit is clocked at the same frequency but a different phase relative to other D2D interface circuits on different circuit dies. This is advantageous as it allows advanced clocking techniques that deliberately modulate a clock frequency, e.g. spread spectrum clocking, to be used because the frequency modulation is applied to all D2D interfaces of all circuit dies simultaneously.

[00285] FIGs. 15A and 15B illustrate one way in which data frames and control frames can be routed around the four-die unidirectional ring shown in FIG. 14. This routing technique is just one way in which this can be achieved and thus is not limiting on the scope of this disclosure.

[00286] FIG. 15A illustrates the path taken by data frames and control frames that are travelling downstream; that is, from root complex 1305 / switch core 115 to an endpoint (not shown in FIGs. 15A and 15B). FIG. 15B illustrates the path taken by data frames and control frames that are travelling upstream; that is, to root complex 1305 / switch core 115 from an endpoint. In both figures, some components and labels have been omitted relative to earlier figures in the interests of clarity. As earlier in this specification, components have been assigned the same reference sign as in other figures for continuity and ease of understanding.

[00287] Referring first to FIG. 15 A, the downlink MAC circuits of switch core 115 have been split into two groups. The first group is MAC circuits 1320a-f that are coupled to local PHY circuits 1315a-f, respectively. These MAC circuits 1320a-f are also referred to as ‘ports 0-5’ of the switch core 115. The coupling between these MAC circuits and PHY circuits does not involve the multi-wire D2D communication link as they are located on the same circuit die (leader circuit die 105). Thus, this coupling is conventional and not discussed further here.

[00288] The second group is MAC circuits 1335a-x are coupled to off-die PHY circuits. Specifically, MAC circuits 1335a-h (ports 6-13) are respectively coupled to PHY circuits 145a-h on first follower circuit die 110, MAC circuits 1335i-p (ports 14-21) are respectively coupled to PHY circuits 145i-p on second follower circuit die 110’, and MAC circuits 1335q-x (ports 22-29) are respectively coupled to PHY circuits 145q-x on third follower circuit die 110”.

[00289] Switch core 115 is configured to enumerate each of PHY circuits 145a-x in the normal manner for PCIe enumeration. For the purposes of this disclosure, the enumeration process creates a mapping between each port of switch core 115 and a particular one of the PHY circuits 145a-x. This enables the switch core 115 to route PCIe packets by selecting the port corresponding to the PHY circuit that the packet is destined for.

[00290] Protocol converter 305 can be configured to address a control frame or data frame based on the port of switch core 115 that the payload control information or data, respectively, is transmitted on. Referring to FIGs. 8-11, the value stored in the Die ID field of each frame shown in these figures can be set based on the port that the payload control information or data was transmitted on by switch core 115.

[00291] Thus, for example, in the configuration of FIG. 15 A, data transmitted over port 15 can be inserted into payload field 810 of a data frame with a Die ID value corresponding to circuit die 110’. Similarly, control information transmitted over port 28 can be inserted into payload field 910 of a control frame with a Die ID value corresponding to circuit die 110”. These examples are provided purely to aid in understanding and are thus not limiting on the scope of this disclosure.

[00292] Other routing-type fields such as Link ID 815, PHY ID 910b, PHY IDs 1010b- lOlOn, PHY ID 1115 and PHY ID 1215 can be populated in a similar manner based on the port that the corresponding data or control information was transmitted over. Protocol converter 305 can be configured to access PCIe enumeration information and/or other such PCIe routing-type information to assist in the population of these fields.

[00293] As can be seen in FIG. 15 A, each protocol converter 400 in the follower circuit dies 110, 110’ and 110” is configured to use a respective routing circuit (not shown) to determine whether to send an inbound control frame or data frame to a V-MAC circuit that is located to the circuit die or to forward it on to the next circuit die in the ring. So, for example, an incoming data/control frame to follower circuit die 110 that has an identifier associated with follower circuit die 110”’ will be transmitted directly to follower circuit die 100’, this being the next circuit die in the ring. The same will happen at follower circuit die 100’, as protocol converter 400’ recognises that the data/control frame is destined for follower circuit die 110”. This routing process adds relatively little latency compared with processing by a local switch core located on a follower circuit die. In one case, a latency of approximately 7ns for the routing process described above was measured, with a latency for processing a PCIe packet by a local switch core being at least 20ns. Thus, this routing process reduced latency by approximately 65% in this case. This value will vary according to the specifics of the apparatus at hand, but in general it is expected that this routing process will outperform switch core routing at least because the switch core must do PCIe packet decoding and re-encoding whereas routing by a control/data frame circuit die ID field does not require any PCIe decoding and re-encoding.

[00294] FIG. 15B illustrates the reverse to FIG. 15 A, i.e. data and control frames flowing in the receive direction, from PHY circuits 145a-x towards switch core 115. Protocol converter 400, 400’ and 400” can each be configured to set a Die ID field value for control/data frames generated from control information/data incoming via PHY circuits 145a-x to a value corresponding to leader circuit die 105. All incoming data and control information is thus sent to switch core 115 for onward routing, e.g. to root complex 1305 or to one of PHYs 145a-x.

[00295] FIG. 16 sets out a method according to an embodiment. The method can be performed by the components of a follower circuit die like circuit die 110. The method can be a method of locally synchronizing data and control information for interoperation between a multi-wire D2D communication link and a PHY circuit.

[00296] In element 1600, a PHY circuit on a first circuit die sends and receives serial data. The PHY circuit can be PHY circuit 145a and the first circuit die can be circuit die 110.

[00297] In element 1605, a D2D interface circuit on the first circuit die sends and receives time-division multiplexed data frames and control frames over a multi-wire D2D communication link that is connected to the D2D interface circuit. The sending and receiving including exchanging data and control information associated with the data frames and the control frames. The D2D interface circuit can be D2D interface circuit 135, the data frames and control frames can be those described above in connection with FIGs. 8 to 12 and the multi-wire D2D communication link can be multi -wire D2D communication link 150.

[00298] In element 1610, a V-MAC interface circuit on the first circuit die locally synchronizes the data and the control information for interoperation with the PHY circuit. The V-MAC interface circuit is connected to the D2D interface circuit and connected to the PHY circuit. The V-MAC interface circuit can be V-MAC interface circuit 140a.

[00299] FIG. 17 sets out another method according to an embodiment. The method can be performed by the components of a leader circuit die like circuit die 105. The method can be a method of locally synchronizing data and control information for interoperation between a multi-wire D2D communication link and a MAC circuit.

[00300] In element 1700, a MAC circuit on a first circuit die sends and receives data. The MAC circuit can be MAC circuit 120a and the first circuit die can be circuit die 105.

[00301] In element 1705, a D2D interface circuit on the first circuit die sends and receives time-division multiplexed data frames and control frames over a multi-wire D2D communication link that is connected to the D2D interface circuit. The sending and receiving including exchanging data and control information associated with the data frames and the control frames. The D2D interface circuit can be D2D interface circuit 130, the data frames and control frames can be those described above in connection with FIGs. 8 to 12 and the multi-wire D2D communication link can be multi -wire D2D communication link 150.

[00302] In element 1710, a V-PHY interface circuit on the first circuit die locally synchronizes the data and the control information for interoperation with the MAC circuit. The V-PHY interface circuit is connected to the D2D interface circuit and connected to the MAC circuit. The V-PHY interface circuit can be V-PHY interface circuit 125a.

[00303] FIG. 18 sets out yet another method according to an embodiment. The method can be performed by the combination of leader circuit die 105 and follower circuit die 110. The method can be a method of communicating between circuit dies using PIPE compliant signals. [00304] In element 1800, one of a plurality of MAC circuits located on a first circuit die transmits a PIPE compliant signal. The first circuit die can be circuit die 105 and the MAC circuits can be MAC circuits 120a - 120n.

[00305] In element 1805, one of a plurality of virtual PIPE PHY interface circuits located on the first circuit die and connected between respective ones of the plurality of MAC interface circuits and a first D2D interface circuit located on the first circuit die receives the PIPE compliant signal. The virtual PIPE PHY interface circuits can be V- PHY interface circuits 125a - 125n and the first D2D interface circuit can be D2D interface 130.

[00306] In element 1810, one of the plurality of virtual PIPE PHY interface circuits generates a D2D signal based on the PIPE compliant signal and transmits the D2D signal to the first D2D interface circuit and in element 1815, the first D2D interface circuit transmits the D2D signal to a second D2D interface circuit located on a second circuit die. The second D2D interface circuit can be D2D interface circuit 135 and the second circuit die can be circuit die 110.

[00307] In element 1820, the second D2D interface circuit transmits the D2D signal to one of a plurality of virtual PIPE MAC interface circuits located on the second circuit die. The virtual PIPE MAC interface circuits are each coupled between the second D2D interface circuit and respective ones of a plurality of PHY circuits located on the second circuit die. The plurality of virtual PIPE MAC interface circuits can be V-MAC interface circuits 140a - 140n and the plurality of PHY circuits can be PHY circuits 145a - 145n.

[00308] In element 1825, the one of the plurality of virtual PIPE MAC interface circuits generates a second PIPE compliant signal based on the D2D signal and transmits the second PIPE compliant signal to one of the plurality of PHY circuits.

[00309] In element 1830, the one of the plurality of virtual PIPE MAC interface circuits locally generates one or more control signals for controlling the one of the plurality of PHY circuits. The one or more control signals can be any of the locally generated control signals discussed above, such as PclkChangeAck, EIOS / EIEOS symbols and the TxElecIdle signal level. The one or more control signals can be based on the D2D signal and/or the second PIPE compliant signal.

[00310] In element 1835, the one of the plurality of virtual PIPE MAC interface circuits transmits the one or more control signals to the one of the plurality of PHY interface circuits. [00311] In addition to the embodiments described above, this disclosure provides the following additional embodiments.

[00312] In an embodiment, an apparatus is provided, the apparatus comprising a circuit die comprising a die-to-die (D2D) transmitter, a D2D receiver and one or more physical layer (PHY) circuits, the D2D receiver coupled to a first end of a first multi-wire D2D communication link and the D2D transmitter coupled to a first end of a second multiwire D2D communication link; and a router circuit located on the circuit die and coupled to the D2D receiver, the router circuit configured to route an incoming frame received by the D2D receiver from the first multi-wire D2D communication link to at least one of: a PHY circuit of the one or more PHY circuits; and the second D2D transmitter, the routing based on a die identifier value stored in a Die ID field of the frame.

[00313] The router circuit can be configured to route a frame having a die identifier that is different from an identifier associated with the circuit die to the second D2D transmitter.

[00314] The router circuit can be configured to route a frame having a die identifier that is the same as an identifier associated with the circuit die to the PHY circuit of the one or more PHY circuits.

[00315] The router circuit can be configured to route a frame having a die identifier that is equal to a broadcast die identifier to both the PHY circuit of the one or more PHY circuits and the second D2D transmitter.

[00316] The circuit die, D2D transmitter, D2D receiver, one or more PHY circuits, first multi-wire D2D communication link, router circuit and incoming frame can be any of the correspondingly-named entities discussed in this specification. This embodiment is combinable with any feature described in any preceding embodiment.

[00317] In another embodiment, an apparatus is provided, the apparatus comprising: a first circuit die comprising: a receiver coupled to a first die-to-die (D2D) communication link that extends off-die, the receiver configured to receive control frames containing control information relating to a first protocol and data frames containing data formatted according to the first protocol via the first D2D communication link; a transmitter coupled to a second D2D communication link that extends off-die, the transmitter configured to transmit control frames and data frames via the second D2D communication link; and a protocol converter coupled to the receiver and the transmitter, the protocol converter comprising a router circuit configured to selectively forward received control frames and data frames to the transmitter based on an address field of the received control frames and data frames, the address field formatted according to a second, different protocol.

[00318] The first protocol can be a PIPE protocol and the second protocol can be an ODVS protocol.

[00319] The router circuit can be configured to forward received control frames and data frames to the transmitter when the address field contains a die identification value that is not the same as a die identification value associated with the first circuit die.

[00320] The router circuit can be configured to transmit control frames and data frames to a physical layer (PHY) circuit located on the first circuit die when the address field contains a die identification value that is either the same as a die identification value associated with the first circuit die or is equal to a broadcast address value.

[00321] The first circuit die, receiver, first D2D communication link, control frames, data frames, transmitter, second D2D communication link, protocol converter and router circuit can be any of the correspondingly-named entities discussed in this specification. This embodiment is combinable with any feature described in any preceding embodiment.

[00322] It will be apparent to a person skilled in the art having the benefit of the present disclosure that various modifications, extensions, substitutions and the like to the subject matter described herein are possible. Such changes are also within the scope of this disclosure. It is also noted that, where method steps are described, these steps can be performed in any order unless expressly stated otherwise.

Claims

1. An apparatus comprising: a serialiser/deserialiser (SERDES) physical layer (PHY) circuit on a first circuit die, the PHY circuit configured to send and receive serial data; a die-to-die (D2D) interface circuit on the first circuit die, configured to send and receive time-division multiplexed data frames and control frames over a multi-wire D2D communication link, and further configured to exchange data and control information associated with the data frames and the control frames; and a virtual media access control (V-MAC) interface circuit on the first circuit die, the V-MAC interface circuit connected to the D2D interface circuit and connected to the PHY circuit, the V-MAC interface circuit configured to locally synchronize the data and the control information for interoperation with the PHY circuit.

2. The apparatus of any one of claim 1, wherein the V-MAC interface circuit further comprises a receive data buffer, the D2D interface circuit further configured to write data extracted from one or more received data frames into the receive data buffer at a first rate corresponding to a rate of a D2D interface circuit clock signal generated by the D2D interface circuit.

3. The apparatus of claim 2, wherein the D2D interface circuit is further configured to identify a received control frame containing an instruction to generate one or more symbols for insertion into the receive data buffer and to responsively generate and insert said one or more symbols into the receive data buffer.

4. The apparatus of claim 2, wherein the V-MAC interface circuit further includes an electrical idle symbol detection circuit configured to monitor the data extracted from the one or more received data frames for the presence of an electrical idle symbol and an exit electrical idle symbol, the electrical idle symbol and the exit electrical idle symbol respectively causing a transmitter of the PHY circuit to turn off and on.

5. The apparatus of claim 4 wherein the electrical idle symbol detection circuit is configured to, responsive to an electrical idle symbol being read out of the data buffer, transmit a corresponding electrical idle control signal to the PHY circuit.

6. The apparatus of claim 5, wherein the electrical idle symbol detection circuit is configured to, responsive to an exit electrical idle symbol being read out of the data buffer, transmit a corresponding exit electrical idle control signal to the PHY circuit.

7. The apparatus of claim 1, wherein the V-MAC interface circuit is further configured to monitor status information provided by the PHY circuit to detect a receiver electrical idle signal and, responsive to the detection of a receiver electrical idle signal, transmit a standby command to the PHY circuit.

8. The apparatus of claim 1, wherein the data and the control information are in a format that conforms to at least one version of a PHY Interface for Peripheral Component Interconnect Express (PIPE) standard.

9. A method comprising: sending and receiving serial data using a serialiser/deserialiser (serdes) physical layer (PHY) circuit on a first circuit die; sending and receiving, by a die-to-die (D2D) interface circuit on the first circuit die, time-division multiplexed data frames and control frames over a multi-wire D2D communication link that is connected to the D2D interface circuit, the sending and receiving including exchanging data and control information associated with the data frames and the control frames; and locally synchronizing the data and the control information for interoperation with the PHY circuit using a virtual media access control (V-MAC) interface circuit on the first circuit die, the V-MAC interface circuit connected to the D2D interface circuit and connected to the PHY circuit.

10. The method of claim 9, wherein the D2D interface circuit includes a router circuit, the method further comprising the router circuit detecting routing data in inbound data frames of the data frames and inbound control frames of the control frames, the inbound data frames and inbound control frames received on the receive link, and responsively retransmitting the inbound data frames and inbound control frames on the transmit link.

11. The method of 1, further comprising the D2D interface circuit writing data extracted from one or more received data frames into a receive data buffer of the V- MAC interface circuit at a first rate corresponding to a rate of a D2D interface circuit clock signal generated by the D2D interface circuit.

12. The method of claim 11, further comprising the D2D interface circuit identifying a received control frame containing an instruction to generate one or more symbols for insertion into the receive data buffer and responsively generating and inserting said one or more symbols into the receive data buffer.

13. The method of claim 9, further comprising monitoring, by an electrical idle symbol detection circuit of the V-MAC interface circuit, the data extracted from the one or more received data frames for the presence of an electrical idle symbol and an exit electrical idle symbol, the electrical idle symbol and the exit electrical idle symbol respectively causing a transmitter of the PHY circuit to turn off and on.

14. The method of claim 13, further comprising the electrical idle symbol detection circuit, responsive to detection of an electrical idle symbol in the data, setting at least one control bit in the data buffer to a first value to indicate detection of the electrical idle symbol, the at least one control bit being part of a data word in the data buffer that also includes the electrical idle symbol.

15. The method of claim 9, wherein the data and the control information are in a format that conforms to at least one version of a PHY Interface for Peripheral Component Interconnect Express (PIPE) standard.