1. Tx Path QoS Control¶

The following files are provided to tune the QoS of the tx path ofthe PCIe core.

• qos_tx_cpl: weight of Tx completion TLPs

• qos_tx_np: weight of Tx non-posted TLPs

• qos_tx_p: weight of Tx posted TLPs

The weight influences the proportion of certain packets on the PCIe link.For example, for the storage scenario, increase the proportionof the completion packets on the link to enhance the performance asmore completions are consumed.

The available tune data of these events is [0, 1, 2].Writing a negative value will return an error, and out of rangevalues will be converted to 2. Note that the event value justindicates a probable level, but is not precise.

2. Tx Path Buffer Control¶

Following files are provided to tune the buffer of tx path of the PCIe core.

• rx_alloc_buf_level: watermark of Rx requested

• tx_alloc_buf_level: watermark of Tx requested

These events influence the watermark of the buffer allocated for eachtype. Rx means the inbound while Tx means outbound. The packets willbe stored in the buffer first and then transmitted either when thewatermark reached or when timed out. For a busy direction, you shouldincrease the related buffer watermark to avoid frequently posting andthus enhance the performance. In most cases just keep the default value.

The available tune data of above events is [0, 1, 2].Writing a negative value will return an error, and out of rangevalues will be converted to 2. Note that the event value justindicates a probable level, but is not precise.

1. Filter¶

The TLP headers to trace can be filtered by the Root Ports or the Requester IDof the Endpoint, which are located on the same core of the PTT device. You canset the filter by specifying thefilter parameter which is required to startthe trace. The parameter value is 20 bit. Bit 19 indicates the filter type.1 for Root Port filter and 0 for Requester filter. Bit[15:0] indicates thefilter value. The value for a Root Port is a mask of the core port id which iscalculated from its PCI Slot ID as (slotid & 7) * 2. The value for a Requesteris the Requester ID (Device ID of the PCIe function). Bit[18:16] is currentlyreserved for extension.

For example, if the desired filter is Endpoint function 0000:01:00.1 the filtervalue will be 0x00101. If the desired filter is Root Port 0000:00:10.0 thenthen filter value is calculated as 0x80001.

The driver also presents every supported Root Port and Requester filter throughsysfs. Each filter will be an individual file with name of its related PCIedevice name (domain:bus:device.function). The files of Root Port filters areunder $(PTT PMU dir)/root_port_filters and files of Requester filtersare under $(PTT PMU dir)/requester_filters.

Note that multiple Root Ports can be specified at one time, but only oneEndpoint function can be specified in one trace. Specifying both Root Portand function at the same time is not supported. Driver maintains a list ofavailable filters and will check the invalid inputs.

The available filters will be dynamically updated, which means you will alwaysget correct filter information when hotplug events happen, or when you manuallyremove/rescan the devices.

2. Type¶

You can trace the TLP headers of certain types by specifying thetypeparameter, which is required to start the trace. The parameter value is8 bit. Current supported types and related values are shown below:

• 8’b00000001: posted requests (P)

• 8’b00000010: non-posted requests (NP)

• 8’b00000100: completions (CPL)

You can specify multiple types when tracing inbound TLP headers, but can onlyspecify one when tracing outbound TLP headers.

3. Direction¶

You can trace the TLP headers from certain direction, which is relativeto the Root Port or the PCIe core, by specifying thedirection parameter.This is optional and the default parameter is inbound. The parameter valueis 4 bit. When the desired format is 4DW, directions and related valuessupported are shown below:

• 4’b0000: inbound TLPs (P, NP, CPL)

• 4’b0001: outbound TLPs (P, NP, CPL)

• 4’b0010: outbound TLPs (P, NP, CPL) and inbound TLPs (P, NP, CPL B)

• 4’b0011: outbound TLPs (P, NP, CPL) and inbound TLPs (CPL A)

When the desired format is 8DW, directions and related values supported areshown below:

• 4’b0000: reserved

• 4’b0001: outbound TLPs (P, NP, CPL)

• 4’b0010: inbound TLPs (P, NP, CPL B)

• 4’b0011: inbound TLPs (CPL A)

Inbound completions are classified into two types:

• completion A (CPL A): completion of CHI/DMA/Native non-posted requests, except for CPL B

• completion B (CPL B): completion of DMA remote2local and P2P non-posted requests

4. Format¶

You can change the format of the traced TLP headers by specifying theformat parameter. The default format is 4DW. The parameter value is 4 bit.Current supported formats and related values are shown below:

• 4’b0000: 4DW length per TLP header

• 4’b0001: 8DW length per TLP header

The traced TLP header format is different from the PCIe standard.

When using the 8DW data format, the entire TLP header is logged(Header DW0-3 shown below). For example, the TLP header for MemoryReads with 64-bit addresses is shown in PCIe r5.0, Figure 2-17;the header for Configuration Requests is shown in Figure 2.20, etc.

In addition, 8DW trace buffer entries contain a timestamp andpossibly a prefix for a PASID TLP prefix (see Figure 6-20, PCIe r5.0).Otherwise this field will be all 0.

The bit[31:11] of DW0 is always 0x1fffff, which can beused to distinguish the data format. 8DW format is like

bits [                 31:11                 ][       10:0       ]     |---------------------------------------|-------------------| DW0 [                0x1fffff               ][ Reserved (0x7ff) ] DW1 [                       Prefix                              ] DW2 [                     Header DW0                            ] DW3 [                     Header DW1                            ] DW4 [                     Header DW2                            ] DW5 [                     Header DW3                            ] DW6 [                   Reserved (0x0)                          ] DW7 [                        Time                               ]

When using the 4DW data format, DW0 of the trace buffer entrycontains selected fields of DW0 of the TLP, together with atimestamp. DW1-DW3 of the trace buffer entry contain DW1-DW3directly from the TLP header.

4DW format is like

bits [31:30] [ 29:25 ][24][23][22][21][    20:11   ][    10:0    ]     |-----|---------|---|---|---|---|-------------|-------------| DW0 [ Fmt ][  Type  ][T9][T8][TH][SO][   Length   ][    Time    ] DW1 [                     Header DW1                            ] DW2 [                     Header DW2                            ] DW3 [                     Header DW3                            ]

5. Memory Management¶

The traced TLP headers will be written to the memory allocatedby the driver. The hardware accepts 4 DMA address with same size,and writes the buffer sequentially like below. If DMA addr 3 isfinished and the trace is still on, it will return to addr 0.

+->[DMA addr 0]->[DMA addr 1]->[DMA addr 2]->[DMA addr 3]-++---------------------------------------------------------+

Driver will allocate each DMA buffer of 4MiB. The finished bufferwill be copied to the perf AUX buffer allocated by the perf core.Once the AUX buffer is full while the trace is still on, driverwill commit the AUX buffer first and then apply for a new one withthe same size. The size of AUX buffer is default to 16MiB. User canadjust the size by specifying the-m parameter of the perf command.

6. Decoding¶

You can decode the traced data withperf report -D command (currentlyonly support to dump the raw trace data). The traced data will be decodedaccording to the format described previously (take 8DW as an example):

[...perf headers and other information]. ... HISI PTT data: size 4194304 bytes.  00000000: 00 00 00 00                                 Prefix.  00000004: 01 00 00 60                                 Header DW0.  00000008: 0f 1e 00 01                                 Header DW1.  0000000c: 04 00 00 00                                 Header DW2.  00000010: 40 00 81 02                                 Header DW3.  00000014: 33 c0 04 00                                 Time.  00000020: 00 00 00 00                                 Prefix.  00000024: 01 00 00 60                                 Header DW0.  00000028: 0f 1e 00 01                                 Header DW1.  0000002c: 04 00 00 00                                 Header DW2.  00000030: 40 00 81 02                                 Header DW3.  00000034: 02 00 00 00                                 Time.  00000040: 00 00 00 00                                 Prefix.  00000044: 01 00 00 60                                 Header DW0.  00000048: 0f 1e 00 01                                 Header DW1.  0000004c: 04 00 00 00                                 Header DW2.  00000050: 40 00 81 02                                 Header DW3[...]