| Skip Navigation Links | |
| Exit Print View | |
 | man pages section 1M: System Administration Commands Oracle Solaris 11 Information Library |
| Skip Navigation Links | |
| Exit Print View | |
| man pages section 1M: System Administration Commands Oracle Solaris 11 Information Library |
- create fault injection test harness error specifications
th_define [-nname-iinstance|-Ppath] [-aacc_types] [-rreg_number] [-loffset [length]] [-ccount [failcount]] [-ooperator [operand]] [-facc_chk] [-wmax_wait_period [report_interval]]
or
th_define [-nname-iinstance|-Ppath] [-a log [acc_types] [-rreg_number] [-loffset [length]]] [-ccount [failcount]] [-scollect_time] [-ppolicy] [-xflags] [-Ccomment_string] [-efixup_script [args]]
or
th_define [-h]
Theth_define utility provides an interface to thebus_ops fault injectionbofidevice driver for defining error injection specifications (referred to as errdefs). Anerrdef corresponds to a specification of how to corrupt a device driver's accessesto its hardware. The command line arguments determine the precise nature ofthe fault to be injected. If the supplied arguments define a consistenterrdef, theth_define process will store the errdef with thebofi driverand suspend itself until the criteria given by the errdef become satisfied (inpractice, this will occur when the access counts go to zero).
You use theth_manage(1M) command with thestart option to activate theresulting errdef. The effect ofth_manage with thestart option is that thebofi driver acts upon the errdef by matching the number of hardwareaccesses—specified incount, that are of the type specified inacc_types, madeby instance numberinstance—of the driver whose name isname, (or by thedriver instance specified bypath) to the register set (or DMA handle)specified byreg_number, that lie within the rangeoffset tooffset + lengthfrom the beginning of the register set or DMA handle. It thenappliesoperator andoperand to the nextfailcount matching accesses.
Ifacc_types includeslog,th_define runs in automatic test script generation mode,and a set of test scripts (written in the Korn shell) iscreated and placed in a sub-directory of the current directory with thename<driver>.test.<id> (for example,glm.test.978177106). A separate, executable script is generated foreach access handle that matches the logging criteria. The log of accessesis placed at the top of each script as a record of thesession. If the current directory is not writable, file output is writtento standard output. The base name of each test file is thedriver name, and the extension is a number that discriminates between differentaccess handles. A control script (with the same name as the createdtest directory) is generated that will run all the test scripts sequentially.
Executing the scripts will install, and then activate, the resulting error definitions.Error definitions are activated sequentially and the driver instance under test istaken offline and brought back online before each test (refer to the-e option for more information). By default, logging applies to allPIO accesses,all interrupts, and all DMA accesses to and from areas mapped forboth reading and writing. You can constrain logging by specifying additionalacc_types,reg_number,offset andlength. Logging will continue forcount matching accesses, with anoptional time limit ofcollect_time seconds.
Either the-n or-P option must be provided. The other optionsare optional. If an option (other than-a) is specified multiple times,only the final value for the option is used. If an optionis not specified, its associated value is set to an appropriate default, whichwill provide maximal error coverage as described below.
The following options are available:
Specify the name of the driver to test. (String)
Test only the specified driver instance (-1 matches all instances of driver). (Numeric)
Specify the full device path of the driver to test. (String)
Test only the given register set or DMA handle (-1 matches all register sets and DMA handles). (Numeric)
Only the specified access types will be matched. Valid values for theacc_types argument arelog,pio,pio_r,pio_w,dma,dma_r,dma_w andintr. Multiple access types, separated by spaces, can be specified. The default is to match all hardware accesses.
Ifacc_types is set tolog, logging will match allPIO accesses, interrupts and DMA accesses to and from areas mapped for both reading and writing.log can be combined with otheracc_types, in which case the matching condition for logging will be restricted to the specified addionalacc_types. Note thatdma_r will match only DMA handles mapped for reading only;dma_w will match only DMA handles mapped for writing only;dma will match only DMA handles mapped for both reading and writing.
Constrain the range of qualifying accesses. Theoffset andlength arguments indicate that any access of the type specified with the-a option, to the register set or DMA handle specified with the-r option, lie at leastoffset bytes into the register set or DMA handle and at mostoffset +length bytes into it. The default foroffset is 0. The default forlength is the maximum value that can be placed in anoffset_t C data type (seetypes.h). Negative values are converted into unsigned quantities. Thus,th_define-l 0-1 is maximal.
Wait forcount number of matching accesses, then apply an operator and operand (see the-o option) to the nextfailcount number of matching accesses. If the access type (see the-a option) includes logging, the number of logged accesses is given bycount +failcount - 1. The -1 is required because the last access coincides with the first faulting access.
Note that access logging may be combined with error injection iffailcount andoperator are nonzero and if the access type includes logging and any of the other access types (pio,dma andintr) See the description of access types in the definition of the-a option, above.
When thecount andfailcount fields reach zero, the status of the errdef is reported to standard output. When all active errdefs created by theth_define process complete, the process exits. Ifacc_types includeslog,count determines how many accesses to log. Ifcount is not specified, a default value is used. Iffailcount is set in this mode, it will simply increase the number of accesses logged by a furtherfailcount - 1.
For qualifying PIO read and write accesses, the value read from or written to the hardware is corrupted according to the value ofoperator:
operand is returned to the driver.
operand is bitwise ORed with the real value.
operand is bitwise ANDed with the real value.
operand is bitwise XORed with the real value.
For PIO write accesses, the following operator is allowed:
Simply ignore the driver's attempt to write to the hardware.
Note that a driver performs PIO via theddi_getX(),ddi_putX(),ddi_rep_getX() andddi_rep_putX() routines (whereX is 8, 16, 32 or 64). Accesses made usingddi_getX() andddi_putX() are treated as a single access, whereas an access made using theddi_rep_*(9F) routines are broken down into their respective number of accesses, as given by therepcount parameter to these DDI calls. If the access is performed via a DMA handle,operator andvalue are applied to every access that comprises the DMA request. If interference with interrupts has been requested then the operator may take any of the following values:
Aftercount accesses (see the-c option), delay delivery of the nextfailcount number of interrupts foroperand number of microseconds.
Aftercount number of interrupts, fail to deliver the nextfailcount number of real interrupts to the driver.
Aftercount number of interrupts, start deliveringoperand number of extra interrupts for the nextfailcount number of real interrupts.
The default value foroperand andoperator is to corrupt the data access by flipping each bit (XOR with -1).
If theacc_chk parameter is set to 1 orpio, then the driver's calls toddi_check_acc_handle(9F) returnDDI_FAILURE when the access count goes to 1. If theacc_chk parameter is set to 2 ordma, then the driver's calls toddi_check_dma_handle(9F) returnDDI_FAILURE when the access count goes to 1.
Constrain the period for which an error definition will remain active. The option applies only to non-logging errdefs. If an error definition remains active formax_wait_period seconds, the test will be aborted. Ifreport_interval is set to a nonzero value, the current status of the error definition is reported to standard output everyreport_interval seconds. The default value is zero. The status of the errdef is reported in parsable format (eight fields, each separated by a colon (:) character, the last of which is a string enclosed by double quotes and the remaining seven fields are integers):
ft:mt:ac:fc:chk:ec:s:"message" which are defined as follows:
The UTC time when the fault was injected.
The UTC time when the driver reported the fault.
The number of remaining non-faulting accesses.
The number of remaining faulting accesses.
The value of theacc_chk field of the errdef.
The number of fault reports issued by the driver against this errdef (mt holds the time of the initial report).
The severity level reported by the driver.
Textual reason why the driver has reported a fault.
Display the command usage string.
Ifacc_types is given with the-a option and includeslog, the errdef will log accesses forcollect_time seconds (the default is to log until the log becomes full). Note that, if the errdef specification matches multiple driver handles, multiple logging errdefs are registered with thebofi driver and logging terminates when all logs become full or whencollect_time expires or when the associated errdefs are cleared. The current state of the log can be checked with theth_manage(1M) command, using thebroadcast parameter. A log can be terminated by runningth_manage(1M) with theclear_errdefs option or by sending aSIGALRM signal to theth_define process. Seealarm(2) for the semantics ofSIGALRM.
Applicable when theacc_types option includeslog. The parameter modifies the policy used for converting from logged accesses to errdefs. All policies are inclusive:
Userare to bias error definitions toward rare accesses (default).
Useoperator to produce a separate error definition for each operator type (default).
Usecommon to bias error definitions toward common accesses.
Usemedian to bias error definitions toward median accesses.
Usemaximal to produce multiple error definitions for duplicate accesses.
Useunbiased to create unbiased error definitions.
Useonebyte,twobyte,fourbyte, oreightbyte to select errdefs corresponding to 1, 2, 4 or 8 byte accesses (if chosen, the-xr option is enforced in order to ensure thatddi_rep_*() calls are decomposed intomultiple single accesses).
Usemultibyte to create error definitions for multibyte accesses performed usingddi_rep_get*() andddi_rep_put*().
Policies can be combined by adding together these options. See the NOTES section for further information.
Applicable when theacc_types option includeslog. Theflags parameter modifies the way in which thebofi driver logs accesses. It is specified as a string containing any combination of the following letters:
Continuous logging (that is, the log will wrap when full).
Timestamp each log entry (access times are in seconds).
Log repeated I/O as individual accesses (for example, addi_rep_get16(9F) call which has a repcount ofN is loggedN times with each transaction logged as size 2 bytes. Without this option, the default logging behavior is to log this access once only, with a transaction size of twice therepcount).
Applicable when theacc_types option includeslog. It provides a comment string to be placed in any generated test scripts. The string must be enclosed in double quotes.
Applicable when theacc_types option includeslog. The output of a logging errdefs is to generate a test script for each driver access handle. Use this option to embed a command in the resulting script before the errors are injected. The generated test scripts will take an instance offline and bring it back online before injecting errors in order to bring the instance into a known fault–free state. The executablefixup_script will be called twice with the set of optionalargs— once just before the instance is taken offline and again after the instance has been brought online. The following variables are passed into the environment of the called executable:
Identifies the device path of the instance.
Identifies the instance number of the device.
Has the value 1 when the instance is about to be taken offline.
Has the value 1 when the instance has just been brought online.
Typically, the executable ensures that the device under test is in a suitable state to be taken offline (unconfigured) or in a suitable state for error injection (for example configured, error free and servicing a workload). A minimal script for a network driver could be:
#!/bin/kshdriver=xyznetdriverifnum=$driver$DRIVER_INSTANCEif [[ $DRIVER_CONFIGURE = 1 ]]; then ifconfig $ifnum plumb ifconfig $ifnum ... ifworkload start $ifnumelif [[ $DRIVER_UNCONFIGURE = 1 ]]; then ifworkload stop $ifnum ifconfig $ifnum down ifconfig $ifnum unplumbfiexit $?
The-e option must be the last option on the command line.
If the-alog option is selected but the-e option isnot given, a default script is used. This script repeatedly attempts todetach and then re-attach the device instance under test.
th_define -n foo -i 1 -a log
Logs all accesses to all handles used by instance 1 of thefoo driver while running the default workload (attaching and detaching the instance).Then generates a set of test scripts to inject appropriate errdefs whilerunning that default workload.
th_define -n foo -i 1 -a log pio
Logs PIO accesses to each PIO handle used by instance 1 ofthefoo driver while running the default workload (attaching and detaching theinstance). Then generates a set of test scripts to inject appropriate errdefswhile running that default workload.
th_define -n foo -i 1 -p onebyte median -e fixup arg -now
Logs all accesses to all handles used by instance 1 of thefoo driver while running the workload defined in the fixup scriptfixupwith argumentsarg and-now. Then generates a set of test scriptsto inject appropriate errdefs while running that workload. The resulting error definitions arerequested to focus upon single byte accesses to locations that are accessedamedian number of times with respect to frequency of access toI/O addresses.
th_define -n se -l 0x20 1 -a pio_r -o OR 0x4 -c 10 1000
Simulates a stuck serial chip command by forcing 1000 consecutive read accessesmade by any instance of these driver to its command statusregister, thereby returning status busy.
th_define -n foo -i 3 -r 1 -a pio_r -c 0 1 -f 1 -o OR 0x100
Causes 0x100 to be ORed into the next physical I/O read accessfrom any register in register set 1 of instance 3 of thefoo driver. Subsequent calls in the driver toddi_check_acc_handle() returnDDI_FAILURE.
th_define -n foo -i 3 -r 1 -a pio_r -c 0 1 -o OR 0x0
Causes 0x0 to be ORed into the next physical I/O read accessfrom any register in register set 1 of instance 3 of thefoo driver. This is of course a no-op.
th_define -n foo -i 3 -r 1 -l 0x8100 1 -a pio_r -c 0 10 -o EQ 0x70003
Causes the next ten next physical I/O reads from the register atoffset 0x8100 in register set 1 of instance 3 of thefoodriver to return 0x70003.
th_define -n foo -i 3 -r 1 -l 0x8100 1 -a pio_w -c 100 3 -o AND 0xffffffffffffefff
The next 100 physical I/O writes to the register at offset 0x8100in register set 1 of instance 3 of thefoo driver takeplace as normal. However, on each of the three subsequent accesses, the0x1000 bit will be cleared.
th_define -n foo -i 3 -r 1 -l 0x8100 0x10 -a pio_r -c 0 1 -f 1 -o XOR 7
Causes the bottom three bits to have their values toggled for thenext physical I/O read access to registers with offsets in the range0x8100 to 0x8110 in register set 1 of instance 3 of thefoo driver. Subsequent calls in the driver toddi_check_acc_handle() returnDDI_FAILURE.
th_define -n foo -i 3 -a pio_w -c 0 1 -o NO 0
Prevents the next physical I/O write access to any register in anyregister set of instance 3 of thefoo driver from going outon the bus.
th_define -n foo -i 3 -l 0 8192 -a dma_r -c 0 1 -o OR 7
Causes 0x7 to be ORed into eachlong long in the first 8192bytes of the next DMA read, using any DMA handle for instance3 of thefoo driver.
th_define -n foo -i 3 -r 2 -l 0 8 -a dma_r -c 0 1 -o OR 0x7070707070707070
Causes 0x70 to be ORed into each byte of the firstlong longof the next DMA read, using the DMA handle with sequential allocationnumber 2 for instance 3 of thefoo driver.
th_define -n foo -i 3 -l 256 256 -a dma_w -c 0 1 -f 2 -o OR 7
Causes 0x7 to be ORed into eachlong long in the range fromoffset 256 to offset 512 of the next DMA write, using anyDMA handle for instance 3 of thefoo driver. Subsequent calls inthe driver toddi_check_dma_handle() returnDDI_FAILURE.
th_define -n foo -i 3 -r 0 -l 0 8 -a dma_w -c 100 3 -o AND 0xffffffffffffefff
The next 100 DMA writes using the DMA handle with sequential allocationnumber 0 for instance 3 of thefoo driver take place asnormal. However, on each of the three subsequent accesses, the 0x1000 bitwill be cleared in the firstlong long of the transfer.
th_define -n foo -i 3 -a intr -c 0 6 -o LOSE 0
Causes the next six interrupts for instance 3 of thefoo driverto be lost.
th_define -n foo -i 3 -a intr -c 30 1 -o EXTRA 10
When the thirty-first subsequent interrupt for instance 3 of thefoo driveroccurs, a further ten interrupts are also generated.
th_define -n foo -i 3 -a intr -c 0 1 -o DELAY 1024
Causes the next interrupt for instance 3 of thefoo driver tobe delayed by 1024 microseconds.
The policy option in theth_define-p syntax determines how a setof logged accesses will be converted into the set of error definitions.Each logged access will be matched against the chosen policies to determinewhether an error definition should be created based on the access.
Any number of policy options can be combined to modify the generatederror definitions.
These select particular I/O transfer sizes. Specifing a byte policy will excludeother byte policies that have not been chosen. If none of thebyte type policies is selected, all transfer sizes are treated equally. Otherwise,only those specified transfer sizes will be selected.
Create errdefs for one byte accesses (ddi_get8())
Create errdefs for two byte accesses (ddi_get16())
Create errdefs for four byte accesses (ddi_get32())
Create errdefs for eight byte accesses (ddi_get64())
Create errdefs for repeated byte accesses (ddi_rep_get*())
The frequency of access to a location is determined according to theaccess type, location and transfer size (for example, a two-byte read accessto address A is considered distinct from a four-byte read access toaddress A). The algorithm is to count the number of accesses (ofa given type and size) to a given location, and find the locationsthat were most and least accessed (letmaxa andmina be thenumber of times these locations were accessed, andmean the total numberof accesses divided by total number of locations that were accessed). Then arare access is a location that was accessed less than
(mean - mina) / 3 + mina
times. Similarly for the definition of common accesses:
maxa - (maxa - mean) / 3
A location whose access patterns lies within these cutoffs is regarded asa location that is accessed with median frequency.
Create errdefs for locations that are rarely accessed.
Create errdefs for locations that are commonly accessed.
Create errdefs for locations that are accessed a median frequency.
If a transaction is duplicated, either a single or multiple errdefs willbe written to the test scripts, depending upon the following two policies:
Create multiple errdefs for locations that are repeatedly accessed.
Create a single errdef for locations that are repeatedly accessed.
For each location, a default operator and operand is typically applied. For maximal test coverage, this default may be modified using theoperators policy so that a separate errdef is created for each of the possible corruption operators.
kill(1),th_manage(1M),alarm(2),ddi_check_acc_handle(9F),ddi_check_dma_handle(9F)
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.Legal Notices |   |