Path: blob/master/Documentation/arch/sparc/oradax/dax-hv-api.txt
26295 views
Excerpt from UltraSPARC Virtual Machine Specification1Compiled from version 3.0.20+152Publication date 2017-09-25 08:213Copyright © 2008, 2015 Oracle and/or its affiliates. All rights reserved.4Extracted via "pdftotext -f 547 -l 572 -layout sun4v_20170925.pdf"5Authors:6Charles Kunzman7Sam Glidden8Mark Cianchetti91011Chapter 36. Coprocessor services12The following APIs provide access via the Hypervisor to hardware assisted data processing functionality.13These APIs may only be provided by certain platforms, and may not be available to all virtual machines14even on supported platforms. Restrictions on the use of these APIs may be imposed in order to support15live-migration and other system management activities.161736.1. Data Analytics Accelerator18The Data Analytics Accelerator (DAX) functionality is a collection of hardware coprocessors that provide19high speed processoring of database-centric operations. The coprocessors may support one or more of20the following data query operations: search, extraction, compression, decompression, and translation. The21functionality offered may vary by virtual machine implementation.2223The DAX is a virtual device to sun4v guests, with supported data operations indicated by the virtual device24compatibility property. Functionality is accessed through the submission of Command Control Blocks25(CCBs) via the ccb_submit API function. The operations are processed asynchronously, with the status26of the submitted operations reported through a Completion Area linked to each CCB. Each CCB has a27separate Completion Area and, unless execution order is specifically restricted through the use of serial-28conditional flags, the execution order of submitted CCBs is arbitrary. Likewise, the time to completion29for a given CCB is never guaranteed.3031Guest software may implement a software timeout on CCB operations, and if the timeout is exceeded, the32operation may be cancelled or killed via the ccb_kill API function. It is recommended for guest software33to implement a software timeout to account for certain RAS errors which may result in lost CCBs. It is34recommended such implementation use the ccb_info API function to check the status of a CCB prior to35killing it in order to determine if the CCB is still in queue, or may have been lost due to a RAS error.3637There is no fixed limit on the number of outstanding CCBs guest software may have queued in the virtual38machine, however, internal resource limitations within the virtual machine can cause CCB submissions39to be temporarily rejected with EWOULDBLOCK. In such cases, guests should continue to attempt40submissions until they succeed; waiting for an outstanding CCB to complete is not necessary, and would41not be a guarantee that a future submission would succeed.4243The availability of DAX coprocessor command service is indicated by the presence of the DAX virtual44device node in the guest MD (Section 8.24.17, “Database Analytics Accelerators (DAX) virtual-device45node”).464736.1.1. DAX Compatibility Property48The query functionality may vary based on the compatibility property of the virtual device:495036.1.1.1. "ORCL,sun4v-dax" Device Compatibility51Available CCB commands:5253• No-op/Sync5455• Extract5657• Scan Value5859• Inverted Scan Value6061• Scan Range62636450965Coprocessor services666768• Inverted Scan Range6970• Translate7172• Inverted Translate7374• Select7576See Section 36.2.1, “Query CCB Command Formats” for the corresponding CCB input and output formats.7778Only version 0 CCBs are available.798036.1.1.2. "ORCL,sun4v-dax-fc" Device Compatibility81"ORCL,sun4v-dax-fc" is compatible with the "ORCL,sun4v-dax" interface, and includes additional CCB82bit fields and controls.838436.1.1.3. "ORCL,sun4v-dax2" Device Compatibility85Available CCB commands:8687• No-op/Sync8889• Extract9091• Scan Value9293• Inverted Scan Value9495• Scan Range9697• Inverted Scan Range9899• Translate100101• Inverted Translate102103• Select104105See Section 36.2.1, “Query CCB Command Formats” for the corresponding CCB input and output formats.106107Version 0 and 1 CCBs are available. Only version 0 CCBs may use Huffman encoded data, whereas only108version 1 CCBs may use OZIP.10911036.1.2. DAX Virtual Device Interrupts111The DAX virtual device has multiple interrupts associated with it which may be used by the guest if112desired. The number of device interrupts available to the guest is indicated in the virtual device node of the113guest MD (Section 8.24.17, “Database Analytics Accelerators (DAX) virtual-device node”). If the device114node indicates N interrupts available, the guest may use any value from 0 to N - 1 (inclusive) in a CCB115interrupt number field. Using values outside this range will result in the CCB being rejected for an invalid116field value.117118The interrupts may be bound and managed using the standard sun4v device interrupts API (Chapter 16,119Device interrupt services). Sysino interrupts are not available for DAX devices.12012136.2. Coprocessor Control Block (CCB)122CCBs are either 64 or 128 bytes long, depending on the operation type. The exact contents of the CCB123are command specific, but all CCBs contain at least one memory buffer address. All memory locations124125126510127Coprocessor services128129130referenced by a CCB must be pinned in memory until the CCB either completes execution or is killed131via the ccb_kill API call. Changes in virtual address mappings occurring after CCB submission are not132guaranteed to be visible, and as such all virtual address updates need to be synchronized with CCB133execution.134135All CCBs begin with a common 32-bit header.136137Table 36.1. CCB Header Format138Bits Field Description139[31:28] CCB version. For API version 2.0: set to 1 if CCB uses OZIP encoding; set to 0 if the CCB140uses Huffman encoding; otherwise either 0 or 1. For API version 1.0: always set to 0.141[27] When API version 2.0 is negotiated, this is the Pipeline Flag [512]. It is reserved in142API version 1.0143[26] Long CCB flag [512]144[25] Conditional synchronization flag [512]145[24] Serial synchronization flag146[23:16] CCB operation code:1470x00 No Operation (No-op) or Sync1480x01 Extract1490x02 Scan Value1500x12 Inverted Scan Value1510x03 Scan Range1520x13 Inverted Scan Range1530x04 Translate1540x14 Inverted Translate1550x05 Select156[15:13] Reserved157[12:11] Table address type1580b'00 No address1590b'01 Alternate context virtual address1600b'10 Real address1610b'11 Primary context virtual address162[10:8] Output/Destination address type1630b'000 No address1640b'001 Alternate context virtual address1650b'010 Real address1660b'011 Primary context virtual address1670b'100 Reserved1680b'101 Reserved1690b'110 Reserved1700b'111 Reserved171[7:5] Secondary source address type172173174511175Coprocessor services176177178Bits Field Description1790b'000 No address1800b'001 Alternate context virtual address1810b'010 Real address1820b'011 Primary context virtual address1830b'100 Reserved1840b'101 Reserved1850b'110 Reserved1860b'111 Reserved187[4:2] Primary source address type1880b'000 No address1890b'001 Alternate context virtual address1900b'010 Real address1910b'011 Primary context virtual address1920b'100 Reserved1930b'101 Reserved1940b'110 Reserved1950b'111 Reserved196[1:0] Completion area address type1970b'00 No address1980b'01 Alternate context virtual address1990b'10 Real address2000b'11 Primary context virtual address201202The Long CCB flag indicates whether the submitted CCB is 64 or 128 bytes long; value is 0 for 64 bytes203and 1 for 128 bytes.204205The Serial and Conditional flags allow simple relative ordering between CCBs. Any CCB with the Serial206flag set will execute sequentially relative to any previous CCB that is also marked as Serial in the same207CCB submission. CCBs without the Serial flag set execute independently, even if they are between CCBs208with the Serial flag set. CCBs marked solely with the Serial flag will execute upon the completion of the209previous Serial CCB, regardless of the completion status of that CCB. The Conditional flag allows CCBs210to conditionally execute based on the successful execution of the closest CCB marked with the Serial flag.211A CCB may only be conditional on exactly one CCB, however, a CCB may be marked both Conditional212and Serial to allow execution chaining. The flags do NOT allow fan-out chaining, where multiple CCBs213execute in parallel based on the completion of another CCB.214215The Pipeline flag is an optimization that directs the output of one CCB (the "source" CCB) directly to216the input of the next CCB (the "target" CCB). The target CCB thus does not need to read the input from217memory. The Pipeline flag is advisory and may be dropped.218219Both the Pipeline and Serial bits must be set in the source CCB. The Conditional bit must be set in the220target CCB. Exactly one CCB must be made conditional on the source CCB; either 0 or 2 target CCBs221is invalid. However, Pipelines can be extended beyond two CCBs: the sequence would start with a CCB222with both the Pipeline and Serial bits set, proceed through CCBs with the Pipeline, Serial, and Conditional223bits set, and terminate at a CCB that has the Conditional bit set, but not the Pipeline bit.224225226512227Coprocessor services228229230The input of the target CCB must start within 64 bytes of the output of the source CCB or the pipeline flag231will be ignored. All CCBs in a pipeline must be submitted in the same call to ccb_submit.232233The various address type fields indicate how the various address values used in the CCB should be234interpreted by the virtual machine. Not all of the types specified are used by every CCB format. Types235which are not applicable to the given CCB command should be indicated as type 0 (No address). Virtual236addresses used in the CCB must have translation entries present in either the TLB or a configured TSB237for the submitting virtual processor. Virtual addresses which cannot be translated by the virtual machine238will result in the CCB submission being rejected, with the causal virtual address indicated. The CCB239may be resubmitted after inserting the translation, or the address may be translated by guest software and240resubmitted using the real address translation.24124236.2.1. Query CCB Command Formats24336.2.1.1. Supported Data Formats, Elements Sizes and Offsets244Data for query commands may be encoded in multiple possible formats. The data query commands use a245common set of values to indicate the encoding formats of the data being processed. Some encoding formats246require multiple data streams for processing, requiring the specification of both primary data formats (the247encoded data) and secondary data streams (meta-data for the encoded data).24824936.2.1.1.1. Primary Input Format250251The primary input format code is a 4-bit field when it is used. There are 10 primary input formats available.252The packed formats are not endian neutral. Code values not listed below are reserved.253254Code Format Description2550x0 Fixed width byte packed Up to 16 bytes2560x1 Fixed width bit packed Up to 15 bits (CCB version 0) or 23 bits (CCB version2571); bits are read most significant bit to least significant bit258within a byte2590x2 Variable width byte packed Data stream of lengths must be provided as a secondary260input2610x4 Fixed width byte packed with run Up to 16 bytes; data stream of run lengths must be262length encoding provided as a secondary input2630x5 Fixed width bit packed with run Up to 15 bits (CCB version 0) or 23 bits (CCB version264length encoding 1); bits are read most significant bit to least significant bit265within a byte; data stream of run lengths must be provided266as a secondary input2670x8 Fixed width byte packed with Up to 16 bytes before the encoding; compressed stream268Huffman (CCB version 0) or bits are read most significant bit to least significant bit269OZIP (CCB version 1) encoding within a byte; pointer to the encoding table must be270provided2710x9 Fixed width bit packed with Up to 15 bits (CCB version 0) or 23 bits (CCB version272Huffman (CCB version 0) or 1); compressed stream bits are read most significant bit to273OZIP (CCB version 1) encoding least significant bit within a byte; pointer to the encoding274table must be provided2750xA Variable width byte packed with Up to 16 bytes before the encoding; compressed stream276Huffman (CCB version 0) or bits are read most significant bit to least significant bit277OZIP (CCB version 1) encoding within a byte; data stream of lengths must be provided as278a secondary input; pointer to the encoding table must be279provided280281282513283Coprocessor services284285286Code Format Description2870xC Fixed width byte packed with Up to 16 bytes before the encoding; compressed stream288run length encoding, followed by bits are read most significant bit to least significant bit289Huffman (CCB version 0) or within a byte; data stream of run lengths must be provided290OZIP (CCB version 1) encoding as a secondary input; pointer to the encoding table must291be provided2920xD Fixed width bit packed with Up to 15 bits (CCB version 0) or 23 bits(CCB version 1)293run length encoding, followed by before the encoding; compressed stream bits are read most294Huffman (CCB version 0) or significant bit to least significant bit within a byte; data295OZIP (CCB version 1) encoding stream of run lengths must be provided as a secondary296input; pointer to the encoding table must be provided297298If OZIP encoding is used, there must be no reserved bytes in the table.29930036.2.1.1.2. Primary Input Element Size301302For primary input data streams with fixed size elements, the element size must be indicated in the CCB303command. The size is encoded as the number of bits or bytes, minus one. The valid value range for this304field depends on the input format selected, as listed in the table above.30530636.2.1.1.3. Secondary Input Format307308For primary input data streams which require a secondary input stream, the secondary input stream is309always encoded in a fixed width, bit-packed format. The bits are read from most significant bit to least310significant bit within a byte. There are two encoding options for the secondary input stream data elements,311depending on whether the value of 0 is needed:312313Secondary Input Description314Format Code3150 Element is stored as value minus 1 (0 evaluates to 1, 1 evaluates316to 2, etc)3171 Element is stored as value31831936.2.1.1.4. Secondary Input Element Size320321Secondary input element size is encoded as a two bit field:322323Secondary Input Size Description324Code3250x0 1 bit3260x1 2 bits3270x2 4 bits3280x3 8 bits32933036.2.1.1.5. Input Element Offsets331332Bit-wise input data streams may have any alignment within the base addressed byte. The offset, specified333from most significant bit to least significant bit, is provided as a fixed 3 bit field for each input type. A334value of 0 indicates that the first input element begins at the most significant bit in the first byte, and a335value of 7 indicates it begins with the least significant bit.336337This field should be zero for any byte-wise primary input data streams.338339340514341Coprocessor services34234334436.2.1.1.6. Output Format345346Query commands support multiple sizes and encodings for output data streams. There are four possible347output encodings, and up to four supported element sizes per encoding. Not all output encodings are348supported for every command. The format is indicated by a 4-bit field in the CCB:349350Output Format Code Description3510x0 Byte aligned, 1 byte elements3520x1 Byte aligned, 2 byte elements3530x2 Byte aligned, 4 byte elements3540x3 Byte aligned, 8 byte elements3550x4 16 byte aligned, 16 byte elements3560x5 Reserved3570x6 Reserved3580x7 Reserved3590x8 Packed vector of single bit elements3600x9 Reserved3610xA Reserved3620xB Reserved3630xC Reserved3640xD 2 byte elements where each element is the index value of a bit,365from an bit vector, which was 1.3660xE 4 byte elements where each element is the index value of a bit,367from an bit vector, which was 1.3680xF Reserved36937036.2.1.1.7. Application Data Integrity (ADI)371372On platforms which support ADI, the ADI version number may be specified for each separate memory373access type used in the CCB command. ADI checking only occurs when reading data. When writing data,374the specified ADI version number overwrites any existing ADI value in memory.375376An ADI version value of 0 or 0xF indicates the ADI checking is disabled for that data access, even if it is377enabled in memory. By setting the appropriate flag in CCB_SUBMIT (Section 36.3.1, “ccb_submit”) it is378also an option to disable ADI checking for all inputs accessed via virtual address for all CCBs submitted379during that hypercall invocation.380381The ADI value is only guaranteed to be checked on the first 64 bytes of each data access. Mismatches on382subsequent data chunks may not be detected, so guest software should be careful to use page size checking383to protect against buffer overruns.38438536.2.1.1.8. Page size checking386387All data accesses used in CCB commands must be bounded within a single memory page. When addresses388are provided using a virtual address, the page size for checking is extracted from the TTE for that virtual389address. When using real addresses, the guest must supply the page size in the same field as the address390value. The page size must be one of the sizes supported by the underlying virtual machine. Using a value391that is not supported may result in the CCB submission being rejected or the generation of a CCB parsing392error in the completion area.393394395515396Coprocessor services39739839936.2.1.2. Extract command400401Converts an input vector in one format to an output vector in another format. All input format types are402supported.403404The only supported output format is a padded, byte-aligned output stream, using output codes 0x0 - 0x4.405When the specified output element size is larger than the extracted input element size, zeros are padded to406the extracted input element. First, if the decompressed input size is not a whole number of bytes, 0 bits are407padded to the most significant bit side till the next byte boundary. Next, if the output element size is larger408than the byte padded input element, bytes of value 0 are added based on the Padding Direction bit in the409CCB. If the output element size is smaller than the byte-padded input element size, the input element is410truncated by dropped from the least significant byte side until the selected output size is reached.411412The return value of the CCB completion area is invalid. The “number of elements processed” field in the413CCB completion area will be valid.414415The extract CCB is a 64-byte “short format” CCB.416417The extract CCB command format can be specified by the following packed C structure for a big-endian418machine:419420421struct extract_ccb {422uint32_t header;423uint32_t control;424uint64_t completion;425uint64_t primary_input;426uint64_t data_access_control;427uint64_t secondary_input;428uint64_t reserved;429uint64_t output;430uint64_t table;431};432433434The exact field offsets, sizes, and composition are as follows:435436Offset Size Field Description4370 4 CCB header (Table 36.1, “CCB Header Format”)4384 4 Command control439Bits Field Description440[31:28] Primary Input Format (see Section 36.2.1.1.1, “Primary Input441Format”)442[27:23] Primary Input Element Size (see Section 36.2.1.1.2, “Primary443Input Element Size”)444[22:20] Primary Input Starting Offset (see Section 36.2.1.1.5, “Input445Element Offsets”)446[19] Secondary Input Format (see Section 36.2.1.1.3, “Secondary447Input Format”)448[18:16] Secondary Input Starting Offset (see Section 36.2.1.1.5, “Input449Element Offsets”)450451452516453Coprocessor services454455456Offset Size Field Description457Bits Field Description458[15:14] Secondary Input Element Size (see Section 36.2.1.1.4,459“Secondary Input Element Size”460[13:10] Output Format (see Section 36.2.1.1.6, “Output Format”)461[9] Padding Direction selector: A value of 1 causes padding bytes462to be added to the left side of output elements. A value of 0463causes padding bytes to be added to the right side of output464elements.465[8:0] Reserved4668 8 Completion467Bits Field Description468[63:60] ADI version (see Section 36.2.1.1.7, “Application Data469Integrity (ADI)”)470[59] If set to 1, a virtual device interrupt will be generated using471the device interrupt number specified in the lower bits of this472completion word. If 0, the lower bits of this completion word473are ignored.474[58:6] Completion area address bits [58:6]. Address type is475determined by CCB header.476[5:0] Virtual device interrupt number for completion interrupt, if477enabled.47816 8 Primary Input479Bits Field Description480[63:60] ADI version (see Section 36.2.1.1.7, “Application Data481Integrity (ADI)”)482[59:56] If using real address, these bits should be filled in with the483page size code for the page boundary checking the guest wants484the virtual machine to use when accessing this data stream485(checking is only guaranteed to be performed when using API486version 1.1 and later). If using a virtual address, this field will487be used as as primary input address bits [59:56].488[55:0] Primary input address bits [55:0]. Address type is determined489by CCB header.49024 8 Data Access Control491Bits Field Description492[63:62] Flow Control493Value Description4940b'00 Disable flow control4950b'01 Enable flow control (only valid with "ORCL,sun4v-496dax-fc" compatible virtual device variants)4970b'10 Reserved4980b'11 Reserved499[61:60] Reserved (API 1.0)500501502517503Coprocessor services504505506Offset Size Field Description507Bits Field Description508Pipeline target (API 2.0)509Value Description5100b'00 Connect to primary input5110b'01 Connect to secondary input5120b'10 Reserved5130b'11 Reserved514[59:40] Output buffer size given in units of 64 bytes, minus 1. Value of5150 means 64 bytes, value of 1 means 128 bytes, etc. Buffer size is516only enforced if flow control is enabled in Flow Control field.517[39:32] Reserved518[31:30] Output Data Cache Allocation519Value Description5200b'00 Do not allocate cache lines for output data stream.5210b'01 Force cache lines for output data stream to be522allocated in the cache that is local to the submitting523virtual cpu.5240b'10 Allocate cache lines for output data stream, but allow525existing cache lines associated with the data to remain526in their current cache instance. Any memory not527already in cache will be allocated in the cache local528to the submitting virtual cpu.5290b'11 Reserved530[29:26] Reserved531[25:24] Primary Input Length Format532Value Description5330b'00 Number of primary symbols5340b'01 Number of primary bytes5350b'10 Number of primary bits5360b'11 Reserved537[23:0] Primary Input Length538Format Field Value539# of primary symbols Number of input elements to process,540minus 1. Command execution stops541once count is reached.542# of primary bytes Number of input bytes to process,543minus 1. Command execution stops544once count is reached. The count is545done before any decompression or546decoding.547# of primary bits Number of input bits to process,548minus 1. Command execution stops549550551552518553Coprocessor services554555556Offset Size Field Description557Bits Field Description558Format Field Value559once count is reached. The count is560done before any decompression or561decoding, and does not include any562bits skipped by the Primary Input563Offset field value of the command564control word.56532 8 Secondary Input, if used by Primary Input Format. Same fields as Primary566Input.56740 8 Reserved56848 8 Output (same fields as Primary Input)56956 8 Symbol Table (if used by Primary Input)570Bits Field Description571[63:60] ADI version (see Section 36.2.1.1.7, “Application Data572Integrity (ADI)”)573[59:56] If using real address, these bits should be filled in with the574page size code for the page boundary checking the guest wants575the virtual machine to use when accessing this data stream576(checking is only guaranteed to be performed when using API577version 1.1 and later). If using a virtual address, this field will578be used as as symbol table address bits [59:56].579[55:4] Symbol table address bits [55:4]. Address type is determined580by CCB header.581[3:0] Symbol table version582Value Description5830 Huffman encoding. Must use 64 byte aligned table584address. (Only available when using version 0 CCBs)5851 OZIP encoding. Must use 16 byte aligned table586address. (Only available when using version 1 CCBs)58758858936.2.1.3. Scan commands590591The scan commands search a stream of input data elements for values which match the selection criteria.592All the input format types are supported. There are multiple formats for the scan commands, allowing the593scan to search for exact matches to one value, exact matches to either of two values, or any value within594a specified range. The specific type of scan is indicated by the command code in the CCB header. For the595scan range commands, the boundary conditions can be specified as greater-than-or-equal-to a value, less-596than-or-equal-to a value, or both by using two boundary values.597598There are two supported formats for the output stream: the bit vector and index array formats (codes 0x8,5990xD, and 0xE). For the standard scan command using the bit vector output, for each input element there600exists one bit in the vector that is set if the input element matched the scan criteria, or clear if not. The601inverted scan command inverts the polarity of the bits in the output. The most significant bit of the first602byte of the output stream corresponds to the first element in the input stream. The standard index array603output format contains one array entry for each input element that matched the scan criteria. Each array604605606607519608Coprocessor services609610611entry is the index of an input element that matched the scan criteria. An inverted scan command produces612a similar array, but of all the input elements which did NOT match the scan criteria.613614The return value of the CCB completion area contains the number of input elements found which match615the scan criteria (or number that did not match for the inverted scans). The “number of elements processed”616field in the CCB completion area will be valid, indicating the number of input elements processed.617618These commands are 128-byte “long format” CCBs.619620The scan CCB command format can be specified by the following packed C structure for a big-endian621machine:622623624struct scan_ccb {625uint32_t header;626uint32_t control;627uint64_t completion;628uint64_t primary_input;629uint64_t data_access_control;630uint64_t secondary_input;631uint64_t match_criteria0;632uint64_t output;633uint64_t table;634uint64_t match_criteria1;635uint64_t match_criteria2;636uint64_t match_criteria3;637uint64_t reserved[5];638};639640641The exact field offsets, sizes, and composition are as follows:642643Offset Size Field Description6440 4 CCB header (Table 36.1, “CCB Header Format”)6454 4 Command control646Bits Field Description647[31:28] Primary Input Format (see Section 36.2.1.1.1, “Primary Input648Format”)649[27:23] Primary Input Element Size (see Section 36.2.1.1.2, “Primary650Input Element Size”)651[22:20] Primary Input Starting Offset (see Section 36.2.1.1.5, “Input652Element Offsets”)653[19] Secondary Input Format (see Section 36.2.1.1.3, “Secondary654Input Format”)655[18:16] Secondary Input Starting Offset (see Section 36.2.1.1.5, “Input656Element Offsets”)657[15:14] Secondary Input Element Size (see Section 36.2.1.1.4,658“Secondary Input Element Size”659[13:10] Output Format (see Section 36.2.1.1.6, “Output Format”)660[9:5] Operand size for first scan criteria value. In a scan value661operation, this is one of two potential exact match values.662In a scan range operation, this is the size of the upper range663664665520666Coprocessor services667668669Offset Size Field Description670Bits Field Description671boundary. The value of this field is the number of bytes in the672operand, minus 1. Values 0xF-0x1E are reserved. A value of6730x1F indicates this operand is not in use for this scan operation.674[4:0] Operand size for second scan criteria value. In a scan value675operation, this is one of two potential exact match values.676In a scan range operation, this is the size of the lower range677boundary. The value of this field is the number of bytes in the678operand, minus 1. Values 0xF-0x1E are reserved. A value of6790x1F indicates this operand is not in use for this scan operation.6808 8 Completion (same fields as Section 36.2.1.2, “Extract command”)68116 8 Primary Input (same fields as Section 36.2.1.2, “Extract command”)68224 8 Data Access Control (same fields as Section 36.2.1.2, “Extract command”)68332 8 Secondary Input, if used by Primary Input Format. Same fields as Primary684Input.68540 4 Most significant 4 bytes of first scan criteria operand. If first operand is less686than 4 bytes, the value is left-aligned to the lowest address bytes.68744 4 Most significant 4 bytes of second scan criteria operand. If second operand688is less than 4 bytes, the value is left-aligned to the lowest address bytes.68948 8 Output (same fields as Primary Input)69056 8 Symbol Table (if used by Primary Input). Same fields as Section 36.2.1.2,691“Extract command”69264 4 Next 4 most significant bytes of first scan criteria operand occurring after the693bytes specified at offset 40, if needed by the operand size. If first operand694is less than 8 bytes, the valid bytes are left-aligned to the lowest address.69568 4 Next 4 most significant bytes of second scan criteria operand occurring after696the bytes specified at offset 44, if needed by the operand size. If second697operand is less than 8 bytes, the valid bytes are left-aligned to the lowest698address.69972 4 Next 4 most significant bytes of first scan criteria operand occurring after the700bytes specified at offset 64, if needed by the operand size. If first operand701is less than 12 bytes, the valid bytes are left-aligned to the lowest address.70276 4 Next 4 most significant bytes of second scan criteria operand occurring after703the bytes specified at offset 68, if needed by the operand size. If second704operand is less than 12 bytes, the valid bytes are left-aligned to the lowest705address.70680 4 Next 4 most significant bytes of first scan criteria operand occurring after the707bytes specified at offset 72, if needed by the operand size. If first operand708is less than 16 bytes, the valid bytes are left-aligned to the lowest address.70984 4 Next 4 most significant bytes of second scan criteria operand occurring after710the bytes specified at offset 76, if needed by the operand size. If second711operand is less than 16 bytes, the valid bytes are left-aligned to the lowest712address.713714715716717521718Coprocessor services71972072136.2.1.4. Translate commands722723The translate commands takes an input array of indices, and a table of single bit values indexed by those724indices, and outputs a bit vector or index array created by reading the tables bit value at each index in725the input array. The output should therefore contain exactly one bit per index in the input data stream,726when outputting as a bit vector. When outputting as an index array, the number of elements depends on the727values read in the bit table, but will always be less than, or equal to, the number of input elements. Only728a restricted subset of the possible input format types are supported. No variable width or Huffman/OZIP729encoded input streams are allowed. The primary input data element size must be 3 bytes or less.730731The maximum table index size allowed is 15 bits, however, larger input elements may be used to provide732additional processing of the output values. If 2 or 3 byte values are used, the least significant 15 bits are733used as an index into the bit table. The most significant 9 bits (when using 3-byte input elements) or single734bit (when using 2-byte input elements) are compared against a fixed 9-bit test value provided in the CCB.735If the values match, the value from the bit table is used as the output element value. If the values do not736match, the output data element value is forced to 0.737738In the inverted translate operation, the bit value read from bit table is inverted prior to its use. The additional739additional processing based on any additional non-index bits remains unchanged, and still forces the output740element value to 0 on a mismatch. The specific type of translate command is indicated by the command741code in the CCB header.742743There are two supported formats for the output stream: the bit vector and index array formats (codes 0x8,7440xD, and 0xE). The index array format is an array of indices of bits which would have been set if the745output format was a bit array.746747The return value of the CCB completion area contains the number of bits set in the output bit vector,748or number of elements in the output index array. The “number of elements processed” field in the CCB749completion area will be valid, indicating the number of input elements processed.750751These commands are 64-byte “short format” CCBs.752753The translate CCB command format can be specified by the following packed C structure for a big-endian754machine:755756757struct translate_ccb {758uint32_t header;759uint32_t control;760uint64_t completion;761uint64_t primary_input;762uint64_t data_access_control;763uint64_t secondary_input;764uint64_t reserved;765uint64_t output;766uint64_t table;767};768769770The exact field offsets, sizes, and composition are as follows:771772773Offset Size Field Description7740 4 CCB header (Table 36.1, “CCB Header Format”)775776777522778Coprocessor services779780781Offset Size Field Description7824 4 Command control783Bits Field Description784[31:28] Primary Input Format (see Section 36.2.1.1.1, “Primary Input785Format”)786[27:23] Primary Input Element Size (see Section 36.2.1.1.2, “Primary787Input Element Size”)788[22:20] Primary Input Starting Offset (see Section 36.2.1.1.5, “Input789Element Offsets”)790[19] Secondary Input Format (see Section 36.2.1.1.3, “Secondary791Input Format”)792[18:16] Secondary Input Starting Offset (see Section 36.2.1.1.5, “Input793Element Offsets”)794[15:14] Secondary Input Element Size (see Section 36.2.1.1.4,795“Secondary Input Element Size”796[13:10] Output Format (see Section 36.2.1.1.6, “Output Format”)797[9] Reserved798[8:0] Test value used for comparison against the most significant bits799in the input values, when using 2 or 3 byte input elements.8008 8 Completion (same fields as Section 36.2.1.2, “Extract command”80116 8 Primary Input (same fields as Section 36.2.1.2, “Extract command”80224 8 Data Access Control (same fields as Section 36.2.1.2, “Extract command”,803except Primary Input Length Format may not use the 0x0 value)80432 8 Secondary Input, if used by Primary Input Format. Same fields as Primary805Input.80640 8 Reserved80748 8 Output (same fields as Primary Input)80856 8 Bit Table809Bits Field Description810[63:60] ADI version (see Section 36.2.1.1.7, “Application Data811Integrity (ADI)”)812[59:56] If using real address, these bits should be filled in with the813page size code for the page boundary checking the guest wants814the virtual machine to use when accessing this data stream815(checking is only guaranteed to be performed when using API816version 1.1 and later). If using a virtual address, this field will817be used as as bit table address bits [59:56]818[55:4] Bit table address bits [55:4]. Address type is determined by819CCB header. Address must be 64-byte aligned (CCB version8200) or 16-byte aligned (CCB version 1).821[3:0] Bit table version822Value Description8230 4KB table size8241 8KB table size825826827828523829Coprocessor services83083183236.2.1.5. Select command833The select command filters the primary input data stream by using a secondary input bit vector to determine834which input elements to include in the output. For each bit set at a given index N within the bit vector,835the Nth input element is included in the output. If the bit is not set, the element is not included. Only a836restricted subset of the possible input format types are supported. No variable width or run length encoded837input streams are allowed, since the secondary input stream is used for the filtering bit vector.838839The only supported output format is a padded, byte-aligned output stream. The stream follows the same840rules and restrictions as padded output stream described in Section 36.2.1.2, “Extract command”.841842The return value of the CCB completion area contains the number of bits set in the input bit vector. The843"number of elements processed" field in the CCB completion area will be valid, indicating the number844of input elements processed.845846The select CCB is a 64-byte “short format” CCB.847848The select CCB command format can be specified by the following packed C structure for a big-endian849machine:850851852struct select_ccb {853uint32_t header;854uint32_t control;855uint64_t completion;856uint64_t primary_input;857uint64_t data_access_control;858uint64_t secondary_input;859uint64_t reserved;860uint64_t output;861uint64_t table;862};863864865The exact field offsets, sizes, and composition are as follows:866867Offset Size Field Description8680 4 CCB header (Table 36.1, “CCB Header Format”)8694 4 Command control870Bits Field Description871[31:28] Primary Input Format (see Section 36.2.1.1.1, “Primary Input872Format”)873[27:23] Primary Input Element Size (see Section 36.2.1.1.2, “Primary874Input Element Size”)875[22:20] Primary Input Starting Offset (see Section 36.2.1.1.5, “Input876Element Offsets”)877[19] Secondary Input Format (see Section 36.2.1.1.3, “Secondary878Input Format”)879[18:16] Secondary Input Starting Offset (see Section 36.2.1.1.5, “Input880Element Offsets”)881[15:14] Secondary Input Element Size (see Section 36.2.1.1.4,882“Secondary Input Element Size”883884885524886Coprocessor services887888889Offset Size Field Description890Bits Field Description891[13:10] Output Format (see Section 36.2.1.1.6, “Output Format”)892[9] Padding Direction selector: A value of 1 causes padding bytes893to be added to the left side of output elements. A value of 0894causes padding bytes to be added to the right side of output895elements.896[8:0] Reserved8978 8 Completion (same fields as Section 36.2.1.2, “Extract command”89816 8 Primary Input (same fields as Section 36.2.1.2, “Extract command”89924 8 Data Access Control (same fields as Section 36.2.1.2, “Extract command”)90032 8 Secondary Bit Vector Input. Same fields as Primary Input.90140 8 Reserved90248 8 Output (same fields as Primary Input)90356 8 Symbol Table (if used by Primary Input). Same fields as Section 36.2.1.2,904“Extract command”90590636.2.1.6. No-op and Sync commands907The no-op (no operation) command is a CCB which has no processing effect. The CCB, when processed908by the virtual machine, simply updates the completion area with its execution status. The CCB may have909the serial-conditional flags set in order to restrict when it executes.910911The sync command is a variant of the no-op command which with restricted execution timing. A sync912command CCB will only execute when all previous commands submitted in the same request have913completed. This is stronger than the conditional flag sequencing, which is only dependent on a single914previous serial CCB. While the relative ordering is guaranteed, virtual machine implementations with915shared hardware resources may cause the sync command to wait for longer than the minimum required916time.917918The return value of the CCB completion area is invalid for these CCBs. The “number of elements919processed” field is also invalid for these CCBs.920921These commands are 64-byte “short format” CCBs.922923The no-op CCB command format can be specified by the following packed C structure for a big-endian924machine:925926927struct nop_ccb {928uint32_t header;929uint32_t control;930uint64_t completion;931uint64_t reserved[6];932};933934935The exact field offsets, sizes, and composition are as follows:936937Offset Size Field Description9380 4 CCB header (Table 36.1, “CCB Header Format”)939940941525942Coprocessor services943944945Offset Size Field Description9464 4 Command control947Bits Field Description948[31] If set, this CCB functions as a Sync command. If clear, this949CCB functions as a No-op command.950[30:0] Reserved9518 8 Completion (same fields as Section 36.2.1.2, “Extract command”95216 46 Reserved95395436.2.2. CCB Completion Area955All CCB commands use a common 128-byte Completion Area format, which can be specified by the956following packed C structure for a big-endian machine:957958959struct completion_area {960uint8_t status_flag;961uint8_t error_note;962uint8_t rsvd0[2];963uint32_t error_values;964uint32_t output_size;965uint32_t rsvd1;966uint64_t run_time;967uint64_t run_stats;968uint32_t elements;969uint8_t rsvd2[20];970uint64_t return_value;971uint64_t extra_return_value[8];972};973974975The Completion Area must be a 128-byte aligned memory location. The exact layout can be described976using byte offsets and sizes relative to the memory base:977978Offset Size Field Description9790 1 CCB execution status9800x0 Command not yet completed9810x1 Command ran and succeeded9820x2 Command ran and failed (partial results may be been983produced)9840x3 Command ran and was killed (partial execution may985have occurred)9860x4 Command was not run9870x5-0xF Reserved9881 1 Error reason code9890x0 Reserved9900x1 Buffer overflow991992993526994Coprocessor services995996997Offset Size Field Description9980x2 CCB decoding error9990x3 Page overflow10000x4-0x6 Reserved10010x7 Command was killed10020x8 Command execution timeout10030x9 ADI miscompare error10040xA Data format error10050xB-0xD Reserved10060xE Unexpected hardware error (Do not retry)10070xF Unexpected hardware error (Retry is ok)10080x10-0x7F Reserved10090x80 Partial Symbol Warning10100x81-0xFF Reserved10112 2 Reserved10124 4 If a partial symbol warning was generated, this field contains the number1013of remaining bits which were not decoded.10148 4 Number of bytes of output produced101512 4 Reserved101616 8 Runtime of command (unspecified time units)101724 8 Reserved101832 4 Number of elements processed101936 20 Reserved102056 8 Return value102164 64 Extended return value10221023The CCB completion area should be treated as read-only by guest software. The CCB execution status1024byte will be cleared by the Hypervisor to reflect the pending execution status when the CCB is submitted1025successfully. All other fields are considered invalid upon CCB submission until the CCB execution status1026byte becomes non-zero.10271028CCBs which complete with status 0x2 or 0x3 may produce partial results and/or side effects due to partial1029execution of the CCB command. Some valid data may be accessible depending on the fault type, however,1030it is recommended that guest software treat the destination buffer as being in an unknown state. If a CCB1031completes with a status byte of 0x2, the error reason code byte can be read to determine what corrective1032action should be taken.10331034A buffer overflow indicates that the results of the operation exceeded the size of the output buffer indicated1035in the CCB. The operation can be retried by resubmitting the CCB with a larger output buffer.10361037A CCB decoding error indicates that the CCB contained some invalid field values. It may be also be1038triggered if the CCB output is directed at a non-existent secondary input and the pipelining hint is followed.10391040A page overflow error indicates that the operation required accessing a memory location beyond the page1041size associated with a given address. No data will have been read or written past the page boundary, but1042partial results may have been written to the destination buffer. The CCB can be resubmitted with a larger1043page size memory allocation to complete the operation.1044104510465271047Coprocessor services104810491050In the case of pipelined CCBs, a page overflow error will be triggered if the output from the pipeline source1051CCB ends before the input of the pipeline target CCB. Page boundaries are ignored when the pipeline1052hint is followed.10531054Command kill indicates that the CCB execution was halted or prevented by use of the ccb_kill API call.10551056Command timeout indicates that the CCB execution began, but did not complete within a pre-determined1057limit set by the virtual machine. The command may have produced some or no output. The CCB may be1058resubmitted with no alterations.10591060ADI miscompare indicates that the memory buffer version specified in the CCB did not match the value1061in memory when accessed by the virtual machine. Guest software should not attempt to resubmit the CCB1062without determining the cause of the version mismatch.10631064A data format error indicates that the input data stream did not follow the specified data input formatting1065selected in the CCB.10661067Some CCBs which encounter hardware errors may be resubmitted without change. Persistent hardware1068errors may result in multiple failures until RAS software can identify and isolate the faulty component.10691070The output size field indicates the number of bytes of valid output in the destination buffer. This field is1071not valid for all possible CCB commands.10721073The runtime field indicates the execution time of the CCB command once it leaves the internal virtual1074machine queue. The time units are fixed, but unspecified, allowing only relative timing comparisons1075by guest software. The time units may also vary by hardware platform, and should not be construed to1076represent any absolute time value.10771078Some data query commands process data in units of elements. If applicable to the command, the number of1079elements processed is indicated in the listed field. This field is not valid for all possible CCB commands.10801081The return value and extended return value fields are output locations for commands which do not use1082a destination output buffer, or have secondary return results. The field is not valid for all possible CCB1083commands.1084108536.3. Hypervisor API Functions108636.3.1. ccb_submit1087trap# FAST_TRAP1088function# CCB_SUBMIT1089arg0 address1090arg1 length1091arg2 flags1092arg3 reserved1093ret0 status1094ret1 length1095ret2 status data1096ret3 reserved10971098Submit one or more coprocessor control blocks (CCBs) for evaluation and processing by the virtual1099machine. The CCBs are passed in a linear array indicated by address. length indicates the size of1100the array in bytes.1101110211035281104Coprocessor services110511061107The address should be aligned to the size indicated by length, rounded up to the nearest power of1108two. Virtual machines implementations may reject submissions which do not adhere to that alignment.1109length must be a multiple of 64 bytes. If length is zero, the maximum supported array length will be1110returned as length in ret1. In all other cases, the length value in ret1 will reflect the number of bytes1111successfully consumed from the input CCB array.11121113Implementation note1114Virtual machines should never reject submissions based on the alignment of address if the1115entire array is contained within a single memory page of the smallest page size supported by the1116virtual machine.11171118A guest may choose to submit addresses used in this API function, including the CCB array address,1119as either a real or virtual addresses, with the type of each address indicated in flags. Virtual addresses1120must be present in either the TLB or an active TSB to be processed. The translation context for virtual1121addresses is determined by a combination of CCB contents and the flags argument.11221123The flags argument is divided into multiple fields defined as follows:112411251126Bits Field Description1127[63:16] Reserved1128[15] Disable ADI for VA reads (in API 2.0)1129Reserved (in API 1.0)1130[14] Virtual addresses within CCBs are translated in privileged context1131[13:12] Alternate translation context for virtual addresses within CCBs:11320b'00 CCBs requesting alternate context are rejected11330b'01 Reserved11340b'10 CCBs requesting alternate context use secondary context11350b'11 CCBs requesting alternate context use nucleus context1136[11:9] Reserved1137[8] Queue info flag1138[7] All-or-nothing flag1139[6] If address is a virtual address, treat its translation context as privileged1140[5:4] Address type of address:11410b'00 Real address11420b'01 Virtual address in primary context11430b'10 Virtual address in secondary context11440b'11 Virtual address in nucleus context1145[3:2] Reserved1146[1:0] CCB command type:11470b'00 Reserved11480b'01 Reserved11490b'10 Query command11500b'11 Reserved11511152115311545291155Coprocessor services115611571158The CCB submission type and address type for the CCB array must be provided in the flags argument.1159All other fields are optional values which change the default behavior of the CCB processing.11601161When set to one, the "Disable ADI for VA reads" bit will turn off ADI checking when using a virtual1162address to load data. ADI checking will still be done when loading real-addressed memory. This bit is only1163available when using major version 2 of the coprocessor API group; at major version 1 it is reserved. For1164more information about using ADI and DAX, see Section 36.2.1.1.7, “Application Data Integrity (ADI)”.11651166By default, all virtual addresses are treated as user addresses. If the virtual address translations are1167privileged, they must be marked as such in the appropriate flags field. The virtual addresses used within1168the submitted CCBs must all be translated with the same privilege level.11691170By default, all virtual addresses used within the submitted CCBs are translated using the primary context1171active at the time of the submission. The address type field within a CCB allows each address to request1172translation in an alternate address context. The address context used when the alternate address context is1173requested is selected in the flags argument.11741175The all-or-nothing flag specifies whether the virtual machine should allow partial submissions of the1176input CCB array. When using CCBs with serial-conditional flags, it is strongly recommended to use1177the all-or-nothing flag to avoid broken conditional chains. Using long CCB chains on a machine under1178high coprocessor load may make this impractical, however, and require submitting without the flag.1179When submitting serial-conditional CCBs without the all-or-nothing flag, guest software must manually1180implement the serial-conditional behavior at any point where the chain was not submitted in a single API1181call, and resubmission of the remaining CCBs should clear any conditional flag that might be set in the1182first remaining CCB. Failure to do so will produce indeterminate CCB execution status and ordering.11831184When the all-or-nothing flag is not specified, callers should check the value of length in ret1 to determine1185how many CCBs from the array were successfully submitted. Any remaining CCBs can be resubmitted1186without modifications.11871188The value of length in ret1 is also valid when the API call returns an error, and callers should always1189check its value to determine which CCBs in the array were already processed. This will additionally1190identify which CCB encountered the processing error, and was not submitted successfully.11911192If the queue info flag is used during submission, and at least one CCB was successfully submitted, the1193length value in ret1 will be a multi-field value defined as follows:1194Bits Field Description1195[63:48] DAX unit instance identifier1196[47:32] DAX queue instance identifier1197[31:16] Reserved1198[15:0] Number of CCB bytes successfully submitted11991200The value of status data depends on the status value. See error status code descriptions for details.1201The value is undefined for status values that do not specifically list a value for the status data.12021203The API has a reserved input and output register which will be used in subsequent minor versions of this1204API function. Guest software implementations should treat that register as voltile across the function call1205in order to maintain forward compatibility.1206120736.3.1.1. Errors1208EOK One or more CCBs have been accepted and enqueued in the virtual machine1209and no errors were been encountered during submission. Some submitted1210CCBs may not have been enqueued due to internal virtual machine limitations,1211and may be resubmitted without changes.1212121312145301215Coprocessor services121612171218EWOULDBLOCK An internal resource conflict within the virtual machine has prevented it from1219being able to complete the CCB submissions sufficiently quickly, requiring1220it to abandon processing before it was complete. Some CCBs may have been1221successfully enqueued prior to the block, and all remaining CCBs may be1222resubmitted without changes.1223EBADALIGN CCB array is not on a 64-byte boundary, or the array length is not a multiple1224of 64 bytes.1225ENORADDR A real address used either for the CCB array, or within one of the submitted1226CCBs, is not valid for the guest. Some CCBs may have been enqueued prior1227to the error being detected.1228ENOMAP A virtual address used either for the CCB array, or within one of the submitted1229CCBs, could not be translated by the virtual machine using either the TLB1230or TSB contents. The submission may be retried after adding the required1231mapping, or by converting the virtual address into a real address. Due to the1232shared nature of address translation resources, there is no theoretical limit on1233the number of times the translation may fail, and it is recommended all guests1234implement some real address based backup. The virtual address which failed1235translation is returned as status data in ret2. Some CCBs may have been1236enqueued prior to the error being detected.1237EINVAL The virtual machine detected an invalid CCB during submission, or invalid1238input arguments, such as bad flag values. Note that not all invalid CCB values1239will be detected during submission, and some may be reported as errors in the1240completion area instead. Some CCBs may have been enqueued prior to the1241error being detected. This error may be returned if the CCB version is invalid.1242ETOOMANY The request was submitted with the all-or-nothing flag set, and the array size is1243greater than the virtual machine can support in a single request. The maximum1244supported size for the current virtual machine can be queried by submitting a1245request with a zero length array, as described above.1246ENOACCESS The guest does not have permission to submit CCBs, or an address used in a1247CCBs lacks sufficient permissions to perform the required operation (no write1248permission on the destination buffer address, for example). A virtual address1249which fails permission checking is returned as status data in ret2. Some1250CCBs may have been enqueued prior to the error being detected.1251EUNAVAILABLE The requested CCB operation could not be performed at this time. The1252restricted operation availability may apply only to the first unsuccessfully1253submitted CCB, or may apply to a larger scope. The status should not be1254interpreted as permanent, and the guest should attempt to submit CCBs in1255the future which had previously been unable to be performed. The status1256data provides additional information about scope of the restricted availability1257as follows:1258Value Description12590 Processing for the exact CCB instance submitted was unavailable,1260and it is recommended the guest emulate the operation. The1261guest should continue to submit all other CCBs, and assume no1262restrictions beyond this exact CCB instance.12631 Processing is unavailable for all CCBs using the requested opcode,1264and it is recommended the guest emulate the operation. The1265guest should continue to submit all other CCBs that use different1266opcodes, but can expect continued rejections of CCBs using the1267same opcode in the near future.1268126912705311271Coprocessor services127212731274Value Description12752 Processing is unavailable for all CCBs using the requested CCB1276version, and it is recommended the guest emulate the operation.1277The guest should continue to submit all other CCBs that use1278different CCB versions, but can expect continued rejections of1279CCBs using the same CCB version in the near future.12803 Processing is unavailable for all CCBs on the submitting vcpu,1281and it is recommended the guest emulate the operation or resubmit1282the CCB on a different vcpu. The guest should continue to submit1283CCBs on all other vcpus but can expect continued rejections of all1284CCBs on this vcpu in the near future.12854 Processing is unavailable for all CCBs, and it is recommended1286the guest emulate the operation. The guest should expect all CCB1287submissions to be similarly rejected in the near future.12881289129036.3.2. ccb_info12911292trap# FAST_TRAP1293function# CCB_INFO1294arg0 address1295ret0 status1296ret1 CCB state1297ret2 position1298ret3 dax1299ret4 queue13001301Requests status information on a previously submitted CCB. The previously submitted CCB is identified1302by the 64-byte aligned real address of the CCBs completion area.13031304A CCB can be in one of 4 states:130513061307State Value Description1308COMPLETED 0 The CCB has been fetched and executed, and is no longer active in1309the virtual machine.1310ENQUEUED 1 The requested CCB is current in a queue awaiting execution.1311INPROGRESS 2 The CCB has been fetched and is currently being executed. It may still1312be possible to stop the execution using the ccb_kill hypercall.1313NOTFOUND 3 The CCB could not be located in the virtual machine, and does not1314appear to have been executed. This may occur if the CCB was lost1315due to a hardware error, or the CCB may not have been successfully1316submitted to the virtual machine in the first place.13171318Implementation note1319Some platforms may not be able to report CCBs that are currently being processed, and therefore1320guest software should invoke the ccb_kill hypercall prior to assuming the request CCB will never1321be executed because it was in the NOTFOUND state.1322132313245321325Coprocessor services132613271328The position return value is only valid when the state is ENQUEUED. The value returned is the number1329of other CCBs ahead of the requested CCB, to provide a relative estimate of when the CCB may execute.13301331The dax return value is only valid when the state is ENQUEUED. The value returned is the DAX unit1332instance identifier for the DAX unit processing the queue where the requested CCB is located. The value1333matches the value that would have been, or was, returned by ccb_submit using the queue info flag.13341335The queue return value is only valid when the state is ENQUEUED. The value returned is the DAX1336queue instance identifier for the DAX unit processing the queue where the requested CCB is located. The1337value matches the value that would have been, or was, returned by ccb_submit using the queue info flag.1338133936.3.2.1. Errors13401341EOK The request was processed and the CCB state is valid.1342EBADALIGN address is not on a 64-byte aligned.1343ENORADDR The real address provided for address is not valid.1344EINVAL The CCB completion area contents are not valid.1345EWOULDBLOCK Internal resource constraints prevented the CCB state from being queried at this1346time. The guest should retry the request.1347ENOACCESS The guest does not have permission to access the coprocessor virtual device1348functionality.1349135036.3.3. ccb_kill13511352trap# FAST_TRAP1353function# CCB_KILL1354arg0 address1355ret0 status1356ret1 result13571358Request to stop execution of a previously submitted CCB. The previously submitted CCB is identified by1359the 64-byte aligned real address of the CCBs completion area.13601361The kill attempt can produce one of several values in the result return value, reflecting the CCB state1362and actions taken by the Hypervisor:13631364Result Value Description1365COMPLETED 0 The CCB has been fetched and executed, and is no longer active in1366the virtual machine. It could not be killed and no action was taken.1367DEQUEUED 1 The requested CCB was still enqueued when the kill request was1368submitted, and has been removed from the queue. Since the CCB1369never began execution, no memory modifications were produced by1370it, and the completion area will never be updated. The same CCB may1371be submitted again, if desired, with no modifications required.1372KILLED 2 The CCB had been fetched and was being executed when the kill1373request was submitted. The CCB execution was stopped, and the CCB1374is no longer active in the virtual machine. The CCB completion area1375will reflect the killed status, with the subsequent implications that1376partial results may have been produced. Partial results may include full1377137813795331380Coprocessor services138113821383Result Value Description1384command execution if the command was stopped just prior to writing1385to the completion area.1386NOTFOUND 3 The CCB could not be located in the virtual machine, and does not1387appear to have been executed. This may occur if the CCB was lost1388due to a hardware error, or the CCB may not have been successfully1389submitted to the virtual machine in the first place. CCBs in the state1390are guaranteed to never execute in the future unless resubmitted.1391139236.3.3.1. Interactions with Pipelined CCBs13931394If the pipeline target CCB is killed but the pipeline source CCB was skipped, the completion area of the1395target CCB may contain status (4,0) "Command was skipped" instead of (3,7) "Command was killed".13961397If the pipeline source CCB is killed, the pipeline target CCB's completion status may read (1,0) "Success".1398This does not mean the target CCB was processed; since the source CCB was killed, there was no1399meaningful output on which the target CCB could operate.1400140136.3.3.2. Errors14021403EOK The request was processed and the result is valid.1404EBADALIGN address is not on a 64-byte aligned.1405ENORADDR The real address provided for address is not valid.1406EINVAL The CCB completion area contents are not valid.1407EWOULDBLOCK Internal resource constraints prevented the CCB from being killed at this time.1408The guest should retry the request.1409ENOACCESS The guest does not have permission to access the coprocessor virtual device1410functionality.1411141236.3.4. dax_info1413trap# FAST_TRAP1414function# DAX_INFO1415ret0 status1416ret1 Number of enabled DAX units1417ret2 Number of disabled DAX units14181419Returns the number of DAX units that are enabled for the calling guest to submit CCBs. The number of1420DAX units that are disabled for the calling guest are also returned. A disabled DAX unit would have been1421available for CCB submission to the calling guest had it not been offlined.1422142336.3.4.1. Errors14241425EOK The request was processed and the number of enabled/disabled DAX units1426are valid.142714281429143014315341432143314341435