- Notifications
You must be signed in to change notification settings - Fork556
Java OTF User Guide
Some applications, such as network sniffers, need to process messages dynamically and thus have to use theIntermediate Representation to decode the messages on-the-fly (OTF). An example of using the OTF API can be foundhere.
The Java OTF decoder follows thedesign principles of the generated codec stubs, and it is thread safe to be reused concurrently across multiple threads for memory efficiency.
Note: Due to the dynamic nature of OTF decoding, the stubs generated by the SBE compiler will yield greater relative performance.
Before messages can be decoded it is necessary to retrieve the IR for the schema describing the messages and types. This can be done by reading the encoded IR into aByteBuffer and then decoding it using the providedIrDecoder.
privatestaticIntermediateRepresentationdecodeIr(finalByteBufferbuffer)throwsIOException {finalIrDecoderirDecoder =newIrDecoder(buffer);returnirDecoder.decode(); }
Once the IR is decoded you can then create the OTF decoder for the message header:
// From the IR we can create OTF decoder for message headers.finalOtfHeaderDecoderheaderDecoder =newOtfHeaderDecoder(ir.headerStructure());
You are now ready to decode messages as they arrive. This can be done by first reading the message header then looking up the appropriate template to decode the message body.
// Now we have IR we can read the message headerintbufferOffset =0;finalUnsafeBufferbuffer =newUnsafeBuffer(encodedMsgBuffer);finalinttemplateId =headerDecoder.getTemplateId(buffer,bufferOffset);finalintactingVersion =headerDecoder.getTemplateVersion(buffer,bufferOffset);finalintblockLength =headerDecoder.getBlockLength(buffer,bufferOffset);bufferOffset +=headerDecoder.encodedLength();
Note: Don't forget to increment thebufferOffset to account for the message header size!
Once you have decoded the header you can lookup the IR for the appropriate message body then begin decoding.
intbufferOffset =0;finalUnsafeBufferbuffer =newUnsafeBuffer(encodedMsgBuffer);finalinttemplateId =headerDecoder.getTemplateId(buffer,bufferOffset);finalintschemaId =headerDecoder.getSchemaId(buffer,bufferOffset);finalintactingVersion =headerDecoder.getSchemaVersion(buffer,bufferOffset);finalintblockLength =headerDecoder.getBlockLength(buffer,bufferOffset);bufferOffset +=headerDecoder.encodedLength();// Given the header information we can select the appropriate message template to do the decode.// The OTF Java classes are thread safe so the same instances can be reused across multiple threads.finalList<Token>msgTokens =ir.getMessage(templateId);bufferOffset =OtfMessageDecoder.decode(buffer,bufferOffset,actingVersion,blockLength,msgTokens,newExampleTokenListener(newPrintWriter(System.out,true)));
The eagle eyed will have noticed theTokenListener. If you are wondering what this is then wonder no longer and read on.
As messages are decoded a number of callback events will be generated as the structural elements of the message are encountered. The callbacks are received by implementing theTokenListener interface. If you only want to receive some of the callbacks then extendAbstractTokenListener.
Primitive fields are the most common data element to be decoded. These are simple types such as integers, floating point numbers, or characters. Primitive field encodings can be a single value or a fixed length array of the same type. To receive primitive values override the following method:
publicvoidonEncoding(finalTokenfieldToken,finalDirectBufferbuffer,finalintindex,finalTokentypeToken,finalintactingVersion) {finalCharSequencevalue =readEncodingAsString(buffer,index,typeToken,actingVersion);printScope();out.append(fieldToken.name()) .append('=') .append(value) .println(); }privatestaticCharSequencereadEncodingAsString(finalDirectBufferbuffer,finalintindex,finalTokentypeToken,finalintactingVersion) {finalPrimitiveValueconstOrNotPresentValue =constOrNotPresentValue(typeToken,actingVersion);if (null !=constOrNotPresentValue) {returnconstOrNotPresentValue.toString(); }finalStringBuildersb =newStringBuilder();finalEncodingencoding =typeToken.encoding();finalintelementSize =encoding.primitiveType().size();for (inti =0,size =typeToken.arrayLength();i <size;i++) {mapEncodingToString(sb,buffer,index + (i *elementSize),encoding);sb.append(", "); }sb.setLength(sb.length() -2);returnsb; }privatelongreadEncodingAsLong(finalDirectBufferbuffer,finalintbufferIndex,finalTokentypeToken,finalintactingVersion) {finalPrimitiveValueconstOrNotPresentValue =constOrNotPresentValue(typeToken,actingVersion);if (null !=constOrNotPresentValue) {returnconstOrNotPresentValue.longValue(); }returngetLong(buffer,bufferIndex,typeToken.encoding()); }
The above code will output the values as strings to the console.
Note: Constant and optional fields are handled by using the metadata provided in thetypeToken.
Enums are encoded on the wire as simple integers or characters. It is necessary to lookup the encoded representation via the metadata tokens to understand the wire encoded value.
publicvoidonEnum(finalTokenfieldToken,finalDirectBufferbuffer,finalintbufferIndex,finalList<Token>tokens,finalintbeginIndex,finalintendIndex,finalintactingVersion) {finalTokentypeToken =tokens.get(beginIndex +1);finallongencodedValue =readEncodingAsLong(buffer,bufferIndex,typeToken,actingVersion);Stringvalue =null;for (inti =beginIndex +1;i <endIndex;i++) {if (encodedValue ==tokens.get(i).encoding().constValue().longValue()) {value =tokens.get(i).name();break; } }printScope();out.append(fieldToken.name()) .append('=') .append(value) .println(); }
BitSets are represented on the wire as an integer with a bit set in the position indicating true or false for the choice value.
publicvoidonBitSet(finalTokenfieldToken,finalDirectBufferbuffer,finalintbufferIndex,finalList<Token>tokens,finalintbeginIndex,finalintendIndex,finalintactingVersion) {finalTokentypeToken =tokens.get(beginIndex +1);finallongencodedValue =readEncodingAsLong(buffer,bufferIndex,typeToken,actingVersion);printScope();out.append(fieldToken.name()).append(':');for (inti =beginIndex +1;i <endIndex;i++) {out.append(' ').append(tokens.get(i).name()).append('=');finallongbitPosition =tokens.get(i).encoding().constValue().longValue();finalbooleanflag = (encodedValue & (1L <<bitPosition)) !=0;out.append(Boolean.toString(flag)); }out.println(); }
A little bitwise manipulation is required to determine if a each choice is true or false as in the example above.
A composite is a reusable collection of fields to simplify the assembly of messages. The collection of fields usually has a semantic significance. Fields within a composite are decoded just like normal fields. Composites are signalled via callbacks to indicate the beginning and end of the composite. In the example, the begin and end are captured to scope fields by adding the scope to a stack in the exampleTokenListener.
publicvoidonBeginComposite(finalTokenfieldToken,finalList<Token>tokens,finalintfromIndex,finalinttoIndex) {namedScope.push(fieldToken.name() +"."); }publicvoidonEndComposite(finalTokenfieldToken,finalList<Token>tokens,finalintfromIndex,finalinttoIndex) {namedScope.pop(); }
Fields can be semantically bound into a repeating group. On the wire the repeating group has a header that defines the size in bytes of the block of fields and a count of how many times the block will repeat. Repeating groups are signalled by callbacks to indicate the beginning and end of block of fields with counter details for the iteration count and the number of times it will repeat in total.
publicvoidonBeginGroup(finalTokentoken,finalintgroupIndex,finalintnumInGroup) {namedScope.push(token.name() +"."); }publicvoidonEndGroup(finalTokentoken,finalintgroupIndex,finalintnumInGroup) {namedScope.pop(); }
Note: Repeating groups can nest so it is necessary to be prepared to handle this scope recursively.
At the end of a message it is possible to encode variable length strings or binary blobs. Strings are binary data that uses a schema defined character encoding.
publicvoidonVarData(finalTokenfieldToken,finalDirectBufferbuffer,finalintbufferIndex,finalintlength,finalTokentypeToken) {finalStringvalue;try {buffer.getBytes(bufferIndex,tempBuffer,0,length);value =newString(tempBuffer,0,length,typeToken.encoding().characterEncoding()); }catch (finalUnsupportedEncodingExceptionex) {ex.printStackTrace();return; }printScope();out.append(fieldToken.name()) .append('=') .append(value) .println(); }