Padding

Use case:

Telecommunications network elements produce call data records (CDRs). The network element that produces the CDRs uses an internal fixed-sized buffer, for example, an 8 kB buffer that collects several CDRs and send them as a chunk. Each chunk of CDRs starts with a header and ends with a trailer structure.

In most cases, there is a gap after the trailer structure, but that gap is too small to keep a call detail record. These bytes are set to, for example, 255 (0xFF).

Padding, or skipping a block of 0xFF values, is allowed between trailer and header only. The StructureParse operator must be able to skip blocks of useless data.

Solution:

A padding variable is declared and set to true. In other words, the first allowed structure is the header structure, and in front of the header, padding is allowed.
A detected header disables padding by setting the variable to false.
A detected trailer enables padding by setting the variable to true again.
A padding structure is detected if the variable is set to true and if only the uint8 field has the padding value. The padding structure skips all consecutive padding bytes (value 0xFF).

For more information, review the teda.sample.StructureParse.Padding sample application.


<?xml version="1.0" encoding="UTF-8"?>
<structures
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.ibm.com/software/data/infosphere/streams/parser StructureParseStructure.xsd"
	xmlns="http://www.ibm.com/software/data/infosphere/streams/parser"
>
	<!--
		The variable indicates whether padding is allowed (true) or forbidden
		(false).
	-->
	<variable name="padding" type="boolean" value="true"/>
	<!--
		The header
		After detecting the header, padding is not allowed anymore until a
		trailer is detected,
	-->
	<structure name="header">
		<condition>
			<cmp op="equal">
				<field name="RECORD_TYPE"/>
				<value>0</value> <!--header -->
			</cmp>
		</condition>
		<action>
			<set value="false" as="padding"/>
			<submit/>
		</action>
		<field name="RECORD_TYPE" type="uint8"/>
		<field name="INDEX" type="uint8"/>
	</structure>
	<!--
		The trailer
		After it detects the trailer, padding is allowed again.
	-->
	<structure name="trailer">
		<condition>
			<cmp op="equal">
				<field name="RECORD_TYPE"/>
				<value>1</value> <!-- trailer -->
			</cmp>
		</condition>
		<action>
			<set value="true" as="padding"/>
			<submit/>
		</action>
		<field name="RECORD_TYPE" type="uint8"/>
		<field name="INDEX" type="uint8"/>
	</structure>
	<!--
		The padding structure detection is active only between trailers and
		headers. A padding structure has at least one byte, but it can be of
		any length. In addition, padding byte sequences cross tuple borders.
	-->
	<structure name="padding">
		<condition>
			<logicOp op="and">
				<cmp op="equal">
					<variable name="padding"/>
					<value>true</value>
				</cmp>
				<cmp op="equal">
					<field name="PADDING"/>
					<value>255</value>
				</cmp>
			</logicOp>
		</condition>
		<action>
			<skip>
				<uint8 value="255"/>
			</skip>
		</action>
		<field name="PADDING" type="uint8"/>
	</structure>
	<!--
		Synchronize in case of failures.
	-->
	<sync>
		<structure name="header"/>
	</sync>
</structures>