Punctuation processing

Punctuation markers in SPL create boundaries. Window punctuation create boundaries within a stream that can be used for creating user-defined windows. Final punctuation is used to mark the end of streams.

void MY_OPERATOR::process(Punctuation const & punct, uint32_t port) {
  if(punct==Punctuation::WindowMarker) {
    // do window punctuation processing
    // forward if necessary
  } else if(punct==Punctuation::FinalMarker) {
    // do final punctuation processing
  }
}

An operator's window marker punctuation behavior is specified in the operator model, through window punctuation modes of input and output ports of the operator.

Depending on the semantics of the operator, a window marker might or might not be of interest to the operator. If it is of interest, the punctuation processing function enables the operator developer to act on that event. One example is when an operator has a window punctuation-expecting input port and wants to act when a window punctuation is received. Another example is when an operator has a punctuation-preserving output port, in which case the operator developer is responsible for forwarding the window punctuation received. The operator can call the submit(punct, oport) for the output port in question, or call the forwardWindowPunctuation(punct) member function of the Operator class, which forwards the punctuation to all output ports (in case that is the behavior that is defined in the operator model). It is important to note that the window punctuation forwarding behavior of the operator is specified in the operator model. The implementation of this behavior is provided by the operator developer and cannot be automated, since the forwarding of punctuation might require operator-specific semantics, such as merging punctuation or queuing them up within the operator.

As a concrete example, the Split operator defines its output ports as forwarding window punctuation from the single input port it has. The Split operator implements this forwarding as follows:

void MY_OPERATOR::process(Punctuation const & punct, uint32_t port) {
  if(punct==Punctuation::WindowMarker)
    forwardWindowPunctuation(punct);
}

The final marker punctuation, unlike the window punctuation, are automatically handled by the SPL language run time, which is based on the wanted behavior that is expressed in the operator model. In other words, the operator model is used to tell the SPL compiler and the language run time what the operator wants the run time to do on the operator's behalf. Final punctuation is delivered to the operator through the process punctuation function, in case the operator wants to process upon receiving a final punctuation on an input port. As enforced by the SPL language run time, no further tuples are received on an input port for which the final punctuation was received. Common use cases for handling final punctuation include signaling threads that are processing tuples asynchronously and flushing pending buffers or windows. The final punctuation is forwarded to operators downstream through the finalPunctuationPortScope element on the output port automatically by the SPL language run time.

It is important to note that any necessary final punctuation forwarding will take place after the call to the process punctuation function returns from the operator-specific handler (if it exists). For source operators, which do not have input ports, the final punctuation is sent on output ports, automatically by the run time, when all operator threads complete their execution. An operator might also submit final punctuation on its output ports to indicate the end of a stream.