In NuvIoT Basics Part Three, A Good Plan, we came up with the plan for how the messages from the IoT device should be processed based on the Device ID and Message ID. The plan is actually a route that the message will flow through to be processed by modules in the pipeline. To make this process as efficient as possible, we take the original message, extract all the relevant data, and put that into something called a Pipeline Execution Message (PEM). This PEM contains everything necessary to process the message, including all the current information about the device in the form of a device twin. The PEM also contains the plan for the route.
After the plan is developed, the message is handed off to the runtime engine. The runtime engine determines the current step within the plan, finds the queue for the runtime module that knows how to process that step, and enqueues the PEM for execution. Once that module completes its work, it determines the next module in the plan and hands it off for execution. After the message hits the final module, the message is then marked as completed and any changes to the device are stored.
If any steps along the way fail, the message is then “dead-lettered” and changes to the device are not applied. Using this sort of pipeline processing strategy is very powerful and lends itself to building both reliable and scalable solutions. Some modules may have heavier work loads then others, and can be scaled accordingly. In addition, using technologies like Kubernetes, the processing modules can be made redundant and self- healing. In addition, built-in analytics track things like how long it took to process the message as well as how long it sat in the queue waiting to be processed.
NuvIoT has a predefined set of core modules that almost any IoT application will need. But the advantage of our solution is that it allows you to create custom modules, either by providing a REST endpoint or by creating a new module from scratch by following a simple specification.
Be sure to check out our full IoT Basics series: