Shattering the silos

So let’s backtrack to the beginning of the project when we were just making the first building blocks for the platform and we were depicting the southbound part of the architecture. We began by looking at Open Data Portals and ways to integrate data into the platform to better understand which standards and protocols we should support. Little did we know, there were many more implemented protocols than what we were expecting in the wild. After understanding this we changed the focus of our strategy and began by implementing and testing each of the protocols.

Right now, the main focus was to tests the platform with an ever-growing array of protocols (HTTP, MQTT, CoAP), IoT standards (OneM2M, NGSI, LwM2M) and mobility standards (GTFS, GTFS-RT, DATEX-II, etc).

When we had all of the protocols implemented we ended up with a strong interoperability layer that is key to develop any mobility (and Smart City) platform. And it was time to start the load testing on the instances, to define the limits of the platform, to understand the performance differences of the protocols (if you have ever designed a platform to integrate with sensors you might be familiar with the pains of choosing protocols).

The load testing began and we were starting to see many issues with the southbound part of the platform and to be fair, the design wasn’t bad, but the technologies that we had chosen were, unfortunately, not the best. It was not too late, we still had little data on the platform and really no deployments (at this point) so we just rebuilt it.

This would turn out to be one of the key decisions that made the project a success.

We did not go for a full platform rebuild, though, we just re-designed the interoperability layer to make it more scalable and stable. The first step was to change the amount of protocols we were able to integrate and include more. Then, since we were having different types of integrations (e.g. push-based systems, poll-based systems, datasets, etc) we created a pathway for anyone to be able to include their datasets on board. All in all, we went on a new architectural path that would turn out to be one of the cornerstones for the project.

By rebuilding this layer we were on a slightly different path because we had field experience on the domains, protocols, standards and types of integration (push or pull). This completely changed the way we looked at the integration and interoperability layer and made us implement an additional component on this layer that would harmonise any dataset into a single standard and protocol to ease the later processing made on the platform. After this was implemented we repeated (and increased) load testing on the platform to understand its limitations and deploy it with those in mind.

All the problems that we’ve faced helped us to build a multi-standard central platform that would shatter every silo in the IoT/mobility scene. Different flavours for integration are available, depending on the current scenario of the city/service provider/developer, but the main goal remains the same, the platform is central and able to integrate with most of the known standards and protocols while harmonising them into the central platform for processing and delivering to final users.

Note: This article is the second of a series of articles to be published about the course of the EMBERS project.

This project has received funding from the EU´s Horizon 2020 research and innovation programme under grant agreement No 687992