Disease surveillance is an essential tool used by public health officials to monitor disease activities, predict outbreaks, and inform actions. With a growing need for early detection of infectious diseases, public health officials are challenged with a huge volume of data from disparate sources that require advanced data integration to be ready for further analysis.
BlueDot, a Toronto-based start-up, sent out the first warning of COVID-19 on December 31, 2019 through their AI-powered early warning system for disease outbreaks. It would be another seven days before the World Health Organization announced the emergence of a novel coronavirus. How did BlueDot do it?
BlueDot uses AI to track, locate and conceptualize infectious disease spread. Its proprietary machine learning algorithms detect credible information from hundreds of thousands of sources globally. How do they get the data ready for AI? The answer is: data integration, and to do it, they chose FME, the spatial data integration platform.
“FME has been an integral solution for orchestrating data processing at BlueDot,” says Adriano Nicolucci, BlueDot’s Senior Manager of Data Engineering. With their ambitious project which relies on a huge array of datasets, BlueDot needed an enterprise integration solution that supports diverse data types and applications, meets constantly changing data requirements, and works with both cloud and on-premise databases.
Integrating a huge Array of Data Sources
BlueDot’s products are powered by a huge volume of data from diverse sources, such as official health notifications, health forums, news articles, travel data, and remote sensing data. They leverage FME to integrate health-related data into their system and automate data processing workflows by having them either run on schedules or triggered by specific events. For example, they use FME to orchestrate the data processing job to scrape disease related information from websites. This way, they can capture and process tons of data in real-time.
Managing Peaks in Processing Demands
Besides the huge volume of diverse datasets, BlueDot must manage constantly changing data requirements. As a result of the pandemic, an explosive amount of information related to COVID-19 entered their system, which dramatically increased their data processing needs. As an early adopter of Dynamic Engines in FME Server when the pilot was introduced in March this year, BlueDot was ready for the challenge. They are able to spin up as many engines as needed across multiple machines to manage peaks in processing demands.
Scaling Through Enterprise Integration
Working with different types of applications, BlueDot leverages FME to build enterprise integration patterns as they provide standardized data integration logic and are easy to set up. With automation capabilities in FME, they can easily set up data processing jobs to be triggered by events in decoupled applications or services. Since the patterns are built into the automation with no coding needed, they can easily build complex enterprise integration solutions in a more efficient and scalable way.
Making Data Accessible On-Premise and On-Cloud
BlueDot uses FME to feed data into their data warehouse where artificial intelligence takes over to analyze the data and detect credible information, and then synchronizes processed data with their products so that their health expert team can further assess the data and identify potential health threats to the public. With FME’s support for cloud databases, they are able to support their clients by delivering data and analytics insights to clients’ preferred cloud data storage locations, such as Amazon S3, ArcGIS Online and Snowflake. Those analytics insights include social distancing metrics from the neighbourhood to the national level as well as risk of infectious disease importation from known hot spots.
“FME is so versatile that we found it was not just useful in one project but rather many projects,” Adriano says. “Our workflows for data integration are complex and FME workspaces are visual which makes it easier for other teammates to quickly understand the data journey.”
With an enterprise integration solution in place, BlueDot is able to monitor nearly 200 diseases around the world on a daily basis in order to identify disease activities and contextualize outbreaks. By automating data processing workflows, they can free up time for the team to focus on other projects. Being data-prepared at the beginning of an outbreak is vital to timely emergency response, helping governments, healthcare organizations, businesses and individuals mitigate risks and safeguard lives.
Are you researching or working on projects related to COVID-19? We are offering free FME licenses to help you find solutions and keep your communities safe and healthy.
Vivian Tang
Vivian is a Product Marketer who’s passionate about aligning values with customers and bringing to market products users need and desire. Outside the office, you can find her wandering in museums and art galleries, searching good food and coffee in the city, and exploring the great outdoors.
