Collecting data in the field has typically been done by pen and paper. But, the methods are changing. Tools like Survey123 from Esri are now being used to collect data electronically, making data collection and synchronization easier.
Each survey form is highly customizable so you have the flexibility to design a form that best fits your organizations needs. The form can include useful features like a calendar pop up when submitting a date or a map to submit geographical data. These features ensure that dates and locations are submitted correctly, saving time and energy correcting them manually later.
The best part is all this data you’ve collected will automatically sync with other ArcGIS Platform products that you have. Data no longer has to be manually entered into the ArcGIS platform like back in the paper form days. This saves time and the tedious task of data entry.
Survey123 is used across many organizations from the National Park Board using it to enhance citizen science or emergency services using it to assess damage after a storm. Survey123 can really be used by any organization collecting geospatial data out in the field.
Survey123 and its role in the full ArcGIS Platform can be extremely useful, but there are times where users may want to do more than what the platform allows. With external data integration and transformation tools, Survey123 data can be made even more useful.
Analyzing and Extracting Survey123 Data
After the data has been collected, you may want to export the data from Survey123 for further analysis. Currently, Survey123 data can be exported on demand to CSV, Excel, KML, shapefile, or a file geodatabase. Survey123 also has a premium service called Report Templates that allows you to create reports with their survey data and export it to Word documents.
Exporting data from Survey123
However, there may be times where you want to send data automatically to other applications, like Cityworks or PowerBI. Or, you may want to automatically perform a QA process to ensure that the data being collected is accurate. Or, perhaps you want to receive an email every time a new survey is submitted. Currently, those options don’t exist within Survey123, but with third-party applications, almost anything can happen.
Tools like FME can be very useful in these cases. And it’s the perfect tool to use alongside Survey123 because FME is the full-package version of the Data Interoperability Extension found in ArcGIS.
FME is a really flexible tool for reading, writing, and analyzing data. With FME, users can either read in all the data stored in an ArcGIS Online or Portal feature service. Or they can get data directly from Survey123 as soon as a new record has been submitted.
“Safe Software sets the standard for spatial data interoperability, so it was an obvious choice to use their FME technology for the Esri ArcGIS Data Interoperability Extension.” – Jack Dangermond, Esri President
Automatically Integrate with Other Applications
For me, the fun really starts when data is extracted as soon as a record is submitted in Survey123. This data can be processed immediately and sent to other applications so that your data is synchronized no matter where it is stored. FME supports over 400 different applications out of the box and can connect to APIs as well so the possibilities are really limitless.
This instant transaction of data between Survey123 and FME can be accomplished through the power of webhooks. Webhooks send messages through a URL as soon as an action occurs.
Creating a webhook in Survey123
An FME Server automation where a webhook triggers an action.
Some common Survey123 and FME examples we see are sending information from the survey to a database or an asset management system outside of the ArcGIS platform. This ensures that any external system is automatically updated with the Survey123 data as soon as it’s collected. Time is no longer spent manually entering data from one system to another, instead it’s synchronized immediately!
Run Data Through Data Validation Workflows
Saving time by automatically integrating collected data into your external systems is great. It’s even better when you know all your data is errorless because you’ve been able to apply data validation rules to all datasets.
To validate data from Survey123 using FME, use a data validation workflow to check your Survey123 data before it is written out to any format or application you choose.
An example HTML data validation report created using FME.
Validation in FME is accomplished by using transformers. An AttributeValidator is one transformer used to ensure that the data quality meets your organization’s standards. Using the AttributeValidator you can set up custom rules to ensure your attributes are in the correct format, have a particular value, or are in the correct range. If attributes fail validation you will be informed which attributes fail and why.
Data validation is not limited to text in FME. Geometry can be validated too using, you guessed it, the GeometryValidator. The GeometryValidator lets you select which rules you’d like to check for and even gives an option to repair issues automatically. If these transformers don’t do exactly what you are looking for, there are always many more options in FME. This data validation tutorial is a great place to start.
The best thing about FME is that workflows are always repeatable. After you’ve created your workflow you can choose to run it manually whenever you’d like or automate it with FME Server. For more information on integrating data from ArcGIS Online, I’d highly recommend checking out this webinar or blog for more information.
Keep Stakeholders Informed (Outside Your GIS Team)
Send emails using FME Server.
Putting It All Together
Now, let’s see what this all might look like when put into action.
An example workflow could be a tree hazard report in Survey123. Once a tree hazard has been reported in Survey123, an inspection report could be automatically created in Cityworks by using FME. This workflow could also send a CSV file to Dropbox, or the file can be processed by Qlik to update a dashboard.
Finally, an email could be sent to the user who filed the issue in Survey123 to confirm their survey has been received. A workflow like this would ensure that all key stakeholders have the most up-to-date information and all the systems used have the same synchronized information.
If you are interested in learning more about automating data extraction from Survey123 check out this video on Automating Workflows from Survey123 to ArcGIS using FME Server.
Interested in Learning More?
If you’re interested in learning more, I’d highly recommend checking out the resources below:
- Knowledge Article: Getting Started with Automations
- Knowledge Article: Automating Workflows from Survey123 to ArcGIS using FME Server
- Knowledge Article: How do I write updates to an existing ArcGIS Online feature service?
- Blog: Connect to APIs and Webhooks in No Time (with No Code)
Sienna Emery
Sienna is one of Safe’s FME Server Technology Experts. When not in the office, Sienna can be found competing for the fastest time in spin class or playing board games with friends.
