FME Server Notifications and Alerts: By Mark Ireland
Following up from my last post – a review of FME Server – this post dives into alerts and notifications in as much detail as I can. I’m guessing you’ve seen demonstrations and movies on the subject, but it’s difficult to convey every single detail in a movie – unless you can get funding to spend eight years in New Zealand filming a trilogy of epic proportions.
I did ask, but without luck. So I’m going to use this article to show you how it all comes together, step by step.
NB: This article uses FME2012. This functionality emerged in a 2012 service pack and I haven’t yet covered it. I’m not going to use FME2013 as it is still changeable, but I’ll update you on any changes in the future.
Alerts and Notifications
Before reading further, you might want to revisit my previous post because it shows what alerts are and how they fit into FME Server as a whole……
OK? Now just to clear up a little terminology.
When we talk about alerts and notifications we’re really just talking about messages. There isn’t a specific difference between a notification and an alert. In general terms, I do see Notifications used more for INCOMING messages; i.e. an event occurs and FME Server is notified; and see Alerts used more often as the OUTGOING messages; i.e. FME Server alerts users. But that’s not a particular rule. Just think of these all as spatial messages passing to and from FME Server as they zip around the world:
And, because you read my last post, you’ll know that you don’t always get a notification AND an alert in the same setup. FME might receive a notification and take an action that doesn’t lead to an alert; or it might generate an alert in response to something other than a notification. You don’t have to include both.
While I’m covering terminology, other terms you might hear are Topic and Subscriber. A Topic is like a message subject. It’s so FME knows where (which workspace or subscriber) to send the message to. Think of the FME notification service as a postmaster, with topics being an address. In fact, it’s not a 1:1 relationship between topic and destination; each topic can be passed on to multiple destinations, which Don calls “the ultimate in loose coupling”.
The term Subscriber – which I just used above – simply means the person who is asking to receive alerts about a particular topic. They will need to subscribe to that topic in the same way that you might subscribe to an online forum to receive messages from it.
Here, you might say, each person has subscribed to a single different topic (signified by a different color); though the image doesn’t do justice to the world of FME, where they could subscribe to multiple topics.
And remember, in between receiving and sending messages is an FME workspace for data processing. That workspace might just write data as a “normal” translation, or it might trigger a message of its own. In 2013 we have a special function to trigger an alert once a workspace is completed, with different messages for translation success and failure.
For example, an incoming message might contain a neighbourhood boundary (in format X) which triggers the workspace to find a list of coffee shops (what else?) within that boundary and send it out as an outgoing message (in format Y).
NB: Technically speaking, yes, FME Server can input and output messages without a workspace. But a workspace is the key functionality that FME brings to the table: the ability to carry out spatial data transformations on messages of nearly any format. A blend of live messaging with Spatial ETL that is unique.
Meet RAFA
So I’m going to create an example project for notifications and alerts, and it will use the FME iPhone apps.
Did you know that Safe Software has FME iPhone apps in the Apple store? They are a proof of concept for the idea of mobile apps for FME Server notifications, and they are going to be the key to my project.
So, what I want to do today is put together a little application called RAFA. RAFA is short for Random Acts of FME Awesomeness. The idea is that whenever FME does something great, a user can notify the whole world, using the FME Reporter app. Anyone who has subscribed to these notifications will therefore receive an alert (on the FME Alerter app) with the message about FME awesomeness.
Basic Structure
The basic structure of my demo project is this:
The FME Reporter app (it says “notifier” in the image, but that’s wrong) will record an event (FME awesomeness) and notify FME Server. FME Server, listening for such an event, passes on the information to a workspace. The workspace will process the information and then trigger a notification. The FME Alerter app will receive the message and alert the end user to another act of FME awesomeness.
Notice that we need two notification components/topics. They are each set up to pass information in a certain direction and if we tried to re-use the same one twice we’d get into a mess. When you design a system you need to be really clear with yourself which topics are for publishing information to, and which are for issuing alerts.
So, we have a workflow of topic – workspace – topic. In this case the topics are making connections to the FME apps, but in reality they could connect through multiple protocols and send emails, tweet, write to FTP, etc all being triggered by a single topic.
First Steps
Perhaps most confusing is the order in which we create the links in the above chain. I like to start with the output side of things, because I think it’s easier to verify it’s working. But I also need to begin with a link that is independent; for example I couldn’t set up the FME Alerter app first because I don’t yet know what the topic is that I’m going to connect to.
So my first step here is to create the alert notification (the second of the two notification objects). I can do that without reference to any other item:
I will do this in the FME Server administrator web interface.
I click Notifications and straightaway notice that a notification is made up of two parts: a subscription and a topic. The subscription is the connection information and action to take, while the topic just a theme or title. They are separate because it allows an action to be triggered by multiple topics (i.e. the same connection can be used to alert you to lots of different events).
So I click Topics and then click the New button to create the Topic (in the above screenshot it is already created). The creation dialog is really just giving the topic a name. Here I call it RAFAAlerts.
Now I click on the Subscriptions tab, create one of those, and this is where the main configuration takes place:
Notice that I’ve defined this as an Apple Push Notification to occur in response to the RAFAAlerts topic. So when that topic is triggered, this notification will fire off an alert to the FME Alerter application (defined by the keystore path and password).
FME Alerter App
So that’s the notification defined, now step 2: Setting up the FME Alerter app to receive it:
I download and install the app onto my iPhone, fire it up and enter the settings like this:
All I’m entering is a server name, username and password, and then selecting the subscription (topic) to monitor.
Notification Trigger
So now my alert mechanism is set up and I just need a way to trigger the RAFAAlerts topic. There are many ways to do this, but for this project it will be by the RAFA workspace:
I’ll start out with the simplest workspace possible:
You can see there’s just a Creator transformer to start things off, an AttributeCreator to create a message (FME ROX!), and an FMEServerNotifier to trigger the topic. Notice in the FMEServerNotifier parameters that I’m connecting to my FME Server and sending a message to the RAFAAlerts topic.
I then upload this workspace to FME Server.
And that’s it for phase 1 of my project. I can run this workspace now (on FME Server or even FME Desktop) and it will send a message to the FME Alerter app on my iPhone. Here I’m using it as part of a larger project, but you could use even just these components, to send out an alert when a translation finishes or perhaps when bad data is encountered; many users use Twitter or email to do the same.
The Second Notification
For my project, the next task is to start the process out from the FME Notifier app, rather than just running a workspace:
This is done in the FME Server interface by creating a second topic and a second notification object, like this:
Notice that the type is now a simple Push notification, the target is the URL to my workspace, and the topic is one I just created called RAFA. So, when this topic (RAFA) is triggered, the workspace will run, trigger the other topic (RAFAAlerts), and send an alert. This is why we need different topics for notification and alert; if it were the same the incoming alert would bypass the workspace.
FME Reporter App
Now I’ll set up the FME Reporter app:
Again I download and install it, and enter the settings like so:
This time the topic is RAFA, the one being handled by the first notification component. The screenshot shows a different account here, but that’s just me getting confused! There is space for a message, but for now it’s not going to do anything (because my workspace isn’t – yet – set up to handle it).
OK. I click Done and return to the main screen:
From now on, I will click the Send button whenever FME commits an act of awesomeness for me. The report will go through the first notification, trigger the workspace, cause the second notification to engage, and a message to pop up in the FME Alerter app for anyone following RAFAAlerts.
If I clicked Start, by the way, I would be sending a stream of RAFA notifications, one every second. I know FME is awesome and all, but that might be overdoing it a bit! That functionality is more designed for sending notifications to track moving objects.
Once set up, a user could download the Notifier app, fill in the parameters as shown, and send messages when FME does something awesome. And I can monitor the RAFAAlert notification to get alerted when it does. I think that’s pretty awesome in itself.
Updated Workspace
However, one more task remains. At the moment the workspace is just sending out alerts with a fixed text message. I’d prefer it to send me the message you enter into the FME Reporter, and tag it with a date and lat/long. To do this I need to update the workspace to receive content from the first notifier. It’s pretty easy though, as you will see, it does have one little quirk!
All I need to do is add a reader to the workspace and choose to have the incoming message sent to it. I’ll use a Text File reader because the message content is plain text. Then when I publish the workspace to FME Server, I’ll open the parameters for the Notification Service and tell it this is the reader to send data to:
So now the message from FME Reporter is going into the workspace. But, of course, I have to handle it properly and use it as the message to send out as a notification. That’s going to be tricky. The incoming message will be in JSON format and I have no idea what the content or attribute names will be.
Problem is, I can’t get this info by running the workspace locally because it won’t have the incoming notification. So what I do is this: I add a Logger transformer after the reader and publish it to FME Server. Then I trigger the workspace with the FME Reporter app, look through the Server job history, pick out the job it just ran, and download the log file:
I can now see in the log what the contents of the format attribute text_line_data are. Now I can decide which transformers I need to decipher this content. The JSONFlattener is a good one. I just keep doing this, run-log-refine, until I have the content I want to send to the FME Alerter app. The final workspace looks like this:
I’m pulling out the information I need with the JSONFlattener and StringReplacers, and pasting together a suitable message in the StringConcatenator. Incidentally, I don’t think it’s documented but the FME Alerter expects a message in the form:
<Title>|<Message>|<Lat>,<Long>
…which is easy to create.
Now, whenever anyone sends a message about FME awesomeness with the FME Reporter app, I can get an alert with the FME Alerter app:
Notice the custom message (from the Notifier) and the date, time, and location. I can even tap on the map icon to see where this event occurred (hey, we are spatial you know).
Email In?
You’re probably now thinking this is great, but you’d like to send/receive info another way – maybe by email. So let’s see how we can do that.
This will be a different project from RAFA. This one is going to send an alert to subscribers to let them know when a new FME beta build is released. The information will come from an email notification from our Build and Release team.
To have an FME Server notification receive notification by email requires 2013 beta. The interface isn’t complete yet, so it looks fairly sparse, but it is very simple anyway; you get a list of topics and simply click a button to enable a particular topic to receive info by email. The email address is <topicname>@<servername>
For example, here I’ve created a topic called FMEBetaBuilder:
…by enabling email I can trigger that topic by sending an email to FMEBetaBuilder@dondemo.safe.com
That topic can, again, fire a workspace that sends a notification to the FME Alerter. So the FME build process sends an email to that address, it runs the workspace, the workspace sends out a notification, and you get it on the iPhone. The trick is just deciphering the email message (which has a different structure than notifications from the FME Reporter app).
Email Out?
To send the notification by email to the recipient (rather than use the Alerter app) is not much more difficult; you just need the subscription for that topic to be of the email type, rather than the Apple app. Like this:
Of course, the nice thing is that you can have multiple notifications for a single topic; like several postmasters sending messages to the same postcode. So I can have an Apple notification for iPhone users and an email notification for anyone else, and trigger both of them at once by tying them both to the same topic.
And FME Server now has an email server built into it, so you can set up incoming/outgoing messages with the minimum of configuration.
Conclusion
So there you are; that’s how you can create alerts and notifications using FME Server, in particular for the apps in the Apple store.
Although these apps are “proof of concept” you are free to download and use them. Provided you have FME Server 2012 (SP2 I think) you’ll be able to use them to create your own FME-based mobile tools. It didn’t take me long (say 8 hours) to pull this example together, and I didn’t have this article as a guide! Hopefully you’ll be able to do something like this even more quickly.
The one thing you are missing is the Apple Push Notification setup on FME Server. You’ll need an Apple developer account I believe, to get your own unique ID. Get in touch with us via our support team and we can help you with setting up this side of things. If you don’t have an iPhone, well I hope we’ll have equivalent apps for Android in the near future.
And if there is particular functionality you’d like to see then let us know; we might be able to add it in for you. As my next Server post intends to show, there really is no limit to what you can do with tools like these!
Regards
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
