FME Evangelist’s State of the Server Address: By Mark Ireland
It might be the “run-up” to FME2013, but for FME Server I’m going to have to sprint to keep up with the many updates that have been implemented. In fact, there’s such an incredible amount of new functionality that I’m going to dedicate three posts to this cool and ultra-useful product.
This first post is going to review FME Server as a whole, and note some of the 2013 updates. Like a US president, I’ll call this my “State of the Server” speech.
Another post will dive into the technical aspect of alerts and notifications. I’m guessing you’ve seen demonstrations and movies on the subject, but I’m going to show you how it all comes together, step by step.
Finally, I’m planning a post on suggestions for FME Server projects. Ideas often pop into my head while watching our Friday developer demos, and I want to share a few with you. I hope you’ll either want to take on one of these, or be inspired with a project idea of your own.
Why do I need to write a State of the Server speech?
Well, as an analogy, I went to the Smithsonian Air/Space museum recently. I wanted to see one specific exhibit that I’d heard about (the first balloon to travel around the world non-stop):
But, once there, I found thousands of other exhibits that were just incredible and that I didn’t know existed. And then, by accident, I found they have another – larger – building stuffed to the gills with even more amazing artifacts of flight!
It’s the same with FME Server. At Safe we tend to get very excited about new capabilities (for example, the 2012 alerts/notifications) but neglect to promote the older stuff at the same time. So users – both old and new – might use a set of specific functionality without realizing all the other incredible supersonic, rocket-powered, record-breaking tools that are also available.
This article is intended to review the state of FME Server to remedy that problem. It’s a brief overview of everything that Server does.
I’ll just add the qualifier that this is entirely my view of things. It’s not a technical specification, and it’s not the sort of description you’ll find in our sales or marketing materials. I’m targeting existing FME’ers who need to update their mental image of how this product is pieced together.
Once you read this you’ll know – in general – everything that FME Server is able to do. And when we release new functionality, you’ll be able to understand how it all fits in to the larger picture.
What is Server?
In FME terms, Server is just another way to run FME workspaces. When you click the Run button in Workbench, you’re using the same engine as on Server and processing a workspace in the same way as Server would….
But (and I can’t really put a “but” that is large enough), it’s not what you do, but how you do it. And FME Server is almost unlimited in how it can run a workspace. I put these options into three categories:
- How to start a workspace
- How to structure the process
- How to output the data
How to Start a Workspace, Part1
The simplest way to run a workspace on FME Server is to log in to the user interface and hit the run button. This is what it will look like in 2013:
Yes, there’s the run button. Not a green arrow like in Workbench, but it does the same thing. But you know there’s more than that. Another way to run a workspace on Server is with a URL. Pressing “Show Request” in 2012 would have told you the equivalent URL to pressing Run. In 2013 you just need to expand the “Developer Information” section:
So, there’s the URL that will run the workspace. That method has been available since FME Server was first released in 2008 (or circa 2004 if we include its forebear SpatialDirect) and was made RESTful in 2010. By RESTful we mean it’s structured in a consistent way, i.e.
http://<hostname>/<service>/<repository>/<workspace>
With that information you could run any workspace by pasting its URL into a browser, or attaching it to a hyperlink (on a web page, in an email, etc), or using it within another application.
For example, if you had a tool to browse spatial metadata – either a web site or a standalone application – you could add a button that says “Download me in Geodatabase”. The button would be a link to run an FME workspace that translates data to Geodatabase. As simple as that, and the user doesn’t need to even know that FME is involved in any way, like this:
It’s a great front-end but, when you get right down to it, clicking the Order Data button just runs a Server workspace that clips the input data and writes the output to AutoCAD DWG in a UTM coordinate system. The FME side to this is incredibly simple.
An additional method for running a workspace on Server is to use the FMEServerJobSubmitter transformer (I show an example in a later section) and, finally, FME2011 introduced scheduling for translations. The schedules are created in the FME Server interface like so (again, 2013 style):
In this case I’m running my workspace every day at 12pm, but the interval can be anything from seconds to years! The nice thing is I can set up the schedule and forget about it, knowing that my FME Server will run the workspace at the set time and date.
How to Start a Workspace, Part2
The previous methods are all about running a workspace on demand (or a set time interval). But 2012 (specifically SP2) introduced services to run a workspace in response to an event. We call these “notifications” and a key characteristic of these is that they handle smaller pieces of data than regular FME translations. Think of them as spatial messages.
Setting up a workspace to be triggered by an event is easy. When I publish it to FME Server I just create a new “topic” for it to listen for:
Now, whenever that topic is triggered, then the workspace is run. Ah, you say, but how do you trigger the topic? Well there are a number of ways; we call each way a “protocol”.
In FME2012 the protocols were UDP, email, and HTTP. UDP is an internet protocol similar to TCP and fairly specialized, so I won’t go into detail. The email protocol let you trigger a topic by sending an email. You may have seen Don’s YouTube video on the subject. He’s simply starting a workspace by sending an email to trigger its topic. What’s really cool is the data for the workspace is attached to the email – but it doesn’t have to be.
The HTTP protocol lets you trigger a topic via a HTTP POST command. You could do this many different ways, for example a form on a web browser, a trigger on a database, or from a mobile application. This – I’m told – is how the FME Reporter app in the iTunes store works.
When the talk is about sensors, this is usually what they are doing. A sensor will send a HTTP post (sometimes directly, sometimes via its own website like COSM) and trigger an FME Server topic.
In FME2013 we’ve added a couple of new protocols.
The Directory Watcher is still a work in progress and for 2013 we’ll just include a sample and an FMEpedia article to go with it. Anyway, what happens is that Server watches a directory and – when a file appears – a topic is triggered and its workspace started. Don again made a video demonstration of the idea. He’s using DropBox but we hope to build the functionality to operate on any folder.
The other 2013 protocol will be JMS. JMS means Java Message Service (thank you Wikipedia). Although I’d never heard of it, Aaron, our FME Server product manager, thinks this is going to be the surprise hit for 2013. That’s because it allows us to tie into applications that include a JMS provider, and it appears there are many enterprise-level applications that include a JMS provider. Two that I can see are IBM WebSphere and SAP. So an event on one of those systems could send a message to FME Server, that triggers a topic and starts a workspace.
So there you have it. A lot of different technologies, but – remember – these are simply ways to run a workspace. Some are more exotic than others, but the outcome is always the same: running a workspace.
How to Structure the Process
Now let’s see the different ways in which a workspace process can be structured.
The most obvious way is to just run a single workspace, or maybe you’ll use a custom format (FME Server 2013 will support encrypted custom formats, by the way).
But sometimes I think we lose sight of FME Server being a scalable solution: it’s not all about running a single workspace on a single dataset. Because FME Server allows multiple engines it is very well suited to doing things like batch processing, and for load balancing lots of jobs over a number of engines.
Safe’s scenario expert, Dmitri, has a good example of this when generating raster maps. He has a simple workspace on Server to rasterize his vector data. But he doesn’t read his entire vector dataset all at once; instead he chops it into tiles and runs the server workspace once for each tile. He does this by using the ServerJobSubmitter transformer on an FME Desktop workspace. This way he can process multiple tiles simultaneously using multiple Server engines. The more engines, the faster it goes:
There’s another good example on the WeoGeo blog, where they scale their FME Server up to 64 engines dynamically by using an Amazon “constellation”.
Another way to structure a process is to tie particular translations to a specific engine, and have those engines distributed to different computers. For example, you might run all Geodatabase translations on a certain engine because that’s the one that has ArcObjects installed, or run all large-scale translations on a particular engine, so small-scale jobs can be run at once on a different engine, without having to wait.
FME2013 adds the capability for the system administrator to assign all jobs from a specific repository to a specific engine (look for the TM_REPOSITORY_ tag in the FMEServerConfig file). This not only gives the administrator another way to manage load balancing, but also provides a (controlled) way for the workspace author to decide which engine to use for a particular job.
NB: In the above diagram, SLB means Software Load Balancing. FME Server has built-in load balancing capabilities, above what you might want to manually implement.
So those are ways to structure a running workspace. Now consider how you could combine that with the different ways to kick off a workspace. For example, you could send an email to start a batch process, rather than just one workspace.
Or maybe when an end user uploads data to a database, it triggers an export to KML for public use – but the engine it is run on is determined by that user’s department. In 2013 you’d just need to have a different repository for each department and tie that to a specific engine. That way each departments’ processes are kept separate and don’t interfere with each other.
How to Output the Data
Now let’s decide how to output data. Of course there are many ways.
As with a workspace in Desktop, you can just write the output to a set folder location, and with a batch process you probably would. Or you might write to a database, or directly to a cloud location like Google Docs.
But, of course, there are all the different services that data can be written to. You probably know many of these. There is Data Download, where the data is zipped up and made available as a link (optionally by email), and Data Streaming where data is fired directly to a particular application. KML Network Link is another output service, and it is similar in action to data streaming, with the ability to refresh at set intervals. There are also services for FME to act as a WFS or WMS Server.
With FME2012 we added the new Subscription Service. As you might know, this is related to the concept of notifications and alerts. The output is essentially an alert to someone that an event has occurred. Similar to running a workspace, these alerts are triggered by topics in a number of ways known as “protocols”.
In 2012 (again, SP2) we included protocols to send alerts by email, and to push information to an Apple mobile device. The latter is how the FME Alerter application (on iTunes) works. Running a workspace triggers an alert and information is sent to the mobile app that is set up to listen for alerts on a particular topic.
With FME2013, we’re renaming the Subscription Service to Notification Service. It’s an attempt to get our terminology all lined up (you’ll see this in my next post). But we’re also adding a new set of protocols. One of these new protocols is for FTP. When an alert is triggered information about the alert is published to an FTP site.
Another new protocol is for Google Cloud Messaging. Put simply, this is the Google equivalent to the Apple Push protocol, and means we can send alerts directly to mobile devices that use an Android OS.
Finally, we also have a notification protocol for JMS; as you now know, this is the Java Message Service but, instead of receiving messages from enterprise JMS providers, this time it is sending messages to them. So there is the potential for two-way messaging communication between FME Server and your other enterprise applications.
Additionally – and I don’t know if this counts as a protocol or not – FME2013 will introduce the ability to trigger a topic automatically when a workspace is complete. Previously (and in the example in my next post) we’d trigger a topic using a workspace transformer, but FME will trigger them in a similar way to a shutdown script. There are even different topics that can be used depending on whether the translation succeeded or failed.
You’ll even be able to designate which writer (from several) is to be used to handle the output.
NOW consider the combinations!
This is turning into an example of those mix-n-match kids games (where you can switch head, feet, and bodies to create a variety of different monsters) or the adult equivalent, a slot machine:
Pull the lever and get a different combination each time, of Start-Process-Output. All those demos you see on YouTube are really just variations of these three items.
It might be a Run-Batch-Folder combination, or Folder-Workspace-Email, or Sensor-Custom Format-Cloud, or Schedule-Specific Engine-Folder, or Email-Workspace-Email, or just about any combination of the above.
And that, dear reader, is a review of FME Server as it stands. Hopefully you’ll now see where all the input and output components – especially the alerts and notifications – fit into place.
And if I said that in FME2014 we may add support for FME to receive notifications from the Advanced Message Queuing Protocol, you’ll be able to tell at once where that fits into the product, even if you don’t know what AMQP is (I don’t! But if it’s sending FME a notification, it’s just another way to run a workspace).
In my next post I’ll cover in detail how to use the protocols I’ve mentioned to receive notifications and issue alerts. But, in the meantime, if you have any questions about FME Server, or would like a demonstration of any of the items I’ve mentioned, then please don’t hesitate to get in touch. You can contact either myself or our Server Product Manager, Aaron and we’ll be glad to help.
Regards
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
