Hi FME’ers,
This post covers a little-known writer format in FME and its usefulness as part of an FME Server Data Download service. In particular I’m going to show a use I can make of it – for downloading the standard FME sample dataset from our web site.
Data Downloads for Training
Did you know that when you take an online training course with Safe Software you get your own instance of an Amazon Web Services virtual machine, on which FME and the required data is pre-installed? You simply access it through a Remote Desktop connection and away you go! We find it’s the easiest way to allow users to carry out online training.
But, sometimes, the user wishes to carry out the course on their own computer, in which case they will need to manually download the sample data.
However, the capability to download data is not as efficient as it should be. The data is held in a zip file – which is simple enough – but that means the user ends up downloading nearly 200mb of data, regardless of whether they are taking a course that uses it all.
Given that Safe is the creator of FME Server, which includes a Data Download service, I’m quite embarrassed at not yet having produced a better solution.
Well, for 2014, I plan to do just that. Here’s how.
FileCopy Basics
First, let me explain what FileCopy is about.
The FileCopy writer is a little-known piece of FME functionality and, because of its obscurity, probably not used as much as it should be. It’s not a format of data, as such. Instead, each incoming feature triggers a file system copy (or move) command:
As you can see, the writer Feature Type includes format attributes that define a source file and a destination. The actual attributes are:
- filecopy_source_dataset
- filecopy_dest_dataset
- filecopy_dest_filename
I simply set the format attribute for a source file/folder, and one each for the destination folder/filename. Then, at run time, the data is simply copied from one to another.
An example use might be a file of metadata. For example, if you have a Shape writer to create myData.shp, then you might use a FileCopy writer to copy myMetadataTemplate.xml to myData.xml. That way you automatically create a metadata file by copying a standard file you have available.
FileCopy can also be used to copy the output from one Writer to a different location. For example, I write the Shape data to a location on my local file system, but use FileCopy to automatically copy that output to a central data store on a remote file system, where other people can access it.
But my plan is to use this format on FME Server, which is where it becomes more interesting.
FileCopy and Data Download
Why is FileCopy the best method for me?
Well, in most FME Server Data Download services, we want to give the user the ability to select the format of data (Generic Writer) or the set of layers (Feature Types to Read) or the Coordinate System (Reprojector Transformer), like so:
But in training we don’t want that. I don’t want to give users the chance to change anything because then the data would be incompatible with the course exercises. The data content isn’t meant to be translated or transformed. All I want is to let the user pick which course they are about to take, and for FME Server to provide the data on that basis, like this:
The FileCopy writer is perfect for that, because the data already exists in the required format. I can simply map the chosen training course to a list of files and pass them to the FileCopy writer. On an FME Server Data Download, the FileCopy output is sent to the zip file being downloaded.
So the user will get an exact copy of whatever source files I choose to give them. The next section shows how I’ll set it up.
Training Downloads Workspace
My workspace starts by scanning my training file system (C:FMEDataData) and looking for data folders. I use the Directory and File Pathnames Reader to do this. This – like the FileCopy Writer – is another unique piece of FME functionality, which returns a list of files and folders for a specific location.
It might seem odd to do this – shouldn’t I know what folders exist? – but over time the data will change and more folders get added. This way my process is dynamic. If a new folder is added I don’t need to change the workspace to handle it.
The next part of the workspace maps the chosen course to a list of data folders. This, at the moment, is hard-coded; but I hope to make this more dynamic too by referring to an external lookup table instead.
For example the Esri training course needs the Parks, Zones, and Airports data folders. It does not need the Raster folder, which is not used in that course.
Then I simply do a test to check whether the folder I have read is in this list of values, using a Tester transformer:
Finally, the FileCopy writer is provided with a list of data folders that pass the test:
When I run this on Server I am asked which course I wish to download data for, and then it is delivered as a zip file ready to install:
It’s as simple as that… nearly!
Final Workspace
In fact the final workspace is a little more complex and looks like this (click to enlarge):
It’s more complex for a couple of reasons.
Firstly the content I wish to deliver includes not just data (the pink bookmark) but example workspaces (yellow bookmark) and a readme file (green bookmark).
Secondly, I want to be able to handle multiple courses at once. For example, I want a user to be able to choose both the KML and Esri courses.
To do this I end up using FME Lists. The selected courses are presented as a comma-separated string that I split up into a list and then explode into individual features. Then I match the list of datasets for each course as separate features. A DuplicateRemover transformer ensures I don’t try and deliver the same folder twice, even if it is used in both courses.
Examples
We do already have a couple of FME users (that I know of) that use FileCopy in this way, so this isn’t just a theoretical project.
Additionally there are some examples on FMEpedia you can check out.
This example explains how to use FileCopy to include metadata as part of your Data Download service.
This example is even more interesting and shows how you can use this to write data to an existing File Geodatabase that you add to the zip file using FileCopy.
All that is required there is the FileCopy writer to be prioritized about the Esri writer. Simple!
In Summary
In Summary, I’m trying to get across a few ideas here.
Firstly take a look at the FileCopy writer. It may be something that you weren’t aware of and can use in your translations, Desktop or Server.
Secondly, if you do want to deliver data via a Data Download service, be aware that using FileCopy means you can simply move data about – and insert it into the downloaded zip file – without having to try and read/write it.
Additionally, I can see me implementing this on FME Cloud, because this is a perfect example of the type of project FME Cloud should be used for. If you didn’t know, FME Cloud is a virtual, on-demand version of FME Server. We’re approaching beta testing and the pricing model is now published on the web site.
In fact, check out my next post when I’ll give you all a sneak peak of how it all works.
Regards
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
