One of the more stressful tasks in life is moving everything you own from one place to another. I have often wished there was a button I could press and everything could be magically moved to exactly the right place in my new home so I could immediately find it.
I am reminded of the stress of this type of situation whenever I hear a story like that of Riley Marsden, Head of IT at Barnsley Metropolitan Borough Council (MBC). He and his team were given a task of migrating over 5 million documents from one system to another.
The Decision to Find a New Home
Barnsley MBC serves a region in South Yorkshire, UK. Over the past eight years it has faced a budget reduction of approximately 40%. They are constantly evaluating ways to introduce more efficiency into their processes which, in some cases, means adopting new technology in place of legacy systems.
They made the decision to implement Microsoft SharePoint in favour of their existing document management system. They felt that SharePoint offered a multi-tool solution for their needs. It would not only provide a platform for efficient content management, but its functionally-rich environment would serve as an intranet system that promotes collaboration and business application development.
To implement SharePoint there would be some hurdles to clear. The most immediate: the migration of ten years’ worth of documents – approximately 5 million – within a hard deadline of 8 months, the time when the license for the legacy system would expire.
Finding the Right Movers
At first they were looking for a point-to-point migration solution and evaluated several options. While useful, each had shortcomings, the biggest of which was lack of precise control over how the documents were migrated.
Riley had been introduced to FME in 2013 by software solutions provider and global leader in managing geospatial data, 1Spatial Group Ltd. They use it daily to extract, transform and load data from various systems into GIS applications and the thought didn’t escape him that it could be used for the big SharePoint move.
FME was capable of the fine-tuned “transform” part of this project’s ETL requirements. It could create folders and hierarchy structures, update metadata and assign permissions to ensure documents could be easily located in their new home.
After extracting the documents to a file share they used FME’s HTTPCaller tool to load them into SharePoint. This flexible transformer connects to a web service’s API and allows users to perform any function that it supports.
FME not only enabled them to migrate many of the 5 million documents in the exact way they required, but its time to production was also well within the 8 month moving day deadline. The added bonus was not having to purchase any additional software.
“Show me a problem FME doesn’t solve”
Riley is often called upon to defend this statement now. Since the SharePoint document migration his team continues to find new ways to use FME for document and list management, particularly with FME’s ability to read and write SharePoint lists. Some examples are:
- Creating project status reports from SharePoint lists
- Embedded a web map in SharePoint that highlights the location of asset types stored in SharePoint lists
- Maintaining changes in employee metadata (from SAP HR) in SharePoint
- Automating content publication to SharePoint
Beyond GIS and SharePoint they’ve also used it to handle other large migration projects. They used it to migrate data from a legacy waste management system to SAP ERP and to merge and migrate data from three proprietary systems into their CRM.
Riley, his team at Barnsley, and 1Spatial continue to demonstrate FME’s value both with geospatial data and beyond. We can’t wait to see what they achieve next!
Riley Greene
