Crafting Complete Spatial Data Distribution Models

September 24, 2008 By: Mary Jo Wagner

Since the turn of this century, traditional business has become all about the "e" — e-commerce, e-procurement, e-government — and the very real benefits technology has delivered to public agencies, private businesses, and citizens. For spatial data managers and providers, the "e" has typically equated to geoportals: online anchor sites offering a wealth of spatial data and map layers for users to access, view, and order.

Though the data transparency and collaborative environments created by online spatial distribution systems can improve productivity and efficiency, organizations have often implemented their self-service concept with a notable "e" — "e-delivery" -— missing entirely. Instead, online spatial "distribution" has typically been limited to online ordering, while fulfillment remains offline.

The offline fulfillment model has dominated, in part, because there has been a lack of Web and GIS tools robust enough to eliminate the human component of extracting, transforming, integrating, and distributing data. Particularly problematic are interoperability problems stemming from diverse data formats.

"Allowing users to view spatial data online and request needed datasets is a fairly easy proposition," said Sean Simpson, GIS manager at the City of Surrey's engineering department in British Columbia, Canada. "However, manually fulfilling those common, repetitive requests can tax resources and inhibit your ability to serve customers efficiently."

Supply and Demand

Thanks to the heightened visibility of spatial information shown by high-profile aggregators and distributors like Google Earth, customers from various user communities now expect that same immediacy from their local governments and other providers of spatial data. They want to be able to view data online, choose only the datasets and formats they need, order them, and then download the files.

Indeed, the unrelenting demand for online information "drive-throughs" has challenged organizations to turn their partly self-service online distribution systems into full-service models, from viewing to e-delivery. Thanks to substantial improvements in Web technology, and GIS tools such as ETL (extract, transform, and load) software, organizations in Canada, Europe, the United States, and New Zealand are creating automated information distribution systems that meet the needs of both internal colleagues and external customers.

Surrey's COSMOS Web GIS system allows users to search for data, order only the specific dataset they need, pay for it, and then download it. Europe's transnational LoG-IN system enables a host of users in Belgium, Germany, and the UK to securely share data in any format. In the United States, the Coast Survey Office (OSC) of the National Oceanic and Atmospheric Administration offers GIS-ready nautical charts for immediate download and integration into GIS workflows through its ENC Direct service. And New Zealand's Landcare Research's geoportal provides environmental data for automatic download in a range of formats and coordinate systems.

Through the OSC’s ENC Direct service, users have access to more than 650 GIS-ready nautical charts that can be completely customized for immediate download. Data layers include coastlines along California.

Easing Administrative Burdens

Although the managers of these systems did not all experience the same development challenges, they were all motivated by one primary driver: shifting the time-consuming administrative tasks of processing data requests away from their limited personnel resources.

"In conducting research at the beginning of the LoG-IN project, we discovered that the typical local GIS professional spends 80 percent of their time in processing information requests and only 20 percent in actually updating and improving data layers," said Filip Meuris, head of the e-government department of Flanders, Belgium-based Intercommunale Leiedal, a regional development agency and primary partner for the partly European Union-funded LoG-IN project.

The same administrative burden was also wearing on the GIS data providers at Landcare Research, OSC, and the City of Surrey. Spurred by the desire to develop a technical solution to a resource challenge, they all chose to capitalize on the connectivity of the Web and the data transformation capabilities of ETL software to offer a better way to create, manage, and distribute spatial data.

"We can now supply requested raster and vector data on the fly," said Simpson. "The system restructures it and presents it to users in the way that they can immediately use it. That helps minimize the burden on our department to service the requests. It's a far more responsive and effective way to service customers."

Reaching this e-freedom comes with its share of technical and infrastructure challenges, but the managers say that with internal commitment, focused partners, and political buy-in, an automated Web-based spatial distribution model is a rewarding reality.

Dealing with Data

One of the first hurdles to address is data "readiness" — knowing what data is available, where and how it is stored, and how much of it will be made available to particular user groups. Understanding end users' data needs will also help create a more targeted offering.

"Knowing and understanding your data and what users want to use the data for are challenging requirements to resolve," said Matt Austin, a physical scientist with the OSC. "Often systems are developed without talking to or involving the end users, so the service is not what they want or is far more complex than what it needs to be."

Closely linked to the data-understanding exercise is the data conversion challenge: how to serve a multitude of unique data format preferences and software systems. The 35 local authorities involved in the LoG-IN project, for example, work with more than 80 different formats and seven different software systems. And that doesn't include external customers' partiality to certain formats. Trying to find a GIS tool that could handle all that variety automatically and also cater to users' myriad format and projection needs and coordinate systems can be a daunting proposition.

According to Meuris, however, using a spatial ETL platform makes the data format a non-issue. Since the solution handles all data conversions automatically, local authorities can maintain their spatial data in their own native formats and coordinate systems. And because the LoG-IN's Generic Information Infrastructure is based on open standards, XML/GML, and Web services, users can integrate custom-made data to build Web applications using only a browser.

Surrey, CSO, and Landcare Research were also faced with conversion challenges posed by the significant volumes of data they managed and ultimately wanted to offer. For Landcare, meeting this challenge was an exercise in keystrokes, said Niels Hoffmann, a GIS specialist with the agency's Hamilton branch. Staff designed the ETL coordinate reprojection and dataset clipping workflow by selecting four data "transformers" and connecting these to create a data-flow diagram that automatically controls the data transformation.

Mapping a Model

The distribution model may look good on paper, but physically building it may be another story. Indeed, getting diverse data object models, software solutions, and hardware systems to interoperate — particularly among various countries — requires some technical ingenuity.

For the LoG-IN system, local authorities needed all their diverse formats and systems to work as one. That meant the data-sharing pipeline had to be chameleon-like in its structure, enabling users to freely request data in any format, projection, or coordinate system and responding accordingly.

"We are very different as organizations," said Meuris. "We have high demands for data manipulation and we need to serve many different user groups — GIS and CAD alike," said Meuris.

Although Landcare's existing geospatial portal allowed users to view available data layers and query map features, it did not offer a provision for automatic download. Hoffmann also chose an ETL platform to complete the distribution process.

For Surrey's COSMOS system, Simpson faced the additional challenge of integrating e-commerce functionality — a payment process that previously entailed acquiring the customer's credit card number by telephone. "For us, there was no precedent to follow for an e-commerce mechanism — no one else was doing this," said Simpson.

The City of Surrey received more than 600 data requests last year, many of which were for orthophoto images; COSMOS offers more than 140 GB of orthophotography at 10-cm resolution.

The "E" in Distribution

Clearly, providing an open window to spatial data demands that managers also install sufficient security to quell data policy concerns. ENC Direct, COSMOS, LoG-IN, and Landcare Research all adopted security measures to maintain the integrity of the available data. Users of COSMOS, LoG-IN, and Landcare Research must first apply for a valid login. Security features are all built in, enabling managers to add new users and to limit data availability based on user clearances.

Once on the site, however, a new challenge presented itself — providing a complete system that allows users to find the data they want, extract and order specific data layers, and receive that data electronically. The challenge for developers was to find an effective way to displace what had been an intensive human task with an automated back-office function. The common solution for all of these systems was to employ ETL technology to provide the electronic "clip, zip, and ship" functionality.

"Although allowing users to only request a specific subset of our entire data is complex by nature, we needed to make the user experience as easy as possible and support that with an intelligent back-end to foster loyalty to the site," said Simpson.

"The 80 percent manual work done by local authorities previously is now done automatically or in seconds with a few keystrokes," added Meuris. "Removing the time-consuming data-processing tasks from authorities allows them to be far more efficient and productive and affords them the time to develop applications or services with far more confidence."

Having met their share of challenges, these program managers — who had no internal precedent to help guide them — have created end-to-end Web-based spatial distribution systems. Though many different lessons were learned, they all imparted similar wisdom on what elements are essential to success:

Political and key stakeholder buy-in (both internally and externally)
Willingness to think out of the box and try new things
Strong project management
A willingness to partner with other organizations to create a best-practices approach
A clear understanding of what users want.

Breaking through the traditional online ordering and offline fulfillment pattern, their self-service online spatial distribution systems truly emphasize the "e."

Manufacturers

All four spatial data providers employed Safe Software's FME ETL platform.