Geospatial Data Science at Scale: GeoAnalytics Engine on Amazon EMR
In a January 2024 AI Connect webinar, experts from Esri and AWS explained how to run spatial analysis natively in a Spark and Amazon EMR environment. They covered common pitfalls, available functions and a Chicago ride share demand demo with more than 50 million records.
By The AI Enterprise Magazine Team5 min read
Key takeaways
- Bring analytics to the data. GeoAnalytics Engine runs inside Spark on Amazon EMR rather than moving data to a separate GIS.
- Spatial is not just maps. Battersby cited an estimate that 80% of business data has a spatial component.
- Watch coordinate systems. Mismatched systems can return zero counts; the engine reprojects automatically when it knows each system.
- Rich toolset. About 140 spatial functions plus aggregation, joins, hot spots, movement analysis and machine learning tools.
- Scales to big data. The demo processed over 50 million ride share records; Battersby said cloud scale matters when a model must finish within the hour.
In a January 2024 AI Connect webinar on geospatial data science at scale, experts from Esri and Amazon Web Services showed how to run spatial analysis inside a cloud big data environment instead of moving data out to a separate GIS. The speakers were Sarah Battersby, a product manager at Esri; Damion Harrylal, a senior software architect on the AWS side; and Arif Masrur, a senior solution engineer at Esri. The moderator noted at the start that industry analysts estimated the global geospatial analysis market would grow to more than $34 billion by 2032.
The session centered on Esri's ArcGIS GeoAnalytics Engine, a Spark-native library, running on Amazon EMR. It ended with a demonstration that modeled ride share demand in Chicago.
What does geospatial data science at scale involve?
Harrylal described the AWS and Esri partnership, which he said began in 2008, and a four-step workflow: configure, connect, analyze and publish. Users connect to data in places such as Amazon S3, run analysis in Jupyter notebooks, and write results back to S3 or into tools such as Amazon QuickSight.
Battersby said spatial work covers data engineering, visualization, analysis, modeling and sharing. "Spatial analytics doesn't always mean making a map," she said. She cited an estimate she had seen that 80% of business data has a spatial component, since any record linked to an address, ZIP code or place name can be tied to other data. Demographic attributes attached by location, for example, could feed a model predicting spending behavior.
Why do spatial analyses fail without anyone noticing?
Battersby gave a common pitfall: points in latitude and longitude and hexagonal bins in a local coordinate system. Many tools then show the bins in the middle of the ocean, and a count of points inside each bin returns zero.
GeoAnalytics Engine, she said, reprojects data behind the scenes when it knows each data set's coordinate system. Users do not need duplicate copies of the data or an extra line in a script. She described this as letting people focus on asking the question.
How does GeoAnalytics Engine run on Amazon EMR?
Battersby said traditional spatial workflows move data to the GIS, which isn't efficient for big data. Instead, "we're going to pick up our analytics and move them into your environment." The library is imported and authenticated like any Python library in an EMR notebook, she said, and can read anything Spark can read, plus formats such as shapefiles, GeoJSON, GeoParquet and feature services.
She said users do not have to build their own spatial indexes, and that the engine includes about 140 spatial functions at the row level. Spatial analysis tools cover aggregation, spatiotemporal joins, hot spot and density analysis, movement analysis and some machine learning.
What can teams do with it?
Battersby walked through several building blocks. A table of roughly 2.5 million insurance points could be turned into geometry and aggregated into square, hexagonal or Uber H3 bins to show patterns, such as total insured value per bin. Customer points can be enriched with US Census attributes through a spatial join. Functions cover proximity, hot spots and snapping GPS points to roads.
What did the Chicago ride share demand demo show?
Masrur used 2020 ride share data with more than 50 million records, read as CSV directly from S3, plus EPA smart location data, Census data and Chicago crime data. He turned the latitude and longitude columns into geometry, plotted them over a base map, then aggregated the points into Census tract polygons and later into Uber H3 hexagons.
The downtown core and the airport stood out as the places with the most pickups, he said. He noted that demand fell from March and never returned to January and February levels, a pattern consistent with the COVID-19 outbreak.
For modeling, Masrur added three supplemental data sets and clipped them to Chicago with a GeoAnalytics Engine tool. He calibrated on data from the start of the pandemic through October and predicted November and December demand. He used the open-source MGWR library for multiscale geographically weighted regression, with violent crime, employment density and population density as predictors and log-scaled trip count as the target.
Comparing predictions with the actual November and December counts, he said the model "performed reasonably well," best over the downtown area. He called it the simplest scenario. Harrylal then showed results pushed into Amazon QuickSight, which he described as a unified business intelligence tool with natural-language queries.
How does it compare with ArcGIS alone, and what else came up in Q&A?
Asked why someone would use this rather than ArcGIS, Battersby pointed to scale. If "you need a model to run every hour and it takes 70 minutes to run in ArcGIS," she said, that will not work, whereas EMR can spin up larger compute. She also said it suits organizations centralizing data in S3 buckets. Harrylal said managed services and per-second billing ease setup.
Running the notebooks requires a GeoAnalytics Engine license, Battersby said, and the notebooks were not yet posted. Harrylal said EMR and QuickSight are available in AWS GovCloud, and Battersby said results can be written where Power BI or Tableau can read them.
On moving objects, she said GeoAnalytics Engine is not designed for real-time data, and pointed to Esri's Velocity product, though it can analyze already-collected trajectories. Hexagonal bins work for point data, she said, if teams consider map projection and area error. Masrur said raster export is not supported directly but a future release should extract raster values to points or polygons.
Hear more from data and AI leaders in 2027
AI Connect has 24 free live webinars on its 2027 schedule, including AI Demo Days and the Plug & Play Series. Details are at /webinars/. The AI Enterprise Conference 2027 takes place Tuesday, August 31, 2027, at Pier Sixty in New York City (/events/ai-enterprise-conference/).
Frequently asked questions
What is GeoAnalytics Engine?
Speakers described it as a cloud-native, Spark-native library that adds spatial analysis to Spark environments such as Amazon EMR.
Does spatial analysis require a map?
No. Sarah Battersby said location can enrich data for non-map models, such as attaching Census attributes to customers by location.
Why did the hexagon counts show zero in the example?
The points and the hexagons used different coordinate systems, so many tools see no overlap; the engine reprojects behind the scenes.
What was the ride share demo?
Arif Masrur aggregated more than 50 million 2020 Chicago ride share records to Census tracts and H3 hexagons to model demand patterns.
This article is based on remarks by Sarah Battersby, Product Manager, Esri; Damion Harrylal, Senior Software Architect, AWS; Arif Masrur, Senior Solution Engineer, Esri at AI Connect webinar, January 2024 on January 5, 2024. More coverage: Webinars.
Filed under: Data & AI Infrastructure, AI Use Cases & Applications
