2.3.1 Parsing JSON and Accessing GIS Data via REST

2.3.1 Parsing JSON and Accessing GIS Data via REST

Python includes several built in methods for handling JSON (or json). As you remember from Lesson 1, json is widely used to transfer data from one language to another or from one data source to another. For this section we will demonstrate loading it into a Python Dictionary and dumping it to a json string. The official documentation can be found here(link is external).

Loading json means to convert a formatted string into a json object. This is useful for converting string representations of dictionaries that come from various sources, or from API’s results. It is important to note that there is also json.load(…) which takes an file or bytes like object as an input parameter whereas json.loads(…) takes a json string. The s at the end of loads denotes that the method is for a string input, whereas the load method without the s is for objects such as files. The dump and dumps follows the same convention, except it outputs to a file writer type object or a json string so beware of what type of data you are working with. If you do forget, the Python interpreter will kindly let you know.

A simple json loads example:

To write the python dictionary back to a json string, you would use the .dumps() method.

And a simple json dumps example:

Equivalent json string of input dictionary: '{"1": "Welcome", "2": "to", "3": "Cheyenne", "4": "Wyoming"}'

You will notice that the JSON dumps converts the keys and values to strings. The deserialization process converts the value to its datatype, but it doesn't always get it right, so sometimes we are left with adding custom casting.

Accessing the properties of the json object when loaded from json.loads() is the same as accessing them via Python dictionary.

Now that we know some methods for requesting data from the web and parsing JSON data, let’s look at a REST service from esri and how we can use it to generate an url for our code. There are four parameters that we need to fill out for it to return a result. Using the query UI endpoint for the service at 2023_BLM_AZ_Fire_Restrictions_view(link is external). These four parameters we will work with are the Where, Out Fields, Return Geometry, and Return format. If you need a more specific result, you can enter more parameters as needed.

The base url for the query will start out as:

https://services3.arcgis.com/T4QMspbfLg3qTGWY/ArcGIS/rest/services/2023_BLM_AZ_Fire_Restrictions_view/FeatureServer/0/query?

As described in the last section, this ? is starting of the passing of the parameters, which are separated by & sign. Note that the url and the parameters we are passing does not contain spaces.

The complete request’s parameters are added to the URL when the request is submitted:

?where=1=1&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&resultType=none&distance=0.0&units=esriSRUnit_Meter&relationParam=&returnGeodetic=false&outFields=*&returnGeometry=true&featureEncoding=esriDefault&multipatchOption=xyFootprint&maxAllowableOffset=&geometryPrecision=&outSR=&defaultSR=&datumTransformation=&applyVCSProjection=false&returnIdsOnly=false&returnUniqueIdsOnly=false&returnCountOnly=false&returnExtentOnly=false&returnQueryGeometry=false&returnDistinctValues=false&cacheHint=false&orderByFields=&groupByFieldsForStatistics=&outStatistics=&having=&resultOffset=&resultRecordCount=&returnZ=false&returnM=false&returnExceededLimitFeatures=true&quantizationParameters=&sqlFormat=none&f=json&token=

Building our query that will return all features and all fields in json format will look like:

https://services3.arcgis.com/T4QMspbfLg3qTGWY/ArcGIS/rest/services/2023_BLM_AZ_Fire_Restrictions_view/FeatureServer/0/query?where=1=1&outFields=*&returnGeometry=true&f=json

With this url string, we can use requests to retrieve data from services and save them locally. There are a few ways of doing this such as using pandas to read the url, converting the json to a dataframe and then to a featureclass using built methods, converting the JSON result to a dictionary and using an insert/update cursor, or creating a new featureclass using the arcpy method JSONToFeatures_conversion(…) method. It is important to note an important aspect of this method noted in the Summary section of the documentation:

Converts feature collections in an Esri JSON formatted file (.json) or a GeoJSON formatted file (.geojson) to a feature class.

This means that the method is expecting a file as the input and trying to pass the response json will result in an error. If you do not need to manipulate the data before outputting to a featureclass, this method might be the simplest to implement with the least amount of packages. If you need to work on the data, such as convert the datetimes for a field, converting the JSON to a dataframe or dictionary would be the preferred process.

Below is an example of the process.

You can also separate the parameters into a dictionary and let requests do the url formatting, making for a cleaner code and this method seems to also format the returned date fields, if there are any:

As we said earlier about searching for packages that perform an ETL process, we saved a hidden gem for last. Compared to the previous methods of retrieving data from a service that we went over, the few lines of code this process requires is welcoming from a managerial and readability standpoint.

A hidden capability of arcpy’s conversion FeatureclassToFeatureclass(link is external) (deprecated in favor of ExportFeatures(link is external)) is that it can take a service endpoint as an input and make short work of this conversion to a Featureclass. However, as promising as it seems, some services do not transform. Since it is only a few lines of code, it is worth giving it a try and saving some time.

To add a definition query, you can set the where_clause parameter of the method.

ExportFeautures contains different parameters so it looks a little different, but works the same: