2.7.1 Overview of the software tool

The idea of the little software tool we are going to build is that sometimes you would like to be able to quickly create a feature class with points for certain places based on their names. Let’s say you want to produce a shapefile showing a few places in the city you are living in. What you could do is open a basemap in ArcGIS or some other GIS software and then digitize point features for these particular locations. Or you could look up the coordinates of the places in some external application such as Google Maps and then create new point features based on these coordinates manually or by exporting the coordinates in a format that can be imported by your GIS. Both these options are relatively time consuming and require several steps. We therefore want to exploit the fact that there exist many web services that allow for querying for geographic places based on name, type, and other properties, and getting back the coordinates. The two web services we are going to work with are the Nominatim query interface to OpenStreetMaps (OSM) and the place query interface of the online geographic gazetteer GeoNames. We will discuss these services in more detail below. Our tool should allow us to enter a query term such as a place name, pick a query service and set some additional query options, and then run the query to get a list of candidate places with coordinates. We can then look at the result list and pick one or more candidates from the list and either

• add them to a layer currently open in ArcGIS Pro (assuming that the software is run as a script tool inside ArcGIS),
• add them to a shapefile on disk, or
• add the result to a CSV text file (basically an alternative that still works even when arcpy is not available).

Figure 2.20 and the video below show the GUI of the software and further illustrate how the tool works. The use case in the shown example is that we use the tool to populate a layer open in ArcGIS Pro with some of the main sightseeing locations in Paris including the Eiffel Tower, the Louvre, etc.

Figure 2.25 The interface of the Locations from Web Services tool

Please watch the following video, Location from Web Service Tool Overview (5:25min):

The GUI is organized into four main parts:

• The text field for entering the query term. We are currently querying for "Eiffel tower".
• The section where you pick a web query service and set some additional options. The figure shows the interface for the Nominatim OSM service. In addition to Nominatim and GeoNames, we also have a “Direct Input” tab for adding places directly by providing a name and lat/lon coordinates.
• Since the query has already been run, the "Results" section shows different candidate toponyms returned by the Nominatim service for our query. The third one looks like the result we want, so we selected just that one. The selection can be changed with the help of the buttons below or by directly checking or unchecking the checkboxes. Next to the result list is a browser widget that is used to display a Leaflet [2] based web map of the results.
• The final section is for adding the selected features to some dataset. The figure shows the tab for adding them to a currently open layer in ArcGIS Pro.  

We can run as many queries as we wish with the tool and collect the results we want to keep in one of the possible output options. While we will only present a basic version of the tool in this walkthrough supporting two different query interfaces and direct input, the tool could easily be extended to provide access to other web portals so that it can be used for running queries to, for instance, get locations of all Starbucks located in a particular city or create locations from a list of addresses.

Before we continue with exploring the GUI and code of the application, let us talk about the two web services we are going to use:

Nominatim – Surely, you know about OpenStreetMaps (OSM) [3] and how it collects geographic data from volunteers all over the world to create a detailed map of the world. OSM data is freely available and can be directly exported from the OSM web site. In addition, there exist quite a few web services built around the OSM data, some of them created with the purpose of allowing for querying the data to only obtain information about particular features in the data. One such example is the Nominatim web service provided by OSM themselves. The website for Nominatim Open Street Maps [4] provides an easy to use interface to the Nominatim query engine. You can type in a search term at the top and then will get to see a list of results displayed on the left side of the page and an OSM based map on the right that shows the currently selected entity from the result list. Give it a try, for instance by running a query for “Eiffel tower, France”.

Note: With some recent changes, it seems Nominatim has become much more restrictive and will often only return a single result rather than multiple options. If you leave out the 'France' in the query, the only result returned will actually not be the Eiffel Tower in Paris. However, you will still get multiple results if you, for instance, enter 'Washington' as the query string. Due to these changes, the results you will get when using Nominatim in the Locations from Web Services Tool will partially deviate from what is shown in the figures and videos in this section (for instance, the list of options shown in Figure 2.25 when only querying for 'Eiffel tower' without country name.

Nominatim provides an HTTP based web API that can be used for running queries from your own code and getting the results back, for instance as JSON or XML data. The web API is explained on this wiki page here [5]. Query terms and additional parameters are encoded in a URL that has the general format:

https://nominatim.openstreetmap.org/search?

where parameters are specified as <parameter name>=<parameter value> pairs and multiple parameters have to be separated by an & symbol. The parameter name for the query string is simply 'q' (so q=...). To run the query, the client sends an HTTP GET request with this URL to the Nominatim server who processes the query and parameters, derives the result, and sends back the results to the client. The format parameter controls how the result is presented and encoded. Without specifying that parameter, you will get the kind of HTML page that you already saw above with the result list and map. When using format=json as we will do in the following, we get the result as a list of entities encoded as JSON. Here is an example query URL, querying for “Eiffel tower, France” again, that you can test out simply by clicking on the link to open the URL in your browser:

https://nominatim.openstreetmap.org/search?q=Eiffel%20tower%2c%20France&format=json [6]

Have a look at the result shown in your browser. As we explained at the beginning of the lesson, JSON uses [...] to denote lists of entities

where each entity is described by its properties like in a Python dictionary:

Due to the changes, we mentioned above, the result will be a list with just a single entity, looking like this  ...

[{
    "place_id":115316817,
    "licence":"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright",
    "osm_type":"way",
    "osm_id":5013364,
    "lat":"48.8582599",
    "lon":"2.2945006358633115",
    "class":"man_made",
    "type":"tower",
    "place_rank":30,
    "importance":0.5868325701744196,
    "addresstype":"man_made",
    "name":"Eiffel Tower",
    "display_name":"Eiffel Tower, 5, Avenue Anatole France, Quartier du Gros-Caillou, 7th Arrondissement, Paris, Ile-de-France, Metropolitan France, 75007, France",
    "boundingbox":["48.8574753","48.8590453","2.2933119","2.2956897"]
}]

We can see that the most important properties for us will be the ‘display_name’ property and the ‘lat’ and ‘lon’ properties (appearing in bold above) in order to create point features and add them to an existing data set with our tool. Feel free to compare this result to what you get when querying for 'Washington' where you will get a list of multiple results.

The following query uses a few more parameters to query for places called London in Canada (countrycodes=CA) and asking for only a single entity be returned (limit=1).

https://nominatim.openstreetmap.org/search?q=London&format=json&countrycodes=CA&limit=1 [7]

If you look at the result, you will see that it lists London in Ontario as the only result. Without using the ‘countrycodes’ parameter the result would have been London in the UK because Nominatim uses a ranking scheme to order entities by likelihood/prominence. Without the 'limit' parameter, we would get a list of multiple options in the JSON result. ‘format’, ‘countrycodes’ and ‘limit’ will be the only parameters we will be using in our tool but please have a look at the other parameters and examples given on the Nominatim wiki page to get an idea of what other kinds of queries could be implemented.

GeoNames – You have already seen the Nominatim examples, so we can keep the section about GeoNames a bit shorter, since the URLs for running queries are somewhat similar. GeoNames is essentially an online geographic gazetteer, so a directory of geographic place names with associated information including coordinates. The main page [8] can be used to type in queries directly but we will be using their REST web API that is documented here [9]. Instead of a parameter for specifying the output format, the API uses a special URL for running queries that are replied to using JSON, as in the following example:

http://api.geonames.org/searchJSON?name=Springfield&maxRows=10&username=demo [10]

Please note that a parameter (name=) is used to specify the query term. In addition, GeoNames requires a user name to be provided with the ‘username’ parameter. In case you tried out the link above, you probably got the reply that the daily request limit for user ‘demo’ has been reached. So you will have to create your own account at http://www.geonames.org/login and then use that user name instead of ‘demo’ in the query. The JSON sent back by GeoNames as the result will start like this:

{
    "totalResultsCount": 3308,
    "geonames": [
        {
            "adminCode1": "IL",
            "lng": "-89.64371",
            "geonameId": 4250542,
            "toponymName": "Springfield",
            "countryId": "6252001",
            "fcl": "P",
            "population": 116565,
            "countryCode": "US",
            "name": "Springfield",
            "fclName": "city, village,...",
            "adminCodes1": {
                "ISO3166_2": "IL"
            },
            "countryName": "United States",
            "fcodeName": "seat of a first-order administrative division",
            "adminName1": "Illinois",
            "lat": "39.80172",
            "fcode": "PPLA"
        },
        {
            "adminCode1": "MO",
            "lng": "-93.29824",
            "geonameId": 4409896,
            "toponymName": "Springfield",
            "countryId": "6252001",
            "fcl": "P",
            "population": 166810,
            "countryCode": "US",
            "name": "Springfield",
            "fclName": "city, village,...",
            "adminCodes1": {
                "ISO3166_2": "MO"
            },
            "countryName": "United States",
            "fcodeName": "seat of a second-order administrative division",
            "adminName1": "Missouri",
            "lat": "37.21533",
            "fcode": "PPLA2"
        }
    ]
}

Here the list of entities is stored under the property called ‘geonames’. Each entity in the list has the properties ‘toponymName’ with the entity name, ‘lng’ with the longitude coordinate, and ‘lat’ with the latitude coordinate. Query parameters we will be using in addition to ‘name’ and ‘username’ are ‘maxRows’ to determine the number of results sent back, ‘country’ to restrict the search to a single country, and ‘featureClass’ to look only for features of a particular type (codes A,H,L,P,R,S,T,U,V of the GeoNames feature class codes defined here [11]).

2.7.2 Creating the GUI in QT Designer

The GUI of our software application is not too complex but still uses many of the most common GUI elements such as a toolbar with a tool button, normal push buttons, group boxes, tabs, labels, text input fields, checkboxes, combo boxes, list views, and a status bar. You already got to know most of these in Section 2.6 [12] but there are also some new ones. The GUI consists of two parts, the GUI for the main window and the GUI for the dialog box that is shown when the user clicks on the “Create new shapefile…” button. Therefore, we will be producing two different .ui files with QT Designer called gui_main.ui and gui_newshapefile.ui. The GUI of the main window uses a vertical layout to organize the different elements into four different rows as shown in the figure below.

Each of the rows is formed by a QGroupBox widget and then other widgets are arranged hierarchically within these group boxes using a combination of grid and horizontal layouts. The tool doesn’t have a menu bar but a toolbar at the top with a button to exit the program and a status bar at the bottom. When creating the GUI in QT Designer, it will be important to name the widgets we need to refer to from our main code as indicated by the orange labels in the two figures below. As we already pointed out, the object names given to the widgets in QT Designer will be the names of the variables used for storing the widgets when the .ui file is compiled into a .py file.

Figure 2.26 Location from Web Services main GUI with labels showing the object names of the different widgets

Figure 2.27 GUI of the "Create shapefile" dialog

The series of videos linked below shows the process of creating the GUI in QT Designer. A zip file with the resulting .ui files is available for download here [13]. We recommend that you work along with the video, pausing it as needed, to create the GUI yourself to get some more practice with QT Designer. The downloadable .ui files are mainly intended as a fallback position in case you experience any difficulties while creating the GUI or later on when compiling the .ui files and using the produced Python code. If you cannot replicate what is shown in the video, please ask for help on the discussion forums.

Create the GUI along with this series of videos [~45 min of video materials] [14]

At this point, you either have saved your own two .ui files or, if you ran into any issues, will continue with these files downloaded from the link posted above. We now need to compile the files into Python code with pyuic5. We do this by running the ArcGIS Pro python.exe from the Windows command line with the pyuic module. The python.exe file is located in the folder “C:\Users \<username>\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\" (unless the ArcGIS Pro Python environment you are using is located in a different folder). So open a command shell, navigate to the folder containing the .ui files and then run the following two commands (again picking the correct version depending on where your default Python environment is installed and replacing <username> as needed):

"C:\Users\<username>\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\python.exe" -m PyQt5.uic.pyuic gui_main.ui -o gui_main.py

"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" -m PyQt5.uic.pyuic gui_main.ui -o gui_main.py

and

"C:\Users\<username>\AppData\Local\ESRI\conda\envs\arcgispro-py3-clone\python.exe" -m PyQt5.uic.pyuic gui_newshapefile.ui -o gui_newshapefile.py

"C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe" -m PyQt5.uic.pyuic gui_newshapefile.ui -o gui_newshapefile.py

Figure 2.28 Pyuic5 commands to compile the two .ui files into Python files

Note: If you get a file not found error such as  ...\\–m': [Errno 2] No such file or directory after running the command, remove the " - " symbol from the m and o and type them again. These symbols may come across as a different encoding and cause the command to fail.

The parameters given to pyuic5 are the name of the input .ui file and then –o followed by the name of the output file. You should now have the two files gui_main.py and gui_newshapefile.py in the project folder. Let us have a quick look at the produced code. Open the produced file gui_main.py in your preferred Python IDE and see whether you recognize and understand how the different elements are created and how their properties are set. Without going into the details, the code defines a class Ui_MainWindow with a method setupUi(…). The parameter MainWindow is for passing a QMainWindow widget to the method. The rest of the code of the method then either...

• changes properties of MainWindow,
• creates new widgets and layouts storing them as attributes of the Ui_MainWindow object and sets their properties, or
• adds widgets to MainWindow or to other widgets to create the hierarchical organization of the widgets.

This all means that we can create a new QMainWindow widget in our code with ...

... create an object of the UI_MainWindow class with

... and then create the GUI for the main window by calling ui.setupUi(…) with mainWindow as the parameter:

Whenever we need to access a widget created by setupUi(…), we can do so by using the expression

where the object name is the name we gave the widget in QT Designer, e.g.

for the QLineEdit widget we created for entering the query term.

2.7.3 Main code of the tool and how it works

Now we are going to develop the main code for our tool that imports the gui_main.py and gui_newshapefile.py files, sets up the application, and connects the different GUI elements with event handler functions and functions that realize the actual functionality of the tool. We will organize the code into several modules. In particular, we will keep the functions that realize the main functionality, such as querying the different web portals, creating a new shapefile, etc. in a separate Python script file called core_functions.py . These functions are completely independent of the GUI of our tool and the few global variables we will need, so we might want to use them in other projects. Separating the project cleanly into GUI dependent and GUI independent code fosters reusability of the GUI independent code. Overall, the project will consist of the following Python files:

• gui_main.py – compiled version of gui_main.ui for setting up the GUI of the main window,
• gui_newshapefile.py – compiled version of gui_newshapefile.ui for setting up the GUI of the dialog box for creating a new shapefile,
• core_functions.py – contains definitions of functions for main functionality that are completely independent of GUI and global variables
• main.py – contains the event handler functions and the code for wiring up the GUI as well as setting up and running the application

In the following, we will focus on the code from main.py but we will start with a brief look at core_functions.py, so please download the file core_functions.py [15] and open it so that you can read the code for the different functions. Most of the functions defined in the script should be rather easy to understand from the comments in the code and from your experience with using arcpy to work with shapefiles. Here is an overview of the functions with a few additional explanations:

• queryNominatim(query, limit=10, countryCodes=''): The purpose of this function is to query the Nominatim query interface of OSM for a query string and parameters for the limit of results returned and country code for the country to search in as discussed in Section 2.7.1 [16]. The code should be easy to understand: urllib.parse(…) is used to encode the query string to be used as part of a URL, before the final query URL is put together. The get(…) method of the requests package is used to run the actual query and the returned result is translated into JSON before being returned. Since no error handling is done in this function, we will have to deal with potential exceptions raised within the function in the code that calls this function.
• queryGeonames(query, maxRows = 10, username='', country='', featureClass=''): This function does the same as queryNominatim(...) but for GeoNames and for a different set of query parameters. Since GeoNames returns JSON code with the list of candidate features stored under the attribute 'geonames', we return json['geonames'] at the end.
• getStringFieldsForDescribeObject(desc): This is the first of a few auxiliary functions for getting information from shapefiles and layers. An arcpy.Describe object of a data set needs to be passed to it and it then returns a list of all editable string fields of that layer.
• getValidFieldsForShapefile(fileName): This function is used to get a list of editable string fields for a shapefile whose name is provided as a parameter. It relies on getStringFieldsForDescribeObject(…) to do most of the work but before that checks that the shapefile exists and is a Point vector data set. If not, it will return an empty list.
• createPointWGS1984Shapefile(fileName, fieldName): This function creates a new Point shapefile with the name provided as parameter, using WGS 1984 coordinates, and with a single additional string field whose name is also provided as a parameter.
• getPointLayersFromArcGIS(): This function is for getting a list of Layer objects for the layers currently open in ArcGIS Pro but only including Point layers.
• importArcpyIfAvailable(): This function returns True if arcpy is available for import, else False. It attempts to import arcpy within a try-except construct so that, if the operation fails, the resulting exception is caught and False is returned.
• runningAsScriptTool(): This function returns True if the program is run as a script tool inside ArcGIS, else False. This is accomplished by calling arcpy.mp.ArcGISProject("CURRENT") and catching the resulting exception if this operation fails, meaning the program is run as a standalone program outside of ArcGIS.
• webMapFromDictionaryList(features): This function produces and returns the HTML code for displaying the web map as part of the GUI. It gets the features that should be displayed on the map as parameter in the form of a list of dictionaries with name, lat, and lon attributes.

Now that you know the functions we have available for realizing the different operations that we will need, let’s develop the code for main.py together. Open a new file main.py in your IDE, then follow the steps listed on the next few pages.