PrintWe will look at data structures and how you can use Python to transform it from one structure to another. We will look at how to access data on the web from within a Python program, and introduce some python packages that help with Extracting, Transforming, and Loading (ETL) data from CSV’s, SQL, JSON, and KML/KMZ’s.
Learning Outcomes
By the end of this lesson, you should be able to:
- Traverse data formats in different formats
- Use cursors to iterate over data
- Access and work with data from the web
- Access GIS data within an Enterprise or AGOL.
Lesson Roadmap
To finish this lesson, you must complete the activities listed below. You may find it useful to print this page out first so that you can follow along with the directions.
| Step | Activity | Access/Directions |
|---|---|---|
| 1 | Engage with Lesson 2 Content | Begin with 2.1 Reading and parsing text using the Python csv module. |
| 2 | Programming Assignment and Reflection | Submit your code for the programming assignment and 400 words write-up with reflections |
| 3 | Quiz 2 | Complete the Lesson 2 Quiz |
| 4 | Questions/Comments | Remember to visit Canvas to post/answer any questions or comments pertaining to Lesson 2 |
Downloads
The following is a list of datasets that you will be prompted to download through the course of the lesson. They are divided into two sections: Datasets that you will need for the assignments and Datasets used for the content and examples in the lesson.
Required:
- None for this lesson!
Suggested:
- GPS_track.txt <- opens a text file in the browser.
- KML and KMZ sample files
- Pennsylvania State Data
Assignments
In this homework assignment, the task is to use requests and any other packages you may need to query the service and parse the features out of the returned JSON and save as a csv file or shapefile. You can select a service from esri, you can be adventurous and select a service from the Pennsylvania Spatial Data Access PASDA hub, or test your skill and see if you can extract vector data from a service of your choosing.
The url for the services are:
esri fire related services https://services3.arcgis.com/T4QMspbfLg3qTGWY/ArcGIS/rest/services/_BLM_...
Pennsylvania Spatial Data Access (PASDA) https://www.pasda.psu.edu/
Hint to return all features of the service, you still need to provide a valid sql query. For this assignment, you can use OBJECTID IS NOT NULL or 1=1 for the where clause. If you want to try other sql statements for other results, please feel free to do so.
If you are having issues with the url string, you can use the service query UI to see how it should be formatted.
Write-up
Produce a 400-word write-up on how the assignment went for you; reflect on and briefly discuss the issues and challenges you encountered and what you learned from the assignment.
Deliverable
Submit a single .zip file to the corresponding drop box on Canvas; the zip file should contain:
• The script
• Your 400-word write-up
Questions?
If you have any questions, please send a message through Canvas. I will check daily to respond. If your question is one that is relevant to the entire class, I may respond to the entire class rather than individually.