There are many interesting offerings of cloud GIS SaaS. This week, we'll try CARTO [1] (formerly CartoDB). Worth noting about CARTO is that it is an open-source project. You could completely replicate what they have, using your own Linux server. At the same time, CARTO is able to operate a business by selling their services to the many folks who would rather focus on simple mapping on the cloud instead of deploying the entire software themselves on their own hardware.

The source code is available at the GitHub website [2]. They are using a fantastic set of technologies, although it might be quite a job keeping up with all the dependent projects if you wanted to work on the source. Fortunately, there's no need, as we can use their free pricing tier to get a feel for their cloud offerings.

Lesson Objectives

At the successful completion of this lesson you should be able to:

• understand CARTO software and how it enables thematic mapping;
• analyze spatial data patterns using CARTO aggregation tools;
• visualize complex data in CARTO using time-series animations; and
• understand how to upload datasets to CARTO.

Deliverables

• Complete: L06: Assignment
• Participate: L06: Discussion

CartoDB was officially launched in 2012 as a web mapping front end to a PostgreSQL + PostGIS back end database. The software was open source and could run on one's own hardware, but at the same time CartoDB offered an online subscription service wherein customers could upload datasets and make maps without having to touch the source code or configure anything themselves. In 2015, one of the best-known PostGIS masterminds, Paul Ramsey, joined CartoDB [3]. In 2016, the company changed its name to CARTO [4] and repositioned itself as a "location intelligence" tool rather than just a basic web mapping interface and online database. As such, it now also offers geodemographic analysis, routing, proximity, and address finding services.

Some of the services, you will note, are similar to those offered by the other SaaS offerings we are studying in this course: Mapbox and ArcGIS Online. This is inevitable because these companies have have found an eager market for the kinds of services they offer, and competition is a byproduct. In these course lessons, we have tried to focus the walkthroughs on some of the unique strengths of each platform or the technologies that they pioneered. One of CARTO's unique points is its variety of thematic mapping options and its appealing basemap and thematic styling options. The color schemes use the ColorBrewer [5] ramps which were developed at Penn State and are based on scientific color theory. Cartographers using CARTO can aggregate point data to tesselated regions such as hexbins or their own boundary files that they upload. They can also make time series maps, rasterized heatmap-style density surfaces, proportional symbol maps, etc.

CARTO offers a "Builder" app for web-based design and an "Engine" piece consisting of APIs. CARTO services can perhaps be considered as either PaaS or SaaS. How can we distinguish between them? One way is to consider how they will be used. If the service is being used as a source and combined with others, then it is probably a platform service. If it is being consumed directly by the end user, then it's a software service. Along the same lines, if you access the service programmatically, it's more likely to be a platform service than if you access it with a GUI.

So, if we used CARTO as a source of web maps that we pass along to end users, then it's a software service. If we use CARTO as a "table in the cloud" then we would be using it as a platform. CARTO's provision of spatial data tables on the Internet, along with both GUI and programmatic access for users and programmers, makes them a good example of a cloud GIS.

Let's go through the steps of uploading a basic dataset to CARTO and making a web map.

First, download the data for this lesson [6]. This folder contains datasets that I derived from Portland Maps Open Data [7] and OpenDataPhilly [8]. They are stored in GeoJSON, a popular format based on JavaScript syntax that is used for interchanging vector data on the web.

Then follow these steps:

1. To log into Carto, in a web browser, visit carto.com
   1. Click the Try For Free button to create a 14-day trial account.
   2. Login to the Carto Platform through the login page [9].
   3. If you are promtped to create an Organization, create a new unique one with your name or Penn State Access ID.
      
      (There is an option to create a free student account via GitHub, but I have had almost no success getting that to work in previous semesters due to GitHub's verification process. They seem to have issues with us being in an online program and not physically at University Park, PA.)
       
2. After you log in, you will see options along the left of the CARTO homepage including Maps and Data Explorer. We'll use the Data Explorer link to upload and view our datasets, and the Maps link to design some maps from them.
3. Click Data Explorer and click the Import Data button.
   1. Browse to csa_farm_dropoffs.geojson on your computer.
   2. Click Continue and select the Private folder within your Organization Data as the destination for the dataset.
   3. Click Continue and let CARTO automatically define the schema.
   4. Import the data.
      
      You should now see the dataset in the Data Explorer list. These are locations where local farms around Portland will bring their produce into the city. They drop it off at these points for urban residents who have signed up for community-supported agriculture (CSA) subscriptions. Mapping these points is one way to see which areas of Portland are currently being served by CSAs (and which ones are not).
       
4. In the main menu along the left, enter the Map section and click the button to Create Your First Map.
5. In the map window, use the Add Source From button to add your farm dataset to the map. You'll see the points overlayed on a basemap of Portland. You're now in CARTO Builder, which is a map design center similar to the ArcGIS.com map viewer or Mapbox Studio (although Builder is more for styling thematic maps rather than basemaps).
6. At the very top of the window, give your map a name, and notice the Updated indicator beside it. Your map will be saved automatically as you work.
7. Click the Basemap link and explore the variety of basemaps offered by CARTO. For example, try the Dark Matter layer. These are built from OpenStreetMap data. You can choose your favorite one for use in this exercise.
8. Go back to the layer list, and click on the farm points layer.
9. Play around with changing the fill, outline, and other properties of the points.
   
   Sometimes it's easier to visually make sense of a bunch of dense points by aggregating them to polygons of uniform size. Hexagons are a popular choice because they are compact and tessellate (fit together) easily. Let's aggregate these points to hexagons, or "hexbins" as cartographers sometimes call them.
    
10. Still viewing the properties of the csa_farm_dropoffs layer, choose the Hexbin (H3) option.
11. In the Fill section, choose UNIQUEID and COUNT in the dropdown list. This specifies that each hexbin will indicate the total number of farm points within each area.
12. Select a color ramp for the hexbins that appeals to you.
13. Click the Advanced Fill Options button to reveal the Color Scale dropdown. Change the Classification Method to Custom and set each color to indicate a range of just one. For example, set the lightest color to a value range of 0 - 0, the next color to 0 - 1, the next color to 1 - 2, and so on. Since our hexbins will only contain a few csa points, these color breaks should show us a reasonable distribution on the map.
14. In the color ramp section, notice the options to change the number of classes that the data is divided into. You can also change the classification type. 
15. Set a classification type and number of buckets that you feel best tells the story about where Portland is serviced by CSAs. Notice how much power you have over the message implicit in the map patterns. This is true with all cartography and not just online tools, yet interactive web GIS of this nature brings the power of the cartographer to the forefront.
16. Use the Share button at the top of the window to Publish your map and generate a Public Share Link. Paste that url link into a new browser window to see how your map may be viewed by other users. You'll also include this link as part of your assignment submission.

Let's try one more kind of map that CARTO does very well: the animated time series map. This type of map is used when your data has a date and/or time field representing when an event occurred. The data we'll use represent incidents of gun violence in Philadelphia. Each point is a shooting with a field noting when the event took place. Animating these events over time within a map can show temporal and spatial patterns of violence throughout the city.

1. Following the techniques you learned in previous lessons, upload shootings.geojson to CARTO.
2. Look at the table in the Data Preview section of Data Explorer.
3. Note the different fields that contain a date and/or time. Which one do you think represents the date the shooting occurred? If you said date, you're right. What we don't want to animate is the date that the point was added or updated within the GIS itself (unless we have no better temporal information), therefore we'll ignore the created_at and updated_at fields.
4. Create a new map with the shootings data.
5. Use the tools in the CARTO Builder to add a Time Series widget using the shootings data you uploaded. 
6. Be sure that the field the widget is using is the Date field.
7. In the Behavior section, select the Filter by Viewport option. This enables the Animation controls.
8. Turn the Animation Controls slider on.
9. Explore the widget in the panel below the map. You should see a Play button that will animate the map and show events over time.
10. You can continue to apply other thematic styling to the layer beyond the animation. Let's change the officer-involved shootings to a red color and leave all the other ones blue.
11. Use the layer style tools to create a custom color scheme with two colors: red indicating that, yes, an officer was involved and blue indicating, no, an officer was not involved. In the Fill section, select the officer_involved field, and use the filter button to set the Color Scale to Ordinal. Now you should be seeing the occasional red dot appear among the animated incidents. You may want to increase the size of the point symbols to see them more easily.
    The animation may be playing a bit fast for your taste at this point. There are lots of settings you can adjust to determine how fast the dots appear and how long they stay on the screen.
12. Use the Share button at the top of the window to Publish your map and generate a Public Share Link. Paste that url link into a new browser window to see how your map may be viewed by other users. Explore the functionality, in particular, the ability to click and draw a range of times in the animaion window to aggregate activity temporally. You'll include the link to your animation map as part of your lesson deliverables.

In this week's assignment, you'll continue getting some experience with CARTO's thematic mapping services. Please assemble a document with all of the following:

1. A hyperlink to your CSA map aggregated to Hexbins.
2. A hyperlink to your time-series map of Philadelphia shootings.
3. A map of one of your own datasets that appropriately applies either the aggregation or time series techniques covered in the walkthroughs. Write a paragraph explaining what you are showing and justifying your design decisions regarding styling, classification, etc.
4. A map of your own datasets made using some new technique you've learned by experimenting in CARTO Builder and reading its documentation. Add a paragraph explaining what technique you are using and why it's appropriate for showing these datasets.
   
   Items 3 and 4 can use data from your final project or some other data that you've obtained from an open data website. Please be sensitive to data ownership and licensing when uploading data to CARTO; don't put your organization's proprietary data on CARTO unless you've received permission to do so.

This week's cloud computing discussion covers Service Oriented Architecture (SOA) and Hadoop-style massively parallel data processing systems. SOA is interesting because this is how new Internet services are being developed. It is also a huge engineering challenge.

An epic blog post that helped me understand the importance of this was written by software engineer Steve Yegge [10], it is known as Stevey's Google Platforms Rant [11]. Yegge used to work for Amazon and now works for Google. Apparently, it was meant to be internal to Google, but it was accidentally published to great acclaim. Please read it for his passionate advocacy of a service oriented architecture and developer tools, and for his rather humorous, if somewhat salty and irreverent, description of life while working at these software companies.

Hadoop is an amazing system that was started by Doug Cutting, who wanted to provide the means to be able to index the entire Internet overnight, which at the time, only Google was doing effectively. Please read the Wikipedia entry on Apache Hadoop [12] for background. Hadoop is quite powerful, but also notoriously tricky to get working. Amazon has an interesting service called Elastic MapReduce [13] which claims to take a way a lot of the pain of setting up and maintaining such systems.