30:T168e, md `## Working with geodata We need to load our geodata in file formats, data formats and coordinate systems that d3 suppports. The file formats natively supported by D3 are [GeoJSON](http://geojson.org/) and TopoJson along with vanilla csv.I'll discuss each and use examples from a GeoJSON file containing all the disability parking spots in Vancouver (my hometown) I downloaded from the Vancouver Open Data catalogue.
But geodata are more complicated than just the file format!
To understand how mapping data work, we need to briefly identify key concepts that arise when we take a location on a spheroid (the Earth) and transform it into a 2D location on a plane (our maps). We call this kind of spatial data geodata and deriving them is a fundamental element in Geographic Information Systems, or GIS. Geodata are coded according to:

a Geographic Coordinate System format (GCS) (sometimes ccalled a Coordinate Reference System (CRS) is a reference framework that defines the locations of features on a model of the earth. It’s shaped like a globe—spherical. Its units are angular, usually degrees. A GCS uniquely defines all locations on Earth with coordinates. The World Geodetic System, or WGS , is currently the standard for spatial reference, based on a vast network of satellites.
A geodetic datum is the part of the GCS that determines which model (spheroid) is used to represent the earth’s surface and where it is positioned relative to the surface. Since the earth’s surface is not perfectly smooth or round, there are many different datums designed for different parts of the world. Its units are in meters, as distance from a reference pont.
A Projected Coordinate System (PCS) contains a GCS and converts it into a flat surface, using a projection algorithm and other parameters. A projection\ is the mathematical algorithm that defines how to present the round earth on a flat map. Its resulting units are coordinates in latitude and longitude.

In short,

A GCS defines where the data is located on the earth’s surface. The datum defines the actual coordinates.
A PCS tells the data how to draw on a flat surface, like on a paper map or a computer screen.

### Why do you need to know all this? Isn't it automatic? Well no, it's not. Because these datums were designed for different parts of the world, and different degrees of precision and projection, it's very common to have to transform your data from one coordinate system to another. Or you may find a location that you thought should be in Vancouver is showing up on the other side of the world! In particular, you need to explicilty verify that the whatever tool or code you are using has the right datum and projection. d3 expects the WGS84 datum, but can work with a number of different projections. #### Datum Let's talk about the datum first. Each is assigned an EPSG code. GIS systems and libraries use these EPSG definition data for identifying coordinate reference systems, projections, and performing transformations between these systems. The EPSG code is a specification ID that you can find at http://spatialreference.org/. The standard longitude and latitude references we use in d3, for example, use the EPSG:4326 datum. (This is commonly called WGS84). HOwever, many geodata come in different coordinate systems: Census Canada, for example, uses Lambert Conformal Conic (EPSG:3347). The City of Vancouver uses EPSG:26910. The geodetic coordinate system used in the data you have should be listed in an accompanying file (in the .prj file if the original data format is a shapefile directory). So you have to need to find the coordinate system (there may or may not be an EPSG number), and then convert the coordinates from the original form to what d3 expects: EPSG:4326.
The bad news is that sometimes you will need to wrangle these data into the right PCS. The good news is that there are a number of easy ways to do this.

You can download the free qGIS tool (http://www.qgis.org) (if you want to dive into the full spectrum of GIS and geodata) and export the layers to the apporpriate datum in the LAyers menu. You can also use the arcGIS tools if you have access to them (it's an expensive license, but most universities have full access).
Or if you only have a couple of coordinates you can use an online pair conversion tool like https://mygeodata.cloud/cs2cs/.

#### Projection d3 (and by extension the Vega and Vega-Lite grammars) supports many projections (take a look at them in this workbook: https://observablehq.com/@vega/vega-lite-cartographic-projections). We can typically use the \`\`\`Mercator\`\`\` projection for most cases. But you will probably need to play with the right scaling factor and centering to position your map as you want it. ### More on Mapping, GIS and geodata There are many great resources if you're interested. I drew from some of these:

the ESRI ArcGIS blog https://www.esri.com/arcgis-blog/products/arcgis-pro/mapping/coordinate-systems-difference/#datum);
qGIS guides https://docs.qgis.org/3.16/en/docs/
Google's geocoding API docs https://developers.google.com/maps/documentation/geocoding/overview

Finally - Here is another VERY POORLY documented fact: a d3 geoProjection takes coordinates in an [x,y] or [long,lat] format. In other words, longitude and latitude are switched from the normal way they will be expressed in many datasets - even after you have done the datum conversion. Typically the most useful workaround is simply to rename your data fields. Urk.` 31:T70d,md ` The VanAreaValues2 dataset is in topojson format. A TopoJSON file is similar, but instead of storing the latitudes and longitudes directly, it describes the topology (geometrical properties and spatial relations of figures) of a group of features, meaning the borders between them.So its structure is a little more complicated: its key categories are in the \`objects\` array. In the usa dataset below in this file, there are three objects:
\` objects: Object {counties: Object, states: Object, land: Object}\`.
In our Vancouver case, there is one object in the array - \`VancouverAreaSize\`). TopoJSON files are usually much smaller, but require a little bit of extra processing in the browser when you want to display one on a map, and can potentially lose details from the original latitudes/longitude.

d3 knows how to process these file types. The first visualization below shows vega maps geoshapes from a geojson file. OF course, you can always bring your data in in csv form as well, but rendering it will take a little more work. The second map shows data rendered from a csv file, where the longitude have to be explcitly identified as fields.

All GIS utilities support exporting geoJSON and topojson. (In either qGIS or arcGIS you typically do this using the \`Layers\` menu.) There are also online tools: I used [mapshaper](http://mapshaper.org). Mapshaper lets you load and inspect your geographic data and export it to geoJSON or topoJSON formats. It does NOT however let you convert your coordinate system. Once you have the data, you can set it up as a gist or host it somewhere and load the file using the regular \`d3.json()\` function. Warning: geodata files can be very big and gist will typically fail on many. Consider hosting on a more robust webserver. `32:T7bf,embed({ width: 900, height: 600, padding: { top: 0, left: 0, right: 0, bottom: 0 }, data: [ { name: "parkingVan", values: dparking1, //"format": {"type": "csv", "parse": "auto"}, "transform": [ { "type": "geopoint", "projection": "projection", "fields": ["longitude", "latitude"] }, { "type": "filter", "expr": "datum.x != null && datum.y != null" } ] }], scales: [], "signals": [ { "name": "hover", "value": null, "on": [ {"events": "symbol:mouseover", "update": "datum"}, {"events": "symbol:mouseout", "update": "null"} ] }, { "name": "dparkspots", "value": "Vancouver Disabled PArking Spaces", "update": "hover ? hover.NOTES + ' (' + hover.DESCRIPTN + ')' + ': ' + hover.LOCATION: 'Vancouver Disabled Parking'" } ], projections: [ { name: "projection", type: "mercator", scale: 200000, center: d3.geoCentroid(dparking) //"translate": [{"signal": "width / 2"}, {"signal": "height / 2"}] } ], "marks": [ { "type": "symbol", "from": {"data": "parkingVan"}, "encode": { "enter": { "size": {"value": 60}, "fill": {"value": "green"}, "fillOpacity": {"value": 0.8}, "stroke": {"value": "black"}, "strokeWidth": {"value": .5} }, "update": { "x": {"field": "x"}, "y": {"field": "y"}, "fill": {"value": "green"} }, "exit": { "x": {"field": "x"}, "y": {"field": "y"}, "fill": {"value": "green"} }, "hover": { "fill": {"value": "red"} } } }, { "type": "text", "interactive": false, "encode": { "enter": { "x": {"signal": "width", "offset": -5}, "y": {"value": 0}, "fill": {"value": "black"}, "fontSize": {"value": 10}, "align": {"value": "right"} }, "update": { "text": {"signal": "dparkspots"} } } } ] })33:Ta0d,embed({ width: 900, height: 560, padding: { top: 0, left: 0, right: 0, bottom: 0 }, data: [ { name: "streetsVan", url: "https://gist.githubusercontent.com/drlynb/234c622e2f98da7c61393cbb33573bf8/raw/8e8fa5ae3dc76d94dcca3a50e3dce41b89eb7185/vanstreets.json", format: { type: "json", feature: "vanstreets" }, transform: [ { type: "geopath", projection: "projection" } ] }, { name: "parkingVan", values: dparking1, //"format": {"type": "csv", "parse": "auto"}, "transform": [ { "type": "geopoint", "projection": "projection", "fields": ["longitude", "latitude"] }, { "type": "filter", "expr": "datum.x != null && datum.y != null" } ] }], scales: [], "signals": [ { "name": "hover", "value": null, "on": [ {"events": "symbol:mouseover", "update": "datum"}, {"events": "symbol:mouseout", "update": "null"} ] }, { "name": "dparkspots", "value": "Vancouver Disabled PArking Spaces", "update": "hover ? hover.NOTES + ' (' + hover.DESCRIPTN + ')' + ': ' + hover.LOCATION: 'Vancouver Disabled Parking'" } ], projections: [ { name: "projection", type: "mercator", scale: 200000, center: d3.geoCentroid(Streets) //"translate": [{"signal": "width / 2"}, {"signal": "height / 2"}] } ], marks: [ { type: "path", from: { data: "streetsVan" }, encode: { enter: { stroke: { value: "#bbbbbb" } }, update: { path: { field: "path" } } } }, { "type": "symbol", "from": {"data": "parkingVan"}, "encode": { "enter": { "size": {"value": 60}, "fill": {"value": "green"}, "fillOpacity": {"value": 0.8}, "stroke": {"value": "black"}, "strokeWidth": {"value": .5} }, "update": { "x": {"field": "x"}, "y": {"field": "y"}, "fill": {"value": "green"} }, "exit": { "x": {"field": "x"}, "y": {"field": "y"}, "fill": {"value": "green"} }, "hover": { "fill": {"value": "red"} }}}, { "type": "text", "interactive": false, "encode": { "enter": { "x": {"signal": "width", "offset": -5}, "y": {"value": 0}, "fill": {"value": "black"}, "fontSize": {"value": 12}, "align": {"value": "right"} }, "update": { "text": {"signal": "dparkspots"} } } } ] })34:T976,parking1 = embed({ $schema: 'https://vega.github.io/schema/vega/v5.json', data:[ { name: "streetsVan1", url: "https://gist.githubusercontent.com/drlynb/234c622e2f98da7c61393cbb33573bf8/raw/8e8fa5ae3dc76d94dcca3a50e3dce41b89eb7185/vanstreets.json", format: { type: "json", feature: "vanstreets" }, transform: [ { type: "geopath", projection: "projection" } ] }, { name: "disabilities1", url: "https://gist.githubusercontent.com/StanisNowak/7432a7e657f857e1e98d2f0f10015db7/raw/8a211fcffbea75f7caa8583736048e096679727b/disabilityParking.geojson", format: { type: "json", property: "features" } }], "signals": [ { "name": "hover", "value": null, "on": [ {"events": "symbol:mouseover", "update": "datum.properties"}, {"events": "symbol:mouseout", "update": "null"} ] }, { "name": "dparkspots", "value": "Vancouver Disabled Parking Spaces", "update": "hover ? hover.properties.NOTES + ' (' + hover + ')' + ': ' + hover: 'Vancouver Disabled Parking Spaces'" } ], projections: [{ name: "projection", fit: {signal: "data('disabilities1')"}, size: {signal: "[800, 600]"}, //scale: 190000, //center: [-123.149648,49.265815], type: "mercator" }], marks: [ { type: "shape", from: {data: "disabilities1"}, encode: { enter: { opacity: {value: 0.50}, fill: {value: "blue"}, "tooltip": {"signal": "datum.properties.NOTES" } }, update: { opacity: {value: 0.50}, fill: {value: "blue"} }, "fill": {"value": "red"} }, transform: [ { type: "geoshape", projection: "projection", pointRadius: 6 } ] }, { type: "path", from: { data: "streetsVan1" }, encode: { enter: { stroke: { value: "#aaaaaa" } }, update: { path: { field: "path" } } } }, { "type": "text", "interactive": false, "encode": { "enter": { "x": {"signal": "width", "offset": -5}, "y": {"value": 0}, "fill": {"value": "black"}, "fontSize": {"value": 14}, "align": {"value": "right"} }, "update": { "text": {"signal": "dparkspots"} } } } ] })35:[{"id":"8839234162abeaa37988fcfac9a2943a1ae3090f55f760352abc9b204a02b984a6c3a3327fe3487dc56558bfbfd49bf7a66547f7ff95ae243e8a1a1b101aff62","href":"https://static.observableusercontent.com/files/8839234162abeaa37988fcfac9a2943a1ae3090f55f760352abc9b204a02b984a6c3a3327fe3487dc56558bfbfd49bf7a66547f7ff95ae243e8a1a1b101aff62","name":"Geojson.png","type":"image/png","lastModified":1672965923942.124,"size":445640},{"id":"c8f2ae0b3c66f0a48f16470a395b054227df49539766132b49f13fb183f0409a63f6d9181d5dd6bc3bb0218998aab9f618fac271b734c1bdc7bcd2139517cf04","href":"https://static.observableusercontent.com/files/c8f2ae0b3c66f0a48f16470a395b054227df49539766132b49f13fb183f0409a63f6d9181d5dd6bc3bb0218998aab9f618fac271b734c1bdc7bcd2139517cf04","name":"Topojson.png","type":"image/png","lastModified":1672965923942.124,"size":429199}] 2e:{"title":"[IAT355] Tutorial 7.1: Geodata wrangling for Vega-lite and d3","cells":[{"id":0,"value":"# [IAT355] Tutorial 7.1: Geodata wrangling for Vega-lite and d3\n^{Copied from Lyn Bartram's [Geodata wrangling for Vega-lite and d3](https://observablehq.com/@drlynb/lyn-tries-maps-again)}\n","pinned":false,"hidden":false,"mode":"md"},{"id":19,"value":"md `I noticed that while I can get and process the \\\"states\\\" json file that is packaged as one of the vega-lite datasets (and subsequently use the \"values\" property as the processed array), I CAN'T do the same thing with regular files that i bring in from other sources. How do we process spatial data so that d3 and Vega can understand it? To do that, here is a little introduction to some geocoding getarounds.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":323,"value":"$30","pinned":false,"hidden":false,"mode":"ojs"},{"id":361,"value":"md `## File Formats\nOnce you have your coordinate system set to [lat,long] you need to get it into the right file format. d3 recognizes two geo file formats: \\`\\`\\`geojson\\`\\`\\` and \\`\\`\\`topojson\\`\\`\\`. Each of these file formats is organized slightly differently and you need to examine your data to find the key level of the hierarchy to tell d3 what features to extract and process: often geofiles will have several feature categories, and you need to choose which to render.\n

\nThe \\`dparking\\` dataset in this workbook is in geojson format. A GeoJSON file is a text file with a list of features and, for each one, a bunch of latitudes and longitudes describing its geometry. The dataset we use here contains a single \\`FeatureCollection\\` object. The individual data objects are in the \\`features\\` array and the \\`features.geometry\\` property contains the [long and lat] array of individual parking spots - explore the object hierarchy to get a feel for the structure of the data and the information. \n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":467,"value":"FileAttachment(\"Geojson.png\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":470,"value":"md ` Figure 1. GeoJSON data structure in Observable`","pinned":false,"hidden":false,"mode":"ojs"},{"id":457,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":452,"value":"FileAttachment(\"Topojson.png\").image()","pinned":false,"hidden":false,"mode":"ojs"},{"id":476,"value":"md `Figure 2. TopoJSON data structure in Observable`","pinned":false,"hidden":false,"mode":"ojs"},{"id":482,"value":"md `## Using geodata to make maps in Vega-Lite`","pinned":false,"hidden":false,"mode":"ojs"},{"id":68,"value":"md `And now we try this with the Vancouver disabled parking spot data.\nThese spots plot perfectly in qGIS and mapshaper.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":145,"value":"md `### Vancouver Parking Spots mapped the easy way as geoshapes.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":147,"value":"viewof parking = embed({\n $schema: 'https://vega.github.io/schema/vega/v5.json',\n data:[{\n name: \"disabilities\",\n url: \"https://gist.githubusercontent.com/StanisNowak/7432a7e657f857e1e98d2f0f10015db7/raw/8a211fcffbea75f7caa8583736048e096679727b/disabilityParking.geojson\",\n format: {\n type: \"json\",\n property: \"features\"\n }\n }],\n projections: [{\n name: \"projection\",\n fit: {signal: \"data('disabilities')\"},\n size: {signal: \"[800, 600]\"},\n //scale: 190000,\n //center: [-123.149648,49.265815],\n type: \"mercator\"\n }],\n marks: [\n {\n type: \"shape\",\n from: {data: \"disabilities\"},\n encode: {\n update: {\n opacity: {value: 0.25},\n fill: {value: \"blue\"}\n }\n },\n transform: [\n {\n type: \"geoshape\",\n projection: \"projection\",\n pointRadius: 6\n }\n ]\n }\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":113,"value":"md `### Mapped the gnarlier way with csv and geopoint`","pinned":false,"hidden":false,"mode":"ojs"},{"id":227,"value":"md `Here because i have the properties associated with each point at the same level in the data structure I can easily access them using the \"hover\" event, so i can use the text mark to show them. Mouse over a point and see. There's an odd aspect that causes some of the points to disappear after they have been hovered, so I needed to add in the exit{} event to make sure that the points re-rendered.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":77,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":116,"value":"md `And now finally let's bring in the streets map and see if I can plot the spaces on top of the streets. I added the hover code to see the data associated with each point.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":118,"value":"$33","pinned":true,"hidden":false,"mode":"ojs"},{"id":163,"value":"md `Finally I am going to do this using the geoshapes - they look much nicer! and they are not so gnarly. Some things to notice here with these data:\n

I have to scale the map to the view. There seem to be a couple of ways to do this. I can use d3.geoCentroid(Streets) but then I need to set d3 parameters for the projection:\n\n d3.geoMercator() \n .fitSize([100, 100], Streets) \n .scale()
\nor I can use a signal that takes the data and fits to it:
\n\nprojections: [{ \n name: \"projection\", \n fit: {signal: \"data('disabilities1')\"}, \n size: {signal: \"[800, 600]\"}, \n\n
I have added a tooltip that shows the NOTES field of the point data on mouseover. But for the life of me I can't see how to access additional properties in the same depth in the json file such as LOCATION (I want to concatenate the string). Next step.

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":175,"value":"$34","pinned":false,"hidden":false,"mode":"ojs"},{"id":129,"value":"md `## Data Modules and Libraries`","pinned":false,"hidden":false,"mode":"ojs"},{"id":133,"value":"d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":439,"value":"html`\n\n`","pinned":false,"hidden":false,"mode":"ojs"},{"id":136,"value":"embed = require('vega-embed@3')","pinned":false,"hidden":false,"mode":"ojs"},{"id":190,"value":"dp2 = d3.json(\"https://gist.githubusercontent.com/StanisNowak/7432a7e657f857e1e98d2f0f10015db7/raw/8a211fcffbea75f7caa8583736048e096679727b/disabilityParking.geojson\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":223,"value":"dparking = d3.json(\"https://gist.githubusercontent.com/drlynb/6dffc7ccdaf774ee031dc46c909721b2/raw/8a8b8fb6b6b60e66980cbb9f897b198bac23dbf7/testparking.geojson\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":219,"value":"dparking1 = d3.csv(\"https://gist.githubusercontent.com/drlynb/2b4ac0221b8138ade06cd91886b95f69/raw/a45e5883da217c0c1b3237df307136b5ee86a7e9/disparking2.csv\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":369,"value":"vanAreaValues2 = d3.json(\"https://www.sfu.ca/~lyn/data/Urban/VancouverAreaSizeTopo.json\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":391,"value":"usa = d3.json('https://vega.github.io/vega-datasets/data/us-10m.json')","pinned":true,"hidden":false,"mode":"ojs"},{"id":229,"value":"Streets = d3.json(\"https://gist.githubusercontent.com/drlynb/234c622e2f98da7c61393cbb33573bf8/raw/8e8fa5ae3dc76d94dcca3a50e3dce41b89eb7185/vanstreets.json\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":265,"value":"","pinned":false,"hidden":false,"mode":"ojs"}],"files":"$W35","stdlib":"1","titling":"auto"}