31:Tadd,md `## Working with geodata We need to load our geodata in file formats, data formats and coordinate systems that d3 suppports. The file format natively supported by D3 is [GeoJSON](http://geojson.org/), so I'll use a GeoJSON file containing all the disbaility parking spots in Vancouver (my hometown) I downloaded from the Vancouver Open Data catalogue.
But geodata are more complicated than just the file format!

Geodata are coded using

a geodetic coordinate system format called a "datum" as a spatial reference;
and a projection.

Datum: Let's talk about the "datum" format first (DO NOT mix this up with the convention of referring to the current object in Vega or d3 as a \`\`\`datum\`\`\`: the d3 \`\`\`datum\`\`\` is quite different.) These all have a particular specification ID that you can find at http://spatialreference.org/. The standard longitude and latitude references we use in d3, as far as I can tell, are WGS84 format - EPSG:4326. HOwever, many geodata come in different coordinate systems: Census Canada, for example, uses Lambert Conformal Conic. The City of Vancouver uses EPSG:26910. The geodetic coordinate system used in the odata you have should be listed in an accompanying file (in the .prj file if the original data format is a shapefile directory). So you have to need to find the coordinate system (there may or may not be an EPSG number), and then convert the coordinates from the original form to the standard EPSG:4326(lat,long). There are a number of ways to do this. You can download the free qGIS tool (if you want to dive into the full spectrum of GIS and geodata) and export the layers to the apporpriate datum in the LAyers menu. You can also use the arcGIS tools if you have access to them (it's an expensive license, but most universities have full access). Or if you only have a couple of coordinates you can use an online pair conversion tool like https://mygeodata.cloud/cs2cs/.

Projection: d3 (and by extension the Vega and Vega-Lite grammars) supports many projections (take a look at them in this workbook: https://observablehq.com/@vega/vega-lite-cartographic-projections). We can typically use the \`\`\`Mercator\`\`\` projection for most cases. But you will probably need to play with the right scaling factor and centering to position your map as you want it. Here is another VERY POORLY documented fact: a d3 geoProjection takes coordinates in an [x,y] or [long,lat] format. In other words, longitude and latitude are switched from the normal way they will be expressed in many datasets - even after you have done the datum conversion. Typically the most useful workaround is simply to rename your data fields. Urk.` 32:Ta6f,md `## File Formats Once you have your coordinate system set to [lat,long] you need to get it into the right file format. d3 recognizes two geo file formats: \`\`\`geojson\`\`\` and \`\`\`topojson\`\`\`. Each of these file formats is organized slightly differently and you need to examine your data to find the key level of the hierarchy to tell d3 what features to extract and process: often geofiles will have several feature categories, and you need to choose which to render.
The \`vanCrime\` dataset in this workbook is in geojson format. A GeoJSON file is a text file with a list of features and, for each one, a bunch of latitudes and longitudes describing its geometry. The dataset we use here contains a single \`FeatureCollection\` object. The individual data objects are in the \`features\` array and the \`features.geometry\` property contains the [long and lat] array of individual parking spots - explore the object hierarchy to get a feel for the structure of the data and the information.

The \`VanAreas\` dataset is in topojson format. A TopoJSON file is similar, but instead of storing the latitudes and longitudes directly, it describes the topology (geometrical properties and spatial relations of figures) of a group of features, meaning the borders between them.So its structure is a little more complicated: in our data here, its key object is in the \`objects\` array (in this case, there is one object in the array - \`VancouverAreaSize\`). TopoJSON files are usually much smaller, but require a little bit of extra processing in the browser when you want to display one on a map, and can potentially lose details from the original latitudes/longitude.

d3 knows how to process these file types. The first visualization below shows vega maps geoshapes from a geojson file. OF course, you can always bring your data in in csv form as well, but rendering it will take a little more work. The second map shows data rendered from a csv file, where the longitude have to be explcitly identified as fields.

All GIS utilities support exporting geoJSON and topojson. (In either qGIS or arcGIS you typically do this using the \`Layers\` menu.) There are also online tools: I used [mapshaper](http://mapshaper.org). Mapshaper lets you load and inspect your geographic data and export it to geoJSON or topoJSON formats. It does NOT however let you convert your coordinate system. Once you have the data, you can set it up as a gist or host it somewhere and load the file using the regular \`d3.json()\` function. Warning: geodata files can be very big and gist will typically fail on many. Consider hosting on a more robust webserver.`33:T695,CrimesMap = VegaLite({ width: 600, height: 400, layer: [{ data: { values: VanAreas, format: { type: "topojson", feature: "VancouverAreaSize" } }, projection: { type: "mercator" }, mark: "geoshape", encoding: { fill: { field: "properties.density", type: "quantitative", title: "Density", scale: {"scheme": "greys"} }, stroke: { value: "black" }, tooltip: [ {field: "properties.NAME", title: "Name"}, {field: "properties.density", title: "#persons/500sqft"} ] } }, { data: { values: vanCrime, format: { type: "json", property: "features", } }, projection: { type: "mercator" }, mark: "circle", encoding: { longitude: { field: "properties.lng", type: "quantitative" }, latitude: { field: "properties.lat", type: "quantitative" }, size: {value: 12}, color: { scale: { domain: ['Non-vehicular', 'Vehicular'], range: ['#ff0000', '#00aaff'] }, field: "properties.Offense Type", title: "Offense Type", type: "nominal" }, tooltip: [ {field: "properties.Offense Type", title: "Offense Type",type: "nominal"}, {field: "properties.Offense", title: "Offense",type: "nominal"}, {field: "properties.NEIGHBOURHOOD", title: "Neighbourhood", type: "nominal"} ] } } ] })34:T9be,TimeOfCrimes = VegaLite({ vconcat:[ { width: 600, height: 400, layer: [{ data: { values: VanAreas, format: { type: "topojson", feature: "VancouverAreaSize" } }, projection: { type: "mercator" }, mark: "geoshape", encoding: { fill: { field: "properties.density", type: "quantitative", title: "Density", scale: {"scheme": "greys"} }, stroke: { value: "black" }, tooltip: [ {field: "properties.NAME", title: "Name"}, {field: "properties.density", title: "#persons/500sqft"} ] } }, { data: { values: vanCrime2, }, projection: { type: "mercator" }, mark: "circle", transform: [ {filter: {selection: "brush"}} ], encoding: { longitude: { field: "lng", type: "quantitative", }, latitude: { field: "lat", type: "quantitative" }, size: {value: 12}, color: { scale: { domain: ['Non-vehicular', 'Vehicular'], range: ['#ff0000', '#00aaff'] }, field: "Offense Type", title: "Offense Type", type: "nominal" }, detail: { field: "date", type: "nominal" }, tooltip: [ {field: "Offense Type", title: "Offense Type",type: "nominal"}, {field: "Offense", title: "Offense",type: "nominal"}, {field: "NEIGHBOURHOOD", title: "Neighbourhood", type: "nominal"}, {field: "Date", type: "nominal"} ] } } ] }, { width: 600, height: 200, data: { values: vanCrime2, }, selection: { brush: { type: "interval", encodings: ["x"]} }, mark: "line", encoding: { x: {timeunit: "monthday", field: "Date", type: "temporal"}, y: {aggregate: "count" , field: "*", type: "quantitative"}, color: { field: "Offense Type", title: "Offense Type", scale: { domain: ['Non-vehicular', 'Vehicular'], range: ['#ff0000', '#00aaff'] } } } } ] })35:T431,ChimaCrime = VegaLite({ width: 500, height: 300, layer: [ { data: { values: ChicagoTopoJson, format: { type: "topojson", feature: "chicago-community-areas" } }, projection: { type: "mercator" }, mark: "geoshape", encoding: { fill: { field: "properties.shape_area", type: "quantitative", scale: {"scheme": "greys"} }, stroke: { value: "black" }, } }, { data: { values: crimes2, }, transform: [ // {filter: {field: "Longitude", grt: 0}} ], projection: { type: "mercator" }, mark: {type: "circle", color: "black"}, encoding: { longitude: { field: "Longitude", type: "quantitative", }, latitude: { field: "Latitude", type: "quantitative" }, size: {value: 12}, } } ] })36:[] 2f:{"title":"GeoVis in Vega-Lite","cells":[{"id":0,"value":"md`# GeoVis in Vega-Lite`","pinned":false,"hidden":false,"mode":"ojs"},{"id":129,"value":"$31","pinned":false,"hidden":false,"mode":"ojs"},{"id":135,"value":"$32","pinned":false,"hidden":false,"mode":"ojs"},{"id":8,"value":"vanCrime","pinned":true,"hidden":false,"mode":"ojs"},{"id":22,"value":"VanAreas","pinned":true,"hidden":false,"mode":"ojs"},{"id":28,"value":"md` remember from our tutorial on how to handle data structures in javascript that you will need drill down through this structure to access the data in your visualizations.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":31,"value":"md` **Challenge**: retrieve the population value of the first neighbourhood in the topojson`","pinned":false,"hidden":false,"mode":"ojs"},{"id":36,"value":"VanAreas.objects.VancouverAreaSize.geometries[0].properties.population","pinned":true,"hidden":false,"mode":"ojs"},{"id":48,"value":"md` Let's make a map of vancouvers population.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":57,"value":"PopMap = VegaLite({\n width: 500,\n height: 300,\n data: {\n values: VanAreas,\n format: {\n type: \"topojson\",\n feature: \"VancouverAreaSize\"\n }\n },\n projection: {\n type: \"mercator\"\n },\n mark: \"geoshape\",\n encoding: {\n fill: {\n field: \"properties.density\",\n type: \"quantitative\",\n scale: {\"scheme\": \"greys\"}\n },\n stroke: {\n value: \"black\"\n },\n tooltip: [\n {field: \"properties.NAME\", title: \"Name\"},\n {field: \"properties.density\", title: \"#persons/500sqft\"}\n ]\n }\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":60,"value":"md` Now let's add crimes as a layer on top.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":64,"value":"$33","pinned":true,"hidden":false,"mode":"ojs"},{"id":72,"value":"md` Let's add a simple time scrubber for the month. Lets use Vconcat for that. Unfortunately, Vega-Lite selections and interactions wont work on nested hierarchical data, so we have to convert them to a flat structure. You can find the code for this in the imports sections for the dataset VanCrimes2.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":91,"value":"$34","pinned":true,"hidden":false,"mode":"ojs"},{"id":153,"value":"md` ## Exercise\nCreate a map for each type of json file.`","pinned":false,"hidden":false,"mode":"ojs"},{"id":192,"value":"Chimap = VegaLite({\n width: 500,\n height: 300,\n data: {\n values: ChicagoTopoJson,\n format: {\n type: \"topojson\",\n feature: \"chicago-community-areas\"\n }\n },\n projection: {\n type: \"mercator\"\n },\n mark: \"geoshape\",\n encoding: {\n fill: {\n field: \"properties.shape_area\",\n type: \"quantitative\",\n scale: {\"scheme\": \"greys\"}\n },\n stroke: {\n value: \"black\"\n },\n }\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":157,"value":"ChicagoTopoJson = d3.json(\"https://raw.githubusercontent.com/RandomFractals/ChicagoCrimes/master/data/chicago-community-areas.topojson\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":155,"value":"ChicagoGeoJson = d3.json(\"https://raw.githubusercontent.com/RandomFractals/ChicagoCrimes/master/data/chicago-community-areas.geojson\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":183,"value":"md` **Challenge** now create a layer for the crimes at their locations and color by type`","pinned":false,"hidden":false,"mode":"ojs"},{"id":206,"value":"crimes2 = Crimes.filter( d => d.Longitude < 0)","pinned":true,"hidden":false,"mode":"ojs"},{"id":180,"value":"Crimes = d3.csv(\"https://gist.githubusercontent.com/emanueles/13dfe2c1d43e11b5207bb4e6125d71bf/raw/41d14f8fbd9d6cd9e8ad074f8417cff1329ab339/chicago_crimes_sept_2019.csv\").then(function(data){\n \n data.forEach(d => {\n \n d.dtg = new Date(d['Date']);\n \n });\n\n return data;\n \n\n \n});","pinned":false,"hidden":false,"mode":"ojs"},{"id":198,"value":"$35","pinned":true,"hidden":false,"mode":"ojs"},{"id":187,"value":"md `**Challenge** make it interactive using other data within the crimes dataset`","pinned":false,"hidden":false,"mode":"ojs"},{"id":160,"value":"","pinned":false,"hidden":false,"mode":"ojs"},{"id":10,"value":"md` ## Imports and Libraries`","pinned":false,"hidden":false,"mode":"ojs"},{"id":85,"value":"import {vanCrime as vanCrime} from '@sfu-iat355/vancouver-crime-data'","pinned":true,"hidden":false,"mode":"ojs"},{"id":122,"value":"vanCrime2 = {\n let i=vanCrime.features.length;\n let j=0;\n let k=0;\n let temp ={};\n let newCrime=[];\n for(k=0;k