31:[] 2f:{"title":"Flight Paths Data Processing","cells":[{"id":0,"value":"# Flight Paths Data Processing","pinned":false,"hidden":false,"mode":"md"},{"id":7,"value":"This notebook demonstrates how to process flight path data in preparation for visualization. For the visualizations, visit the [Flight Paths Edge Bundling](https://observablehq.com/@sjengle/flight-paths-edge-bundling) notebook.","pinned":false,"hidden":false,"mode":"md"},{"id":32,"value":"This notebook uses datasets from the [Vega Datasets](https://github.com/vega/vega-datasets) repository and the [U.S. Geographic Data](https://observablehq.com/@observablehq/us-geographic-data) Observable notebook. The Vega datasets are imported below:","pinned":false,"hidden":false,"mode":"md"},{"id":47,"value":"vegaData = (await import('https://cdn.jsdelivr.net/npm/vega-datasets@3/+esm')).default","pinned":false,"hidden":false,"mode":"ojs"},{"id":1224,"value":"The following sections provide more details on these datasets and handles preprocessing to make our visualizations easier to create.","pinned":false,"hidden":false,"mode":"md"},{"id":182,"value":"## Geospatial Data","pinned":false,"hidden":false,"mode":"md"},{"id":893,"value":"

The United States geospatial data is loaded from the U.S. Geographic Data Observable notebook.

\n\n

Description: Commonly used contextual data for geographic visualizations.
Additional Details: https://observablehq.com/@observablehq/us-geographic-data
Original Source: U.S. Atlas TopoJSON
License: ISC License

","pinned":false,"hidden":false,"mode":"md"},{"id":902,"value":"The data can be loaded as follows:","pinned":false,"hidden":false,"mode":"md"},{"id":138,"value":"import {nation,statesMesh} from \"@observablehq/us-county-datasets\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":904,"value":"We can test it out using Observable Plot:","pinned":false,"hidden":false,"mode":"md"},{"id":906,"value":"Plot.plot({\n projection: \"albers-usa\",\n width: 500,\n marks: [\n Plot.geo(nation, {stroke: \"none\", fill: \"whitesmoke\"}),\n Plot.geo(statesMesh, {stroke: \"white\", strokeWidth: 1.5}),\n Plot.geo(nation, {stroke: \"gainsboro\", strokeWidth: 0.75, fill: \"none\"})\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":191,"value":"The United States TopoJSON boundary data has already been projected. We must use the [exact same projection settings](https://github.com/topojson/us-atlas) to overlay data on top:","pinned":false,"hidden":false,"mode":"md"},{"id":184,"value":"topoProjection = d3.geoAlbersUsa()\n .scale(1300)\n .translate([487.5, 305])","pinned":true,"hidden":false,"mode":"ojs"},{"id":222,"value":"topoPath = d3.geoPath(topoProjection);","pinned":true,"hidden":false,"mode":"ojs"},{"id":197,"value":"The [`viewBox` attribute](https://developer.mozilla.org/en-US/docs/Web/SVG/Reference/Attribute/viewBox) is another way to specify the width and height of an SVG, but allows for the entire SVG to scale. Again, we must use the same dimensions as the original TopoJSON projection:","pinned":false,"hidden":false,"mode":"md"},{"id":213,"value":"topoWidth = 975;","pinned":false,"hidden":false,"mode":"ojs"},{"id":214,"value":"topoHeight = 610;","pinned":false,"hidden":false,"mode":"ojs"},{"id":193,"value":"topoViewBox = `0 0 ${topoWidth} ${topoHeight}`","pinned":false,"hidden":false,"mode":"ojs"},{"id":864,"value":"We can create a function that will return an SVG with this `viewbox` already set, along with any other (optional) attributes:","pinned":false,"hidden":false,"mode":"md"},{"id":867,"value":"function topoSVG(attrs) {\n const svg = d3.create(\"svg\").attr(\"viewBox\", topoViewBox);\n\n // set attributes if provided\n for (let property in attrs) {\n svg.attr(property, attrs[property]);\n }\n \n return svg;\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":911,"value":"## Airport Data","pinned":false,"hidden":false,"mode":"md"},{"id":58,"value":"

This dataset is loaded from the Vega Datasets repository.

\n\n

Description: Airports in the United States and its territories, including major commercial, regional, and municipal airports. Contains information about each airport's location (latitude/longitude coordinates), identification codes, name, city, state, and country.
Details: https://github.com/vega/vega-datasets/blob/main/datapackage.md#airports
Original Source: Federal Aviation Administration (FAA)
License: U.S. Government Works

","pinned":false,"hidden":false,"mode":"html"},{"id":1391,"value":"Below is an example of the raw data:","pinned":false,"hidden":false,"mode":"md"},{"id":1241,"value":"```\niata,name,city,state,country,latitude,longitude\n00M,Thigpen,Bay Springs,MS,USA,31.95376472,-89.23450472\n```","pinned":false,"hidden":false,"mode":"md"},{"id":303,"value":"Each airport as a unique [IATA airport code](https://en.wikipedia.org/wiki/IATA_airport_code). We need to be careful that the IATA airport codes are not erroneously interpreted as numbers. For example, both `SFO` and `0E0` should be interpreted as text. As such, we will load this using [d3.csv](https://d3js.org/d3-fetch#csv) with a row conversion function:","pinned":false,"hidden":false,"mode":"md"},{"id":51,"value":"airports = d3.csv(vegaData['airports.csv'].url, row => ({\n // include original data\n ...row,\n \n // overwrite converted values\n iata: `${row.iata}`,\n latitude: parseFloat(row.latitude),\n longitude: parseFloat(row.longitude)\n}))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1258,"value":"airports","pinned":true,"hidden":false,"mode":"ojs"},{"id":310,"value":"Let's do a quick sanity check of the data. We can get all of the unique IATA values:","pinned":false,"hidden":false,"mode":"md"},{"id":1415,"value":"airportCodes = new Set(airports.map(airport => airport.iata));","pinned":true,"hidden":false,"mode":"ojs"},{"id":1426,"value":"We should have the same the number of unique IATA values as airports:","pinned":false,"hidden":false,"mode":"md"},{"id":312,"value":"airportCodes.size == airports.length","pinned":true,"hidden":false,"mode":"ojs"},{"id":1289,"value":"## Flights Data","pinned":false,"hidden":false,"mode":"md"},{"id":86,"value":"

This dataset is loaded from the Vega Datasets repository.

\n\n

Description: Flight information for the year 2008. Each record consists of an origin airport (identified by IATA id), a destination airport, and the count of flights along this route.
Details: https://github.com/vega/vega-datasets/blob/main/datapackage.md#flights_airport
Original Source: U.S. Bureau of Transportation Statistics (BTS)
License: U.S. Government Works

\n\n\n\n","pinned":false,"hidden":false,"mode":"html"},{"id":1405,"value":"Below is an example of the raw data:","pinned":false,"hidden":false,"mode":"md"},{"id":1407,"value":"```\norigin,destination,count\nABE,ATL,853\nABE,BHM,1\n```","pinned":false,"hidden":false,"mode":"md"},{"id":1410,"value":"We will load this dataset similarly as the airports to make sure the IATA values do not get converted to numbers:","pinned":false,"hidden":false,"mode":"md"},{"id":89,"value":"flights = d3.csv(vegaData['flights-airport.csv'].url, row => ({\n // include original data\n ...row,\n \n // overwrite converted values\n origin: `${row.origin}`,\n destination: `${row.destination}`,\n count: parseInt(row.count)\n}))","pinned":true,"hidden":false,"mode":"ojs"},{"id":92,"value":"flights","pinned":false,"hidden":false,"mode":"ojs"},{"id":286,"value":"Again, we should do a quick sanity check. Let's make sure all of our flights are for known airport IATA codes:","pinned":false,"hidden":false,"mode":"md"},{"id":284,"value":"!flights.some((flight) => { \n !airportCodes.has(flight.origin) || \n !airportCodes.has(flight.destination)})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1291,"value":"## Data Processing","pinned":false,"hidden":false,"mode":"md"},{"id":1444,"value":"We need to do some processing to prepare our airports and flights into nodes and links for visualization.\n\n","pinned":false,"hidden":false,"mode":"md"},{"id":1463,"value":"### Processing Nodes","pinned":false,"hidden":false,"mode":"md"},{"id":1441,"value":"First, we need to add the number of incoming and outgoing flights from each airport. We can use [d3.rollup](https://d3js.org/d3-array/group#rollup) for these calculations:","pinned":false,"hidden":false,"mode":"md"},{"id":1446,"value":"incomingFlights = d3.rollup(\n flights, \n values => d3.sum(values, flight => flight.count),\n flight => flight.origin\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":281,"value":"outgoingFlights = d3.rollup(\n flights, \n values => d3.sum(values, flight => flight.count),\n flight => flight.destination\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1626,"value":"We will have a configurable limit of the number of top airports we will keep. This can be changed when the data is imported in other notebooks:","pinned":false,"hidden":false,"mode":"md"},{"id":1623,"value":"limit = airports.length","pinned":false,"hidden":false,"mode":"ojs"},{"id":327,"value":"We only want to keep airports with non-zero incoming and outgoing flights, and airports that fall within our projection.","pinned":false,"hidden":false,"mode":"md"},{"id":333,"value":"nodes = airports.map(airport => {\n // get number of flights for this airport\n const incoming = incomingFlights.get(airport.iata);\n const outgoing = outgoingFlights.get(airport.iata);\n\n // filter out airports with no incoming or outgoing flights\n if (!incoming || !outgoing) {\n return undefined;\n }\n\n // calculate the positions using the TopoJSON projection\n const position = topoProjection([airport.longitude, airport.latitude]);\n\n // filter out airports that cannot be projected on map\n if (!position) {\n return undefined;\n }\n\n // otherwise return airport with incoming and outgoing flights\n return {\n ...airport,\n incoming,\n outgoing,\n position\n };\n})\n// filter out undefined values\n.filter(airport => airport)\n// sort so larger airports are first\n.sort((a, b) => d3.descending(a.outgoing, b.outgoing))\n// limit the number of results to return\n.slice(0, limit);","pinned":true,"hidden":false,"mode":"ojs"},{"id":1469,"value":"We lost ${formatThousands(flights.length - nodes.length)} total airports in this processing step. We can take a peek at our largest and smallest airports:","pinned":false,"hidden":false,"mode":"md"},{"id":1475,"value":"nodes[0]","pinned":true,"hidden":false,"mode":"ojs"},{"id":1477,"value":"nodes[nodes.length - 1]","pinned":true,"hidden":false,"mode":"ojs"},{"id":1497,"value":"It will also help to have a quick way to look up the object associated with nodes by the airport IATA code:","pinned":false,"hidden":false,"mode":"md"},{"id":336,"value":"nodeLookup = new Map(nodes.map(node => [node.iata, node]));","pinned":true,"hidden":false,"mode":"ojs"},{"id":1567,"value":"nodeLookup.get(\"ONT\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":1486,"value":"### Processing Links","pinned":false,"hidden":false,"mode":"md"},{"id":341,"value":"We only want to keep flights for the airports we kept above. We also want to calculate the distance between our nodes for use in edge bundling later on.","pinned":false,"hidden":false,"mode":"md"},{"id":851,"value":"// calculates the distance between two nodes\n// sqrt( (x2 - x1)^2 + (y2 - y1)^2 )\nfunction calcDistance(source, target) {\n const dx2 = Math.pow(target.position[0] - source.position[0], 2);\n const dy2 = Math.pow(target.position[1] - source.position[1], 2);\n return Math.sqrt(dx2 + dy2);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":344,"value":"links = flights.map(flight => {\n // use nodeLookup (not airportCodes) for post-filter data\n const source = nodeLookup.get(flight.origin);\n const target = nodeLookup.get(flight.destination);\n\n // if origin or destination is not a valid airport, skip this row\n if (!source || !target) {\n return undefined;\n }\n\n // create a unique path identifier\n const code = `${flight.origin}-${flight.destination}`;\n\n // calculate the distance between source and target\n const distance = calcDistance(source, target);\n\n // create array of points for drawing\n const positions = [source.position, target.position];\n \n // return link with source and target\n return {\n ...flight,\n code, \n distance,\n positions\n };\n})\n// filter out undefined values\n.filter(flight => flight);","pinned":true,"hidden":false,"mode":"ojs"},{"id":354,"value":"We lost ${formatThousands(flights.length - links.length)} flights with this processing and filtering step.","pinned":false,"hidden":false,"mode":"md"},{"id":1554,"value":"We can now create a handy lookup of links by their origin node:","pinned":false,"hidden":false,"mode":"md"},{"id":1560,"value":"linkLookup = d3.group(links, link => link.origin);","pinned":true,"hidden":false,"mode":"ojs"},{"id":1563,"value":"linkLookup.get(\"ONT\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":30,"value":"## Exporting Data","pinned":false,"hidden":false,"mode":"md"},{"id":1570,"value":"We can bundle our data and utility functions together:","pinned":false,"hidden":false,"mode":"md"},{"id":1577,"value":"graph = ({\n nodes, \n links,\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1587,"value":"lookup = ({\n nodes: nodeLookup,\n links: linkLookup\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1572,"value":"topo = ({\n projection: topoProjection,\n path: topoPath,\n svg: topoSVG,\n width: topoWidth,\n height: topoHeight\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":1598,"value":"The above can be imported separately as follows:\n\n```javascript\nimport {graph, lookup, topo} from \"@sjengle/flight-paths-data-processing\"\n```","pinned":false,"hidden":false,"mode":"md"},{"id":1631,"value":"If you want to set the limit differently, try instead:","pinned":false,"hidden":false,"mode":"md"},{"id":1633,"value":"```javascript\nimport {graph, lookup, topo} with {limit} from \"@sjengle/flight-paths-data-processing\"\n```","pinned":false,"hidden":false,"mode":"md"},{"id":1603,"value":"Or, it can be imported as an all-in-one bundle:","pinned":false,"hidden":false,"mode":"md"},{"id":1599,"value":"data = ({graph, lookup, topo})","pinned":false,"hidden":false,"mode":"ojs"},{"id":1607,"value":"```javascript\nimport {data} from \"@sjengle/flight-paths-data-processing\"\n```","pinned":false,"hidden":false,"mode":"md"},{"id":19,"value":"## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":1471,"value":"formatThousands = d3.format(\",\");","pinned":false,"hidden":false,"mode":"ojs"},{"id":76,"value":"","pinned":false,"hidden":false,"mode":"html"}],"files":"$W31","stdlib":"1","titling":"auto"}