31:T419,## About At [Periscopic](https://periscopic.com), we often use repositories to take in raw data, process it, and output data files for different purposes. We wanted an easier way to get data we have stored on our repos into Observable for data visualization, exploration, and more. By using an API, we can make updates with less hassle. Instead of modifying file attachments as data files change, we can commit changes to update a data file repo-side, which the API can pull in. The `gh()` helper function takes links from navigating a GitHub repo in the browser and turns them into calls to the GitHub API. This makes it easy to go from repo to data source. We hope this makes working with your data that much easier. There are helpers for [Bitbucket repos](https://observablehq.com/@periscopic/bitbucket-api-in-observable) or [GitHub repos](https://observablehq.com/@periscopic/github-api-in-observable), depending on your preference. Got feedback? Feel free to comment and help us make this better. Made with love by the Periscopic Data Team32:T5f6,## Authentication: Private Repos If you are trying to access a private GitHub repository, you will need to provide a [Personal access token](https://github.com/settings/tokens) (PAT). For convenience, this notebook defaults to filling the `ghToken` field with an [Observable Secret](https://observablehq.com/settings/secrets) called `GH_API_PAT`. Below you can check the secret is accessible on your account. Secrets are not accessible in public notebooks, but you can fork this notebook as a template to create your own version for private repo access. ```js import {gh} from "${currentSpecifier}" ``` ${hasPassword ? "🚀 Ready to Access Private Repos!" : "⚠️ Provide Github PAT as Observable Secret to Access Private Repos"} - `GH_API_PAT` ${hasPassword ? "✅ Secret found" : "⚠️ Secret not found"} Regarding permissions, your personal access token should at least have read-level access to your repositories. For more info, checkout the [GitHub Support Page on personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) and the [Introduction to Secrets](https://observablehq.com/@observablehq/secrets).33:T9e0,async function gh( link // chosen link in github repo , { endpoint = "repos", // endpoint string ghToken = credentials[0].ghToken, // GitHub Personal Access Token, see https://docs.github.com/en/rest debug = false // if true, return API url } = {}) { // main url endpoint for API, see https://docs.github.com/en/rest/repos/contents#custom-media-types-for-repository-contents const URL_PREFIX = 'https://api.github.com' // create header for accepting responses, see https://docs.github.com/en/rest/overview/media-types?apiVersion=2022-11-28 const headers = { Accept: `application/vnd.github.raw` } if (ghToken !== null) { headers["Authorization"] = `token ${ghToken}` } // parse data link: url → path if (link == undefined) { return "Include GitHub Link (Docs: https://observablehq.com/@periscopic/github-api-in-observable)"} // parsing the github link let urlLink = "" if (link !== "") { const rawLink = String(link) let inputLink = rawLink // get url components to ping api let unpack = inputLink.split("/").slice(3) let owner = unpack.slice(0, 1) let repo = unpack.slice(1, 2) let flowType = unpack.slice(2, 3) // the kind of link, tree or blob let branch = unpack.slice(3, 4) let path = unpack.slice(4) // get the default branch via the api let fetchMeta = `${URL_PREFIX}/repos/${[owner, repo].join("/")}` let resMeta = await fetch(fetchMeta, { headers: headers }).then((response) => response.json()) let defaultBranch = resMeta.default_branch // define branch parameter let branchRef = branch.slice(0, 1).length === 0 | branch.slice(0, 1)[0] === defaultBranch ? "" : "?ref=" + branch.slice(0, 1)[0] // define the owner/repo/path let linkArray = [owner, repo, "contents", path, branchRef] let linkCollapsed = linkArray.flat() // finish creating url link urlLink = linkCollapsed.slice(0, -1).join("/") + linkCollapsed.slice(-1) if (!urlLink.startsWith("/") & urlLink !== "") { urlLink = '/' + urlLink } } // parse the endpoint const rawEndpoint = String(endpoint) let urlEndpoint = rawEndpoint if (!urlEndpoint.startsWith("/")) { urlEndpoint = '/' + urlEndpoint } // make full link to ping api let fetchQuery = `${URL_PREFIX}${urlEndpoint}${urlLink}` if (debug) { return fetchQuery } // get response, use username/password if provided const response = await fetch(fetchQuery, { headers: headers }) // return response return response }34:[] 2f:{"title":"Github API in Observable","cells":[{"id":0,"value":"# Github API in Observable\n#### _Getting Data from Private & Public GitHub Repos into Observable_\n\nThis notebook provides a helper function for getting data from private or public ${textcolor(\"GitHub repos\", \"#228636\")} into Observable via the GitHub API, streamlining the data ingestion process so you can focus on working with your data.","pinned":false,"hidden":false,"mode":"md"},{"id":265,"value":"```js\nimport {gh} from \"${currentSpecifier}\"\n```","pinned":false,"hidden":false,"mode":"md"},{"id":512,"value":"toc({headers: \"h2,h3\"})","pinned":false,"hidden":false,"mode":"ojs"},{"id":859,"value":"## Quickstart\nReturns a response, which you can unpack into desired data formats.","pinned":false,"hidden":false,"mode":"md"},{"id":858,"value":"response = gh(\"https://github.com/periscopic/github-api-demo/blob/main/penguins.json\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":960,"value":"dataJson = (await response).json()","pinned":true,"hidden":false,"mode":"ojs"},{"id":1041,"value":"Below we get penguins data from the repo and parse as JSON all in one cell.","pinned":false,"hidden":false,"mode":"md"},{"id":1035,"value":"repoPenguins = gh(\"https://github.com/periscopic/github-api-demo/blob/main/penguins.json\")\n .then(response => response.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":856,"value":"And voilà, your repo data is in Observable!","pinned":false,"hidden":false,"mode":"md"},{"id":965,"value":"Plot.plot({\n grid: true,\n nice: true,\n x: {\n label: \"Flipper Length (mm) →\"\n },\n y: {\n tickFormat: \"d\",\n label: \"↑ Body Mass (g)\"\n },\n color: {\n legend: true\n },\n marks: [\n Plot.dot(repoPenguins, {\n x: \"flipper_length_mm\",\n y: \"body_mass_g\",\n fill: \"species\",\n title: \"species\"\n })\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":233,"value":"## Usage\n```js\nimport {gh} from \"${currentSpecifier}\"\n```\n- **Copy GitHub Links, Get Data**: Use links when navigating [a GitHub repo in the browser](https://github.com/periscopic/github-api-demo) as access points to your data.\n- **Data Updates Work Repo-Side**: if the requested data file is updated in the repo, changes will propagate to your notebook without having to update attachments. Can also lock to a commit, see Examples below.","pinned":false,"hidden":false,"mode":"md"},{"id":1006,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":853,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":632,"value":"---\n## Examples\nBelow are some examples of getting data from [a public demo repo on GitHub](https://github.com/periscopic/github-api-demo).","pinned":false,"hidden":false,"mode":"md"},{"id":250,"value":"### Repo File List\n_[→ Link in Repo](https://github.com/periscopic/github-api-demo/tree/main)_\n\nGet all of the files in the repo at selected level and branch. Useful for iterating through data files.","pinned":false,"hidden":false,"mode":"md"},{"id":2241,"value":"files = gh(\"https://github.com/periscopic/github-api-demo/tree/main\")\n .then(response => response.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":275,"value":"### Data: JSON\n_[→ Link in Repo](https://github.com/periscopic/github-api-demo/blob/main/penguins.json)_\n\nAccess a data file and parse as JSON.","pinned":false,"hidden":false,"mode":"md"},{"id":2239,"value":"json = gh(\"https://github.com/periscopic/github-api-demo/blob/main/aapl.json\")\n .then(response => response.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":286,"value":"### Data: CSV\n_[→ Link in Repo](https://github.com/periscopic/github-api-demo/blob/main/aapl.csv)_\n\nUse [`d3.csvParse`](https://github.com/d3/d3-dsv) to unpack the response as text. Can also use [`d3.autoType`](https://observablehq.com/@d3/d3-autotype) to format column values.","pinned":false,"hidden":false,"mode":"md"},{"id":1074,"value":"csv = gh(\"https://github.com/periscopic/github-api-demo/blob/main/aapl.csv\")\n .then(res => res.text())\n .then(text => d3.csvParse(text, d3.autoType))","pinned":true,"hidden":false,"mode":"ojs"},{"id":533,"value":"### Data: SQLite\n_[→ Link in Repo](https://github.com/periscopic/github-api-demo/blob/main/chinook.db)_\n\nAccess as a `SQLiteDatabaseClient`. See [Observable/SQLite](https://observablehq.com/@observablehq/sqlite) for more info.","pinned":false,"hidden":false,"mode":"md"},{"id":2235,"value":"db = gh(\"https://github.com/periscopic/github-api-demo/blob/main/chinook.db\")\n .then(SQLiteDatabaseClient.open)","pinned":true,"hidden":false,"mode":"ojs"},{"id":2234,"value":"db.describe(\"tracks\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":716,"value":"### Data: Arrow Table, Arquero, & DuckDB\n_[→ Link in Repo](https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/flights-200k.arrow)_\n\nParse the `arrayBuffer`. Then can make an arrow table:","pinned":false,"hidden":false,"mode":"md"},{"id":1072,"value":"arrowTable = gh(\"https://github.com/periscopic/github-api-demo/blob/main/flights-200k.arrow\")\n .then(res => res.arrayBuffer())\n .then(buffer => Arrow.Table.from(new Uint8Array(buffer)))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2308,"value":"You can go further and parse the arrow table as an [Arquero](https://observablehq.com/@uwdata/introducing-arquero) table:","pinned":false,"hidden":false,"mode":"md"},{"id":1071,"value":"aqArrow = gh(\"https://github.com/periscopic/github-api-demo/blob/main/flights-200k.arrow\")\n .then(res => res.arrayBuffer())\n .then(buffer => Arrow.Table.from(new Uint8Array(buffer)))\n .then(arrowTable => aq.fromArrow(arrowTable))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1070,"value":"You could also store the Arquero objects using [DuckDB](https://observablehq.com/@observablehq/duckdb).","pinned":false,"hidden":false,"mode":"md"},{"id":2306,"value":"duckDb = gh(\"https://github.com/periscopic/github-api-demo/blob/main/flights-200k.arrow\")\n .then(res => res.arrayBuffer())\n .then(buffer => Arrow.Table.from(new Uint8Array(buffer)))\n .then(x => DuckDBClient.of({demo: aq.fromArrow(x).objects()}))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1068,"value":"duckData = duckDb","pinned":false,"hidden":false,"mode":"ojs"},{"id":2297,"value":"### Data: Excel\n_[→ Link in Repo](https://github.com/periscopic/github-api-demo/blob/main/Data%20Analyst%20Excel%20Meetup.xlsx)_\n\nParse the ArrayBuffer using [sheetjs](https://docs.sheetjs.com/), then can extract the desired sheet from the workbook object.","pinned":false,"hidden":false,"mode":"md"},{"id":2296,"value":"xlsx = require(\"xlsx@0.17.0/dist/xlsx.core.min.js\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":2295,"value":"workbook = gh(\"https://github.com/periscopic/github-api-demo/blob/main/Data%20Analyst%20Excel%20Meetup.xlsx\")\n .then(res => res.arrayBuffer())\n .then(buffer => xlsx.read(new Uint8Array(buffer), { type: \"array\" }))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2294,"value":"// example function to parse sheet from workbook object\n// adapted from Fetch an Excel sheet by @neocarto\n// - https://observablehq.com/@neocartocnrs/fetch-an-excel-sheet\nfunction parseSheet(workbook, sheetname) {\n let sheet = workbook.Sheets[sheetname];\n // get the data\n let range = xlsx.utils.decode_range(sheet[\"!ref\"]);\n range.s.r = 0; // skip rows\n range.s.c = 0; // skip cols\n sheet[\"!ref\"] = xlsx.utils.encode_range(range);\n return xlsx.utils.sheet_to_json(sheet);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":2293,"value":"excel = parseSheet(workbook, \"StreetEasy Data\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":1067,"value":"### Images\nParse as a [blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob), then can create object url as image source.","pinned":false,"hidden":false,"mode":"md"},{"id":1077,"value":"image = gh(\"https://github.com/periscopic/github-api-demo/blob/main/li-zhang-HYwU-Fl6IoM-unsplash.jpeg\")\n .then(res => res.blob())","pinned":true,"hidden":false,"mode":"ojs"},{"id":1076,"value":"

","pinned":true,"hidden":false,"mode":"html"},{"id":345,"value":"### Files by a specific commit\n_[→ Link in Repo](https://github.com/periscopic/github-api-demo/blob/9c954753a3548a1e2830eacde445cb95e099e404/secretPenguins.json)_\n\nWe can extract data from a specific commit. Notice if we simply try to get the `secretPenguins.json` file, we see it's not there in the repo (it's been removed in a subsequent commit):","pinned":false,"hidden":false,"mode":"md"},{"id":358,"value":"gh(\"https://github.com/periscopic/github-api-demo/blob/main/secretPenguins.json\")\n .then(res => res.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":366,"value":"But, we can access a [specific commit](https://github.com/periscopic/github-api-demo/blob/9c954753a3548a1e2830eacde445cb95e099e404/secretPenguins.json) to get at the data that is stored in the git tree:","pinned":false,"hidden":false,"mode":"md"},{"id":354,"value":"gh(\"https://github.com/periscopic/github-api-demo/blob/9c954753a3548a1e2830eacde445cb95e099e404/secretPenguins.json\")\n .then(res => res.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":837,"value":"## Advanced","pinned":false,"hidden":false,"mode":"md"},{"id":836,"value":"### Parameters\nHere's a look at the parameter defaults. For more info, checkout the [function definition](https://observablehq.com/d/6a64c014db4045f7#gh).","pinned":false,"hidden":false,"mode":"md"},{"id":835,"value":"```js\ngh(\"link\" // GitHub url link, \n{\n endpoint: \"repositories\", // endpoint\n username: Secret(\"GH_API_PAT\") | null, // GitHub Personal access token\n debug: false // return as [\"url\", Response()]\n})\n```","pinned":false,"hidden":false,"mode":"md"},{"id":452,"value":"### Custom Endpoints\nIf you want to access a custom endpoint, you can specify the `endpoint` field and simply provide the `link` as an empty string.\n\n**`gh(``link``, {endpoint:``endpoint``})` → `https://api.github.com/``endpoint``/``link`**\n\nFor example, you could get back the commits for a repo: ","pinned":false,"hidden":false,"mode":"md"},{"id":1084,"value":"recentCommits = gh(\"\", {endpoint: \"repos/periscopic/github-api-demo/commits\"})\n .then(res => res.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":1086,"value":"Inputs.table(recentCommits.map(d => ({\n date: d.commit.author.date,\n message: d.commit.message,\n hash: d.sha\n})), {\n columns: [\"date\", \"message\", \"hash\"]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":2217,"value":"Checkout the [GitHub REST API Documentation](https://docs.github.com/en/rest/) for more information on possible endpoints.","pinned":false,"hidden":false,"mode":"md"},{"id":833,"value":"### Note on Refreshing Cells\nIn general, it is recommended to do both fetching and parsing all in one cell.","pinned":false,"hidden":false,"mode":"md"},{"id":1049,"value":"allInOne = {\n let res = gh(\"https://github.com/periscopic/github-api-demo\")\n return (await res).json()\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1065,"value":"If you choose to store the response source and wrangling of the response as separate cells, you will likely update how you wrangle the `Response()`. This will throw an error about the data already being read. Simply re-run the source cell to refresh things.","pinned":false,"hidden":false,"mode":"md"},{"id":832,"value":"viewof refreshButton1 = Inputs.button(\"Update Wrangling Cell (throws error)\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":831,"value":"viewof refreshButton2 = Inputs.button(\"Re-Run Source Cell (Fixed)\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":830,"value":"resRefresh = { // source cell\n refreshButton2\n return gh(\"https://github.com/periscopic/github-api-demo\")\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":829,"value":"{ // wrangling cell\n refreshButton1\n return resRefresh.json()\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":828,"value":"### Debugging\nSetting `debug` to `true` will return the API url, which can be helpful in confirming the query is being generated correctly.","pinned":false,"hidden":false,"mode":"md"},{"id":827,"value":"gh(\"https://github.com/periscopic/github-api-demo\", {debug: true})","pinned":true,"hidden":false,"mode":"ojs"},{"id":76,"value":"---\n## Function","pinned":false,"hidden":false,"mode":"md"},{"id":1994,"value":"$33","pinned":true,"hidden":false,"mode":"ojs"},{"id":823,"value":"### Observable Secret Handling","pinned":false,"hidden":false,"mode":"md"},{"id":822,"value":"// pack up Observable Secrets, or return empty\ncredentials = {\n try {\n let ghToken = Secret(\"GH_API_PAT\")\n return [{ghToken}]\n } catch (error) {\n return [{ghToken: null}]\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":225,"value":"---\n## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":820,"value":"hasPassword = {\n try {\n Secret(\"GH_API_PAT\");\n return true;\n } catch {\n return false;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1029,"value":"// from https://observablehq.com/@d3/example-components\ncurrentSpecifier = {\n const {pathname} = new URL(document.baseURI);\n return pathname.slice(pathname.startsWith(\"/d/\") ? 3 : 1);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":226,"value":"#### Imports","pinned":false,"hidden":false,"mode":"md"},{"id":222,"value":"// encoding the header values\nbase64 = require('base-64@1.0.0/base64.js') ","pinned":false,"hidden":false,"mode":"ojs"},{"id":506,"value":"#### Notebook Imports","pinned":false,"hidden":false,"mode":"md"},{"id":507,"value":"import {toc} from \"@nebrius/indented-toc\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":2285,"value":"import {textcolor} from \"@observablehq/text-color-annotations-in-markdown\"\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":617,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":618,"value":"Photo by Li Zhang on Unsplash\n ","pinned":false,"hidden":false,"mode":"html"}],"files":"$W34","stdlib":"1","titling":"auto"}