31:T41c,## About At [Periscopic](https://periscopic.com), we often use Bitbucket to take in raw data, process it, and output data files for different purposes. We wanted an easier way to get data we have stored on our repos into Observable for data visualization, exploration, and more. By using an API, we can make updates with less hassle. Instead of modifying file attachments as data files change, we can commit changes to update a data file repo-side, which the API can pull in. The `bb()` helper function takes links from navigating a Bitbucket repo in the browser and turns them into calls to the Bitbucket API. This makes it easy to go from repo to data source. We hope this makes working with your data that much easier. There are helpers for [Bitbucket repos](https://observablehq.com/@periscopic/bitbucket-api-in-observable) or [GitHub repos](https://observablehq.com/@periscopic/github-api-in-observable), depending on your preference. Got feedback? Feel free to comment and help us make this better. Made with love by the Periscopic Data Team32:T78b,## Authentication: Private Repos If you are trying to access a private Bitbucket repository, you will need to provide your Bitbucket username and an [App Password](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/). For convenience, this notebook defaults to filling the `username` and `password` fields with [Observable Secrets](https://observablehq.com/settings/secrets) called `BB_API_USERNAME` and `BB_API_PASSWORD`. Below you can check the secrets are accessible on your account. Secrets are not accessible in public notebooks, but you can fork this notebook as a template to create your own version for private repo access. ```js import {bb} from "${currentSpecifier}" ``` ${hasUsername & hasPassword ? "🚀 Ready to Access Private Repos!" : "⚠️ Provide Bitbucket Credentials as Observable Secrets to Access Private Repos"} - `BB_API_USERNAME` ${hasUsername ? "✅ Secret found" : "⚠️ Secret not found"} - `BB_API_PASSWORD` ${hasPassword ? "✅ Secret found" : "⚠️ Secret not found"} Regarding [permissions](https://confluence.atlassian.com/bitbucketserver/using-repository-permissions-776639771.html), your App Password should at least have read-level access to your repositories. For more info, checkout the [Bitbucket Support Page on App Passwords](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/) and the [Introduction to Secrets](https://observablehq.com/@observablehq/secrets).33:T604,async function bb( link // chosen link in bitbucket repo , { endpoint = "repositories", // endpoint string username = credentials[0].username, // Bitbucket username password = credentials[0].password, // Bitbucket App Password, see https://bitbucket.org/account/settings/app-passwords/ debug = false // if true, return API url } = {}) { // main url endpoint for API, see https://developer.atlassian.com/cloud/bitbucket/rest/intro/ const URL_PREFIX = 'https://api.bitbucket.org/2.0' // parse data link: url → path if (link == undefined) { return "Include Bitbucket Link (Docs: https://observablehq.com/@periscopic/bitbucket-api-in-observable)"} const rawLink = String(link) let urlLink = rawLink if (urlLink.startsWith("https://bitbucket.org/")) { urlLink = urlLink.split('/').slice(3).join("/") } if (!urlLink.startsWith("/") & urlLink !== "") { urlLink = '/' + urlLink } // parse the endpoint const rawEndpoint = String(endpoint) let urlEndpoint = rawEndpoint if (!urlEndpoint.startsWith("/")) { urlEndpoint = '/' + urlEndpoint } // make full link to ping api const fetchQuery = `${URL_PREFIX}${urlEndpoint}${urlLink}` if (debug) { return fetchQuery } // create header for authentication const headers = (username !== null & password !== null) ? {Authorization: 'Basic ' + base64.encode(username + ':' + password)} : {} // get response, use username/password if provided const response = await fetch(fetchQuery, { headers: headers }) return response }34:[] 2f:{"title":"Bitbucket API in Observable","cells":[{"id":0,"value":"# Bitbucket API in Observable\n#### _Getting Data from Private & Public Bitbucket Repos into Observable_\n\nThis notebook provides a helper function for getting data from private or public ${textcolor(\"Bitbucket repos\", \"#0053CC\")} into Observable via the Bitbucket API, streamlining the data ingestion process so you can focus on working with your data.","pinned":false,"hidden":false,"mode":"md"},{"id":265,"value":"```js\nimport {bb} from \"${currentSpecifier}\"\n```","pinned":false,"hidden":false,"mode":"md"},{"id":512,"value":"toc({headers: \"h2,h3\", skip: [\"Getting Data from Private Bitbucket Repos\"]})","pinned":false,"hidden":false,"mode":"ojs"},{"id":859,"value":"## Quickstart\nReturns a response, which you can unpack into desired data formats.","pinned":false,"hidden":false,"mode":"md"},{"id":858,"value":"response = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/penguins.json\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":960,"value":"dataJson = (await response).json()","pinned":true,"hidden":false,"mode":"ojs"},{"id":1041,"value":"Below we get penguins data from the repo and parse as JSON all in one cell.","pinned":false,"hidden":false,"mode":"md"},{"id":1035,"value":"repoPenguins = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/penguins.json\")\n .then(response => response.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":856,"value":"And voilà, your repo data is in Observable!","pinned":false,"hidden":false,"mode":"md"},{"id":965,"value":"Plot.plot({\n grid: true,\n nice: true,\n x: {\n label: \"Flipper Length (mm) →\"\n },\n y: {\n tickFormat: \"d\",\n label: \"↑ Body Mass (g)\"\n },\n color: {\n legend: true\n },\n marks: [\n Plot.dot(repoPenguins, {\n x: \"flipper_length_mm\",\n y: \"body_mass_g\",\n fill: \"species\",\n title: \"species\"\n })\n ]\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":233,"value":"## Usage\n```js\nimport {bb} from \"${currentSpecifier}\"\n```\n- **Copy Bitbucket Links, Get Data**: Use links when navigating [a Bitbucket repo in the browser](https://bitbucket.org/periscopic/bitbucket-api-demo) as access points to your data.\n- **Data Updates Work Repo-Side**: if the requested data file is updated in the repo, changes will propagate to your notebook without having to update attachments. Can also lock to a commit, see Examples below.","pinned":false,"hidden":false,"mode":"md"},{"id":1006,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":853,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":632,"value":"---\n## Examples\nBelow are some examples of getting data from [a public demo repo on Bitbucket](https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/).","pinned":false,"hidden":false,"mode":"md"},{"id":250,"value":"### Repo File List\n_[→ Link in Repo](https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/aapl.csv)_\n\nGet all of the files in the repo at selected level and branch. Useful for iterating through data files.","pinned":false,"hidden":false,"mode":"md"},{"id":253,"value":"files = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/\")\n .then(response => response.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":275,"value":"### Data: JSON\n_[→ Link in Repo](https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/aapl.csv)_\n\nAccess a data file and parse as JSON.","pinned":false,"hidden":false,"mode":"md"},{"id":325,"value":"json = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/aapl.json\")\n .then(response => response.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":286,"value":"### Data: CSV\n_[→ Link in Repo](https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/aapl.csv)_\n\nUse [`d3.csvParse`](https://github.com/d3/d3-dsv) to unpack the response as text. Can also use [`d3.autoType`](https://observablehq.com/@d3/d3-autotype) to format column values.","pinned":false,"hidden":false,"mode":"md"},{"id":1074,"value":"csv = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/aapl.csv\")\n .then(res => res.text())\n .then(text => d3.csvParse(text, d3.autoType))","pinned":true,"hidden":false,"mode":"ojs"},{"id":533,"value":"### Data: SQLite\n_[→ Link in Repo](https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/chinook.db)_\n\nAccess as a `SQLiteDatabaseClient`. See [Observable/SQLite](https://observablehq.com/@observablehq/sqlite) for more info.","pinned":false,"hidden":false,"mode":"md"},{"id":536,"value":"db = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/chinook.db\")\n .then(SQLiteDatabaseClient.open)","pinned":true,"hidden":false,"mode":"ojs"},{"id":541,"value":"db.describe(\"tracks\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":716,"value":"### Data: Arrow Table, Arquero, & DuckDB\n_[→ Link in Repo](https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/flights-200k.arrow)_\n\nParse the `arrayBuffer`. Then can make an arrow table:","pinned":false,"hidden":false,"mode":"md"},{"id":1119,"value":"arrowTable = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/flights-200k.arrow\")\n .then(res => res.arrayBuffer())\n .then(buffer => Arrow.Table.from(new Uint8Array(buffer)))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1122,"value":"You can go further and parse the arrow table as an [Arquero](https://observablehq.com/@uwdata/introducing-arquero) table:","pinned":false,"hidden":false,"mode":"md"},{"id":1071,"value":"aqArrow = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/flights-200k.arrow\")\n .then(res => res.arrayBuffer())\n .then(buffer => Arrow.Table.from(new Uint8Array(buffer)))\n .then(arrowTable => aq.fromArrow(arrowTable))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1070,"value":"You could also store the Arquero objects using [DuckDB](https://observablehq.com/@observablehq/duckdb).","pinned":false,"hidden":false,"mode":"md"},{"id":1069,"value":"duckDb = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/flights-200k.arrow\")\n .then(res => res.arrayBuffer())\n .then(buffer => Arrow.Table.from(new Uint8Array(buffer)))\n .then(arrowTable => DuckDBClient.of({demo: aq.fromArrow(arrowTable).objects()}))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1068,"value":"duckData = duckDb","pinned":false,"hidden":false,"mode":"ojs"},{"id":1100,"value":"### Data: Excel\n_[→ Link in Repo](https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/Data%20Analyst%20Excel%20Meetup.xlsx)_\n\nParse the ArrayBuffer using [sheetjs](https://docs.sheetjs.com/), then can extract the desired sheet from the workbook object.","pinned":false,"hidden":false,"mode":"md"},{"id":1099,"value":"xlsx = require(\"xlsx@0.17.0/dist/xlsx.core.min.js\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":1098,"value":"workbook = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/Data%20Analyst%20Excel%20Meetup.xlsx\")\n .then(res => res.arrayBuffer())\n .then(buffer => xlsx.read(new Uint8Array(buffer), { type: \"array\" }))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1097,"value":"// example function to parse sheet from workbook object\n// adapted from Fetch an Excel sheet by @neocarto\n// - https://observablehq.com/@neocartocnrs/fetch-an-excel-sheet\nfunction parseSheet(workbook, sheetname) {\n let sheet = workbook.Sheets[sheetname];\n // get the data\n let range = xlsx.utils.decode_range(sheet[\"!ref\"]);\n range.s.r = 0; // skip rows\n range.s.c = 0; // skip cols\n sheet[\"!ref\"] = xlsx.utils.encode_range(range);\n return xlsx.utils.sheet_to_json(sheet);\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1096,"value":"excel = parseSheet(workbook, \"StreetEasy Data\")","pinned":true,"hidden":false,"mode":"ojs"},{"id":1067,"value":"### Images\nParse as a [blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob), then can create object url as image source.","pinned":false,"hidden":false,"mode":"md"},{"id":1077,"value":"image = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/simon-lee-KDFUR-sG2Ls-unsplash.jpeg\")\n .then(res => res.blob())","pinned":true,"hidden":false,"mode":"ojs"},{"id":1076,"value":"

","pinned":true,"hidden":false,"mode":"html"},{"id":345,"value":"### Files by a specific commit\n_[→ Link in Repo](https://bitbucket.org/periscopic/bitbucket-api-demo/src/8218285f144755a96b5e08c4adc965427308f885/secretPenguins.json)_\n\nWe can extract data from a specific commit. Notice if we simply try to get the `secretPenguins.json` file, we see it's not there in the repo (it's been removed in a subsequent commit):","pinned":false,"hidden":false,"mode":"md"},{"id":358,"value":"bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/secretPenguins.json\")\n .then(res => res.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":366,"value":"But, we can access a [specific commit](https://bitbucket.org/periscopic/bitbucket-api-demo/src/8218285f144755a96b5e08c4adc965427308f885/secretPenguins.json) to get at the data that is stored in the git tree:","pinned":false,"hidden":false,"mode":"md"},{"id":354,"value":"bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/8218285f144755a96b5e08c4adc965427308f885/secretPenguins.json\")\n .then(res => res.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":1083,"value":"Here are all the recent commits to the demo repository:","pinned":false,"hidden":false,"mode":"md"},{"id":1084,"value":"recentCommits = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/commits/\")\n .then(res => res.json())","pinned":true,"hidden":false,"mode":"ojs"},{"id":1086,"value":"Inputs.table(recentCommits.values, {\n columns: [\"date\", \"message\", \"hash\"]\n})","pinned":true,"hidden":false,"mode":"ojs"},{"id":837,"value":"## Advanced","pinned":false,"hidden":false,"mode":"md"},{"id":836,"value":"### Parameters\nHere's a look at the parameter defaults. For more info, checkout the [function definition](https://observablehq.com/@periscopic/bitbucket-api#bb).","pinned":false,"hidden":false,"mode":"md"},{"id":835,"value":"```js\nbb(\"link\" // Bitbucket url link, \n{\n endpoint: \"repositories\", // endpoint\n username: Secret(\"BB_API_USERNAME\") | null, // Bitbucket username\n password: Secret(\"BB_API_PASSWORD\") | null, // Bitbucket password\n debug: false // return as [\"url\", Response()]\n})\n```","pinned":false,"hidden":false,"mode":"md"},{"id":452,"value":"### Custom Endpoints\nIf you want to access a custom endpoint, you can specify the `endpoint` field and simply provide the `link` as an empty string.\n\n**`bb(``link``, {endpoint:``endpoint``})` → `https://api.bitbucket.org/2.0/``endpoint``/``link``**\n\nFor example, you could use the [user endpoint](https://developer.atlassian.com/cloud/bitbucket/rest/api-group-users/#api-user-get) to get back JSON on the current user: \n\n```js\n(await bb(\"\", {endpoint: \"user\"})).json() // requires \"Account\" permission\n```\n\nCheckout the [Bitbucket REST API Documentation](https://developer.atlassian.com/cloud/bitbucket/rest/intro/) for more information on possible endpoints.","pinned":false,"hidden":false,"mode":"md"},{"id":833,"value":"### Note on Refreshing Cells\nIn general, it is recommended to do both fetching and parsing all in one cell.","pinned":false,"hidden":false,"mode":"md"},{"id":1049,"value":"allInOne = {\n let res = bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/\")\n return (await res).json()\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1065,"value":"If you choose to store the response source and wrangling of the response as separate cells, you will likely update how you wrangle the `Response()`. This will throw an error about the data already being read. Simply re-run the source cell to refresh things.","pinned":false,"hidden":false,"mode":"md"},{"id":832,"value":"viewof refreshButton1 = Inputs.button(\"Update Wrangling Cell (throws error)\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":831,"value":"viewof refreshButton2 = Inputs.button(\"Re-Run Source Cell (Fixed)\")","pinned":false,"hidden":false,"mode":"ojs"},{"id":830,"value":"resRefresh = { // source cell\n refreshButton2\n return bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/\")\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":829,"value":"{ // wrangling cell\n refreshButton1\n return resRefresh.json()\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":828,"value":"### Debugging\nSetting `debug` to `true` will return the API url, which can be helpful in confirming the query is being generated correctly.","pinned":false,"hidden":false,"mode":"md"},{"id":827,"value":"bb(\"https://bitbucket.org/periscopic/bitbucket-api-demo/src/master/\", {debug: true})","pinned":true,"hidden":false,"mode":"ojs"},{"id":76,"value":"---\n## Function","pinned":false,"hidden":false,"mode":"md"},{"id":824,"value":"$33","pinned":true,"hidden":false,"mode":"ojs"},{"id":823,"value":"### Observable Secret Handling","pinned":false,"hidden":false,"mode":"md"},{"id":822,"value":"// pack up Observable Secrets, or return empty\ncredentials = {\n try {\n let username = Secret(\"BB_API_USERNAME\")\n let password = Secret(\"BB_API_PASSWORD\")\n return [{username, password}]\n } catch (error) {\n return [{username: null, password: null}]\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":225,"value":"---\n## Appendix","pinned":false,"hidden":false,"mode":"md"},{"id":821,"value":"hasUsername = {\n try {\n Secret(\"BB_API_USERNAME\");\n return true;\n } catch {\n return false;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":820,"value":"hasPassword = {\n try {\n Secret(\"BB_API_PASSWORD\");\n return true;\n } catch {\n return false;\n }\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":1029,"value":"// from https://observablehq.com/@d3/example-components\ncurrentSpecifier = {\n const {pathname} = new URL(document.baseURI);\n return pathname.slice(pathname.startsWith(\"/d/\") ? 3 : 1);\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":226,"value":"#### Imports","pinned":false,"hidden":false,"mode":"md"},{"id":222,"value":"// encoding the header values\nbase64 = require('base-64@1.0.0/base64.js') ","pinned":false,"hidden":false,"mode":"ojs"},{"id":506,"value":"#### Notebook Imports","pinned":false,"hidden":false,"mode":"md"},{"id":507,"value":"import {toc} from \"@nebrius/indented-toc\"","pinned":false,"hidden":false,"mode":"ojs"},{"id":1093,"value":"import {textcolor} from \"@observablehq/text-color-annotations-in-markdown\"\n","pinned":false,"hidden":false,"mode":"ojs"},{"id":617,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":618,"value":"Photo by SIMON LEE on Unsplash","pinned":false,"hidden":false,"mode":"html"}],"files":"$W34","stdlib":"1","titling":"auto"}