31:[{"id":"5a6c462b7e9dc002e6b4b310ca311f73f54f1c958369e6b482ecfd5aabf2604031d60851b3a304a13ffcd8eda35fa3cbd289c648cf59ecad7f47e89ef05ffeab","href":"https://static.observableusercontent.com/files/5a6c462b7e9dc002e6b4b310ca311f73f54f1c958369e6b482ecfd5aabf2604031d60851b3a304a13ffcd8eda35fa3cbd289c648cf59ecad7f47e89ef05ffeab","name":"p1-vanilla-get.png","type":"image/png","lastModified":1644006899585.217,"size":77993},{"id":"e6aa0c833b762387e969ac2d6d468d416d7e324e5b9d2222e2b8202fc67635bad724fb26fc6f331bf523068f1a08c4871ac85b5a0ad00b8417f28f89a7c27907","href":"https://static.observableusercontent.com/files/e6aa0c833b762387e969ac2d6d468d416d7e324e5b9d2222e2b8202fc67635bad724fb26fc6f331bf523068f1a08c4871ac85b5a0ad00b8417f28f89a7c27907","name":"p1-status.png","type":"image/png","lastModified":1644006899586.273,"size":31438},{"id":"fec8b22b81a93bcb1996a9cb2779a9831188407c7fc2e4d3f30a1899676049afc1aa49a8f193a6697acca81a5ebfd25e0b787fbff24a9b9253ee02934cb7d949","href":"https://static.observableusercontent.com/files/fec8b22b81a93bcb1996a9cb2779a9831188407c7fc2e4d3f30a1899676049afc1aa49a8f193a6697acca81a5ebfd25e0b787fbff24a9b9253ee02934cb7d949","name":"p1-json-locations.png","type":"image/png","lastModified":1644006901155.708,"size":196117},{"id":"13539edd84071260b6dfe505a023558589a170323e28b96bf659bdf6e189a7386838d8258378bc1d80d8e1329a0b0d7c9ab48ba8e6ece209dcefa75354d200ce","href":"https://static.observableusercontent.com/files/13539edd84071260b6dfe505a023558589a170323e28b96bf659bdf6e189a7386838d8258378bc1d80d8e1329a0b0d7c9ab48ba8e6ece209dcefa75354d200ce","name":"p1-batches-geo.png","type":"image/png","lastModified":1644006900865.253,"size":157858},{"id":"f16ef7d4de37fcb4f2770ec786bca807bc6701786554654f1622ec14fbe55dc325de0352f8309f118dc262b242b031c35b285f8e1d1a6ccf9784e024793e7ef1","href":"https://static.observableusercontent.com/files/f16ef7d4de37fcb4f2770ec786bca807bc6701786554654f1622ec14fbe55dc325de0352f8309f118dc262b242b031c35b285f8e1d1a6ccf9784e024793e7ef1","name":"p1-json-weather.png","type":"image/png","lastModified":1644006900963.783,"size":186533},{"id":"e02e9b581dadb7a3e00ebf70a1aad5a1afe77135d9a6395511d657841a31edab782e3a44e902b7e9d818add4494672133d3179e871d6369914b1886e6f73d9e5","href":"https://static.observableusercontent.com/files/e02e9b581dadb7a3e00ebf70a1aad5a1afe77135d9a6395511d657841a31edab782e3a44e902b7e9d818add4494672133d3179e871d6369914b1886e6f73d9e5","name":"p1-types.png","type":"image/png","lastModified":1644006900729.604,"size":52312},{"id":"f70a48bfe70ed91dfe109a3a25766ca0da2e6bd8c840f3f8fd32e2069498427f4024508c50e4434cc6d82641d981663672262b7fd881f49c45b9387bc542f0e0","href":"https://static.observableusercontent.com/files/f70a48bfe70ed91dfe109a3a25766ca0da2e6bd8c840f3f8fd32e2069498427f4024508c50e4434cc6d82641d981663672262b7fd881f49c45b9387bc542f0e0","name":"jiawei-zhao-BsXeYX3efOI-unsplash.jpeg","type":"image/jpeg","lastModified":1644006923285.464,"size":437256},{"id":"d39af1a91dce5aa7e461a97e5c23e746d57e1babeed02149b7b5fc48ebfc7225bc7e24303b1eaa855e5d6844ef02fc5b42a1eb61596e1c65f3caa8f4cfd41251","href":"https://static.observableusercontent.com/files/d39af1a91dce5aa7e461a97e5c23e746d57e1babeed02149b7b5fc48ebfc7225bc7e24303b1eaa855e5d6844ef02fc5b42a1eb61596e1c65f3caa8f4cfd41251","name":"p1-param-df.png","type":"image/png","lastModified":1644006899802.676,"size":70355},{"id":"2092d4dbad54cb7418f1b9412c4b306c588b8f25cdc19e71cb08b8652b6a005ae82f4d13d0945d1b7d14db083b5837c4979bc8685d76bedd1214f9beb998cab0","href":"https://static.observableusercontent.com/files/2092d4dbad54cb7418f1b9412c4b306c588b8f25cdc19e71cb08b8652b6a005ae82f4d13d0945d1b7d14db083b5837c4979bc8685d76bedd1214f9beb998cab0","name":"p1-get-by-query.png","type":"image/png","lastModified":1644006900868.68,"size":107450},{"id":"36420d441661eca278a2540cbb0206884cda9b407f52f070b22c7fec7c591cb70ac9f508979b9cc0f82150c70224e8d6e6681d8bd3c1b1bcc24ca7872f16246a","href":"https://static.observableusercontent.com/files/36420d441661eca278a2540cbb0206884cda9b407f52f070b22c7fec7c591cb70ac9f508979b9cc0f82150c70224e8d6e6681d8bd3c1b1bcc24ca7872f16246a","name":"p1-locations-list.png","type":"image/png","lastModified":1644006900580.985,"size":84988},{"id":"35e875850fba06e086d14aebfc32cbd4f22809d0ee0e3e2528e4c9a2573185117349fcdace7480ce23b88df16826fc5a42325c3b8e64e380dd2ec310ecd2b860","href":"https://static.observableusercontent.com/files/35e875850fba06e086d14aebfc32cbd4f22809d0ee0e3e2528e4c9a2573185117349fcdace7480ce23b88df16826fc5a42325c3b8e64e380dd2ec310ecd2b860","name":"charles-deluvio-AT5vuPoi8vc-unsplash@1.jpeg","type":"image/jpeg","lastModified":1644272130243.071,"size":196229}] 2f:{"title":"Cozy Collecting, Part 1: Pinging an API & Creating JSON in R","cells":[{"id":0,"value":"# Cozy Collecting, Part 1: Pinging an API & Creating JSON in R","pinned":false,"hidden":false,"mode":"md"},{"id":5,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":8,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":11,"value":"APIs are a data-lover's dream, making a lot of information widely accessible, usually in the form of JSON. This is super useful, but it can sometimes be a pain to figure out how to work with the data in R. Tutorials often use existing packages specific to one API or use examples where the returned object is not heavily nested. In our experience, parsing nested JSON data in R has been challenging and quality online resources or guides are surprisingly hard to find, so we'd like to share one approach we've used to tackle it.","pinned":false,"hidden":false,"mode":"md"},{"id":13,"value":"This series explains **a workflow for getting data from an API in R**: accessing the JSON output and then working with the results, whether they are nested or not. Hopefully this method will help you tackle APIs you come across, regardless of whether they have helper functions.","pinned":false,"hidden":false,"mode":"md"},{"id":17,"value":"The Overall Process:\n\n- Consult the documentation of the API for required parameters\n- Create a tibble of any query parameters or additional parameter fields for desired API calls\n- Use httr to grab responses from API endpoint\n- Inspect and unpack responses to a nested R list\n- Export as JSON or work in R as a nested tibble with rectangling methods","pinned":false,"hidden":false,"mode":"md"},{"id":21,"value":"Here we go with Part 1 of the 3 part series:\n\n1. ➡️ 👋 **Pinging an API, Creating JSON**: Get data from an API and extract as JSON\n2. 🔨 **[Working with JSON](https://observablehq.com/@zachperiscopic/cozy-collecting-part-2)**: Use extracted JSON data in R\n3. 🚀 **[Case Study: Getting a Decade's Worth of Weather Data](https://observablehq.com/@zachperiscopic/cozy-collecting-case-study)**: Example of method with a larger data request","pinned":false,"hidden":false,"mode":"md"},{"id":30,"value":"---\n# 👋 Part 1: Pinging an API, Creating JSON","pinned":false,"hidden":false,"mode":"md"},{"id":35,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":38,"value":"Let's go get some JSON data from an API in R! We're going to get the weather at different zip codes from the [OpenWeather API](https://openweathermap.org/api). To start, we need to figure out how to create the calls. Think of them like digital handshakes. But the OpenWeather API wants to know who it's shaking hands with, so we need to provide it an API key. We can [sign up for a key on OpenWeather's site](https://home.openweathermap.org/users/sign_up) and [store our key in R](https://cran.r-project.org/web/packages/rromeo/vignettes/setting_up_api_key.html) using `Sys.setenv():`","pinned":false,"hidden":false,"mode":"md"},{"id":42,"value":"```\nSys.setenv(\"OW_API_KEY\" = \"\")\n```","pinned":false,"hidden":false,"mode":"md"},{"id":44,"value":"OpenWeather, like any API, has its own special handshake, but luckily for us, it's not overly secretive about how to do it. There's [plenty of documentation](https://openweathermap.org/api) explaining how to set it up. In this case, we'll [get the lat/long coordinates for our zip codes](https://openweathermap.org/api/geocoding-api#direct_zip) and then use those to [get the current weather data](https://openweathermap.org/current).","pinned":false,"hidden":false,"mode":"md"},{"id":50,"value":"*Note: To clarify where code is coming from, we will often include the longer format of package function names (`httr::GET()` versus `GET()`).*","pinned":false,"hidden":false,"mode":"md"},{"id":54,"value":"```\n# imports\nlibrary(httr) # accessing APIs\nlibrary(glue) # pass variables into strings\nlibrary(tibble) # nicer dataframes\nlibrary(purrr) # apply functions to lists\nlibrary(dplyr) # data wrangling\nlibrary(tidyr) # creating tidy data\nlibrary(jsonlite) # JSON parsing\n```","pinned":false,"hidden":false,"mode":"md"},{"id":58,"value":"## ✏️ One at a time","pinned":false,"hidden":false,"mode":"md"},{"id":62,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":66,"value":"Our first thought might be to construct the URL by-hand, inserting all of the [query parameters](https://en.wikipedia.org/wiki/Query_string) we need with the glue package:","pinned":false,"hidden":false,"mode":"md"},{"id":70,"value":"```\n# create the URL by hand with glue\nzip_code <- '90210'\ncountry_code <- 'US' # Alpha-2 ISO 3166 country code\n\n# reference api key using \"Sys.getenv('OW_API_KEY')\"\napi_url <- glue::glue(\"http://api.openweathermap.org/geo/1.0/zip?zip={zip_code},{country_code}&appid={Sys.getenv('OW_API_KEY')}\")\n```","pinned":false,"hidden":false,"mode":"md"},{"id":74,"value":"We can then use httr to get a response from the API:","pinned":false,"hidden":false,"mode":"md"},{"id":77,"value":"```\n# send formatted URL to GET\nhttr::GET(api_url)\n```","pinned":false,"hidden":false,"mode":"md"},{"id":82,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":86,"value":"Using glue works pretty well, but it is a bit of a hassle to define the entire URL, especially if the parameter options change. To clean things up, we can define the API as just the endpoint, then pass the query parameters as a named list, where the names are the API fields for each query parameter:","pinned":false,"hidden":false,"mode":"md"},{"id":90,"value":"```\n# define the endpoint URL\napi_endpoint_geo <- \"http://api.openweathermap.org/geo/1.0/zip\"\n\n# make list of query parameters\napi_query_list <- list(\n zip = paste(zip_code, country_code, sep = \",\"),\n appid = Sys.getenv('OW_API_KEY')\n)\n\n# send as endpoint and query parameters\nhttr::GET(api_endpoint_geo, query = api_query_list)\n```","pinned":false,"hidden":false,"mode":"md"},{"id":94,"value":"

Get response using base URL and query parameter list

","pinned":false,"hidden":false,"mode":"html"},{"id":103,"value":"Much cleaner! But, this only gets us the coordinates for one zip code, and we want to get data for multiple locations. Let's build on this method to call an API multiple times.","pinned":false,"hidden":false,"mode":"md"},{"id":106,"value":"## ⚙️ Mechanize it with a dataframe","pinned":false,"hidden":false,"mode":"md"},{"id":110,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":114,"value":"To satisfy our insatiable weather data needs, we can make a dataframe of the parameters and iterate through them to get multiple responses. Essentially, we are expanding the method above, making each row of a dataframe the necessary items to call the API. First, let's make a dataframe of the parameters we need for each call. We can combine the location codes to generate the combined query parameter OpenWeather expects (the zip field):","pinned":false,"hidden":false,"mode":"md"},{"id":118,"value":"```\n# make a dataframe of the api parameters\nlocation_codes <- tibble::tribble(\n ~zip_code, ~country_code,\n \"90210\", \"US\",\n \"E14\", \"GB\",\n \"99999\", \"US\",\n \"60606\", \"US\",\n \"97210\", \"US\",\n \"01010\", \"GB\"\n)\n\n# manipulate data to define parameters\nparam_df <- location_codes %>% \n mutate(\n combined_code = paste(zip_code, country_code, sep = ',')\n ) %>% \n select(combined_code)\n\nparam_df\n```","pinned":false,"hidden":false,"mode":"md"},{"id":124,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":128,"value":"We now have a dataframe where every row includes the query parameters for a call to the API (excluding any authentication methods). This works well for seeing all the options for more complicated calls.","pinned":false,"hidden":false,"mode":"md"},{"id":132,"value":"We can then use `purrr::pmap` to get responses for every zip code. Here we pass a named list of all the query parameters that we want to include in the URL (the values after the question mark in the by-hand URL above). Passing a named list inside of [`with`](https://www.journaldev.com/47815/with-and-within-function-in-r) using `...` ([dot-dot-dot](https://adv-r.hadley.nz/functions.html#fun-dot-dot-dot)), we can [access the dataframe names inside of pmap](https://stackoverflow.com/questions/51122773/harnessing-f-list-names-with-purrrpmap/51123520#51123520) to build up the query list in a readable way (for example, using column names like `combined_code` rather than something like `..3`, which can be harder to parse when revisiting scripts). So, we are taking the columns from our parameter dataframe and using them as inputs to the query list for GET. We define the parameter field names to match the desired API fields.","pinned":false,"hidden":false,"mode":"md"},{"id":136,"value":"Also notice we start each call by waiting for just over a second. OpenWeather doesn't like you shaking their hand too quickly, so it puts [rate limits](https://openweathermap.org/price) on how often you can make calls. By taking a beat in between calls, we can ensure we don't stress the API out.","pinned":false,"hidden":false,"mode":"md"},{"id":140,"value":"```\n# ping api\nbatches_geo <- purrr::pmap(param_df, ~with(list(...), {\n # wait a bit before each call\n Sys.sleep(1.1)\n \n # get response from API, pass query parameters from dataframe\n return(\n httr::GET(api_endpoint_geo, \n query = list(\n zip = combined_code, \n appid = Sys.getenv('OW_API_KEY')\n )\n )\n )\n}\n))\n\nbatches_geo\n```","pinned":false,"hidden":false,"mode":"md"},{"id":144,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":150,"value":"We end up with a list of [httr responses](https://httr.r-lib.org/reference/response.html): these are the results from all our handshakes. Now we'd like to parse the results and get at the underlying data.","pinned":false,"hidden":false,"mode":"md"},{"id":154,"value":"## 🤔 Checking Responses","pinned":false,"hidden":false,"mode":"md"},{"id":157,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":161,"value":"We can use `purrr::map` to sift through the httr responses and assess how our calls went. Let's take a look at what we got back. We can use `httr::status_code` to get the [http response status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) from every response, then flatten the nested list to a simple vector with integers.","pinned":false,"hidden":false,"mode":"md"},{"id":165,"value":"```\n# for all responses, get status codes\nstatus_codes <- purrr::map(batches_geo, ~ httr::status_code(.)) %>% \n purrr::flatten_int()\n\ntable(status_codes)\n```","pinned":false,"hidden":false,"mode":"md"},{"id":169,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":173,"value":"Whoops, looks like we got back some error codes. There were two invalid inputs that the API didn't like (`01010` and `99999` aren't valid zip codes). No worries! Let's only keep those results with a valid status code:","pinned":false,"hidden":false,"mode":"md"},{"id":177,"value":"```\n# keep valid responses only\nvalid_results <- batches_geo %>% \n purrr::keep(~ httr::status_code(.) == 200)\n```","pinned":false,"hidden":false,"mode":"md"},{"id":181,"value":"## 📦 Unpack the Responses","pinned":false,"hidden":false,"mode":"md"},{"id":185,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":189,"value":"We can now unpack the valid responses, taking their raw contents and reading them in as JSON. Again, httr helps with this, although it's good to note that [`content`](https://httr.r-lib.org/reference/content.html) does some interpretation based on what the API returns, so you may want to check the type of the responses using [`http_type`](https://httr.r-lib.org/reference/http_type.html), ensuring failed responses don't return something in an unexpected format.","pinned":false,"hidden":false,"mode":"md"},{"id":193,"value":"```\n# check the http types, all JSON\npurrr::map(valid_results, ~ httr::http_type(.)) %>% \n purrr::flatten_chr() %>%\n table()\n```","pinned":false,"hidden":false,"mode":"md"},{"id":197,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":201,"value":"In this case, they're all JSON, meaning `content` will use jsonlite to read in the raw data, making a larger nested list in R.","pinned":false,"hidden":false,"mode":"md"},{"id":205,"value":"```\n# extract content from responses\nlocations_list <- purrr::map(valid_results, ~ httr::content(.))\n```","pinned":false,"hidden":false,"mode":"md"},{"id":209,"value":"

Location responses as a nested R list (truncated)

","pinned":false,"hidden":false,"mode":"html"},{"id":214,"value":"Finally, we can convert the nested list of content to JSON and export it. By default, jsonlite \"boxes\" all fields in an array, even vectors of length one. If you are working within R this shouldn't affect things, but if you are expecting a specific format of JSON output, you can set `auto_unbox` to `TRUE` to unpack all atomic vectors of length one.","pinned":false,"hidden":false,"mode":"md"},{"id":218,"value":"```\n# make into JSON, pretty to expand out with indents/whitespace\njson_locations <- locations_list %>% \n jsonlite::toJSON(pretty = T, auto_unbox = T)\n\n# export JSON\njson_locations %>%\n write(\"locations.json\")\n\njson_locations\n```","pinned":false,"hidden":false,"mode":"md"},{"id":222,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":230,"value":"## ☀️ Go get weather data","pinned":false,"hidden":false,"mode":"md"},{"id":233,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":237,"value":"We now have the lat/long values for our valid zip codes, so let's find out what the weather's like in those places. To start, we can read in the JSON and use it as our parameter dataframe. This works in this case because the JSON is not nested (it's just an array of objects with single fields), but we will discuss using tibbles to work on nested JSON output in [Part 2](https://observablehq.com/@zachperiscopic/cozy-collecting-part-2) (a more robust approach).","pinned":false,"hidden":false,"mode":"md"},{"id":241,"value":"```\n# read in JSON\nlocations <- json_locations %>% \n jsonlite::fromJSON()\n```","pinned":false,"hidden":false,"mode":"md"},{"id":245,"value":"We can go through the same process to get the weather data from a different API endpoint. It's just like before, but with a few tweaks to the handshake. We simply change the endpoint, input dataframe, and query list. Notice the updated URL and different query parameters being extracted from our new dataframe.","pinned":false,"hidden":false,"mode":"md"},{"id":249,"value":"```\n# getting weather from lat/long this time\napi_endpoint_weather <- \"http://api.openweathermap.org/data/2.5/weather\"\n\nbatches_weather <- purrr::pmap(locations, ~with(list(...), {\n # wait a bit before each call\n Sys.sleep(1.1)\n \n # pass lat/long to new endpoint\n return(\n httr::GET(api_endpoint_weather, \n query = list(\n lat = lat, \n lon = lon,\n appid = Sys.getenv('OW_API_KEY')\n )\n )\n )\n}\n))\n\n# convert the content of the responses, combine into a nested list\nweather_list <- purrr::map(batches_weather, ~ httr::content(.))\n\n# make into JSON, pretty to expand out structure\njson_weather <- weather_list %>% \n jsonlite::toJSON(pretty = T, auto_unbox = T)\n\njson_weather\n```","pinned":false,"hidden":false,"mode":"md"},{"id":253,"value":"

Resulting JSON of weather at desired locations (truncated)

","pinned":false,"hidden":false,"mode":"html"},{"id":258,"value":"Nice handshaking!","pinned":false,"hidden":false,"mode":"md"},{"id":262,"value":"

","pinned":false,"hidden":false,"mode":"html"},{"id":266,"value":"## 🗄️ Process Summary\n- **Figure out what the API needs**: Consult the API documentation, register for a key/auth/etc., and figure out how to collect the data you want. What are the endpoints and what are the parameters needed for the call?\n- **Create Dataframe of Parameters**: Use classic dplyr methods to make a dataframe of all the parameters you need\n- **Purrr to Ping API**: Iterate through your dataframe, getting back responses\n- **Inspect and Convert**: Check the status of the responses, keeping valid ones or reassessing your inputs, then unpacking the content and converting the nested list to JSON for export","pinned":false,"hidden":false,"mode":"md"},{"id":270,"value":"## 🤖 Wrap Up\nWe have successfully gotten back JSON weather data from an API for our desired zip codes. However, here we encountered a special case with the zip code data having no nesting, meaning we could simply use `jsonlite::fromJSON` to parse the data into a `data.frame` to use as our parameter input for getting the weather data. However, we will usually get back *nested* JSON data and we would like to use [tibbles](https://tibble.tidyverse.org/) instead since they have a nicer print method for larger datasets plus variable checking. Since we will be making tibbles from *nested* JSON data (as this is often what you’ll encounter in the wild) we need to handle it differently. ","pinned":false,"hidden":false,"mode":"md"},{"id":274,"value":"---\n# ✨ Keep Going\n\nIn [Part 2](https://observablehq.com/@zachperiscopic/cozy-collecting-part-2), we will explore using \"rectangling\" methods to navigate our newly generated JSON output as a nested tibble and transform it into tidy rows and columns.","pinned":false,"hidden":false,"mode":"md"},{"id":294,"value":"[Cozy Collecting, Part 2: Working with Nested JSON in R](https://observablehq.com/@zachperiscopic/cozy-collecting-part-2)","pinned":false,"hidden":false,"mode":"md"}],"files":"$W31","stdlib":"1","titling":"auto"}