---
title: "Comp/Stat 112: API"
author: "INSERT STUDENT NAME HERE (COLLABORATOR NAMES HERE)"
output:
  bookdown::html_document2:
    split_by: none
    toc: yes
    toc_depth: 3
    toc_float:
      toc_collapsed: true
    number_sections: false
    code_download: true
---

```{r 14_setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, error = TRUE)
library(tidyverse)
library(ggplot2)
library(lubridate)
```

# Public Web APIs

## Learning Goals {-}

- Understand the difference between acquiring data through web scraping vs. a web API
- Set up an API key for a public API
- Develop comfort in using a wrapper package or url-method of calling a web API
- Recognize the structure in a url for a web API and adjust for your purposes



## APIs {-}

In this lesson you'll learn how to collect data from websites such as The New York Times, Zillow, and Google. While these sites are primarily known for the information they provide to humans browsing the web, they (along with most large websites) also provide information to computer programs.

Humans use browsers such as Firefox or Chrome to navigate the web. Behind the scenes, our browsers communicate with web servers using a technology called [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) or Hypertext Transfer Protocol. 

Programming languages such as R can also use HTTP to communicate with web servers. We have seen how it is possible for R to "scrape" data from almost any static web page. However, it's easiest to interact with websites that are specifically designed to communicate with programs. These [Web APIs](https://en.wikipedia.org/wiki/Web_API), or Web Application Programming Interfaces, focus on transmitting data, rather than images, colors, or other appearance-related information.

An large variety of web APIs provide data accessible to programs written in R (and almost any other programming language!). Almost all reasonably large commercial websites offer APIs. Todd Motto has compiled an excellent list of [Public Web APIs](https://github.com/toddmotto/public-apis) on GitHub. Browse the list to see what kind of information is available.


## Wrapper Packages {-}

Possible readings: <br>
1. [NY Times API](https://developer.nytimes.com/?mcubz=3) <br>
2. [NY Times Blog post announcing the API](https://open.blogs.nytimes.com/2009/02/04/announcing-the-article-search-api/?mcubz=3&_r=0) <br>
3. [Working with the NY Times API in `R`](https://www.storybench.org/working-with-the-new-york-times-api-in-r/)   
4. [nytimes pacakge for accessing the NY Times' APIs from `R`](https://github.com/mkearney/nytimes) <br>
5. [Video showing how to use the NY Times API](https://www.youtube.com/watch?v=3at3YTAFbxs) <br>
6. [rOpenSci](https://ropensci.org/packages/) has a good collection of wrapper packages

In R, it is easiest to use Web APIs through a *wrapper package*, an R package written specifically for a particular Web API. The R development community has already contributed wrapper packages for most large Web APIs. To find a wrapper package, search the web for "R Package" and the name of the website. For example, a search for "R Reddit Package" returns [RedditExtractor](https://cran.r-project.org/web/packages/RedditExtractoR/index.html) and a search for "R Weather.com Package" surfaces [weatherData](https://ram-n.github.io/weatherData/).

This activity will build on the [New York Times Web API](https://developer.nytimes.com/), which provides access to news articles, movie reviews, book reviews, and many other data. Our activity will specifically focus on the 
[Article Search API](https://developer.nytimes.com/docs/articlesearch-product/1/overview), which finds information about news articles that contain a particular word or phrase. 

We will use the [nytimes](https://github.com/mkearney/nytimes) package that provides functions for some (but not all) of the NYTimes APIs. First, install the package by copying the following two lines into your console (you just need to run these once):

```
install.packages("devtools")
devtools::install_github("mkearney/nytimes")
```

Next, take a look at the Article Search API example on the [package website](https://github.com/mkearney/nytimes) to get a sense of the syntax.

```{exercise} 
What do you think the nytimes function below does? How does it communicate with the NY Times? Where is the data about articles stored?

```

```{r eval=FALSE}
# Don't try to run the code yet, just look at it
#res <- nyt_search(q = "gamergate", n = 20, end_date = "20150101")
```

> ANSWER: 

To get started with the NY Times API, you must [register and get an **authentication key**](https://developer.nytimes.com/accounts/create). Signup only takes a few seconds, and it lets the New York Times make sure nobody abuses their API for commercial purposes. It also **rate limits** their API and ensures programs don't make too many requests per day. For the NY Times API, this limit is 1000 calls per day. Be aware that most APIs do have rate limits --- especially for their free tiers.

Once you have signed up, verified your email, log back in to https://developer.nytimes.com. Under your email address, click on Apps and Create a new App (call it First API) and enable Article Search API, then press Save. This creates an **authentication key**, which is a 32 digit string with numbers and the letters a-e. 

Store this in a variable as follows (this is just an example ID, not an actual one):

```{r}
# Change value to your personal API key
times_key <- "c935b213b2dc1218050eec976283dbbd"
```


Now, let's use the key to issue our first API call. We'll adapt the code we see in the vignette to do what we need.

```{r}
library(nytimes)

# Tell nytimes what our API key is
Sys.setenv(NYTIMES_KEY = times_key)

# Issue our first API call
res <- nyt_search(q = "gamergate", n = 20, end_date = "20150101")

# Convert response object to data frame
res <- as.data.frame(res)
```

Something magical just happened. Your computer sent a message to the New York Times and asked for information about 20 articles about [Gamergate](https://en.wikipedia.org/wiki/Gamergate_controversy) starting at January 1, 2015 and going backwards in time. Thousands of public Web APIs allow your computer to tap into almost any piece of public digital information on the web. 

Let's take a peek at the structure of the results. You can also look at the data in the "Environment" tab in one of the windows of RStudio:

```{r }
colnames(res)
head(res$web_url)
head(res$headline)
head(res$pub_date)
```


## Accessing Web APIs{-}

Wrapper packages such as `nytimes` provide a convenient way to interact with Web APIs. However, many Web APIs have incomplete wrapper packages, or no wrapper package at all. Fortunately, most Web APIs share a common structure that `R` can access relatively easily. There are two parts to each Web API: the **request**, which corresponds to a function call, and the **response**, which corresponds to the function's return value.^[Although we imply that a Web API call corresponds to a single function on the webserver, this is not necessarily the case. Still, we use this language because the analogy fits well.]

As mentioned earlier, a Web API call differs from a regular function call in that the request is sent over the Internet to a webserver, which performs the computation and calculates the return result, which is sent back over the Internet to the original computer.


**Web API Requests**

Possible readings: <br>
1. [Understanding URLs](https://www.tutorialspoint.com/html/understanding_url_tutorial.htm) <br>
2. [urltools Vignette](https://cran.r-project.org/web/packages/urltools/vignettes/urltools.html)

The request for a Web API call is usually encoded through the [URL](https://www.tutorialspoint.com/html/understanding_url_tutorial.htm), the web address associated with the API's webserver. Let's look at the URL associated with the first `nytimes` `nyt_search` example we did. Open the following URL in your browser (you should replace MY_KEY with the api key you were given earlier).

    http://api.nytimes.com/svc/search/v2/articlesearch.json?q=gamergate&api-key=MY_KEY
    

The text you see in the browser is the response data. We'll talk more about that in a bit. Right now, let's focus on the structure of the URL. You can see that it has a few parts:

* `http://` --- The **scheme**, which tells your browser or program how to communicate with the webserver. This will typically be either `http:` or `https:`.
* `api.nytimes.com` --- The **hostname**, which is a name that identifies the webserver that will process the request.
* `/svc/search/v2/articlesearch.json` --- The **path**, which tells the webserver what function you would like to call.
* `?q=gamergate&api-key=MY_KEY` --- The **query parameters**, which provide the parameters for the function you would like to call. Note that the query can be thought of as a table, where each row has a key and a value (known as a key-value pair). In this case, the first row has key `q` and value `gamergate` and the second row has value `MY_KEY`. The query parameters are preceded by a `?`. Rows in the key-value table are separated by '&', and individual key-value pairs are separated by an `=`.

Typically, each of these URL components will be specified in the API documentation. Sometimes, the scheme, hostname, and path (`http://api.nytimes.com/svc/search/v2/articlesearch.json`) will be referred to as the [endpoint](https://en.wikipedia.org/wiki/Web_API#Endpoints) for the API call.

We will use the `urltools` module to build up a full URL from its parts. We start by creating a string with the endpoint and then add the parameters one by one using `param_set` and `url_encode`:

```{r eval=FALSE}
library(urltools)

url <- "http://api.nytimes.com/svc/search/v2/articlesearch.json"
url <- param_set(url, "q", url_encode("marlon james"))
url <- param_set(url, "api-key", url_encode(times_key))
url
```

Copy and paste the resulting URL into your browser to see what the NY Times response looks like!

```{exercise}
You may be wondering why we need to use `param_set` and `url_encode` instead of writing the full url by hand. This exercise will illustrate why we need to be careful. 
```

 a) Repeat the above steps, but create a URL that finds articles related to `Ferris Bueller's Day Off` (note the apostrophe). What is interesting about how the title appears in the URL?
 
> ANSWER: 

```{r}
library(urltools)

url <- "http://api.nytimes.com/svc/search/v2/articlesearch.json"
url <- param_set(url, "q", url_encode("???"))
url <- param_set(url, "api-key", url_encode(times_key))
url
```
 
 b) Repeat the steps above for the phrase `Nico & Vinz` (make sure you use the punctuation mark `&`). What do you notice?

> ANSWER: 

```{r}


```
 
 c) Take a look at the Wikipedia page describing [percent encoding](https://en.wikipedia.org/wiki/Percent-encoding). Explain how the process works in your own words.

> ANSWER: 


## Web API Responses {-}

Possible readings: <br>
1. [A Non-Programmer's Introduction to JSON](https://blog.scottlowe.org/2013/11/08/a-non-programmers-introduction-to-json/) <br>
2. [Getting Started With JSON and jsonlite](https://cran.r-project.org/web/packages/jsonlite/vignettes/json-aaquickstart.html) <br>
3. [Fetching JSON data from REST APIs](https://cran.r-project.org/web/packages/jsonlite/vignettes/json-apis.html)

We now discuss the structure of the web response, the return value of the Web API function. Web APIs generate string responses. If you visited the earlier New York Times API link in your browser, you would be shown the string response from the New York Times webserver:

```
{"status":"OK","copyright":"Copyright (c) 2021 The New York Times Company. All Rights Reserved.","response":{"docs":[{"abstract":"Here’s what you need to know.","web_url":"https://www.nytimes.com/2019/08/16/briefing/rashida-tlaib-gamergate-greenland.html","snippet":"Here’s what you need to know.","lead_paragraph":"(Want to get this briefing by email? Here’s the sign-up.)","source":"The New York Times","multimedia":[{"rank":0,"subtype":"xlarge","caption":null,"credit":null,"type":"image","url":"images/2019/08/16/world/16US-AMBRIEFING-TLAIB-amcore/merlin_158003643_c67928bc-e547-4a2e-9344-5f0209ca024d-articleLarge.jpg","height":400,"width":600,"legacy":{"xlarge":"images/2019/08/16/world/16US-AMBRIEFING-TLAIB-amcore/merlin_158003643_c67928bc-e547-4a2e-9344-5f0209ca024d-articleLarge.jpg","xlargewidth":600,"xlargeheight":400},"subType":"xlarge","crop_name":"articleLarge"},...
```

If you stared very hard at the above response, you may be able to interpret it. However, it would be much easier to interact with the response in some more structured, programmatic way. The vast majority of Web APIs, including the New York Times, use a standard called JSON (Javascript Object Notation) to take data and encode it as a string. 
To understand the structure of JSON, take the NY Times web response in your browser, and copy and paste it into an online [JSON formatter](https://jsonformatter.curiousconcept.com/). The formatter will add newlines and tabs to make the data more human interpretable. You'll see the following:

```
{  
   "status":"OK",
   "copyright":"Copyright (c) 2021 The New York Times Company. All Rights Reserved.",
   "response":{  
      "docs":[  
      
        # A HUGE piece of data, with one object for each of the result articles
        
      ],
      "meta":{  
         "hits":128,
         "offset":0,
         "time":93
      }
   }
}     
```

You'll notice a few things in the JSON above:

* Strings are enclosed in double quotes, for example `"status"` and `"OK"`.
* Numbers are written plainly, like `2350` or `72`.
* Some data is enclosed in square brackets `[` and `]`. These data containers can be thought of as R lists.
* Some data is enclosed in curly braces `{` and `}`. These data containers are called  *Objects*. An  object can be thought of as a single observation in a table. The columns or variables for the observation appear as **keys** on the left (`hits`, `offset`, etc.). The **values** appear after the specific key separated by a colon (`2350`, and `0`, respectively).
Thus, we can think of the `meta` object above as:

```{r, results='as.is', echo=FALSE}
library(kableExtra)

knitr::kable(data.frame(hits = c(128), offset = c(0), time = c(93)), "html") %>%
  kable_styling(full_width = F)
```

Let's repeat the NY Times search for gamergate, but this time we will peform the Web API call by hand instead of using the `nytimes` wrapper package. We will use the `jsonlite` package to retrieve the response from the webserver and turn the string response into an `R` object. The `fromJson` function sends our request out over and across the web to the NY Times webserver, retrieves it, and turns it from a JSON-formatted string into R data.


```{r cache=TRUE}

library(jsonlite)

# Rebuild the URL
url <- "http://api.nytimes.com/svc/search/v2/articlesearch.json"
url <- param_set(url, "q", url_encode("gamergate"))
url <- param_set(url, "api-key", url_encode(times_key))

# Send the request to the webserver over the Internet and
# retrieve the JSON response. Turn the JSON response into an
# R Object.
response_js <- fromJSON(url)
```


The  `jsonlite` makes the keys and values of an object available as attributes. For example, we can fetch the status:

```{r}
response_js$status
```

While some keys in the object are associated with simple values, such as `"status"`, others are associated with more complex data. For example, the key `"response"` is associated with an object that has two keys: `"docs"`, and `"meta"`. `"meta"` is another object: `{  "hits":128, "offset":0, "time":19 }`. We can retrieve these *nested* attributes by sequentially accessing the object keys from the outside in. For example, the inner `"hits"` attribute would be accessed as follows:

```{r}
response_js$response$meta$hits
```


```{exercise} 
Retrieve the data associated with 

1) the `copyright` key of the `response_js` object, and 
2) the `time` attribute nested within the `meta` object.

```

> ANSWER:

```{r}

```

The majority of the data is stored under `response`, in `docs`. Notice that `docs` is a list, where each element of the list is a JSON object that looks like the following:

```
 {  
  "web_url":"https://www.nytimes.com/2017/06/27/arts/milkshake-duck-meme.html",
  "snippet":"Oxford Dictionaries is keeping a close eye on a term that describes someone who rapidly gains and inevitably loses the internet’s intense love.",  
  "blog":{  },  
  "source":"The New York Times",  
  "multimedia":[    
      ... A LIST OF OBJECTS ...  
  ],  
  "headline":{    
     "main":"How a Joke Becomes a Meme: The Birth of ‘Milkshake Duck’",  
     "print_headline":"How a Joke Becomes a Meme: The Birth of ‘Milkshake Duck’"  
  },  
  "keywords":[    
      ... A LIST OF OBJECTS ...  
  ],  
  "pub_date":"2017-06-27T12:24:20+0000",  
  "document_type":"article",  
  "new_desk":"Culture",  
  "byline":{    
     "original":"By JONAH ENGEL BROMWICH"  
  },  
  "type_of_material":"News",  
  "_id":"59524e7f7c459f257c1ac39f",  
  "word_count":1033,  
  "score":0.35532707,  
  "uri":"nyt://article/a3e5bf4a-6216-5dba-9983-73bc45a98e69"  
},
```

`jsonlite` makes lists of objects available as a data frame, where the columns are the keys in the object (`web_url`, `snippet`, etc.)

```{r}
docs_df <- response_js$response$docs
class(docs_df)
colnames(docs_df)
dim(docs_df)
```

```{exercise, name="Your own article search"}
Consider the following:
```

 a) Select your own article search query (any topic of interest to you). You may want to play with NY Times online search or the [API web search console](https://developer.nytimes.com/docs/articlesearch-product/1/routes/articlesearch.json/get) to find a query that is interesting, but not overly popular. You can change any part of the query you would like. Your query should have at least 30 matches.

> ANSWER:

```{r}

``` 
 
 b) Retrieve data for the first three pages of search results from the article search API, and create a data frame that joins together the `docs` data frames for the three pages of results.  *Hint: The example in the section below shows how to get different pages of results and use `bind_rows` to combine them.*   
 

> ANSWER:

```{r}

``` 
 
 c) Visualize the number of search results per day or month in your result set.

> ANSWER:

```{r}

``` 

### A Note on Nested Data Frames {-}

Here is some code to generate queries on NY Times articles about the Red Sox. It fetches the first thirty entries in batches of 10.

```{r cache=TRUE}
url <- "http://api.nytimes.com/svc/search/v2/articlesearch.json"
url <- param_set(url, "q", url_encode("Red Sox"))
url <- param_set(url, "api-key", url_encode(times_key))
url <- param_set(url, "page", 0)
Sys.sleep(1)
res1 <- fromJSON(url)

# This pauses for 1 second.
# It is required when knitting to prevent R from issuing too many requests to
# The NY Times API at a time. If you don't have it you will get an error that
# says "Too Many Requests (429)"
Sys.sleep(1)
url <- param_set(url, "page", 1)
res2 <- fromJSON(url)

Sys.sleep(1)
url <- param_set(url, "page", 2)
res3 <- fromJSON(url)

docs1 <- res1$response$docs
docs2 <- res2$response$docs
docs3 <- res3$response$docs
```

Each of these docs variables is a table with ten entries (articles) and the same 18 variables:
```{r}
names(docs1)
```

Now we want to stack the tables on top of each other to get a single table with 30 rows and 18 variables. If you try the following command:

```
bind_rows(docs1,docs2,docs3)
```

then you will get an error saying "Error in bind_rows_(x, .id) : Argument 4 can't be a list containing data frames."

What is happening???

Let's check out the first column of the `docs1` table:
```{r}
docs1$web_url
```

It lists the web addresses of the first ten sites returned in the search. It is a vector of ten character strings, which is just fine for one column of data in our table.

Now let's check out the `headline` variable:
```{r}
docs1$headline
```

The `headline` variable is actually a data frame that contains three variables: `main`, `kicker`, and `print_headline`. That is, we have **nested data frames**. This is a common problem when scraping data from JSON files, and it is why we are not able to directly bind the rows of our three tables on top of each other.

We can check out the type of variable in each column with the `class` function:

```{r}
sapply(docs1, class)
```

We see that `blog`, `headline`, and `byline` are the three problem columns that each contain their own data frames.

The solution is to **flatten** these variables, which generates a new column in the outer table for each of the columns in the inner tables.

```{r}
docs1_flat <- jsonlite::flatten(docs1)
names(docs1_flat)
sapply(docs1_flat, class)
```

The `headline` variable is now replaced with seven separate columns for `headline.main`, `headline.kicker`, `headline.content_kicker`, `headline.print_headline`, `headline.name`, `headline.seo`, and `headline.sub`. The `byline` variable is replaced with three separae columns. The `blog` variable contained an empty data frame, so it has been removed. The overall result is a new flat table with 25 columns, and no more nested data frames.

Once the data is flattened, we can bind rows:

```{r}
all_docs <- bind_rows(jsonlite::flatten(docs1), jsonlite::flatten(docs2), jsonlite::flatten(docs3))
dim(all_docs)
```

## Additional Practice {-}


```{exercise, name="Choose-your-own public API visualization"}
 
 Browse [toddomotos' list of Public APIS](https://github.com/toddmotto/public-apis#science) and [abhishekbanthia's list of Public APIs](https://github.com/abhishekbanthia/Public-APIs). Select one of the APIs from the list. Here are a few criteria you should consider:

 * You must use the JSON approach we illustrated above; not all APIs support JSON.^[If you want to use an API that does not support JSON, you can check if there is an `R` wrapper package.]
 * Stay away from APIs that require OAuth for Authorization unless you are prepared for extra work before you get data! Most of the large social APIs (Facebook, LinkedIn, Twitter, etc.) require OAuth. toddomoto's page lists this explicitly, but you'll need to dig a bit if the API is only on abhishekbanthia's list.
 * You will probably need to explore several different APIs before you find one that works well for your interests and this assignment.
 * Beware of the `rate limits` associated with the API you choose. These determine the maximimum number of API calls you can make per second, hour or day. Though these are not always officially published, you can find them by Google (for example) `GitHub API rate limit`. If you need to slow your program down to meet the API insert calls to `Sys.sleep(1)` as is done in the example below.
 * Sketch out one interesting visualization that relies on the public API you selected earlier. Make sure the exact data you need is available. If it's not, try a new visualization or API.
 * If a wrapper package is available, you may use it, but you should also try to create the request URL and retrieve the JSON data using the techniques we showed earlier, without the wrapper package.
 * Visualize the data you collected and describe the results.

```

> ANSWER:

```{r}

``` 

