API as a package: Testing

Introduction

This is part the final part of our three part series

• Part 1: API as a package: Structure
• Part 2: API as a package: Logging
• Part 3: API as a package: Testing (this post)

This blog post is a follow on to our API as a package series, which looks to expand on the topic of testing {plumber} API applications within the package structure leveraging {testthat}. As a reminder of the situation, so far we have an R package that defines functions that will be used as endpoints in a {plumber} API application. The API routes defined via {plumber} decorators in inst simply map the package functions to URLs.

The three stages of testing

The intended structure of the API as a package setup is to encourage a particular, consistent, composition of code for each exposed endpoint. That is:

• A plumber decorator that maps a package function to a URL
• A wrapper function that takes a request object, deals with any serialization of data and dispatches to a “business logic” function
• The “business logic” function, or core functionality of the purpose of a particular endpoint

With that, we believe that this induces three levels of testing to consider:

• Does the running API application successfully return an appropriate response when we make a request to an endpoint?
• Does the wrapper function handle behaviour matching your expectations?
• Is my logic correct?

Example: Sum

Consider a POST endpoint that will sum the numeric contents of objects. For simplicity, we will consider only requests that send valid JSON objects, however there are a few scenarios that might arise:

1. A JSON array

   # array.json
   [1, 2]
   # expected sum: [3]
   

2. A single JSON object

   # single_object.json
   {
     "a": 1,
     "b": 2
   }
   # expected sum: [3]
   

3. An array of JSON objects

   # array_objects.json
   [
     {
       "a": 1,
       "b": 2
     },
     {
       "a": 1,
       "b": 2
     }
   ]
   # expected sum: [3, 3]
   

R code solution

Writing some R code to ensure that we calculate the expected sums for each of these is fairly simple, keeping in mind that when parsing JSON objects we would obtain a named list to represent an object and an unnamed list to represent an array:

# R/api_sum.R

# function to check whether the object we 
# receive looks like a json array
is_array = function(parsed_json) {
  is.null(names(parsed_json))
}

# function to sum the numeric components in a list
sum_list = function(l) {
  purrr::keep(l, is.numeric) |> purrr::reduce(`+`, .init = 0)
}

# main sum function which handles lists of lists appropriately
my_sum = function(x) {
  if (is_array(x)) {
    if(is.list(x)) {
      purrr::map(x, sum_list)
    } else {
      sum(x)
    }
  } else {
    sum_list(x)
  }
}

To integrate this into our API service we can then write a wrapper function

# R/api_sum.R

#' @export
api_sum = function(req) {
  # parse the JSON body of the request
  parsed_json = jsonlite::fromJSON(req$postBody)
  # return the sum
  return(my_sum(parsed_json))
}

and add a plumber annotation in inst/extdata/api/routes/example.R

#* @post /sum
#* @serializer unboxedJSON
cookieCutter::api_sum

which exposes our sum function on the URL <root_of_api>/example/sum.

Testing: Setup

With the above example we are now ready to start writing some tests. There are a few elements which are likely to be common when wanting to test endpoints of an API application:

• Start an instance of your API
• Send a request to your local running API
• Create a mock object that looks like a real rook request object

The {testthat} package for R has utilities that make defining and using common structures like this easy. A tests/testthat/setup.R script will run before any tests are constructed, here we can put together the setup and subsequent tear down for a running API instance, for our cookieCutter example package being built as part of this series this might look like

# test/testthat/setup.R

## run before any tests
# pick a random available port to serve your app locally
# note that port will also be available in the environment in which your
# tests run.
port = httpuv::randomPort()

# start a background R process that launches an instance of the API
# serving on that random port
running_api = callr::r_bg(
  function(port) {
    dir = cookieCutter::get_internal_routes()
    routes = cookieCutter::create_routes(dir)
    api = cookieCutter::generate_api(routes)
    api$run(port = port, host = "0.0.0.0")
  }, list(port = port)
)

Sys.sleep(2)
## run after all tests
withr::defer(running_api$kill(), testthat::teardown_env())

With this, as our test suite runs, we can send requests to our API at the following url pattern, http://0.0.0.0:{port}{endpoint}.

Similarly, {testthat} allows for defining helper functions for the purposes of your test-suite. Any file with “helper” at the beginning of the name in your testthat directory will be executed before your tests run. We might use this to define some helper functions which will allow us to send requests easily and create mock objects, as well as some other things.

# tests/testthat/helper-example.R

# convenience function for creating correct endpoint url
endpoint = function(str) {
  glue::glue("http://0.0.0.0:{port}{str}")
}

# convenience function for sending post requests to our test api
api_post = function(url, ...) {
  httr::POST(endpoint(url), ...)
}

# function to create minimal mock request objects
# doesn't fully replicate a rook request, but gives the parts
# we need
as_fake_post = function(obj) {
  req = new.env()
  req$HTTP_CONTENT_TYPE = "application/json"
  req$postBody = obj
  req
}

You might also want to skip the API request tests in cases where the API service did not launch correctly

# tests/testthat/helper-example.R

# skip other tests if api is not alive
skip_dead_api = function() {
  # running_api is created in setup.R
  testthat::skip_if_not(running_api$is_alive(), "API not started")
}

One of the things that we like to do, inspired by the pytest-datadir plugin for the python testing framework, pytest, is have numerous test cases stored as data files. This makes it easy to run your tests against many examples, as well as to add new ones that should be tested in future. With that our final helper function might be

# tests/testthat/helper-example.R

test_case_json = function(path) {
  # test_path() will give appropriate path to running test environment
  file = testthat::test_path(path)
  # read a file from disk
  obj = readLines(file)
  # turn json contents into a single string
  paste(obj, collapse = "")
}

Testing: Tests

With all of the setup work done (at least we only need to do that once) we will finally write tests to address the three types identified earlier in the article. We identified three scenarios for JSON we might receive, so we can go ahead and stick those in a data folder within our test directory.

└── tests
    ├── testthat
    │   ├── example_data
    │   │   ├── array.json
    │   │   ├── array_objects.json
    │   │   └── single_object.json

Our test script for this endpoint then, will iterate through the files in this directory and:

• Send each example as the body of a POST request and ensure we get a success response (200)
• Send a mock request object to the wrapper function, ensuring that data is being parsed correctly and the return object is of the right shape
• Take the data from the example file, run it through the my_sum() function and ensure that the result is correct

# tests/testthat/test-example.R

# iterate through multiple test cases
purrr::pwalk(tibble::tibble(
  # get all files in the test data directory
  file = list.files(test_path("example_data"), full.names = TRUE),
  # expected length (shape) of result
  length = c(2, 1, 1),
  # expected sums
  sums = list(c(3,3), 3, 3)
), function(file, length, sums){

  # use our helper to create the POST body
  test_case = test_case_json(file)

  # test against running API
  test_that("succesful api response", {
    # skip if not running
    skip_dead_api()
    headers = httr::add_headers(
      Accept = "application/json",
      "Content-Type" = "application/json"
    )
    # use our helper to send the data to the correct endpoint
    response = api_post("/example/sum", body = test_case, headers = headers)
    # check our expectation
    expect_equal(response$status_code, 200)
  })

  # test that the wrapper is doing its job
  test_that("successful api func", {
    # use helper to create fake request object
    input = as_fake_post(test_case)
    # execute the function which is exposed as a route directly
    res = api_sum(input)
    # check the output has the expected shape
    expect_length(res, length)
  })

  # test the business logic of the function
  test_that("successful sum", {
    # use the data parsed from the test case
    input = jsonlite::fromJSON(test_case)
    # execute the logic function directly
    res = my_sum(input)
    # check the result equals our expectation
    expect_equal(res, sums)
  })
})

Concluding remarks

With that we have a setup for our test suite that takes care of a number of common elements, which can of course be expanded for other HTTP methods, data types etc; and a consistent approach to testing many cases at the API service level, serialization/parsing and logic level. As with the other posts in this series a dedicated package example is available in our blogs repo.