Lecture notes: Testing tables, plotting

A small correction

At the end of Wednesday’s class, I noticed that the spreadsheet we’ve been pulling data from is formatted differently than the Pyret code implied–the 2000 and 2010 population columns were swapped. I’ve fixed the spreadsheet, which is here. All of the code is still correct, but some of the results I displayed in class were incorrect. Always sanity check your results! See this article for a high-profile example of this kind of mistake.

Testing table programs

Here’s the program we ended up with last time:

include tables
include gdrive-sheets

include shared-gdrive("cs111-2018.arr", "1XxbD-eg5BAYuufv6mLmEllyg28IR7HeX")

ssid = "1jHvn5CPE6RkTTQRIXQbY5n5p4aiOH7fZsnwK2s6s6tc"
spreadsheet = load-spreadsheet(ssid)

all-municipalities = load-table: name :: String, city :: Boolean,
  population-2000 :: Number, population-2010 :: Number
  # true because the sheet has a "header" row
  source: spreadsheet.sheet-by-name("municipalities", true)
end

fun is-town(r :: Row) -> Boolean:
  not(r["city"])
end

fun percent-change(r :: Row) -> Number:
  (r["population-2010"] - r["population-2000"]) /
  r["population-2000"]
end

fun fastest-growing-towns(municipalities :: Table) -> Table:
  towns = filter-by(municipalities, is-town)
  towns-with-percent-change = build-column(towns, "percent-change", percent-change)
  sort-by(towns-with-percent-change, "percent-change", false)
end

We’ve done a bit of a bad thing here: we have written three functions, but we don’t have tests for any of them! Let’s see how we can rectify this.

We can test table programs by using test tables–tables with the same structure as the larger tables we are interested in, but which are smaller and contain data ` that are useful for testing. What would good test data look like for this problem?

test-municipalities = table: name, city, population-2000, population-2010
  row: "City", true, 100, 101
  row: "Town 1", false, 100, 102
  row: "Town 2", false, 100, 99
  row: "Town 3", false, 50, 54
end

Let’s see how we use these test data to test our functions.

First, we can test is-town. This is a boolean function; we want to make sure it works on town and city rows.

fun is-town(r :: Row) -> Boolean:
  not(r["city"])
where:
    is-town(test-municipalities.row-n(0)) is false
    is-town(test-municipalities.row-n(1)) is true
    is-town(test-municipalities.row-n(2)) is true
end

How about percent-change? In the end we’ll only want to use it on towns, but we can test it on city rows, too.

fun percent-change(r :: Row) -> Number:
  (r["population-2010"] - r["population-2000"]) /
  r["population-2000"]
where:
  percent-change(test-municipalities.row-n(0)) is 0.01
  percent-change(test-municipalities.row-n(1)) is 0.02
  percent-change(test-municipalities.row-n(2)) is -0.01
end

Finally, we can test our top-level table function. It produces a table, so we’ll have to construct a new table that it should return.

fun fastest-growing-towns(municipalities :: Table) -> Table:
  towns = filter-by(municipalities, is-town)
  towns-with-percent-change = build-column(towns, "percent-change", percent-change)
  sort-by(towns-with-percent-change, "percent-change", false)
where:
  test-municipalities-after = table: name, city, population-2000, population-2010, percent-change
    row: "Town 3", false, 50, 54, 0.08
    row: "Town 1", false, 100, 102, 0.02
    row: "Town 2", false, 100, 99, -0.01
  end
  fastest-growing-towns(test-municipalities) is test-municipalities-after
end

It’s important to write out the result table before you see the table produced by the function–we’re trying to make sure that the function does what it’s supposed to, which means that we have to predict its behavior! If we just copy in the table it produces, we’re testing the function against its own behavior rather than against our intuitions.

Testing a function against its own previous behavior can be useful if you are planning on changing the function and want to ensure that it still has the same behavior. This is called a regression test.

Plotting data

Plots help us visually understand the shape of data, and are often much more readable than a large table of numbers. Data scientists use plots for both exploratory and explanatory purposes–they are useful for understanding data in preparation for further analysis and in presenting data to a general audience.

Our tables library includes several functions to generate different kinds of plot. Here are a few examples using our municipalities data.

# how is population distributed in the state?
pie-chart(all-municipalities, "name", "population-2010")

# how many municipalities of various sizes are there?
histogram(all-municipalities, "population-2010", 1000)

# hw much, and how, does population vary?
box-plot(all-municipalities, "population-2010")

ft = fastest-growing-towns(all-municipalities)

# visually present the growth data
bar-chart(ft, "name", "population-2010")

# is a town's size (in 2000) correlated with its growth?
scatter-plot(ft, "population-2000", "percent-change")
# linear regression
lr-plot(ft, "population-2000", "percent-change")