CS195Y Lecture 2

1/27/17

---

Announcement: Oracle goes out today (1/27/17) and is due Thursday (2/2/17)

One minute recap of first lecture:
A logic class in the philosophy department will be primarily proof and formalism
This logic class will be predominately application-based. There will be no pencil-paper proofs.
We will be writing code to show what problems we can solve using logic.

Testing Oracles

Strengths of Test Cases

• Make sure your code still performs as expected if you change things
• Edge cases
• To make sure it has the correct functionality
• Writing tests makes you think about what correct functionality is

Weaknesses of Test Cases

• It's hard to come up with all the edge cases
• It's not as easy as it sounds to turn ideas/intuition into test cases
• They're tedious to write
• There are infinite possible inputs, so you can't possibly write them all

Sort
Record = {Name, School}
Test cases for sorting records of this type:

[] -> []
[{"", ""}] -> [{"", ""}]
[{"Tim", "Brown"}, {"Tim", "Not Brown"}]

Are we sorting by Name or School or both?
If we sort just by Name, there are potentially several correct outputs.
[{"Tim", "Brown"}, {"Tim", "Not Brown"}]
If we want order to be preserved, then the only possible output is [{"Tim", "Brown"}, {"Tim", "Not Brown"}]
But, if we don't care whether order is presevered, there are two possible outputs, so it's harder to
write the test case (you can't just have the input and output)
Other problems with multiple correct results:

• Finding a satisfying instance to a boolean formula
• Randomly shuffling an array
• Finding a minimum spanning tree

Issue: Test cases are too rigid. We need to be more flexible in order to be sure our tests are enough to say the
works.

What is the type of Sort?
Sort: Listof<Rec> -> Listof<Rec>
What is the type of a test suite for sort?
Suite: (Listof<Rec> -> Listof<Rec>) -> (Boolean)

What are we going to test our sort against?

• Write another sort
  • If they match, that's great!
  • If they don't match, the first sort could be wrong or the other sort could be wrong or maybe they handle
    edge cases differntly
• Other options?

What properties should be true about our sorting algorithm?
A. all x: Input | x in Output
B. all X: Output| x in Input
C. "sortedness" all x,y : Output | x > y implies pos(x) > pos(y)

Oracle: (Listof<Rec> -> Listof<Rec>) -> (Boolean)
Our goal is to construct a function that we can pass our sort function to, and it will tell us
whether or not the sort function is correct.
It's challenging to be both sound and complete.
Sound: You never say true if the answer is not true.
Complete: You never say false if the answer is true.

Programs are hard to verify. Our goal in this course is to fight that problem.

An oracle is made of two components: an input generator and a validity checker.
valid?: List<Rec>, List<Rec> -> List<Rec> -> Boolean

• Is the list ordered?
• Does the output list contain the same elements as the input list
• We might want to check for stability (items with the same key appear in the same order as they did in the input list)

generate-input: () -> List<Rec>
Break down into a couple problems:

• Generate random strings with letters, numbers, non-alphanumeric characters
• Generate records consisting of these random strings
• Generate lists of varying lengths consisting of these random records

New paradigm for testing: Instead of coming up with inputs and outputs, write code to generate inputs and
test whether the outputs satisfy certain properties.

Integer Factorization
Given an integer, express that integer as the product of primes

4 -> 2^2
7 -> 7^1
1 -> 1
12 -> 2^2 * 3^1

generate-input: () -> Integer

• use a random number generator

valid?: Integer, Integer -> List<Integer> -> Boolean

• all prime
• multiply together to produce the input number
• grouped in a nice way (ie. 2^2, 3 not 2,3,2)
• no extra 1s

Sometimes the properties are themselves difficult to check. For example, how do we determine if the output numbers
are all prime?
Even if we can't test all of the properties, partial oracles can still be helpful.

Generation is not intellectually difficult. Checking validity can be difficult.

Sudoku
Input: A 3x3 board, each cell of which is also divided into a 3x3 grid. Some spaces will be filled in with a number
1-9, but most will be empty
Output: The same board with every space filled in.
valid?:

• cells that were filled in in the input are the same in the output
• no repeats in any row, column, or box
• all cells are filled in
• cells are only filled in with numbers 1-9
• 9 instances each of 1-9

Next week, we will prove implications between these properties.