Link Search Menu Expand Document
CSCI 1600, Fall 2021

Testing and debugging

Table of contents

  1. About
    1. Lab 6 Rubric
    2. Resources
    3. Materials
  2. Prelab (due date: October 19, 2021 at 4pm)
  3. Procedure
  4. Writeup

About

In this lab, you will come up with test cases to test your lab 5 code, based on a reference FSM. You will run these test cases and use the tests to debug your code. You will then be asked to reflect on the assumptions we made in testing. By the end of this lab, you will have seen how FSM-driven design influences unit testing for an embedded project.

Lab 6 Rubric

Resources

Test file

Materials

Included in your kits:

  • Arduino MKR1000 and USB cable
  • 2 small or 1 large breadboard
  • 1 16x2 LCD screen
  • 1 Potentiometer
  • 4 1MΩ resistors
  • 1 220Ω resistor
  • Jumpers/wires

Provided:

  • Materials for gamepad

Prelab (due date: October 19, 2021 at 4pm)

Duplicate the following worksheet and fill it out digitally, or print it, fill it out on paper, and scan it to turn it in. The idea is to be able to refer to this worksheet during the lab.

Procedure

  1. Make sure that you still have your circuit and code from Lab 5, and follow these steps to set your code up for testing:

    1. Copy the testing file the folder that holds your lab 5 code.

    2. In lab5.h, add the following lines (anywhere in the file):

       #define TESTING
 bool test_all_tests();

    3. In lab5.ino, comment out everything in your loop() function, and add the following line to the end of your setup() function:

      test_all_tests()

    4. Compile your code (Sketch -> Verify/Compile or the checkmark button next to the upload button). Your Arduino does not need to be plugged in to complete this step. If there are compilation errors, double-check the steps above.

  2. Read the test_transition function and discuss how it works with your partner. When you have come up with a 2-4 sentence description, get it checked off by a TA.

  3. Work with your partner to combine your test case tables. For most transitions and stutter conditions, choose one partner’s test case to keep. For four transitions of your choice, include both tests, such that you have 23 tests in total. To create more rows in the testing table, highlight the two sheet rows that make up one transition entry. Copy the rows. Right-click on the next blank row and select “Paste special -> Paste formal only”)

  4. Follow the instructions in part 3 of the prelab to turn your test cases into lines of Arduino code. Remember that your spreadsheet shouldn’t have any invalid input errors nor blank cells in order for this code to work. Paste these lines into lab5_tests.ino under the REPLACE THE FOLLOWING 6 LINES WITH YOUR TEST CASES comment.

  5. Connect your Arduino, open the Serial Monitor, and upload and run the code. If you get to “All tests passed!”, congratulations. Get checked off by a TA. Be prepared for your TA to ask you conceptual questions about your test cases. Otherwise, debug your code, based on which transition fails. You may find that there is a flaw in your implementation, or you may find that your test case made an incorrect assumption. In the second case, you should correct your assumptions (asking a TA if need be), modify the test case, and re-parse your test cases as in step 4 above.

  6. Comment out the #define TESTING line in lab5.h; and uncomment your loop() code and remove the test_all_tests() line in lab5.ino. Run your code and verify that the game plays as intended. If not, discuss what might have gone wrong and work on debugging your test cases and code. Once the game is working, get it checked off by a TA.

  7. OPTIONAL if your tests pass well before lab is over, try one or both of the following bonus steps. If you get checked off for the bonus step, you do not have to answer the corresponding question of the lab writeup.

    1. (Replaces writeup question 5) Test of display functions: the tests you ran do not test whether the display is displaying the correct text. Write some code that mocks up the display functions (see the lines under the MOCKED UP FUNCTIONS FOR TESTING comment in lab5_utils.ino) such that they set global variable(s) that can be checked against expected values. You will need to modify the test_transition function signature and contents to check whether the expected outputs were correct. Write two test cases for transitions that use different display functions and explain and demonstrate your code to a TA.

    2. (Replaces writeup question 6) Test of non-deterministic functions: In order to make testing easier, we mocked out the non-determinism of the helper functions random_location_orientation, random_turn, and shrink_bounds (you may refer to the lines under the MOCKED UP FUNCTIONS FOR TESTING comment in lab5_utils.ino). Remove the #define TESTING line in lab5.h. Now test that your test cases pass.

      Hint 1: think of this testing in terms of the “properties” you need to check. For example, if a transition calls shrink_bounds, you know that the value of uxb-lxb before the transition will be 1 larger than the value of uxb-lxb after transition. Similarly, you know that for random_turn, the orientation after the transition will never be the direct opposite of the orientation before the transition.

      Hint 2: you will have to modify the test_transition code to not do a direct comparison in some cases. One way to do this is to hand-create array(s) of flags that tell test_transition whether, for a given test case, it is doing a direct comparison or the check of a property.

      Once you are confident your code works, get it checked off by a TA. Your TA will check that you run your 23 test cases and that #define TESTING does not exist anywhere in your code.

Writeup

In a document, answer the following questions. Make sure the report states your name (first and last) and your partner’s name (first and last)! (if you worked alone, write “no partner”)

1. Think through the following list of concepts you encountered in the lab, and note if you have questions about any of them. Please also include any questions about concepts not listed here. If you have no questions, just write “no questions”

 unit testing, mock functions, testing frameworks in general, testing based on properties, testing assumptions

2. What, if any, frustrations did you encounter while doing the lab?

3. What was your main takeaway from this lab?

4. In the prelab, you were asked to assume well-formed inputs (such as lxb <= uxb).

a) Assuming a correct statechart design, why is this an acceptable assumption for this lab (hint: think of how lxb and uxb, for example, change from initialization as the FSM executes)?

b) How would you change the statechart if you wanted to catch invalid inputs and display an error?

c) How would you test that the error displaying is correct?

5. Your tests do not check that the correct display functions are used with the correct inputs at each transition. Give a 3-6 sentence description of how you would mock out these functions for testing and use the mocks in your testing function.

6. Read the instructions and hints for step 7b.

a) What property would you define to test random_location_orientation?

b) Identify the transitions for which the test_transition code would need to check a property rather than a direct comparison of values.

7. Including the pre-lab and writing this document, how long did this lab take you? You will not be judged on your answer for this question, the numbers are just being used for calibration purposes for later labs and future course offerings.

Turn in the code (entire Arduino project zipped up) and the document (saved as a PDF; separate from the zip of the code) on Canvas.