Assignment: Debugging

Create your github repo using this link

This assignment focuses on methods and strategies for debugging robots.

Most of the time spent writing a program is spent debugging that program. This issue is particularly challenging for robotics because a robot will not work unless everything else works. During the development of this drone project, we have had our robots fail because:

  • a bug in our program
  • a bug in the library we were calling. (The original version of the library we were using to talk to the flight controller had *many * bugs that we fixed.)
  • bad electrical wiring (so the motors spun backwards).
  • inadequate cooling. (Our Raspberry Pis were burning out until we put heat sinks on them. We tried several other fixes before deciding this was probably the issue.)
  • inadequate circuits. We tried using the battery eliminator circuit that comes with the frame. It powered the Pi for a while, but then got really hot when we started driving the motors. It burned out the Pi’s power supply. As a result we are sticking with a separate BEC that can handle more current.

Despite ours and your best efforts, you will encounter bugs when building and flying your drone. We expect this to happen, and part of our goal for this assignment is to teach you methods and strategies for debugging a robot. Fundamentally, debugging is about checking your assumptions and localizing the problem. You need to be systematic and verify each part of the system is working (or not) when finding a bug.

Often, bugs are present in a sequence: one bug masks a second one. So if you fix the first bug, it still doesn’t work, because now a second problem comes into play. Don’t let this get you down! Expect it. As you work on each project, you should expect that you did ten things wrong, that you’ll have to find and fix. So if you find one thing and fix it, expect that there are nine more things you’ll have to fix before you can fly.

Read Richard Feynman’s 1974 Caltech Commencement address, entitled Cargo Cult Science. In many ways carrying out experiments in science is like debugging a robot. The example of the experiment by Young in 1937 about rats running through mazes is a beautiful example of debugging. In both cases you are carrying out experiments in order to test hypotheses to determine the problem.

Below we present four strategies for debugging that are useful to try when you encounter a problem.

Decompose the Problem

Decomposing the problem means breaking it down to smaller components. The robot doesn’t work? Have you verified that each part works? Does the Pi power on? Is your flight controller talking to the motors? Can it connect to your laptop via CleanFlight? Does the IR sensor light turn on?

You want to try to isolate which parts are working and which parts are not working in order to zero in on where the bug is. To decompose the problem it is essential to be systematic and think through ways to check each part of a system that is failing, separately.

Visualize the State

To figure out what is wrong, it helps to visualize the state of the robot and the system. Can you see the output from each sensor? Is the output what you expect? Is there a “human friendly” way to draw what is going on? Often one spends as much time writing visualizers as one does implementing the algorithm or functionality on the robot. An incorrect transform might be impossible to debug if you print out the matrix, but instantly obvious as soon as you draw it in a 3D visualizer.

Break the Abstraction Barriers

Bugs don’t respect abstraction barriers, and you shouldn’t either! The Law of Leaky Abstractions applies here. As you decompose the problem, you might find that all the code you wrote is correct, and the actual bug lies in some other module. In the course of developing the drone, we had to fix bugs caused by insufficient swap space on the Raspberry Pi, incorrect implementation of the MSP serial protocol used to talk to the drone, and more. If decomposition tells you that all your parts are working, then continue working to isolate and find the problem in some other module, even if you didn’t write it.

Slow Things Down

Things happen fast on a robot, often too fast to see. It helps to find ways to slow things down. You can look at a recorded log of what happened and play it back slowly. Or you can write software to carry out each step in isolation and slowly enough that you can verify its correctness.

Problem 1: Decompose the Problem. (25 points)

For each condition, 1) Describe a test to verify it is working. Then 2) carry out your verification on the drone and describe the results of your test. 3) Change something about the drone so that just that component stops working. Describe what you changed to cause it to stop working. 4) Carry out your test again and describe the results.

For example: Verify the Raspberry Pi has power.

Example Answer: 1) Verify the red power light is visible when the Pi is receiving power. Look in the Raspberry Pi manual for the meanings of the various LEDs. 2) When I plug the battery into my drone, the red power LED turns on. 3) When I unplug the header from my drone, it doesn’t receive power. 4) When I plug the battery into the drone but the header is unplugged, the Pi doesn’t receive power, and the red power LED doesn’t turn on.

  1. The motors are powering on.
    Answer: 1) Take the props off and plug the motors into CleanFlight on your laptop. Verify they power on in Cleanflight. 2) When I power up the motors on my drone, they spin. 3) When I unplug power from the motors but still power on CleanFlight, the motors don't receive power. 4) When I power them on in Cleanflight, they don't spin up.
  2. The Pi is receiving data from the camera
    Answer: 1) Use raspistill or raspivid to capture data, and view the output by sending it to the base station. 2) When I do this on my drone I see an image from the camera. 3) When I unplug the camera connector, I get the camera isn't sending data to the Pi. 4) When I try raspistill it gives an error message.

Problem 2: Visualize the state (25 points)

For each condition, use the Javascript interface to visualize the state or output. Describe a procedure to verify the output works as expected. For example, how would you move the drone or under what conditions would you test the sensor? Actually do so using your drone and describe the results.

  1. The output from the camera. How does it work on different lighting conditions?
    Answer: Use the ROS JS interface. Point the camera at something, and verify it looks as expected. Things to pay attention to include aperture, white balance and gain. Try different backgrounds and see how it changes.
  2. The IR sensor reading. How does it work with different materials? What are the minimum and maximum distances?
    Answer: The ROS JS interface, or you could use rostopic echo to print the output. Try putting an object such as your hand close to and far away from the sensor. What are the minimum and maximum ranges of the sensor? Can you find a material that it does not detect?
  3. The IMU. How does it behave when you move the drone? Can you detect a difference before and after calibration?
    Answer: Use the ROS JS interface. Move the drone and verify the IMU shows acceleration in the appropriate directions. Rotate the drone and observe the gyro reading and verify it is the correct orientation. Verify that when the drone is still, the readings are not moving.

Problem 3: Slow Things Down (25 points)

In order to fly, the drone must read sensor data and output motor commands at a very high frame rate. However it is often hard to see what is happening since it is changing so fast. For both of the following assignments, you should not need to write any ROS code. In both cases we are looking for relatively short programs that talk to the respective hardware module.

  1. Write a program to arm the drone, then wait 10 seconds, then disarm the drone. Verify your program runs. Look at for an example of how to talk directly to the controller without using ROS. The flight controller speaks Multiwii Serial Protocol.
  2. Write a program to read a single frame from the camera, save it to a file, and return, without using ROS. Verify your program runs, and include your picture in the project write-up. You can use as a model.

Problem 4: Break the Abstraction Barriers (25 points)

In embedded computing, often the LEDs give important information about the underlying components, as do audible beep codes. Note that the CleanFlight software and the ESCs spin the motor at high frequencies in order to generate audible beeps.

  1. Find the LEDs on the Rasberry Pi. What does each LED mean? What do they mean? What happens to the LEDs if the SD card is not plugged into the Pi?
    Answer: The red LED indicates power. The green LED indicates activity. If there is no SD card plugged in the red power LED will turn on but not the green activity light.

  2. Find the manual for the Skyline 32. What LEDs does it have? What do they mean? What happens if the Skyline is not receiving power?
    Answer: The Skyline Manual has the ESC meanings. It has a green status LED, a red mode LED, and a blue power LED. If the Skyline is not receiving power, the blue power LED will not turn on (and any other LED either).

  3. Find the manual for the ESCs. (We couldn't find the 12A manual so use the one for 30A.) What mechanisms do the ESCs have to indicate their status?
    Answer: The ESC Data Sheet contains information about the beep codes that indicate power up and various error conditions. They also describe the behavior when no input is detected, the ESCs beep to locate in case of a crash. Additionally the LEDs have blinking codes to indicate various states, including whether a hardware check has failed.