Object Seeking
Barbara Korel
bkorel@cs.brown.edu

Introduction

The task to accomplish for the Object Seeking assignment is to design and implement a control client that uses its visual sensing to detect a specified object, drive towards that object, and determine when it is within close proximity to the object. These basic tasks are extended to model moderately complex robot behavior to continuously seek and find objects in a sequential order for a set of different objects. This is a relevant and interesting problem because object seeking aims to build an intelligent robot that is able to extract information from its environment and use this perceived knowledge to move in a meaningful and purposive manner to achieve an end goal. By creating a mechanism for object recognition through Snappy's camera sensor, implementing both a fixed, reactive controller to handle decision making and a motion controller, we propose that in a controlled environment, Snappy has the ability to continuously seek and drive among a set of objects.

Approach and Methods

The problem of seeking and driving towards objects continually is solved using a combination of three approaches: visual color sensing to recognize objects, a finite state machine to handle decision making and a proportional-derivative controller for implementing motion control. Object recognition is achieved using the Player blobfinder proxy which performs color segmentation based on previously defined calibrations of pixel value ranges for specified colors. The proxy returns bounding boxes around solid color image regions that Snappy's camera senses; Snappy uses the detected blobs to determine which (if any) blob represents the currently desired object. Snappy's representation of the object is the blob with the largest area that matches the color description of the current object and is in the current visual stream. The below image is the data (blobs) Snappy senses of each object.

A finite state machine serves as Snappy's decision making controller. A FSM is a fixed, reactive controller that lets Snappy autonomously make decisions based on his current state and specified post conditions that allow him to transition to the next state. Each individual state encapsulates its own behavior and goal. Together the entire FSM is used to exhibit the decision making of moderately complex behavior: to sequentially seek different objects and move purposefully to reach them. The below figure is the internal representation of our FSM.

Desired Object no Longer in Visual Stream

Desired Object in View

On Exit: Seek Next Desired Object in Order

Distance to Desired Object in Close Range

Input:
- object order
- first desired object

Snappy maintains a global variable which represents the current state of the robot. He begins in the Search state and remains so until the desired blob is found; in this state Snappy simply rotates in place. In the MoveToObj state, Snappy moves towards the object by a certain amount determined by the PD controller. If the blob is no longer in sight, Snappy maintains a history of the last seen blob to determine the optimal direction to turn; if the most recent blob was on the right side of the screen before leaving the visual stream then Snappy will rotate right, and vice versa. Within the MoveToObj state is a bump sensor check; if Snappy hits a wall or another object en route to the object, he will reverse slightly at an angle. However in the controlled environment of the 404 lab, Snappy should never hit a wall since he should not perceive any blobs on the bare white walls. This functionality is more practical as a means to move about another robot.

A proportional-derivative controller is the basis for implementing motion control such that when Snappy is driving towards the desired object, he is able to center the blob in his visual stream and drive forward smoothly and accurately towards the object. The feedback controller has a notion of “error” which is essentially how far off Snappy is from his goal (the goal in this case is getting the blob centered in his view). The PD controller measures the distance to the goal and allows for minimizing the error and getting closer to the goal; this must be done without overshooting how far Snappy must rotate, causing him to oscillate at his drives towards the object. In our implementation, the output of the PD mechanism is a control element that represents a corrective action for centering the blob appropriately, and this element is explicitly used to set the rotation speed as Snappy is driving towards the object. Overall the approaches we have used to implement object seeking involve color blob finding, a FSM and a PD controller to successfully handle: object recognition, decision making and motion control. All approaches are fairly straightforward and would be simple to reproduce in terms of the algorithms.

Experiments and Results

The continual object seeking task of Snappy was tested multiple times with the following conditions varying in our controlled test environment: the placement of objects around the room, the input order of object sequences, and stationary object placement verses moving object position. By testing the time it took Snappy to recognize each currently desired object in the visual stream, the time to drive to the desired object, if he stopped before reaching the object and if he was able to follow a moving object (when applicable), Snappy successfully demonstrated his object seeking capabilities. The following test outlines our results:

Test 1 – Test 1 used the five recognizable objects (2 yellow balls were in the test) that were placed around the room and left stationary. This test shows that each object was found, driven to, and the next object was successfully sought after.
Input Order = {green/orange fiducial, yellow ball, pink fiducial, orange/green fiducial}
Object Setup:


Video: YouTube - Snappy Object Seeking 1
Results:

SEGMENT	SEEK TIME	DRIVE TIME	STOPPED BEFORE OJB
1	0 (3 sec)	7 (5 sec)	True
2	10 (3 sec)	14 (4 sec)	True
3	18 (4 sec)	24 (6 sec)	True
4	25 (1 sec)	32 (7 sec)	True
5	33 (1 sec)	44 (11 sec)	True
6	45 (1 sec)	56 (11 sec)	True
7	60 (4 sec)	65 (5 sec)

Test 2 – Test 2 used the same object placement as Test 1, however a different input ordering is specified to show the ability of Snappy to recognize objects in order based on any sequence.
Input Order = {yellow ball, green/orange fiducial, orange/green fiducial , pink fiducial}
Object Setup: same as Test 1
Video: YouTube - Snappy Object Seeking 2
Results:

SEGMENT	SEEK TIME	DRIVE TIME	STOPPED BEFORE OBJ
1	0 (3 sec)	7 (4 sec)	True
2	9 (2 sec)	18 (9 sec)	True
3	23 (5 sec)	30 (7 sec)	True
4	36 (6 sec)	42 (6 sec)	True
5	44 (2 sec)	45 (1 sec)	True
6	48 (3 sec)	56 (8 sec)	True
7	61 (5 sec)	69 (8 sec)	True
8	74 (5 sec)	80 (6 sec)	True
9	82 (2 sec)	83 (1 sec)	True

Test 3 – This test involved objects placed in a different arrangement around the room to demonstrate Snappy’s flexibility to recognize object’s anywhere, independent of location within the room. A new object ordering is specified. Additionally this test shows how Snappy will follow an object even as it is moving. On Snappy’s fifth segment, the yellow ball is picked up, in which Snappy successfully continues to follow it, he smoothly turns 180 degrees around while still keeping the ball in his visual stream, and finally successfully drives towards the ball until he has reached within a close range of the object, in this case, into the goal!
Input Order = {yellow ball, green/orange fiducial, pink fiducial, orange/green fiducial }
Video: YouTube - Object Seeking with Snappy 3, preview of goal scoring ability!
Results:

SEGMENT	SEEK TIME	DRIVE TIME	FOLLOWED MOVING OBJECT	STOPPED
1	0 (3 sec)	7 (4 sec)	n/a	True
2	10 (3 sec)	14 (4 sec)	n/a	True
3	15 (1 sec)	20 (5 sec)	n/a	True
4	30 (10 sec)	37 (7 sec)	n/a	True
5	43 (6 sec)	64 (11 sec)	True	True

Overall Snappy performs successfully in all the experiments; with 100% correctness Snappy recognized and drove towards the correct object and in the correct order. Even though Snappy recognizes 100% of objects, periodically it took Snappy a couple tries to find the correct blob associated with the object. This primarily only happened when the green/orange fiducial was located on the other side of the room from Snappy. In such a case it took a little longer for Snappy to consistently receive the correct blob from the blobfinder proxy; however once he was closer to the object he consistently recognized it. The only other problem encountered was occasionally Snappy would touch the object he was seeking; this is due to the effect of blobs being broken up into many smaller blobs covering the object. Since the stopping criteria is based on the dimensions of the largest blob Snappy sees, this causes Snappy to get a little too close to the object at times. Thus, even in a controlled environment with set lighting, known objects, a defined color calibration and the same implementation, Snappy always performed differently with every run. This is mostly due to the inconsistent nature of the blobs Snappy sees in his visual stream. At every moment Snappy sees a different number of blobs each with different sizes; this determines what state Snappy is in and how Snappy moves. Thus no, even with fairly straightforward algorithms and implementation, our results are not reproducible.

Conclusion and Discussion

Our implementation, which uses color sensing for object recognition, a FSM as the decision making controller and a PD controller to appropriately move Snappy, has successfully given Snappy the autonomous capability to continually seek and reach objects in a controlled environment. Snappy recognizes all objects, in not only a timely, but also efficient manner by keeping track of the relative position of the last blob in the case the sought after blob is removed from his visual stream. The primary strength of this approach lies within the simple representation of a reactive controller to guide Snappy’s decision making; Snappy clearly only handles one task at a time dependent on what state he is in, and he only transitions to the next state after meet clearly defined post conditions. Together the FSM achieves the overall goal of object seeking and successfully models fairly complex behavior. A shortcoming of this approach is the dependence on the controlled environment which Snappy can perform in, causing him to not adapt well to different environments. This is solely due to the specific color calibration in the 404 lab with the set objects and lighting; even if the door to the lab is open and sunlight in shining in, Snappy will not recognize certain objects effectively. The object seeking task could be improved using additional sensing from the camera beyond just color sensing to accurately recognize objects; in such a case Snappy may not be so restricted to the 404 Roomba lab environment.