Erich C. Deise (edeise)

Team Grouchy: (dsblack & jlin)

CS148: assignment 2 – object seeking

Introduction & Problem statement

The problem set for the team, quite generally, was to construct a finite state machine capable of directing a SmURV robot in object seeking behavior. This problem, however, is decomposable into three sub-problems – each presenting its own set of challenges. The first challenge was to calibrate the SmURV's blob-detection device (via interface with its unibrain firewire camera) so that it would be able to adequately detect unique color blobs. So doing provides the SmURV with a (very) rudimentary visual perceptual system which (when successful) enables the SmURV to discern particular objects (by associated blob-color) in its environment. The team next considered the simpler puzzle (I.e., the second sub-problem) of how the SmURV might be controlled such that it would be capable of engaging in the tracking/following of a single object. This, by necessity, requires that the SmURV be capable of both perceiving objects (of the relevant sort) and tracking them as they are moved about in the environment. Finally, and building incrementally upon the previous puzzles, the team considered the third question of how the SmURV might be controlled such that it would be capable of (1) perceiving, recognizing and discriminating among/between a set of distinct (I.e., YUV distinguishable) objects (namely, a yellow ball, a pink fiducial, a green over orange fiducial and an orange over green fiducial) and (2) ultimately “visiting” (I.e., getting the SmURV acceptably near) each object in any order desired.

Given the aim of (mobile) robotics, it is necessary to contend with the general problem of how object-seeking behavior is to be computationally instantiated. The problem is, for that quite general reason (and at that level of granularity) of interest. The sub-problems or composite puzzles/challenges considered are likewise of interest individually. First, the puzzle of how perceptual systems are to be computationally implemented is of obvious interest. Before a system can interact intelligently with its environment it must be capable of (at least somewhat intelligently) perceiving and representing the objects in its environment in ways that are (computationally) exploitable. The problem of single object following/tracking presents a challenge that is of interest for an analogous set of reasons. Specifically, for a robotic system to behave intelligently it must not only be capable of perceiving and representing (aspects of) its environment but it must also be capable of interacting with (objects in) its environment in a reasonably intelligent manner. So doing requires – at the very least – the capacity to discern, track and follow desired objects. Finally, contending with the problem posed by the third challenge is of interest since we do not wish our robots to be functional in environments that are entirely contrived or otherwise significantly constrained – e.g., limited to only one target object. By contending with the problem of how to discern a “target” object from a set of other objects, a greater degree of flexibility might be added to the system.

Approach & Methods

Since the problem is decomposable into three sub-puzzles, approaches and methods for each will be discussed separately. With respect to the first challenge, that of blob-finder calibration, our approach was to obtain YUV data from each of the target objects (specifically, the yellow ball, pink, green over orange and orange over green fiducials) under a limited set of lighting conditions. Specifically, color ranges were obtained for Pink, Orange, Yellow and Green objects (via the Playercam_YUV proxy). YUV ranges were obtained for each object under the lighting conditions in room 404 of the CIT. As the lighting in room 404 is not uniform, (e.g., corners tend to be less illuminated) target objects were placed in various locations within the room and samples taken from each. Likewise, YUV samples were taken of each object from various distances. The teams approach to color calibration was to manually (via the playercam interface) obtain YUV data for each object/color – eliminating any obvious “out-lier” values. Corrected high and low Y, Cb and Cr values were then recorded and ranges set. These ranges were then tested and the successes (or failures) noted. Through multiple iterations of the sampling and testing cycle (under the lighting/distance conditions discussed above) a workable set of YUV ranges were obtained that allowed for the detection of yellow, pink, green and orange objects.

While an initial SmURV controller was constructed for the simple single object following, this approach was ultimately reworked and expanded to account for the multiple object case. However, before considering the teams more fully developed controller, some discussion of the simple single-object following system might be of use. Specifically, in contending with the simpler (single object tracking) challenge, a proportional derivative control mechanism (PD-controller) was devised. The PD-controller enabled the SmURV to correct its approach such that the target object (I.e., the largest blob of a given color) is maintained in the center of the SmURV's camera. Rotation of the SmURV's plant – when attempting to “center” the largest target colored blob (I.e., object) - was controlled by a simple proportional derivative control device. Similarly, in order to prevent the SmURV from over-running the target, a similar PD-control procedure was employed to direct the robots linear velocity. Specifically, the robots forward velocity is a function of its change in distance over time (I.e., error) from the target object. This device results in the SmURV's linear velocity proportionally decreasing as it nears the target object. Taken together, by exploiting PD-control structures to direct the SmURV's rotational and linear speed, stable single object tracking behavior was achieved.

In contending with the multiple object discernment, tracking and seeking challenge, a finite state architecture was decided upon to direct the SmURV. Quite generally, each behavioral state was marked by 1) a set of preconditions (I.e., qualifications) that must be met/satisfied; 2) an action or sequence of actions to be undertaken; 3) a set of postconditions describing the success/satisfaction conditions of that state and 4) a description of the successor state. And so, once the target order is set, the SmURV is directed to scan its environment searching for objects that satisfy the preconditions associated with that target. Specifically the SmURV is directed to scan for persistent (I.e., “seen” at least some number of times greater than a fixed value) blobs of the target color with an area greater than a fixed value. Two scan/search processes are employed – one for single color objects and another for multi-colored object. The former searches for blobs of a single color, returning the centroid coordinates of the largest found. The latter searches for two blobs (of at least some minimal size), the centroids of which are in the proper alignment (e.g., green above orange) and returns an approximation of the centroid of both. Once the target is obtained, the seeking behavioral state is visited. While seeking (moving toward the target), if the SmURV loses sight of the target (I.e., if the area of any target blob perceived falls below a fixed value) it will revisit the scanning state – ultimately either regaining sight of the target or entering into a timed phase of wandering behavior followed by another iteration of scanning. Likewise, if in the process of target seeking the SmURV's bumper is struck, another action state is entered whereby it undertakes a rudimentary obstacle avoidance sequence followed, once again, by another iteration of scanning and target acquisition. Success in “visiting” a target is determined when the set of postconditions set for each target are met. In order to compensate for deficiencies in the camera (notably problems associated with auto white balance to be discussed subsequently) a target was consider visited when the area of its associated blob was greater than a set/fixed value. Once the set of target specific postconditions are satisfied (I.e., the target object is successfully “visited”) the next target is brought to bear – and the SmURV is redirected to undertake a scan of its environment looking for objects satisfying the preconditions set for it in the manner outlined above. In this manner and by means of this architecture, we conjectured that the SmURV should be capable of visiting any number of properly blob-calibrated target objects in any order desired.

With respect to the general control architecture discussed here, there is no reason to think the finite state machine implemented nor any of the particular algorithms employed (e.g., scanning, wandering, obstacle avoidance, blob-detection, proportional control) to be unreproducible. Each state is provided with a full description of preconditions, actions, postconditions and successor states. Likewise, all preconditions, postconditions and actions are themselves fully specified. Finally, successor states are fully described for each behavioral state. Taken together, given the simplicity, transparency and specificity of both the overall control architecture implemented and specific algorithms employed we have every reason to think the control system outlined here to be a highly reproducible one.

Experiments and Results: blob calibration

YcbCr (YUV) values/ranges were obtained from each of the target objects under the conditions present in room 404 of the CIT. As illumination is not uniform, sample data were taken from objects in a number of locations in the room (e.g., directly under the lights, in the corner). Due to technical difficulties with our team's SmURV, an initial set of color range data had to be obtained from the SmURVs “enamored” and “Hefty.” Data gathered from Hefty was eliminated from further consideration once it was noted that its camera differed from that of the SmURV Grouchy. Under the assumption that identical makes and models of camera should provide suitably fungible color data (ranges and values), the data obtained from Enamored was taken as our initial working data set. This proved to be a very poor assumption. Once the data ranges obtained on Enamored's camera were tested on the SmURV Grouchy, it became obvious that this data would need to be rejected and new values obtained.

Calibration proved to be a very difficult task. Specifically, obtaining YUV ranges that would result in robust and reproducible object detection proved quite troublesome. While no tests were undertaken to confirm the hypotheses, it appeared to the team that YUV data ranges for target colors depended on (1) how long the camera had been operating (2) the life/voltage of the battery. Specifically, both of these factors seemed to result in an over application of the camera's auto white balancing feature. This resulted in target/colors coming to be increasingly “washed out” (I.e., being seen as white by the camera) over time. As such, data obtained early in each sampling trial would, after a period of time (approximately 10 minutes), no longer serve to properly discern target objects. This effect, when coupled with the loose USB connection between the camera and the SmURV “Grouchy's” ITX which resulted in sporadic interruption of and/or distortion of the camera feed, resulted in our having significant difficulties in target calibration. Any future use of the blob-finder proxy will require our overcoming the noted auto-white-balancing effect. Further investigation is, with respect to these concerns, needed. However, when fresh (I.e., provided with a charged battery and a “rested” camera), the YUV data ranges adequately mark blobs for all target colors. This is evidenced by the SmURV's successes in both the object seeking and following tasks. (c.f. The video links provided earlier). That noted, we have, given that the YUV range data obtained from the SmURV enamored failed remarkably when applied on Grouchy, good reason to think our calibration data to be - at the very least – suspect and likely unreproducible. Contending with calibration issues will likely be an ongoing enterprise.

Experiments and Results: object tracking and seeking

With respect to the single object tracking task, we set the SmURV the task of following each of the target objects (yellow ball, pink fiducial, orange over green and green over orange fiducials). As noted above, when provided with a “rested” camera and freshly charged battery, the SmURV was capable of tracking each of the target objects. In order, however, to contend with some of the camera difficulties (notably the loose USB connection), we were forced to reduce the SmURVs rotational and linear speed. And so, while the SmURV did succeed in tracking each of the four targets it did so slowly. For video of the SmURV's single object tracking behavior please see http://cs.brown.edu/people/dsblack/csci1480/asgn2_follow.avi

With respect to the multiple object discernment and seeking task, we tested our controller by having the SmURV (controlled in the manner discussed previously) visit each of the four targets in a random order. All tests were conducted under normal lighting conditions in room 404 of the CIT. The SmURV readily acquired all targets and succeeded in all trials in traveling among/between objects. The SmURV's capacity to mark the distinction between (I.e., distinguish visually) the orange over green and green over orange fiducial was next tested. The team found that when the above noted camera troubles were accounted for, the SmURV had no difficulty in differentiating the two. And so, with the exception of the troubles arising from the auto-white balance effect (specifically, the tendency of pink and orange to wash out to white over time) the SmURV controlled in the manner outlined, is remarkably successful in acquiring and visiting any and all targets. Again, as with the object tracking task, in order to compensate for the camera troubles noted, the SmURV was directed to run at a fairly slow (rotational and linear) velocity. No empirical studies were undertaken to determine the speeds (rotational and linear) at which the camera troubles become truly troublesome. That is, no attempt was made to determine how fast is too fast for this particular SmURV. Determining this maximum speed is a topic for future investigation. For video of the SmURV undertaking multiple object seeking please see http://cs.brown.edu/people/dsblack/csci1480/asgn2_seek.avi

Study and testing of the particular controller employed was, as noted, stymied to a significant extent by troubles with the camera. That noted, however, given the successes obtained with the SmURV grouchy, we have reason to think that the controller should result in robust single object tracking and multiple object seeking behaviors in other similarly constituted robots.

Conclusion and Discussion

Given the successes of the controller outlined in directing the SmURV in single object tracking and multiple object seeking behaviors, we have reason to think the architecture proposed and algorithms employed to be a satisfactory way to contend with the challenges posed. Given the robustness of the behaviors (objects seen are always tracked and visited), we have likewise, good reason to think this a suitable way to proceed in contending with similar challenges. Less clear, however, is the success of our calibration technique.

With respect to the strengths of the approach outlined here, a number of points bear mention. First, in spite of the camera difficulties discussed, the controller resulted in robust object seeking and tracking behavior. Similarly, the simplicity and transparency of both the overall architecture and the particular algorithms employed when, taken in conjunction with the practical successes of the implementation, provide additional reason for thinking it a suitable means by which to contend with the particular challenges posed.

However, there are a number of drawbacks to this particular implementation that should be discussed as well. As noted above, the SmURV is directed (in response to camera difficulties) to engage quite slowly in seeking and tracking behaviors. This is a significant problem – particularly in light of the fact that the ultimate goal is one of robot soccer – an activity where speed will presumably be of great importance. Similarly, in light of these same camera troubles, the SmURV is directed to stop (I.e., its postconditions for visiting a target are met) at some distance from some targets. This effect is most apparent when the SmURV “visits” the pink fiducial. This is, we conjecture, most likely due to the camera over zealously auto white balancing the pink (and to a lesser extent orange) objects. So doing results ultimately in these colors/objects being un-perceivable (I.e., seen as white) at ranges under 2 feet. How best to close this “visiting” gap is another topic for further investigation. Furthermore, the particular obstacle avoidance procedure implemented is quite simplistic and can result in the robot becoming trapped by particular environmental configurations. This is easily rectified. Finally, the SmURV is directed to engage in a wandering behavior if it cannot (visually) acquire a given target with a fixed amount of time. This wander is not random and can result in the SmURV overrunning the target. This too is something that can be fairly easily rectified.