The purpose of this project is to develop new techniques for computer-assisted exploration and instruction of formal methods in mathematics and computer science. The primary research objectives are (1) to develop methods for human-computer interaction that facilitate fluid direct manipulation of abstract objects such as diagrams and mathematical symbols, (2) to develop methods by which formal arguments can be developed by gradual refinement of incomplete and/or informal concepts, and (3) to create an electronic environment that is preferable to paper for doing mathematics in many more circumstances than at present.
These goals will entail the combined use of hand gestures and speech, as well as more traditional input mechanisms such as pointing devices. One of the most fundamental tasks in this project is the recognition of handwritten symbols and "gestures" produced by means of a pointing device, a 6 degree-of-freedom position tracker, or unconstrained hand positions and motions. Toward this end, we have developed a character recognition module that achieves a very respectable accuracy rate (better than 90%) on a large alphabet of symbols (more than 60) and across multiple users (at least four). Created as part of a student work-study project, the system was initially intended as a quick prototype with which to experiment. By combining a number of known techniques, it exceeded our expectations rather dramatically, and will provide a solid foundation for online handwritten symbol recognition.
Concurrent with the development of the above software, another project has been developing a flexible set of tools for manipulating mathematical typography, resulting in a "smart text" module that is capable of scaling and mixing multiple fonts, and creating layouts appropriate for mathematics. We are currently working on a first prototype of a system in which simple symbol manipulation is performed exclusively through pen input and hand gestures acquired through passive vision techniques. The manipulation will likely involve only algebraic simplification at first.
The new tools will be integrated together using a prototype system constructed by Jim Arvo and Kevin Novins (and partially supported by the New Zealand Ministry of Research in Science and Technology) as well as a system for gesture-driven vector calculus created as a summer project by a Caltech undergraduate.
Full Research Bibliography