Chen Sun Receives A Salomon Award For Learning From Astronaut Spacewalk Videos
- Posted by Jesse Polhemus
- on April 24, 2023

A new project ("Learning Implicit Structured Neural Network Representation by Watching and Listening to Astronaut Spacewalk Videos") by Brown CS faculty member Chen Sun has just received a Richard B. Salomon Faculty Research Award. This honor, given annually by Brown University’s Office of the Vice President for Research, was established to support excellence in scholarly work by providing funding for selected faculty research projects of exceptional merit, with preference given to junior faculty in the process of building their research portfolio.
"Structured and compositional concept learning," Chen explains, "is essential for principled generalization in Artificial Intelligence, such as deploying a robot to a novel environment. In this proposal, we ask the research question: can we learn implicit structured representations in a generic, multi-task, and end-to-end trained neural network? Our hope is that the implicit representation achieves similar compositional generalization behavior as the explicitly structured concepts. Solving this question would enable us to apply a flexible and unified neural network to jointly perform perception, reasoning, and planning, all of which are crucial for robotics manipulation."
The proposal has three research goals:
-
Learning explicit and implicit structured representation from multimodal data, such as images and captions, or videos and speech, by leveraging existing pretrained foundation models.
-
Collecting a new video-language benchmark with egocentric view of finegrained human activities and detailed speech transcript, based on astronaut spacewalk videos publicly available.
-
Evaluating the proposed learning framework on the spacewalk videos, to discover objects, actions, and skills from the narrated videos.
The proposed research is an important building block and the first concrete step for a multi-PI research collaboration. This project, led by Brown CS faculty member Stefanie Tellex, will include Brown CS faculty members George Konidaris, Ellie Pavlick, Srinath Sridhar, and Chen. Ultimately, they hope to transfer skills discovered from video observations (for example, how astronauts manipulate objects and accomplish tasks following human instructions) to robots (how they can learn policies to manipulate objects and accomplish tasks). The findings from the proposed project will serve as validation and preliminary results for a multi-year, multi-PI, interdisciplinary research proposal.
A full list of Richard B. Salomon Faculty Research awardees is available here. Chen joins multiple previous Brown CS winners, including (most recently) George Konidaris, Anna Lysyanskaya, Ritambhara Singh, and Malte Schwarzkopf.
For more information, click the link that follows to contact Brown CS Communications Manager Jesse C. Polhemus.