Brown CS News

“While natural intelligence often learns sophisticated skills through simple visual observation,” says Brown CS faculty member Chen Sun, “current artificial intelligence (AI) systems largely lack this ability. Most modern AI systems require massive amounts of text or human-labeled data to understand the world, a dependency that limits the ability of machines to perform complex physical tasks that are difficult to describe in words.”

To address these limitations, he’s just received a National Science Foundation (NSF) CAREER Award to create a scientific framework that allows machines to learn directly from passive observations and active interactions with the physical environment. CAREER Awards are given in support of outstanding junior faculty teacher-scholars who excel at research, education, and integration of the two within the context of an organizational mission.

“This research,” Chen explains, “establishes a new paradigm for machine intelligence centered on the concept of adaptable video blueprints. These blueprints function as a representation that allows an agent to translate a visual experience into a sequence of physical actions generalizable across diverse tasks, environments, and embodiments.”

Three integrated thrusts drive the technical approach:

1. Developing visual planners that utilize video generation models to causally predict the future states necessary to reach a target goal

2. Focusing on inverse dynamics to map these predicted visual sequences into specific motor commands

3. Implementing an automatic self-improvement loop, allowing the agent to refine its planning and execution through continuous experience and adaptation 

“The goal,” says Chen, “is to advance the fields of computer vision and AI by grounding visual generation in physical interaction and providing a scalable method for machines to acquire sophisticated skills with minimal human intervention.”

A member of the Brown CS faculty since 2021, Chen leads the PALM research lab, which focuses on learning generalizable temporal dynamics from unlabeled videos, whether as multimodal concepts, human behaviors, or raw pixels, and then exploring applications of these video-centric world models in robotics.

Chen thanks his many mentors, colleagues, and collaborators across Brown CS and Brown for their generous support: “I want to take the opportunity to express my gratitude for our amazing Department. My students have been extremely patient with me, and are brave enough to spearhead fun research projects that they truly love and believe in. I am really lucky to have the opportunity to work with them, and their wonderful research and persistence are the very reasons why a proposal could be written. We have projects that took two years to go from ideas to publication, and we also have projects whose results were so surprising to me that I thought there must be bugs in them!”

Chen joins numerous previous Brown CS winners of the award, including (most recently) Ritambhara Singh, Peihan Miao, Vasileios Kemerlis, Srinath Sridhar, and Malte Schwarzkopf.

For more information, click the link that follows to contact Brown CS Communications Manager Jesse C. Polhemus.