CSCI 2951-I:
Computer Vision for Graphics and Interaction

Fall 2019, MW 15:00 to 16:20, CIT 101

Instructor: James Tompkin

Sign-up | Course | Schedule/Class Material | General Policy | Feedback | Acknowledgements

<-- Form for Paper Questions -->

Contact

Office/TA Hours

James' office hour (Tues 1pm) appointment slot signups are here (top left).

Course Description

How does computer vision enable new interactive graphical applications, and how can we improve them?

Computer vision strives to understand, interpret, and reconstruct information about the real world from image and video data. Computer graphics models dynamic virtual worlds to be synthesized in realistic or stylized ways. In visual computing, these fields are converging since both disciplines create and exploit models describing the visual appearance of objects and scenes. Interaction allows us to explore these worlds, and to use ourselves and our environments directly as interfaces. Machine learning and deep learning allows us to define mappings between domains across vision, graphics, and interaction, and to generate new data such as images from recombining existing databases. Combined, these disciplines enable applications from the seemingly simple, like semantic photo editing, to the seemingly science fiction, like mixed reality.

In this seminar course, we will discover the state of the art algorithmic contributions in computer vision which make these new applications possible. We will concentrate on recent research results that were published at top-tier conferences and journals from problem fields such as reconstruction of static and dynamic 3D scenes, computational photography and videography, multi-view camera systems, generative methods for image formation, and vision-based interaction devices. Each week, we will read state-of-the-art papers, present them, and discuss their contributions, impact, and limitations. Then, we will develop projects which implement and extend these ideas. Beyond computer vision, this course will help us learn how to quickly interpret and assess academic papers, and how to give effective and engaging presentations.

Please join us!

Yo dawg. We herd you like augmented reality, so we put AR on your AR so your pokemon can pokemon go while you pokemon go.

Learning Objectives

Upon completion of this course, students will have:

  1. Practical experience reading academic papers, and the skill to digest them quickly;
  2. Created effective presentations to explain state-of-the-art techniques, by learning how to critique and how to respond to critique;
  3. Formed, discussed, and evaluated many project ideas, and gained experience creating structured research project proposals;
  4. Developed practical research project skills and demonstrated these on an unsolved computer vision problem;
  5. Familiarity with the state of the art in reconstructing and generating images computationally.

Course Structure

The course is spilt into two halves. In the first half, we will read research papers, present them, and discuss their contributions, impact, and limitations. We will build upon our analyses and discussions to propose projects which would further the state of the art. Then, in the second half, we will try and do it: we will break into teams and implement projects which further the state of the art.

First Half—Review, Ideation, and Proposal

Second Half—Implementation

Grading

Time Commitment

Tasks Hours
In class 40
Paper reading 35
Paper presentation 5
Projects:
—Proposals 10
—Discussion 10
—Implementation 80
Total: 180

Capstone

This class can be taken as a capstone. Talk to James about the expected standard and additional work across the course.

Prerequisites

This is a graduate course, but undergraduates are welcome! As a graduate class, we expect you to be somewhat self-guided; be prepared to read beyond the course material, and to explore and discover for yourself. Students should know something about visual computing before taking this course, e.g., having taken an existing vision, graphics, or deep learning course. Any other interested students should get in touch with James!

Late Submissions and Late Days

Due to the form of the class, there are no late submissions or late days. We expect you to attend every session, but let James know if you have any special requirements. For sickness and other issues of wellbeing, please obtain a note from health services and we will accommodate.


Course Notes

Paper Reading

We expect you to read every paper in preparation for the upcoming presentation and discussion. Reading these papers may be difficult initially, and students are not expected to understand everything. However, students are expected to actively engage in discussions to further their understanding of the presented material, with the help of the instructor and the class, within a supportive and creative atmosphere. Ideas that are developed during the seminar discussion are intended to directly influence your projects.

Paper Presentations

One student presents the work each session and is the designated `primary reviewer'. Given that everyone will have already read the paper, the presentation should aim to critically review the work:

Demos are welcome! Code or executables may be available for the techniques, and you should feel free to show them off. Likewise, for video material, but don’t just play it without providing any insight.

Leading the Discussion

Each session, one student will be randomly selected as the discussion leader. They will receive a digest of the submitted questions before the seminar. Their goal is to briefly summarize the strengths and weaknesses of the technique, raise questions appropriately throughout the discussion, covers future work ideas, and keep order.

Reading References

How to Read and Present Academic Papers:

Further reading material:


Tentative Schedule

Date Topic Reading / Slides More info
First half—Review, Ideation, and Proposal
Wed 4 Sep Intro
Mon 9 Sep How to read and present; submitting questions; lab demos; interest declaration Active Appearance Models, Cootes et al., ECCV 1998 (PAMI version 2001)
Wed 11 Sep Paper session A Morphable Model For The Synthesis Of 3D Faces, Blanz and Vetter, 1999
Mon 16 Sep Paper session Single-image Portrait Relighting, Sun et al., 2019
Wed 18 Sep Paper session Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer, Lasinger et al., 2019, arXiv preprint
Code
Mon 23 Sep Paper session KinectFusion: Real-Time Dense Surface Mapping and Tracking, Newcombe et al., ISMAR 2011
Wed 25 Sep Paper session Real-Time Geometry, Albedo, and Motion Reconstruction Using a Single RGB-D Camera, Guo et al., SIGGRAPH 2017
Mon 30 Sep Proposal session
Wed 02 Oct Paper session DC-GAN: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Radford et al., ICLR 2016
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets, Chen et al.
Mon 07 Oct Paper session StyleGAN, Karras et al., CVPR 2019
Wed 09 Oct Paper session MeshCNN, Hanocka et al., SIGGRAPH 2019
Mon 14 Oct Indiginous People's Day—no class
Wed 16 Oct Paper session PlatonicGAN, Henzler et al., arXiv 2019
Mon 21 Oct Paper session 3D-Aware Scene Manipulation via Inverse Graphics, Yao et al., NeurIPS 2018
Wed 23 Oct Final proposal session
Second half—Implementation
Mon 28 Oct Studio session
Wed 30 Oct Project review 1
Mon 04 Nov Studio session
Wed 06 Nov Reading group
Mon 11 Nov Studio session
Wed 13 Nov Project review 2
Mon 18 Nov Studio session
Wed 20 Nov Reading group
Mon 25 Nov Studio session
Wed 27 Nov Thanksgiving—no class
Mon 02 Dec Studio session
Wed 04 Nov Project review 3
Fri 06 Dec New England Computer Vision Workshop @ Brown
Mon 09 Dec Reading period
Wed 11 Dec Reading period
Mon 16 Dec Final presentations

General Policy

Welcome!

Our intent is that this course provide a welcoming environment for all students who satisfy the prerequisites. Our TAs have undergone training in diversity and inclusion, and all members of the CS community, including faculty and staff, are expected to treat one another in a professional manner. If you feel you have not been treated in a professional manner by any of the course staff, please contact any of James (the instructor), John Hughes (Dept. Chair), Tom Doeppner (Director of Undergraduate Studies), David Laidlaw (Director of Graduate Studies), or Laura Dobler (diversity and inclusion staff member). We will take all complaints about unprofessional behavior seriously. Your suggestions are encouraged and appreciated. Please let James know of ways to improve the effectiveness of the course for you personally, or for other students or student groups. To access student support services and resources, and to learn more about diversity and inclusion in CS, please visit http://cs.brown.edu/about/diversity/resources/.

Prof. Krishnamurthi has good notes on this area.

Quiet Hours

This class runs quiet hours from 9pm to 9am every day. Please do not expect a response from us via any channel. Likewise, we won't ask you to do anything between these times, either, like hand in projects.

Academic Integrity, Collaboration, and Citation

Feel free to talk to your friends about the concepts in the projects, and work through the ideas behind problems together, but be sure to always write your own code and perform your own write up. You are expected to implement the core components of each project on your own, but the extra credit opportunties often build on third party data sets or code. Feel free to include results built on other software, as long as you credit correctly in your handin and clearly demark your own work. In general, if you use an idea, text, or code from elsewhere, then cite it.

Brown-wide, academic dishonesty is not tolerated. This includes cheating, lying about course matters, plagiarism, or helping others commit a violation. Plagiarism includes reproducing the words of others without both the use of quotation marks and citation. Students are reminded of the obligations and expectations associated with the Brown Academic and Student Conduct Codes.

Accommodations

Brown University is committed to full inclusion of all students. Please inform me if you have a disability or other condition that might require accommodations or modification of any of these course procedures. You may email me, come to office hours, or speak with me after class, and your confidentiality is respected. We will do whatever we can to support accommodations recommended by SEAS. For more information contact Student and Employee Accessibility Services (SEAS) at 401-863-9588 or . Students in need of short-term academic advice or support can contact one of the deans in the Dean of the College office.

Mental Health

Being a student can be very stressful. If you feel you are under too much pressure or there are psychological issues that are keeping you from performing well at Brown, we encourage you to contact Brown's Counseling and Psychological Services. They provide confidential counseling and can provide notes supporting extensions on assignments for health reasons.

Incomplete Policy

We expect everyone to complete the course on time. However, we certainly understand that there may be factors beyond your control, such as health problems and family crises, that prevent you from finishing the course on time. If you feel you cannot complete the course on time, please discuss with James Tompkin the possibility of being given a grade of Incomplete for the course and setting a schedule for completing the course in the upcoming year.

Electronic Etiquette

Laptops are discouraged, please, except for class-relevant activities, e.g., to help answer questions and show items relevant to discussion. No social media, email, etc., because it distracts not just you but other students as well. Read Shirky on this issue ("Why I Just Asked My Students to Put Their Laptops Away"), or Rockmore ("The Case for Banning Laptops in the Classroom").

We will release course lecture material online. In considering laptop use for note taking, please be aware that research has shown note taking on paper to be more efficient than on a laptop keyboard (Mueller and Oppenheimer), as it pushes you to summarize the content instead of transcribe it.


Feedback

We would appreciate any feedback on how to improve the course. We have created an anonymous form to collect feedback, which is accessible through your Brown Google account (but we do not collect your identity). We will read the feedback regularly. If there is something which you do not feel comfortable addressing with this anonymous form, please consider contacting one of the parties listed in the general policy.


Acknowledgements

Portions of this seminar design are from Stefanie Tellex's CSCI 2951-R course, from Christian Theobalt and his CVfCG course @ Max-Planck-Institute for Informatics, with special thanks to Christian Richardt. Thanks also to James Hays and CSCI2951-T Data-Driven Computer Vision course @ Brown University, with special thanks to Genevieve Patterson.

Thanks to Karras et al. (StyleGAN, CVPR 2019) and to Dmitry Nikitko, whose software I used to make the Brown CS faculty bust teaser.

Thanks to Tom Doeppner and Laura Dobler for the text on accommodation, mental health, and incomplete policy.

Thank you to the previous students who helped to improve this class. Previous course runs: