CSCI 1430: Introduction to Computer Vision

Spring 2020, MW 15:00 to 16:20, MacMillan 117

Course | Office Hours | Projects | Schedule/Slides | General Policy | Feedback | Acknowledgements


Instructor: James Tompkin

HTAs: Isa Milefchik, George Lee

TAs: Joy Zheng, Eliot Laidlaw, Neev Parikh, Trevor Houchens, Katie Friis, Raymond Cao, Isabella Ting, Andrew Park, Qiao Jiang, Mary Dong, Katie Scholl, Jason Senthil, Melis Gokalp, Michael Snower, Yang Jiao, Yuting Liu, Cong Huang, Kyle Cui, Nine Prasersup, Top Piriyakulkij, Eleanor Tursman, Claire Chen, Josh Roy, Megan Gessner, Yang Zhang

ETAs: Katie Friis, Isabella Ting


Computer Vision, art by HarperCollins / Clare Skeats / James Tompkin

Contact

Office/Piazza Hours

We will use Brown's SignMeUp (here) to arrange TA office hour and Zoom for remote hours.

Piazza hours protocol

  1. During hours each TA will have a join link on Signmeup, if you join a link, you will be automatically placed in a waiting room, but please don’t join any TA’s link, wait until you are claimed because each TA will have a different link.
  2. Once you have been claimed in Signmeup, you will be admitted to the call with a TA and off of the waiting list, if we do not see you in the waiting list queue to admit you, you will be marked missing, please make sure you join the Zoom link of the TA who claimed you.
  3. If you are marked missing, join the waiting room of any TA and if they see you, you will be unmarked missing, but please make sure you monitor Signmeup and if you are claimed by a TA not in the meeting room you are in, make sure to switch to the correct meeting link.

James' office hour (Tues 1pm) appointment slot signups are here.

Course Description

This course provides an introduction to computer vision, including fundamentals of image formation, camera imaging geometry, feature detection and matching, stereo, motion estimation and tracking, image classification, scene understanding, and deep learning with neural networks. We will develop basic methods for applications that include finding known models in images, depth recovery from stereo, camera calibration, image stabilization, automated alignment, tracking, boundary detection, and recognition. We will develop the intuitions and mathematics of the methods in class, and then learn about the difference between theory and practice in projects.

This course is based upon James Hays' computer vision course, previously taught at Brown as CS143, and currently taught at Georgia Tech as CS 4476. Significant thanks to him and his staff, across the years, for all their hard work.

Learning Objectives

Upon completion of this course, students will:
  1. Be familiar with both the theoretical and practical aspects of computing with images;
  2. Have described the foundation of image formation, measurement, and analysis;
  3. Have implemented common methods for robust image matching and alignment;
  4. Understand the geometric relationships between 2D images and the 3D world;
  5. Have gained exposure to object and scene recognition and categorization from images;
  6. Grasp the principles of state-of-the-art deep neural networks; and
  7. Developed the practical skills necessary to build computer vision applications.

Prerequisites

No prior experience with computer vision is assumed, although previous knowledge of visual computing or signal processing will be helpful (e.g., CSCI 1230). The following skills are necessary for this class:

Time Commitment

Task Hours
In class 40
Projects: 20 ×6
Office Hours/Piazza:    20
Total: 180

Capstone

This class can be taken as a capstone. You will need to complete 10 points of extra credit in each of the projects, including a commensurate amount of work in the final project.

Reading References

There is no requirement to buy a textbook. The goal of the course is to be self contained, but sections from three textbooks will be suggested for more formalization and information. Two of these books are available free online, with the third available online through Brown's library.

  1. Concise Computer Vision by Reinhard Klette
  2. Computer Vision: Algorithms and Applications by Richard Szeliski.
  3. Deep Learning, by Goodfellow, Bengio, and Courville.

Further reading material:

Software

We will use Python 3 for the course, and we will support editing and debugging Python through Visual Studio Code (vscode). Project 0 includes a tutorial for how to set up a Python environment on your personal computer, or use the CS department machines.

Versions:(Spring 2020) Our Python virtual environment uses Python 3.7.3. Our autograder uses the same virtual environment. Python 2.7 is not supported by the class.

Projects

Projects are released every ~two weeks, with deliverables due each week at Friday at 9pm. Each project has ~two parts: written, and code. You have one week to complete the written part, and two weeks to complete the code part.

Hand-in for both parts is electronic via Gradescope.

Projects 1, 2, 3, 4 and 5 must be completed in Python. Project 6 as a final project presents a free choice.

Projects Gradescope Hand in
0. Python Primer Questions:
GitHub Guide | GitHub Classroom Assignment Link
1. Image Filtering and Hybrid Images Questions + code:
2. Local Feature Matching Questions: ; Code: 14th Feb. 21:00
3. Scene Recognition with Bag of Words Questions: ; Code:
4. Convolutional Neural Nets Questions: ; Code:
5. Camera Calibration and Fundamental Matrix Estimation Questions + code: Please refer to Gradescope.
6. Final Group Project Team + proposal:
Progress report 1:
Progress report 2:
Presentation:
Final submission:

Grading

Grading is A >= 90, B >= 80, C >= 70, with no curve, and with rounding to the nearest integer. Your final grade will be 100% from 7 projects. All projects are graded. We leave ourselves a little flexibility to make minor adjustments. Say, if one project ends up being a little more difficult, then we can tweak that project to be less significant in your final grade.

Project Percent
0 2%
1–5 ~14.6%
6 ~25%

Late Submissions and Late Days

Our projects are split into two parts: questions and code. You will lose 10% from the total possible marks of each project part for each day (24 hours) that it is late. Over the course, we give you free late days for each project part: three question late days and three code late days.

These late days will not be reflected in the initial grade reports for your assignment, but they will be factored into your final grade at the end of the semester.

Late days cover unexpected clustering of due dates, travel commitments, interviews, hackathons, etc. Please do not ask for extensions to due dates—we give you a pool of late days to manage yourself.

For sickness and other issues of wellbeing, please obtain a note from health services and we will accommodate.

Video Lecture Capture

Please look here for lecture capture of the class sessions via video (Brown CSCI 1430 course registration required).

Tentative Schedule

Date Topic Slides More info
Image Formation and Filtering
Wed 22 Jan Introduction to Computer Vision PPTX | PDF Szeliski 1
Fri 24 Jan Due: Project 0 Questions, 21:00, Gradescope
Mon 27 Jan Image Filtering PPTX | PDF Klette 1.1, 2.1; Szeliski 3.2
Wed 29 Jan Thinking in Frequency PPTX | PDF Klette 2.2.2, Szeliski 3.4
Thinking in Frequency, part 2 PPTX | PDF Klette 1.2, Szeliski 3.5.2, 8.1.1
Fri 31 Jan Due: Project 1 Questions + Code, 21:00, Gradescope
Feature Detection and Matching
Mon 03 Feb Edge Detection PPTX | PDF Klette 2.3, 2.4, Szeliski 4.2
Weds 05 Feb Interest Points and Corners PPTX | PDF Klette 2.3, 2.4, Szeliski 4.1.2
Local Image Features and Feature Matching PPTX | PDF Klette 9.2, Szeliski 4.1.3, 4.3.2
Mon 10 Feb Light and Color PPTX | PDF Klette 6.1, Klette 1.3, Szeliski 2.1, esp. 2.1.5, 2.2, 2.3
Machine Learning Crash Course
Wed 12 Feb Machine Learning: Unsupervised Learning PPTX | PDF Szeliski 5.3
Machine Learning: Supervised Learning PPTX | PDF Szeliski 5.3
Fri 14 Feb Due: Project 2 Questions+Code, 21:00, Gradescope
Recognition
Mon 17 Feb No class—University long weekend.
Wed 19 Feb Recognition, Bag of Features, and Large-scale Instance Recognition PPTX | PDF Szeliski 14
Fri 21 Feb Due: Project 3 Questions, 21:00, Gradescope
Mon 24 Feb Large-scale Scene Recognition and Advanced Feature Encoding PPTX | PDF
Wed 26 Feb Detection with Sliding Windows: Dalal Triggs and Viola Jones PPTX | PDF Szeliski 14.1 and 14.2
Wed 26 Feb Descriptor Failure and Big Data PPTX | PDF Szeliski 14.5
Fri 28 Feb Due: Project 3 Code, 21:00, Gradescope
Deep Learning
Mon 02 Mar Neural Networks and Convolutional Neural Networks PPTX | PDF Goodfellow 6, 9
Wed 04 Mar Training Neural Networks PPTX | PDF Goodfellow 6, Goodfellow 7.1-7.5, 7.12
Mon 09 Mar What do CNNs learn? PPTX | PDF
Wed 11 Mar Architectures: ResNets, R-CNNs, FCNs, and UNets PPTX | PDF
Fri 13 Mar Due: Project 4 Questions, 21:00, Gradescope
Computer Vision and Society
Mon 16 Mar Social Good and Dataset Bias PPTX | PDF
Wed 18 Mar COVID-19 cancellation
Fri 20 Mar Due: Project 4 Code, 21:00, Gradescope
Spring break
Mon 23 Mar Spring break!
Wed 25 Mar Spring break!
Fri 27 Mar Spring break!
Cameras, Multiple Views, and Motion
Mon 30 Mar Cameras and Calibration PPTX | PDF Szeliski 2.1, 6.1, 6.2.1
Wed 01 Apr Final Project Explanation and Discussion Google Slides (Brown Internal Only)
Wed 01 Apr Stereo Vision, Epipolar Geometry, and RANSAC PPTX | PDF Szeliski 7, 8.1, 8.4, 11
Mon 06 Apr Stereo Disparity Matching PPTX | PDF Szeliski 6.2.1
Wed 08 Apr Depth Cameras and Real-time Reconstruction PPTX | PDF
Fri 10 Apr Due: Project 5 Questions + Code, 21:00, Gradescope
Due: Project teams and proposal, 21:00, Gradescope
Mon 14 Apr No class
Wed 16 Apr Real-time Depth-based Pose Estimation PPTX | PDF
Fri 17 Apr Due: Project progress report 1, 21:00, Gradescope
'Reading period'
Mon 20 Apr Research paper class
Wed 22 Apr Research paper class
Fri 24 Apr Due: Project progress report 2, 21:00, Gradescope
Mon 27 Apr Final project AMA
Wed 29 Apr Final project AMA
Mon 04 May Due: Final project presentations 'fast forward'! 15:00 in class.
Fri 08 May Due: Final project report and code, 21:00, Gradescope

General Policy

Welcome!

Our intent is that this course provide a welcoming environment for all students who satisfy the prerequisites. Our TAs have undergone training in diversity and inclusion, and all members of the CS community, including faculty and staff, are expected to treat one another in a professional manner. If you feel you have not been treated in a professional manner by any of the course staff, please contact any of James (the instructor), Ugur Cetintemel (Dept. Chair), Tom Doeppner (Vice Chair) or Laura Dobler (diversity and inclusion staff member). We will take all complaints about unprofessional behavior seriously. Your suggestions are encouraged and appreciated. Please let James know of ways to improve the effectiveness of the course for you personally, or for other students or student groups. To access student support services and resources, and to learn more about diversity and inclusion in CS, please visit http://cs.brown.edu/about/diversity/resources/.

Prof. Krishnamurthi has good notes on this area.

Quiet Hours

This class runs quiet hours from 9pm to 9am every day. Please do not expect a response from us via any channel. Likewise, we won't ask you to do anything between these times, either, like hand in projects.

Academic Integrity, Collaboration, and Citation

Feel free to talk to your friends about the concepts in the projects, and work through the ideas behind problems together, but be sure to always write your own code and perform your own write up. You may work with other students to debug code; please keep in mind that debugging is a useful way to learn and is a skill to practice---please spend time debugging independently and come to office hours for help.

You are expected to implement the core components of each project on your own, but the extra credit opportunities often build on third party data sets or code. Feel free to include results built on other software, as long as you credit correctly in your handin and clearly demark your own work. In general, if you use an idea, text, or code from elsewhere, then cite it.

Brown-wide, academic dishonesty is not tolerated. This includes cheating, lying about course matters, plagiarism, or helping others commit a violation. Plagiarism includes reproducing the words of others without both the use of quotation marks and citation. Students are reminded of the obligations and expectations associated with the Brown Academic and Student Conduct Codes.

Accommodations

Brown University is committed to full inclusion of all students. Please inform me if you have a disability or other condition that might require accommodations or modification of any of these course procedures. You may email me, come to office hours, or speak with me after class, and your confidentiality is respected. We will do whatever we can to support accommodations recommended by SEAS. For more information contact Student and Employee Accessibility Services (SEAS) at 401-863-9588 or . Students in need of short-term academic advice or support can contact one of the deans in the Dean of the College office.

Mental Health

Being a student can be very stressful. If you feel you are under too much pressure or there are psychological issues that are keeping you from performing well at Brown, we encourage you to contact Brown's Counseling and Psychological Services. They provide confidential counseling and can provide notes supporting extensions on assignments for health reasons.

Incomplete Policy

We expect everyone to complete the course on time. However, we certainly understand that there may be factors beyond your control, such as health problems and family crises, that prevent you from finishing the course on time. If you feel you cannot complete the course on time, please discuss with James Tompkin the possibility of being given a grade of Incomplete for the course and setting a schedule for completing the course in the upcoming year.

Electronic Etiquette

Laptops are discouraged, please, except for class-relevant activities, e.g., to help answer questions and show items relevant to discussion. No social media, email, etc., because it distracts not just you but other students as well. Read Shirky on this issue ("Why I Just Asked My Students to Put Their Laptops Away"), or Rockmore ("The Case for Banning Laptops in the Classroom").

We will release course lecture material online. In considering laptop use for note taking, please be aware that research has shown note taking on paper to be more efficient than on a laptop keyboard (Mueller and Oppenheimer), as it pushes you to summarize the content instead of transcribe it.


Feedback

We would appreciate any feedback on how to improve the course. We have created an anonymous form to collect feedback, which is accessible through your Brown Google account (but we do not collect your identity). We will read the feedback every two weeks at grading. If there is something more urgent (and not anonymous), please email James or the course staff. If there is something urgent and anonymous, please consider contacting one of the parties listed in the general policy.


Acknowledgements

The materials from this class rely significantly on slides prepared by other instructors, especially James Hays, Derek Hoiem, and Svetlana Lazebnik. Each slide set and assignment contains acknowledgements. Feel free to use these slides for academic or research purposes, but please maintain all acknowledgements.

Thanks to Tom Doeppner and Laura Dobler for the text on accommodation, mental health, and incomplete policy.

Thank you to the previous TAs who helped to teach and improve this class. Previous course runs: