Learning with Limited Labeled Data (Fall 2020)
Course Description
As machine learning is deployed more widely, researchers and practitioners
keep running into a fundamental problem: how do we get enough labeled data?
This seminar course will survey research on learning when only limited
labeled data is available. Topics covered include weak supervision,
semi-supervised learning, active learning, transfer learning, and few-shot
learning. Students will lead discussions on classic and recent research
papers, and work in teams on final research projects.
Essential Info
Class Meetings: Tuesdays and Thursdays, 1-2:20 pm, synchronously on Zoom. Connection information available on
Canvas.
Office Hours by appointment. Email anytime to schedule!
Textbook: None
Prerequisites: Previous experience in machine learning is required through CSCI 1420 or equivalent research experience.
Important Links
Canvas
for discussions, assignment guidelines, and additional class resources
Past years
for previous reading lists (project ideas, etc.)
Contact
For questions, discussion, and other course-related posts, use
Canvas.
If you have an atypical question that you are certain does not belong on
Canvas, email the instructor.
Course Schedule
Introduction
Sep 10
Introductions, an overview of the research
topics we will cover during the semester, how to read a research paper.
Suplemental reading:
- Introduction to Semi-Supervised Learning. Olivier Chapelle, Bernhard
Schölkopf, and Alexander Zien. In Semi-Supervised Learning, MIT Press, 2006.
[PDF]
[Online, requires Brown login]
- Incidental Supervision: Moving beyond Supervised Learning. Dan Roth.
AAAI 2017.
[PDF]
- How to Read a CS Research Paper? Philip W. L. Wong.
[PDF]
- How to Read a Technical Paper. Jason Eisner.
[Online]
- How to Read a Paper. S. Keshav.
[PDF]
Semi-Supervised Learning
Sep 15
Billion-scale semi-supervised learning for image classification.
I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan.
ArXiv 1905.00546 2019.
[PDF]
Suplemental reading:
-
Big Self-Supervised Models are Strong Semi-Supervised Learners.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton.
ArXiv 2006.10029 2020.
[PDF]
[Code]
Sep 17
S4L: Self-Supervised Semi-Supervised Learning.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer.
The IEEE/CVF International Conference on Computer Vision (ICCV) 2019.
[PDF]
[Supplemental]
[Video]
Suplemental reading:
-
Self-Supervised Learning of Pretext-Invariant Representations.
Ishan Misra and Laurens van der Maaten.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020.
[PDF]
Sep 22
Temporal Ensembling for Semi-Supervised Learning.
Samuli Laine and Timo Aila.
International Conference on Learning Representations (ICLR) 2017.
[PDF]
Suplemental reading:
-
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.
Antti Tarvainen and Harri Valpola.
Neural Information Processing Systems (NeurIPS) 2017.
[PDF]
[Supplemental (Zip)]
[Reviews]
Sep 24
Start of course survey due
Self-training with Noisy Student improves ImageNet classification.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020.
[PDF]
[Supplemental]
Suplemental reading:
-
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning.
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness.
ArXiv 1908.02983 2019.
[PDF]
[Reviews]
Transfer/Representation Learning
Sep 29
How transferable are features in deep neural networks?
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.
Neural Information Processing Systems (NeurIPS) 2014.
[PDF]
[Supplemental (Zip)]
[Reviews]
Supplemental reading:
-
Learning Transferable Features with Deep Adaptation Networks.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.
International Conference on Machine Learning (ICML) 2015.
[PDF]
Oct 1
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Meeting of the North American Association for Computational Linguistics (NAACL) 2019.
[PDF]
Supplemental reading:
-
How to Fine-Tune BERT for Text Classification?
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang.
ArXiv 1905.05583 2019.
[PDF]
Oct 6
Unsupervised Feature Learning via Non-Parametric Instance Discrimination.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2018.
[PDF]
[Video]
Supplemental reading:
-
Momentum Contrast for Unsupervised Visual Representation Learning.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020.
[PDF]
[Supplemental]
[Code]
Oct 8
Rethinking Pre-training and Self-training.
Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, and Quoc V. Le.
ArXiv 2006.06882 2020.
[PDF]
Supplemental reading:
-
On Learning Invariant Representations for Domain Adaptation.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon.
International Conference on Machine Learning (ICML) 2019.
[PDF]
[Supplemental]
Weakly Supervised Learning
Oct 13
Snorkel: Rapid Training Data Creation with Weak Supervision. Alexander Ratner,
Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré.
Proceedings of the VLDB Endowment, 11(3):269-282, 2017.
[PDF]
[Code]
Supplemental reading:
-
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm.
A. P. Dawid and A. M. Skene.
Journal of the Royal Statistical Society. Series
C (Applied Statistics), 28(1):20-28, 1979.
[PDF]
Oct 15
Weakly Supervised Sequence Tagging from Noisy Rules.
Esteban Safranchik, Shiying Luo, and Stephen H. Bach.
AAAI Conference on Artificial Intelligence (AAAI) 2020.
[PDF]
[Code]
Supplemental reading:
-
TriggerNER: Learning with Entity Triggers as Explanation for Named Entity Recognition.
Bill Yuchen Lin, Dongho Lee, Ming Shen, Xiao Huang, Ryan Moreno, Prashant Shiralkar, and Xiang Ren.
Meeting of the Association for Computational Linguistics (ACL) 2020.
[PDF]
[Code]
Oct 20
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov.
International Conference on Learning Representations (ICLR) 2020.
[PDF]
Supplemental reading:
-
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, and Ming Zhou.
ArXiv 2002.01808 2020.
[PDF]
Oct 22
Project proposal due
Exploring the Limits of Weakly Supervised Pretraining.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, Laurens van der Maaten.
European Conference on Computer Vision (ECCV) 2018.
[PDF]
Supplemental reading:
-
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition.
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan.
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019.
[PDF]
Data Generation and Augmentation
Oct 27
Semi-Supervised Learning with Deep Generative Models.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling.
Neural Information Processing Systems (NeurIPS) 2014.
[PDF]
[Supplemental (Zip)]
[Reviews]
Supplemental reading:
-
Learning from Simulated and Unsupervised Images through Adversarial Training.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017.
[PDF]
Oct 29
Unsupervised Data Augmentation for Consistency Training.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le.
ArXiv 1904.12848 2019.
[PDF]
[Reviews]
Supplemental reading:
- MixMatch: A Holistic Approach to Semi-Supervised Learning.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel.
Neural Information Processing Systems (NeurIPS) 2019.
[PDF]
[Supplemental (Zip)]
[Reviews]
[Code]
Nov 5
MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation.
Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2020.
[PDF]
[Talk]
[Code]
[Video]
Supplemental reading:
-
Generating Object Stamps.
Youssef Alami Mejjati, Zejiang Shen, Michael Snower, Aaron Gokaslan, Oliver Wang, James Tompkin, and Kwang In Kim.
ArXiv 2001.02595 2020.
[PDF]
Active Learning
Nov 10
Re-active Learning: Active Learning with Relabeling.
Christopher H. Lin, Mausam, and Daniel S. Weld.
AAAI Conference on Artificial Intelligence (AAAI) 2016.
[PDF]
Supplemental reading:
- Variational Adversarial Active Learning.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell.
IEEE/CVF International Conference on Computer Vision (ICCV) 2019.
[PDF]
[Supplemental]
[Code]
[Video]
Few-Shot Learning
Nov 12
Prototypical Networks for Few-shot Learning.
Jake Snell, Kevin Swersky, and Richard Zemel.
In Neural Information Processing Systems (NeurIPS) 2017.
[PDF]
[Supplemental (Zip)]
[Reviews]
Supplemental reading:
-
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.
Chelsea Finn, Pieter Abbeel, and Sergey Levine.
In International Conference on Machine Learning (ICML) 2017.
[PDF]
Nov 17
Project status report due
Rethinking Few-Shot Image Classification: A Good Embedding Is All You Need?
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola.
ArXiv:2003.11539 2020.
[PDF]
[Project]
[Code]
Supplemental reading:
-
A New Meta-Baseline for Few-Shot Learning.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell.
ArXiv:2003.04390 2020.
[PDF]
[Code]
Zero-Shot Learning
Nov 19
DeViSE: A Deep Visual-Semantic Embedding Model.
Andrea Frome, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov.
In Neural Information Processing Systems (NeurIPS) 2015.
[PDF]
[Supplemental (Zip)]
[Reviews]
Supplemental reading:
-
Zero-Shot Learning through Cross-Modal Transfer.
Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng.
In Neural Information Processing Systems (NeurIPS) 2013.
[PDF]
[Reviews]
Nov 24
Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018.
[PDF]
Supplemental reading:
-
Rethinking Knowledge Graph Propagation for Zero-Shot Learning.
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P. Xing.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2019.
[PDF]
Dec 11
Final project report due
(No class)
Learning Goals
Students who complete this course will:
- Acquire a working knowledge of the landscape of research on machine
learning with limited labeled data.
- Practice identifying and critically assessing the claims,
contributions, and supporting evidence in machine learning research papers.
- Develop their ability to share scientific ideas via writing and discussion.
- Gain practical experience with the course's subject matter by applying and
extending it to their own research interests though an open-ended project.
Grading
The following standards will be used to assign grades.
To Earn an A
- Participate actively in class discussions by asking questions, sharing
opinions, and listening carefully to others.
- Meet all deadlines in the course schedule related to the research project.
- Submit two discussion questions to Canvas by 6 PM the evening before class
for assigned readings, missing no more than 3 readings.
- Attend class meetings, missing no more than 3 meetings.
- Fulfill the requirements below to earn a B.
To Earn a B
- Conduct an original research project related to course materials and submit
a written report meeting the assignment guidelines.
- Lead the assigned class discussion demonstrating preparation and inclusion.
- Submit two discussion questions to Canvas by 6 PM the evening before class
for assigned readings, missing no more than 6 readings.
- Attend class meetings, missing no more than 6 meetings.
Estimated Time Commitment
Activity | Hours |
Class Meetings | 28 |
Readings | 65 |
Submitting Discussion Questions | 10 |
Preparing to Lead Discussion | 2 |
Project Research | 60+ |
Project Proposal / Status
| 10 |
Project Final Report | 5 |
Total | 180+ |
General Course Policies
Diversity & Inclusion
The Brown computer science department has made it its mission to create and
sustain a diverse and inclusive environment in which all students, faculty, and
staff can thrive. In this course, that responsibility falls on us all, students and
teaching staff alike. In particular,
Brown's Discrimination and Harassment Policy
applies to all participants.
If you feel you have not been treated in an inclusive
manner by any of the course members, please contact either me (Stephen) or the
department chair (Prof. Cetintemel). Laura Dobler is also available as a resource
for members of underrepresented groups.
Additional resources
are listed on the department's website. We, the computer science department, take all complaints about discrimination, harassment, and other
unprofessional behavior seriously.
In addition, Brown welcomes students from all around the country and the world, and their
unique perspectives enrich our learning community. To empower students whose first
language is not English, an array of support is available on campus, including
language and culture workshops and individual appointments. For more information,
contact the English Language Learning Specialists at
ellwriting@brown.edu.
Academic Integrity
Academic dishonesty will not be tolerated. This includes cheating, lying about
course matters, plagiarism, or helping others commit a violation. Plagiarism
includes reproducing the words of others without both the use of quotation marks
and citation. Students are reminded of the obligations and expectations associated
with the
Brown Academic and Student Conduct Codes.
For project work, feel free to build on third-party software, datasets, or other
resources, as long as you credit them in your report(s) and clearly state what
work is solely your own. As a general policy (for this course and for the rest of
your academic career): if you use any idea, text, code, or data that you did not
create, then cite it.
Accommodations
Brown University is committed to full inclusion of all students. Please inform
me if you have a disability or other condition that might require accommodations
or modification of any of these course procedures. You may email me, come to office
hours, or speak with me after class, and your confidentiality is respected. I
will do whatever I can to support accommodations recommended by SEAS. For more
information contact Student and Employee Accessibility Services
(SEAS)
at 401-863-9588 or SEAS@brown.edu.
Mental Health
Being a student can be very stressful. If you feel you are under too much
pressure or there are psychological issues that are keeping you from performing
well at Brown, I encourage you to contact Brown’s Counseling and Psychological Services
CAPS.
They provide confidential counseling and can provide notes supporting accommodations
for health reasons.
Incomplete Policy
I expect everyone to complete the course on time. However, I understand that
there may be factors beyond your control, such as health problems and family
crises, that prevent you from finishing the course on time. If you feel you cannot
complete the course on time, please discuss with me (Stephen) the possibility
of being given a grade of Incomplete for the course and setting a schedule for
completing the course in the upcoming year.
Thanks to Tom Doeppner, Laura Dobler, and Daniel
Ritchie for borrowed text.