Learning with Limited Labeled Data (Fall 2018)
Course Description

As machine learning is deployed more widely, researchers and practitioners keep running into a fundamental problem: how do we get enough labeled data? This seminar course will survey research on learning when only limited labeled data is available. Topics covered include weak supervision, semi-supervised learning, active learning, transfer learning, and few-shot learning. Students will lead discussions on classic and recent research papers, and work in teams on final research projects.

Essential Info
Instructor: Stephen Bach a.k.a. Steve
Class Meetings: Tuesdays and Thursdays, 1-2:20 pm, CIT 316
Office Hours: Mondays 10 AM-12 Noon, CIT 335
Textbook: None
Prerequisites: Previous experience in machine learning is required through CSCI 1420 or equivalent research experience.

For questions, discussion, and other course-related posts, use Canvas.

If you have an atypical question that you are certain does not belong on Canvas, email the instructor.

Course Schedule
Sep 6
Introductions, an overview of the research topics we will cover during the semester, how to read a research paper.
Suplemental reading:
  • Introduction to Semi-Supervised Learning. Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. In Semi-Supervised Learning, MIT Press, 2006. [PDF] [Online, requires Brown login]
  • Incidental Supervision: Moving beyond Supervised Learning. Dan Roth. AAAI 2017. [PDF]
  • How to Read a CS Research Paper? Philip W. L. Wong. [PDF]
  • How to Read a Technical Paper. Jason Eisner. [Online]
  • How to Read a Paper. S. Keshav. [PDF]
Semi-Supervised Learning: Generative Models of Data
Sep 11
Semi-Supervised Text Classification using EM. Kamal Nigam, Andrew McCallum, and Tom Mitchell. In Semi-Supervised Learning, MIT Press, 2006.
[PDF] [Online, requires Brown login]
Suplemental reading:
  • Risks of Semi-Supervised Learning: How Unlabeled Data can Degrade Performance of Generative Classifiers. Fabio Cozman and Ira Cohen. In Semi-Supervised Learning, MIT Press, 2006. [Online, requires Brown login]
Sep 13
Learning from Simulated and Unsupervised Images through Adversarial Training. Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017.
Supplemental reading:
  • Generative Adversarial Nets. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio. In Neural Information Processing Systems (NIPS) 2014. [PDF] [Reviews]
Sep 18
Combining Labeled and Unlabeled Data with Co-Training. Avrim Blum and Tom Mitchell. Conference on Computational Learning Theory (COLT) 1998.
Suplemental reading:
  • Input-dependent Regularization of Conditional Density Models. Matthias Seeger. Institute for ANC Technical Report, 2000. [PDF]
  • Analyzing the effectiveness and applicability of co-training. Kamal Nigam and Rayid Ghani. International Conference on Information and Knowledge Management (CIKM) 2000. [PDF]
Semi-Supervised Learning: Low-Density Separation
Sep 20
Start of course survey due
Transductive Support Vector Machines. Thorsten Joachims. In Semi-Supervised Learning, MIT Press, 2006.
[PDF] [Online, requires Brown login]
Semi-Supervised Learning: Graph-Based Methods
Sep 25
Learning from Labeled and Unlabeled Data with Label Propagation. Xiaojin Zhu and Zoubin Ghahramani. Carnegie Mellon University Tech Report, 2002.
Supplemental reading:
  • Semi-Supervised Classification with Graph Convolutional Networks. Thomas N. Kipf and Max Welling. International Conference on Learning Representations (ICLR) 2017. [PDF]
Semi-Supervised Learning: Change of Representation
Sep 27
Spectral Methods for Dimensionality Reduction. Lawrence K. Saul, Kilian Q. Weinberger, Fei Sha, Jihun Ham, and Daniel D. Lee. In Semi-Supervised Learning, MIT Press, 2006.
[Online, requires Brown login]
Oct 2
Why Does Unsupervised Pre-training Help Deep Learning? Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Journal of Machine Learning Research (JMLR) 11:625-660, 2010.
Oct 4
GloVe: Global Vectors for Word Representation. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Conference on Empirical Methods in Natural Language Processing (EMNLP) 2014.
Weak Supervision: Noisy Labels
Oct 9
Distant supervision for relation extraction without labeled data. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Annual Meeting of the Association for Computational Linguistics (ACL) 2009.
Weak Supervision: Generative Models of Labels
Oct 11
Reducing wrong labels in distant supervision for relation extraction. Shingo Takamatsu, Issei Sato, and Hiroshi Nakagawa. Annual Meeting of the Association for Computational Linguistics (ACL) 2012.
Oct 16
Learning from Weak Teachers. Ruth Urner, Shai Ben-David, and Ohad Shamir. Artificial Intelligence and Statistics (AISTATS) 2012.
Oct 18
Snorkel: Rapid Training Data Creation with Weak Supervision. Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Proceedings of the VLDB Endowment, 11(3):269-282, 2017.
Supplemental reading:
  • Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. A. P. Dawid and A. M. Skene. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20-28, 1979. [PDF]
Weak Supervision: Biased Labels
Oct 23
Building Text Classifiers Using Positive and Unlabeled Examples. Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, Philip S. Yu. International Conference on Data Mining (ICDM) 2003.
Weak Supervision: Feature Annotation
Oct 25
Project proposal due
Learning from Labeled Features using Generalized Expectation Criteria. Gregory Druck, Gideon Mann, and Andrew McCallum. Conference on Research and Development in Information Retrieval (SIGIR) 2008.
Active Learning
Oct 30
Active Learning with Statistical Models. D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Journal of Artificial Intelligence Research, 4:129-145, 1996.
Supplemental reading:
  • How transferable are the datasets collected by active learners? David Lowell, Zachary C. Lipton, and Byron C. Wallace. ArXiv:1807.04801. [PDF]
  • Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study. Aditya Siddhant and Zachary C. Lipton. ArXiv:1808.05697. [PDF]
Transfer Learning
Nov 1
A Survey on Transfer Learning. Sinno Jialin Pan and Qiang Yang. IEEE Transactions on Knowledge and Data Engineering. 22(10):1345-1359, 2010.
[Online, requires Brown login]
Supplemental reading:
  • Universal Language Model Fine-tuning for Text Classification. Jeremy Howard and Sebastian Ruder. Annual Meeting of the Association for Computational Linguistics (ACL) 2018. [PDF]
Multi-Task Learning
Nov 6
Multi-Task Learning of Keyphrase Boundary Classification. Isabelle Augenstein and Anders Søgaard. Annual Meeting of the Association for Computational Linguistics (ACL) 2017.
Supplemental reading:
  • Multitask Learning. Rich Caruana. Machine Learning 28:41-75, 1997. [PDF]
Few-Shot Learning
Nov 8
One-shot learning of object categories. Li Fei-Fei, Rob Fergus, and Pietro Perona. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594-611, 2006.
[Online, requires Brown login]
Nov 13
Zero-Shot Learning through Cross-Modal Transfer. Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. In Neural Information Processing Systems (NIPS) 2013.
[PDF] [Reviews]
Supplemental reading:
  • Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly. Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. TPAMI 2018. [PDF]
Data Augmentation
Nov 15
AutoAugment: Learning Augmentation Policies from Data. Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. ArXiv:1805.09501.
Supplemental reading:
  • Learning to Compose Domain-Specific Transformations for Data Augmentation. Alexander J. Ratner, Henry R. Ehrenberg, Zeshan Hussain, Jared Dunnmon, Christopher Ré. In Neural Information Processing Systems (NIPS) 2017. [PDF] [Reviews]
Reinforcement Learning
Nov 20
Project status report due
Apprenticeship learning via inverse reinforcement learning. Pieter Abbeel and Andrew Y. Ng. International Conference on Machine Learning (ICML) 2004.
Nov 27
Policy Shaping: Integrating Human Feedback with Reinforcement Learning. Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L. Isbell, and Andrea L. Thomaz. Neural Information Processing Systems (NIPS) 2013.
[PDF] [Reviews]
Nov 29
Learning to Segment Under Various Forms of Weak Supervision. Jia Xu, Alexander G. Schwing, and Raquel Urtasun. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Dec 4
Indirect Supervision for Relation Extraction using Question-Answer Pairs. Ellen Wu, Xiang Ren, Frank F. Xu, Ji Li, and Jiawei Han. International Conference on Web Search and Data Mining (WSDM) 2018.
Dec 6
Exploring the Limits of Weakly Supervised Pretraining. Dhruv Mahajan et al. ArXiv:1805.00932.
Dec 13
Final project report due
(No class)
Learning Goals

Students who complete this course will:


The following standards will be used to assign grades.

To Earn an A
To Earn a B
Estimated Time Commitment
Class Meetings28
Submitting Discussion Questions10
Preparing to Lead Discussion(s)2
Project Research60+
Project Proposal / Status                             10
Project Final Report5
General Course Policies
Diversity & Inclusion

The Brown computer science department has made it its mission to create and sustain a diverse and inclusive environment in which all students, faculty, and staff can thrive. In this course, that responsibility falls on us all, students and teaching staff alike. In particular, Brown's Discrimination and Harassment Policy applies to all participants.

If you feel you have not been treated in an inclusive manner by any of the course members, please contact either me (Stephen) or the department chair (Prof. Cetintemel). Laura Dobler is also available as a resource for members of underrepresented groups. Additional resources are listed on the department's website. We, the computer science department, take all complaints about discrimination, harassment, and other unprofessional behavior seriously.

In addition, Brown welcomes students from all around the country and the world, and their unique perspectives enrich our learning community. To empower students whose first language is not English, an array of support is available on campus, including language and culture workshops and individual appointments. For more information, contact the English Language Learning Specialists at ellwriting@brown.edu.

Academic Integrity

Academic dishonesty will not be tolerated. This includes cheating, lying about course matters, plagiarism, or helping others commit a violation. Plagiarism includes reproducing the words of others without both the use of quotation marks and citation. Students are reminded of the obligations and expectations associated with the Brown Academic and Student Conduct Codes. For project work, feel free to build on third-party software, datasets, or other resources, as long as you credit them in your report(s) and clearly state what work is solely your own. As a general policy (for this course and for the rest of your academic career): if you use any idea, text, code, or data that you did not create, then cite it.


Brown University is committed to full inclusion of all students. Please inform me if you have a disability or other condition that might require accommodations or modification of any of these course procedures. You may email me, come to office hours, or speak with me after class, and your confidentiality is respected. I will do whatever I can to support accommodations recommended by SEAS. For more information contact Student and Employee Accessibility Services (SEAS) at 401-863-9588 or SEAS@brown.edu.

Mental Health

Being a student can be very stressful. If you feel you are under too much pressure or there are psychological issues that are keeping you from performing well at Brown, I encourage you to contact Brown’s Counseling and Psychological Services CAPS. They provide confidential counseling and can provide notes supporting accommodations for health reasons.

Incomplete Policy

I expect everyone to complete the course on time. However, I understand that there may be factors beyond your control, such as health problems and family crises, that prevent you from finishing the course on time. If you feel you cannot complete the course on time, please discuss with me (Stephen) the possibility of being given a grade of Incomplete for the course and setting a schedule for completing the course in the upcoming year.

Thanks to Tom Doeppner, Laura Dobler, and Daniel Ritchie for borrowed text.

Updated Oct. 1, 2018: Substituted two papers for project presentations to accomodate larger course enrollment. Estimated time commitment and grading updated.