CSCI 544: Applied Natural Language Processing — Fall 2019
Prof. Nanyun (Violet) Peng

Announcements | Course Information | Schedule

Announcements

Course Information

Course objectives: Welcome! This course is designed to introduce you to some of the problems and solutions of NLP, and their relation to linguistics and statistics. You need to know how to program and use common data structures.
It might also be nice—though it's not required—to have some previous familiarity with linear algebra and probabilities.
At the end you should agree (I hope!) that language is subtle and interesting, feel some ownership over some of NLP's formal and statistical techniques, and be able to understand research papers in the field.

Lectures: WF 5:30 - 7:20pm
Location: WPH B27.
Prof: Nanyun (Violet) Peng Email: [email protected]
TAs: Rujun Han Email: [email protected]
Graders: Sachin Vorkady Balakrishna, Email: [email protected]; Anisha Jagadeesh Prasad, Email: [email protected], Jiawei Zhang, Email: [email protected]
Office hrs: Prof: Wed. 4:30pm at RTH 512; or by appt
TAs: Fri. 1:00pm - 3:00pm
Discussion site: Piazza https://piazza.com/usc/fall2019/csci544/home ... public questions, discussion, announcements
Web page: https://violetpeng.github.io/cs544_fa19.html
Textbook: Jurafsky & Martin, 3rd ed. (recommended)
Manning & Schütze (recommended)
Policies: Grading: homework 40%, project 20%, midterm 15% or 25%, final 15% or 25%
Honesty: Viterbi integrity code, USC-Viterbi graduate policies

Schedule

Warning: The schedule below may change. Links to future lectures and assignments are just placeholders and will change.


Week Wednesday Friday Suggested Reading
8/26 Introduction
  • Why is NLP hard?
  • Levels of language
  • NLP applications
 Probability concepts
  • Joint & conditional prob
  • Chain rule and backoff
  • Modeling sequences
  • Cross-entropy and perplexity
  • Intro: J&M chapter 1
  • Chomsky hierarchy: J&M 16
  • Prob/Bayes: M&S 2
9/2 Modeling grammaticality; N-gram language models
  • What's wrong with n-grams?
  • Regular expressions, FSAs, CFGs, ...
  • Spelling correction
  • Segmentation
  • Speech recognition
  • Machine translation
 Smoothing n-grams
  • Bias and variance
  • Add-one or add-λ smoothing
  • Cross-validation
  • Smoothing with backoff
  • Good-Turing, Witten-Bell
  • Log-linear models
  • Language models: J&M 3
9/9 Assignment 1 given: Probabilities
Intro to neural language Models
  • Conditional log-linear models
  • Maximum likelihood, regularization
  • Feedforward neural language Models
  • Recurrent neural language Models
 No class (SoCal NLP symposium)
  • You are encouraged to attend
  • On the USC campus
  • Free registration, free food
  • Several companies recruiting
  • Smoothing: J&M 3; Rosenfeld (2000)
  • Neural language models: J&M 7; OpenAI blog post GPT-2 (with paper); BERT paper
9/16 Context-free parsing
  • What is parsing?
  • Why is it useful?
  • Brute-force algorithm
  • CKY algorithms
 Guest Lecture from TA:
  • Deep Learning workshop
  • Intro to PyTorch
  • Attributes: J&M 12
  • Parsing: J&M 13
9/23 Assignment 1 due
Assignment 2 given: Language Models
Probabilistic parsing
  • PCFG parsing
  • Dependency grammar
  • Lexicalized PCFGs
 Dependency Parser
  • Dependency Trees
  • Shift-reduce parser
  • CCG: Steedman & Baldridge; more
  • TAG/TSG: Van Noord, Guo, Zhang 1/2/3
  • Prob. parsing: J&M 14
9/30 Semantics
  • What is understanding?
  • Semantic phenomena and representations
  • WordNet
  • Semantic similarity
 Midterm review
  • Word Sense and WordNet: J&M 19; lambda calculus for kids
10/7 Midterm exam
(5:30-6:30 in classroom)		 Distributional semantics (word embeddings)
  • Compositional semantics
  • Distributional semantics
  • Vector Semantics: J&M 6
10/14 Sequence tagging models
  • Ice cream, weather, words and tags
  • Inferring hidden states
  • Likelihood convergence
 Project proposal due
No class (fall break)
  • The Viterbi Algorithm: J&M 8
10/21 Assignment 2 due
Assignment 3 given: Semantics
 HMM, MEMM, and CRF
  • Generalizing the forward-backward strategy
  • Inside-outside algorithm
  • Posterior decoding
 Neural sequence tagging and relation extraction
  • Recurrent Neural Networks
  • Neural CRFs
  • Relation Extraction
  • Hidden Markov Models: J&M Appendix A; John Lafferty's paper on CRF
10/28 Dialog systems
  • Guest Lecture: Sarik Ghazarian
  • Dialog system
 Text classification
  • Features
  • Linear Classifiers
11/4 Assignment 3 due
Assignment 4 given: Neural Sequence Tagging
 Project proposal revision (if applicable) due
  • Guest Lecture: Anna Farzindar
 Machine Translation
  • introduction
  • history
  • evaluation
11/11 Phrase-based machine translation
  • Phrase-based MT
  • Alignment
  • IBM models
 Sequence to sequence models
  • RNN-based seq-to-seq models
  • Transformers
  • Applications to machine translation, summarization.
  • Morphology: R&S 2
    11/18 NLP Applications: Creative Generation
    • Stemming
    • Compounds, segmentation
    • Two-level morphology
    • Punctuation
    • Rewrite rules
    		 Socially responsible NLP
    • Biases in word embeddings.
    • Biases in coreference resolution.
    • Bias-mitigation algorithms
    • MT: J&M 25, M&S 13, statmt.org; tutorial (2003), workbook (1999), introductory essay (1997), technical paper (1993); tutorial (2006) focusing on more recent developments (slides, 3-hour video part 1, part 2)
    11/25 No class
    (Thanksgiving break)    		 Assignment 4 due
    No class
    (Thanksgiving break)
    12/2 Final Project Due
    Final exam recitation I     		Final exam recitation II


    Final exam: Wed 12/11, in class.