COMSCI 269: Advanced Topics in Natural Language Generation (2020 Fall)

Nanyun (Violet) Peng

Instruction (course introduction, overview of NLG history and basic techniques)


Course overview, Intro to NLG, N-gram language models

Smoothing, log-linear language models, neural networks basics.


Neural language models, MT and Sequence-to-Sequence models (conditional LMs)

Decoding methods (beam search, sampling)

Paper Presentations (3 weeks for methodology, 3 weeks for applications. Each class we present two papers on one topic.)


Topic: Sequence to sequence models

·      Neural Machine Translation by Jointly Learning to Align and Translate. Suggested supplementary readings: Sequence to Sequence Learning with Neural Networks

·      Attention is All You Need


Topic: Autoregressive language models

·      Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

·      XLNet: Generalized Autoregressive Pretraining for Language Understanding Suggested supplementary readings: GPT-2: Language Models are Unsupervised Multitask Learners



Topic: VAE-based generation

·      Auto-regressive Decoding: Generating Sentences from a Continuous Space.

·      Avoiding Latent Variable Collapse with Generative Skip Models.


Topic: Generative Adversarial Networks (GAN) for text

·      SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

·      Language GANs Falling Short: Suggested supplementary readings: Evaluating Text GANs as Language Models.



Topic: Insertion-based generation

·      Insertion Transformer: Suggested supplementary readings: Enabling Language Models to Fill in the Blanks.

·      Non-monotonic Sequential Text Generation:


Topic: Controlled generation

·      Toward Controlled Generation of Text

·      Posterior Control of Blackbox Generation



Topic: Summarization

·      Pointer Generator Network:

·      Text Summarization with BERT:


Topic: Machine Translation

·      Dual Learning for Machine Translation.

·      BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension



Topic: Dialog system

·      Personalizing Dialogue Agents: I have a dog, do you have pets too? Suggested reading: How NOT To Evaluate Your Dialogue System.

·      Task-Oriented Dialogue as Dataflow Synthesis:


Topic: Story generation

·      Hierarchical Neural Story Generation:

·      Content Planning for Neural Story Generation with Aristotelian Rescoring: Suggested reading: Plan-and-Write:


Topic: Figurative language generation

·      Sarcasm generation:

·      Simile generation:

Topic: Poetry generation

·      Chinese Poetry Generation with Recurrent Neural Networks:

·      Generating Topical Poetry:


Final Presentation (each team have 20 mins to present their project)