# cs224n_learning_note **Repository Path**: lin323/cs224n_learning_note ## Basic Information - **Project Name**: cs224n_learning_note - **Description**: Stanford cs224n. 斯坦福2019最新cs224n学习资料 个人学习笔记和解读以及作业解答。 http://web.stanford.edu/class/cs224n/ - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2020-03-02 - **Last Updated**: 2020-12-19 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # cs224n_learning_note 我会将自己的学习笔记以及相关资料都放在这里。时间表安排将会按照斯坦福的时间安排进行。开始时间为2019.6.19 表示我没有做A3.我实在是对这部分内容不太感冒。挣扎了两个礼拜，每次打开就放弃了。后来我想起我也没有在上课，就不做了。随他去吧。我要去做A4了。a5看心情。主要是我要把final做出来 http://web.stanford.edu/class/cs224n/ The timetable is in Schedule.md ## Schedule Lecture **slides** will be posted here shortly before each lecture. If you wish to view slides further in advance, refer to [last year's slides](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1184/syllabus.html), which are mostly similar. This is the timetable from stanford website. I only use it to manage my study schedule to avoid the [procrastination](). Usually, I study in the evening to ensure I have enough time. *This schedule is subject to change*. All the material could be found [here](https://github.com/zhanlaoban/CS224N-Stanford-Winter-2019) | Date | Description | Course Materials | Events | Deadlines | Note | | ----------: | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | ---- | | Tue June 19 | Introduction and Word Vectors [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture01-wordvecs1.pdf)] [[video](https://youtu.be/8rXD5-xhemo)] [[notes](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes01-wordvecs1.pdf)] Gensim word vectors example: [[code](http://web.stanford.edu/class/cs224n/materials/Gensim.zip)] [[preview](http://web.stanford.edu/class/cs224n/materials/Gensim word vector visualization.html)] | Suggested Readings:[Word2Vec Tutorial - The Skip-Gram Model](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/)[Efficient Estimation of Word Representations in Vector Space](http://arxiv.org/pdf/1301.3781.pdf)(original word2vec paper)[Distributed Representations of Words and Phrases and their Compositionality](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) (negative sampling paper) | Assignment 1 **out** [[code](http://web.stanford.edu/class/cs224n/assignments/a1.zip)] [[preview](http://web.stanford.edu/class/cs224n/assignments/a1_preview/exploring_word_vectors.html)] | | done | | Thu June 21 | Word Vectors 2 and Word Senses [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture02-wordvecs2.pdf)] [[video](https://youtu.be/kEMJRjEdNzM)] [[notes](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes02-wordvecs2.pdf)] | Suggested Readings:[GloVe: Global Vectors for Word Representation](http://nlp.stanford.edu/pubs/glove.pdf) (original GloVe paper)[Improving Distributional Similarity with Lessons Learned from Word Embeddings](http://www.aclweb.org/anthology/Q15-1016)[Evaluation methods for unsupervised word embeddings](http://www.aclweb.org/anthology/D15-1036)Additional Readings:[A Latent Variable Model Approach to PMI-based Word Embeddings](http://aclweb.org/anthology/Q16-1028)[Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://transacl.org/ojs/index.php/tacl/article/viewFile/1346/320)[On the Dimensionality of Word Embedding.](https://papers.nips.cc/paper/7368-on-the-dimensionality-of-word-embedding.pdf) | 1 | | done | | Fri June 22 | Python review session [[slides](http://web.stanford.edu/class/cs224n/readings/python-review.pdf)] | 1:30 - 2:50pm Skilling Auditorium [[map](https://maps.google.com/maps?hl=en&q=Skilling Auditorium%2C 494 Lomita Mall%2C Stanford%2C CA 94305%2C USA)] | 1 | | Done | | Tue June 26 | Word Window Classification, Neural Networks, and Matrix Calculus [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture03-neuralnets.pdf)] [[video](https://youtu.be/8CWyBNX6eDo)] [[matrix calculus notes](http://web.stanford.edu/class/cs224n/readings/gradient-notes.pdf)] [[notes (lectures 3 and 4)](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes03-neuralnets.pdf)] | Suggested Readings:[CS231n notes on backprop](http://cs231n.github.io/optimization-2/)[Review of differential calculus](http://web.stanford.edu/class/cs224n/readings/review-differential-calculus.pdf)Additional Readings:[Natural Language Processing (Almost) from Scratch](http://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) | Assignment 2 **out** [[code](http://web.stanford.edu/class/cs224n/assignments/a2.zip)] [[handout](http://web.stanford.edu/class/cs224n/assignments/a2.pdf)] | Assignment 1 **due** done | Done | | Thu June 28 | Backpropagation and Computation Graphs [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture04-backprop.pdf)] [[video](https://youtu.be/yLYHDSv-288)] [[notes (lectures 3 and 4)](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes03-neuralnets.pdf)] | Suggested Readings:[CS231n notes on network architectures](http://cs231n.github.io/neural-networks-1/)[Learning Representations by Backpropagating Errors](http://www.iro.umontreal.ca/~vincentp/ift3395/lectures/backprop_old.pdf)[Derivatives, Backpropagation, and Vectorization](http://cs231n.stanford.edu/handouts/derivatives.pdf)[Yes you should understand backprop](https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b) | | | done | | Tue July 2 | Linguistic Structure: Dependency Parsing [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture05-dep-parsing.pdf)] [[scrawled-on slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture05-dep-parsing-scrawls.pdf)] [[video](https://youtu.be/nC9_RfjYwqA)] [[notes](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes04-dependencyparsing.pdf)] | Suggested Readings:[Incrementality in Deterministic Dependency Parsing](https://www.aclweb.org/anthology/W/W04/W04-0308.pdf)[A Fast and Accurate Dependency Parser using Neural Networks](http://cs.stanford.edu/people/danqi/papers/emnlp2014.pdf)[Dependency Parsing](http://www.morganclaypool.com/doi/abs/10.2200/S00169ED1V01Y200901HLT002)[Globally Normalized Transition-Based Neural Networks](https://arxiv.org/pdf/1603.06042.pdf)[Universal Stanford Dependencies: A cross-linguistic typology](http://nlp.stanford.edu/~manning/papers/USD_LREC14_UD_revision.pdf)[Universal Dependencies website](http://universaldependencies.org/) | Assignment 3 **out** [[code](http://web.stanford.edu/class/cs224n/assignments/a3.zip)] [[handout](http://web.stanford.edu/class/cs224n/assignments/a3.pdf)] | Assignment 2 **due** | done | | Thu July 4 | The probability of a sentence? Recurrent Neural Networks and Language Models [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture06-rnnlm.pdf)] [[video](https://youtu.be/iWea12EAu6U)] [[notes (lectures 6 and 7)](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes05-LM_RNN.pdf)] | Suggested Readings:[N-gram Language Models](https://web.stanford.edu/~jurafsky/slp3/3.pdf) (textbook chapter)[The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)(blog post overview)[Sequence Modeling: Recurrent and Recursive Neural Nets](http://www.deeplearningbook.org/contents/rnn.html)(Sections 10.1 and 10.2)[On Chomsky and the Two Cultures of Statistical Learning](http://norvig.com/chomsky.html) | | 添加中期总结 | | | Tue July 9 | Vanishing Gradients and Fancy RNNs [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture07-fancy-rnn.pdf)] [[video](https://youtu.be/QEw0qEa0E50)] [[notes (lectures 6 and 7)](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes05-LM_RNN.pdf)] | Suggested Readings:[Sequence Modeling: Recurrent and Recursive Neural Nets](http://www.deeplearningbook.org/contents/rnn.html)(Sections 10.3, 10.5, 10.7-10.12)[Learning long-term dependencies with gradient descent is difficult](http://ai.dinfo.unifi.it/paolo//ps/tnn-94-gradient.pdf) (one of the original vanishing gradient papers)[On the difficulty of training Recurrent Neural Networks](https://arxiv.org/pdf/1211.5063.pdf) (proof of vanishing gradient problem)[Vanishing Gradients Jupyter Notebook](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1174/lectures/vanishing_grad_example.html) (demo for feedforward networks)[Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) (blog post overview) | Assignment 4 **out** [[code](http://web.stanford.edu/class/cs224n/assignments/a4.zip)] [[handout](http://web.stanford.edu/class/cs224n/assignments/a4.pdf)] [[Azure Guide](https://docs.google.com/document/d/1MHaQvbtPkfEGc93hxZpVhkKum1j_F1qsyJ4X0vktUDI/edit)] [[Practical Guide to VMs](https://docs.google.com/document/d/1z9ST0IvxHQ3HXSAOmpcVbFU5zesMeTtAc9km6LAPJxk/edit)] | Assignment 3 **due** | | | Thu July 11 | Machine Translation, Seq2Seq and Attention [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture08-nmt.pdf)] [[video](https://youtu.be/XXtpJxZBa2c)] [[notes](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes06-NMT_seq2seq_attention.pdf)] | Suggested Readings:[Statistical Machine Translation slides, CS224n 2015](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1162/syllabus.shtml) (lectures 2/3/4)[Statistical Machine Translation](https://www.cambridge.org/core/books/statistical-machine-translation/94EADF9F680558E13BE759997553CDE5) (book by Philipp Koehn)[BLEU](https://www.aclweb.org/anthology/P02-1040.pdf) (original paper)[Sequence to Sequence Learning with Neural Networks](https://arxiv.org/pdf/1409.3215.pdf) (original seq2seq NMT paper)[Sequence Transduction with Recurrent Neural Networks](https://arxiv.org/pdf/1211.3711.pdf) (early seq2seq speech recognition paper)[Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/pdf/1409.0473.pdf) (original seq2seq+attention paper)[Attention and Augmented Recurrent Neural Networks](https://distill.pub/2016/augmented-rnns/) (blog post overview)[Massive Exploration of Neural Machine Translation Architectures](https://arxiv.org/pdf/1703.03906.pdf)(practical advice for hyperparameter choices) | | | | | Tue July 16 | Practical Tips for Final Projects [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture09-final-projects.pdf)] [[video](https://youtu.be/fyqm8fRDgl0)] [[notes](http://web.stanford.edu/class/cs224n/readings/final-project-practical-tips.pdf)] | Suggested Readings:[Practical Methodology](https://www.deeplearningbook.org/contents/guidelines.html) (*Deep Learning* book chapter) | | | | | Thu July 18 | Question Answering and the Default Final Project [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture10-QA.pdf)] [[video](https://youtu.be/yIdF-17HwSk)] [[notes](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes07-QA.pdf)] | | Project Proposal **out** [[instructions](http://web.stanford.edu/class/cs224n/project/project-proposal-instructions.pdf)] Default Final Project **out**[[handout](http://web.stanford.edu/class/cs224n/project/default-final-project-handout.pdf)] [[code](https://github.com/chrischute/squad)] | Assignment 4 **due** | | | Tue July 23 | ConvNets for NLP [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture11-convnets.pdf)] [[video](https://youtu.be/EAJoRA0KX7I)] [[notes](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes08-CNN.pdf)] | Suggested Readings:[Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882)[A Convolutional Neural Network for Modelling Sentences](https://arxiv.org/pdf/1404.2188.pdf) | | | | | Thu July 25 | Information from parts of words: Subword Models [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture12-subwords.pdf)] [[video](https://youtu.be/9oTHFx0Gg3Q)] | | Assignment 5 **out** [[original code (requires Stanford login)](https://stanford.box.com/s/t4nlmcc08t9k6mflz6sthjlmjs7lip6p)/ [public version](http://web.stanford.edu/class/cs224n/assignments/a5_public.zip)] [[handout](http://web.stanford.edu/class/cs224n/assignments/a5.pdf)] | Project Proposal **due** | | | Tue July 30 | Modeling contexts of use: Contextual Representations and Pretraining [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture13-contextual-representations.pdf)] [[video](https://youtu.be/S-CspeZ8FHc)] | Suggested readings:Smith, Noah A. [Contextual Word Representations: A Contextual Introduction](https://arxiv.org/abs/1902.06006). (Published just in time for this lecture!) | | | | | Thu Aug 1 | Transformers and Self-Attention For Generative Models *(guest lecture by Ashish Vaswaniand Anna Huang)* [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf)] [[video](https://youtu.be/5vcj8kSwBCY)] | Suggested readings:[Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf)[Image Transformer](https://arxiv.org/pdf/1802.05751.pdf)[Music Transformer: Generating music with long-term structure](https://arxiv.org/pdf/1809.04281.pdf) | | | | | Fri Aug 2 | | | Project Milestone **out** [[instructions](http://web.stanford.edu/class/cs224n/project/project-milestone-instructions.pdf)] | Assignment 5 **due** | | | Tue Aug 6 | Natural Language Generation [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture15-nlg.pdf)] [[video](https://youtu.be/4uG1NMKNWCU)] | | | | | | Thu Aug 8 | Reference in Language and Coreference Resolution [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture16-coref.pdf)] [[video](https://youtu.be/i19m4GzBhfc)] | | | | | | Tue Aug 13 | Multitask Learning: A general model for NLP? *(guest lecture by Richard Socher)* [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture17-multitask.pdf)] [[video](https://youtu.be/M8dsZsEtEsg)] | | | Project Milestone **due** | | | Thu Aug 15 | Constituency Parsing and Tree Recursive Neural Networks [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture18-TreeRNNs.pdf)] [[video](https://youtu.be/6Z4A3RSf-HY)] [[notes](http://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes09-RecursiveNN_constituencyparsing.pdf)] | Suggested Readings:[Parsing with Compositional Vector Grammars.](http://www.aclweb.org/anthology/P13-1045)[Constituency Parsing with a Self-Attentive Encoder](https://arxiv.org/pdf/1805.01052.pdf) | | | | | Tue Aug 20 | Safety, Bias, and Fairness *(guest lecture by Margaret Mitchell)* [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture19-bias.pdf)] [[video](https://youtu.be/XR8YSRcuVLE)] | | | | | | Thu Aug 22 | Future of NLP + Deep Learning [[slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture20-future.pdf)] [[video](https://youtu.be/3wWZBGN-iX8)] | | | | | | Sun Aug 24 | | | | **Final Project Report due**[[instructions](http://web.stanford.edu/class/cs224n/project/project-report-instructions.pdf)] | | | Wed Aug 28 | **Final project poster session** [[details](https://www.facebook.com/events/1218481914969541)] | 5:15 - 8:30pm McCaw Hall at the Alumni Center [[map](https://alumni.stanford.edu/get/page/resources/alumnicenter/directions)] | | **Project Poster/Video due**[[instructions](http://web.stanford.edu/class/cs224n/project/project-postervideo-instructions.pdf)] | | Reference: 1. https://github.com/ankit-ai/cs224n-natural-language-processing-winter2019 2. https://github.com/ZacBi/CS224n-2019-solutions 3. https://blog.csdn.net/bqw18744018044/article/details/83120425 4. https://github.com/Observerspy/CS224n