# Reinforcement-Learning-2nd-Edition-by-Sutton-Exercise-Solutions **Repository Path**: terry_tian/Reinforcement-Learning-2nd-Edition-by-Sutton-Exercise-Solutions ## Basic Information - **Project Name**: Reinforcement-Learning-2nd-Edition-by-Sutton-Exercise-Solutions - **Description**: No description available - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 1 - **Created**: 2020-02-27 - **Last Updated**: 2022-05-16 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # Solutions of Reinforcement Learning 2nd Edition ([Original Book](https://www.amazon.co.jp/exec/obidos/ASIN/0262039249/hatena-blog-22/) by Richard S. Sutton,Andrew G. Barto) ## Right now I have not yet finished the rest of Chapter 12. Due to many learners' requests, I will continue working on the solution but please wait for further notification. I plan to finish it in Feburary. ## I had some issue from original author but I dont care any more. This project will be finished. But I have to warn: ## Those students who are using this to complete your homework, stop it. They KNOW this repo and WILL catch you. This is written for serving millions of self-learners who do not have official guide or proper learning environment. Welcome to this project. It is a tiny project where we don't do too much coding (yet) but we cooperate together to finish some tricky exercises from famous RL book Reinforcement Learning, An Introduction by Sutton. You may know that this book, especially the second version which was published last year, has no official solution manual. If you send your answer to the email address that the author leaved, you will be returned a fake answer sheet that is incomplete and old. So, why don't we write our own? Most of problems are mathematical proof in which one can learn the therotical backbone nicely but some of them are quite challenging coding problems. Both of them will be updated gradually but math will go first. Main author would be me and current main cooperater is **Jean Wissam Dupin**, and before was Zhiqi Pan (quitted now). Jean's Work (Error Fix): Chapter 3: Ex 3.8, 3.11, 3.14, 3.23, 3.24, 3.26, 3.28, 3.29, 4.5 ABOUT MISTAKES: Don't even expect the solutions be perfect, there are always mistakes. And, sometimes the problems are just open. Show your ideas and question them in 'issues' at any time! Let's roll'n out! [UPDATE JAN 2020] Future works will NOT be stopped. I will try to finish it in FEB 2020. [UPDATE JAN 2020] Future works will be stopped. Unfinished work of Chapter 12 is published and later work on Chapter 13 and 14 are hidden from public. [UPDATE JAN 2020] Chapter 12's ideas are not so hard but questions are very difficult. (most chanllenging one in this book ). As far, I have finished up to Ex 12.5 and I think my answer of Ex 12.1 is the only valid one on the internet (or not, challenge welcomed!) But because later half is even more challenging (tedious when it is related to many infiite sums), I would release the final version little bit later. [UPDATE JAN 2020] Chapter 11 updated. One might have to read the referenced link to Sutton's paper in order to understand some part. Espeically how and why Emphatic-TD works. [UPDATE JAN 2020] Chapter 10 is long but interesting! Move on! [UPDATE DEC 2019] Chapter 9 takes long time to read thoroughly but practices are surprisingly just a few. So after uploading the Chapter 9 pdf and I really do think I should go back to previous chapters to complete those programming practices. # Chapter 12 Unfinished on purpose. EX12.1 is important for understanding later chapters. [CHAPTER 12 SOLUTION PDF HERE](https://github.com/LyWangPX/Reinforcement-Learning-2nd-Edition-by-Sutton-Exercise-Solutions/tree/master/Chapter%2012) # Chapter 11 Major challenges about off-policy learning. Like Chapter 9, practices are short. [CHAPTER 11 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%2011/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_11_r1.pdf) # Chapter 10 It is a substantial complement to Chapter 9. Still many open problems which are very interesting. [CHAPTER 10 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%2010/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_10.pdf) # Chapter 9 Long chapter, short practices. [CHAPTER 9 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%209/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_9.pdf) # Chapter 8 Finished without programming. Plan on creating additional exercises to this Chapter because many materials are lack of practice. [CHAPTER 8 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%208/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_8.pdf) # Chapter 7 Finished without programming. Thanks for help from Zhiqi Pan. [CHAPTER 7 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%207/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_7_r2.pdf) # Chapter 6 Finished without programming [CHAPTER 6 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%206/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_6_r2.pdf) # Chapter 5 Partially finished. [CHAPTER 5 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%205/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_5_r1.pdf) # Chapter 4 Finished. Ex4.7 Partially finished. Dat DP question will burn my mind and macbook but I encourage any one who cares nothing about that trying to do yourself. Running through it forces you remember everything behind ordinary DP.:) [CHAPTER 4 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%204/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_4_r5.pdf) # Chapter 3 [CHAPTER 3 SOLUTION PDF HERE](https://github.com/LyWangPX/Solutions-of-Reinforcement-Learning-An-Introduction-Sutton-2nd/blob/master/Chapter%203/Solutions_to_Reinforcement_Learning_by_Sutton_Chapter_3_r5.pdf)