Hellow guys, Welcome to my website, and you are watching Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer. and this vIdeo is uploaded by Umar Jamil at 2023-12-26T21:29:08-08:00. We are pramote this video only for entertainment and educational perpose only. So, I hop you like our website.
| Name | Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer |
| Video Uploader | Video From Umar Jamil |
| Upload Date | This Video Uploaded At 27-12-2023 05:29:08 |
| Video Discription | In this video I will be introducing all the innovations in the Mistral 7B and Mixtral 8x7B model: Sliding Window Attention, KV-Cache with Rolling Buffer, Pre-Fill and Chunking, Sparse Mixture of Experts (SMoE); I will also guide you in understanding the most difficult part of the code: Model Sharding and the use of xformers library to compute the attention for multiple prompts packed into a single sequence. In particular I will show the attention computed using BlockDiagonalCausalMask, BlockDiagonalMask and BlockDiagonalCausalWithOffsetPaddedKeysMask. I will also show you why the Sliding Window Attention allows a token to "attend" other tokens outside the attention window by linking it with the concept of Receptive Field, typical of Convolutional Neural Networks (CNNs). Of course I will prove it mathematically. When introducing Model Sharding, I will also talk about Pipeline Parallelism, because in the official mistral repository they refer to microbatching. I release a copy of the Mistral code commented and annotated by me (especially the most difficult parts): https://github.com/hkproj/mistral-src-commented Slides PDF and Python Notebooks: https://github.com/hkproj/mistral-llm-notes Prerequisite for watching this video: https://www.youtube.com/watch?v=bCz4OMemCcA Other material for better understanding Mistral: Grouped Query Attention, Rotary Positional Encodings, RMS Normalization: https://youtu.be/Mn_9W1nCFLo Gradient Accumulation: https://youtu.be/toUSzwR0EV8 Chapters 00:00:00 - Introduction 00:02:09 - Transformer vs Mistral 00:05:35 - Mistral 7B vs Mistral 8x7B 00:08:25 - Sliding Window Attention 00:33:44 - KV-Cache with Rolling Buffer Cache 00:49:27 - Pre-Fill and Chunking 00:57:00 - Sparse Mixture of Experts (SMoE) 01:04:22 - Model Sharding 01:06:14 - Pipeline Parallelism 01:11:11 - xformers (block attention) 01:24:07 - Conclusion |
| Category | Science & Technology |
| Tags | deep learning | pytorch | ai | ml | machine learning | paper review | large language model | mistral | mistral ai | mistral 7b | mistral 8x7b | pipeline parallelism | kv-cache | kv cache | sparse mixture of experts | mixture of experts | moe | smoe | sliding window attention | model sharding | prefill | chunking | xformers | BlockDiagonalCausalMask | BlockDiagonalMask | BlockDiagonalCausalWithOffsetPaddedKeysMask |
