Skip to content

Latest commit

 

History

History
211 lines (181 loc) · 13.7 KB

README.md

File metadata and controls

211 lines (181 loc) · 13.7 KB
Raw

Deep Learning Topics and Resources

description

Resources for DL in General

  1. Blogs
    • Lilian Weng’s Blog [link]
    • AI Summer Blog [link]
    • Colah’s Blog [link]
  2. Books
    • Neural Networks and Deep Learning [link]
    • Deep Learning Book [link]
    • Dive into Deep Learning [link]
    • Reinforcement Learning: An Introduction | Sutton and Barto [link]
  3. Open Courses

Mathematics

  1. Linear Algebra ([notes][practice questions])

    • 3Blue1Brown essence of linear algebra [youtube]
    • Gilbert Strang’s lectures on Linear Algebra [link] [youtube]
    • Topics
      • Linear Transformations
      • Linear Dependence and Span
      • Eigendecomposition - Eigenvalues and Eigenvectors
      • Singular Value Decomposition [blog]

  2. Probability and Statistics ([notes][youtube series])

    • Harvard Statistics 110: Probability [link] [youtube]
    • Topics
      • Expectation, Variance, and Co-variance
      • Distributions
      • Random Walks
      • Bias and Variance
        • Bias Variance Trade-off
      • Estimators
        • Biased and Unbiased
      • Maximum Likelihood Estimation [blog]
      • Maximum A-Posteriori (MAP) Estimation [blog]

  3. Information Theory [youtube]

    • (Shannon) Entropy [blog]
    • Cross Entropy, KL Divergence [blog]
    • KL Divergence
      • Not a distance metric (unsymmetric)
      • Derivation from likelihood ratio (Blog)
      • Always greater than 0
      • Relation with Entropy (Explanation)

Basics

  1. Neural Networks Overview [youtube]
  2. Backpropogation
    • Vanilla [blog]
    • Backpropagation in CNNs [blog]
    • Backprop through time [blog]
  3. Loss Functions
    • MSE Loss
      • Derivation by MLE and MAP
    • Cross Entropy Loss
      • Binary Cross Entropy
      • Categorical Cross Entropy
  4. Activation Functions (Sigmoid, Tanh, ReLU and variants) (blog)
  5. Optimizers
  6. Regularization
    • Early Stopping
    • Noise Injection
    • Dataset Augmentation
    • Ensembling
    • Parameter Norm Penalties
      • L1 (sparsity)
      • L2 (smaller parameter values)
    • BatchNorm [Paper]
      • Internal Covariate Shift
      • BatchNorm in CNNs [Link]
      • Backprop through BatchNorm Layer [Explanation]
    • Dropout Regularization [Paper]

Computer Vision

  1. Convolution [youtube]

    • Cross-correlation
    • Pooling (Average, Max Pool)
    • Strides and Padding
    • Output volume dimension calculation
    • Deconvolution (Transposed Convolution), Upsampling, Reverse Pooling [Visualization]
    • Types of convolution operation [blog]

  2. ImageNet Classification

  3. Object Detection [blog series]

  4. Semantic Segmentation

Natural Language Processing

  1. Recurrent Neural Networks

    • Architectures (Limitations and inspiration behind every model)
    • Vanishing and Exploding Gradients

  2. Word Embeddings [blog_1] [blog_2]

    • Word2Vec
    • CBOW
    • Glove
    • SkipGram, NGram
    • FastText
    • ELMO
    • BERT

  3. Transformers [blog posts] [youtube series]

    • Attention is All You Need [blog] [paper] [annotated transformer]
    • Query-Key-Value Attention Mechanism (Quadratic Time)
    • Position Embeddings [blog]
    • BERT (Masked Language Modelling) [blog]
    • Longe Range Sequence Modelling [blog]
    • ELECTRA (Pretraining Transformers as Discriminators) [blog]
    • GPT (Causal Language Modelling) [blog]
    • OpenAI ChatGPT [blog]

Multimodal Learning

  • Vision Language Models | AI Summer [blog]
  • Open AI DALL-E [blog]
  • OpenAI CLIP [blog]
  • Flamingo [blog]
  • Gato [blog]
  • data2vec [blog]
  • OpenAI Whisper [blog]

Generative Models

  1. Generative Adversarial Networks (GANs) [blog series]
    • Basic Idea
    • Variants
    • Mode Collapse
    • GAN Hacks [link]
  2. Variational Autoencoders (VAEs)
    • Variational Inference [tutorial paper]
    • ELBO and Loss Function derivation
  3. Normalizing Flows
    • Basic Idea and Applications [link]

Stable Diffusion

  • Demos

    • Lexica (Stable Diffusion search engine) [link]
    • Stability AI | Huggingface Spaces [link]

  • Diffusion Models in general [paper]

    • What are Diffusion Models? | Lil'Log [link]

  • Stable Diffusion | Stability AI [blog] [annotated stable diffusion]

  • Illustrated Stable DIffusion | Jay Alammar [blog]

  • Stable Diffusion in downstream Vision tasks

Keeping up with the developments in Deep Learning

  • Youtube Channels
    • Yannic Kilcher [link]
    • Two Minute Papers [link]
  • Blogs
    • DeepMind Blog [link]
    • OpenAI Blog [link]
    • Google AI Blog [link]
    • Meta AI Blog [link]
    • Nvidia - Deep Learning Blog [link]
    • Microsoft Research Blog [link]
  • Trending Reseach Papers
    • labml [link]
    • deep learning monitor [link]

Contributing

We welcome contributions to add resources such as notes, blogs, or papers for a topic. Feel free to open a pull request for the same!