Movatterモバイル変換

[0]ホーム
URL:

YH
Uploaded byYao-Chieh Hu
PPTX, PDF1,413 views

RNN & LSTM: Neural Network for Sequential Data

Recurrent neural networks (RNNs) and long short-term memory (LSTM) networks can process sequential data like text and time series data. RNNs have memory and can perform the same task for every element in a sequence, but struggle with long-term dependencies. LSTMs address this issue using memory cells and gates that allow them to learn long-term dependencies. LSTMs have four interacting layers - a forget gate, input gate, cell state, and output gate that allow them to store and access information over long periods of time. RNNs and LSTMs are applied to tasks like language modeling, machine translation, speech recognition, and image caption generation.

In this document
Powered by AI

Overview of RNN and LSTM as neural networks designed for processing sequential data.

Categorization of machine learning: Supervised, Unsupervised, and Reinforcement Learning with various examples.

Introduction to the K-Nearest Neighbor algorithm for classification.

Discussion on how neural networks can reveal hidden relationships within features.

Intro to RNNs, their sequential process capabilities, and applications in various fields.

Method of training RNNs using Backpropagation Through Time (BPTT).

Different types of RNNs including bidirectional and LSTM networks. Detailed explanation of LSTM architecture, including cell state and gated mechanisms.

Stepwise explanation of LSTM gate operations, including forget, input, and output decisions.

Different variations of LSTM including peephole connections, coupled gates, and GRUs.

Discussion on the effectiveness of RNN and LSTM models.

Examples of multiple applications of RNN models.

Concept of Turing-Completeness in relation to RNNs and their capability to simulate programs.

Application of RNNs on non-sequential data through sequential processing.

Resources showcasing interesting applications of RNN and LSTM architectures.

Compilation of references and resources for deeper understanding of RNNs, LSTMs, and ML.

Embed presentation

Downloaded 108 times
RNN & LSTMNeural Network for Sequential Data- Jeff Hu -
Machine Learning Categories• Supervised• Unsupervised• Reinforcement Learning
Supervised Machine Learning• Training Set: Inputs + Outputs• Learn a link between the inputs and the outputs• Linear and logistic regression• Support vector machine• K-nearest neighbors (k-NN)• Naive Bayes• Neural network• Gradient boosting• Classification trees and random forest
Unsupervised Machine Learning• Training Set: Inputs• Cluster the inputs• K-means• Hierarchical clustering• Mixture models• PCA• ICA• Auto-encoder
Reinforcement Learning• Training Set: N/A• Find the best way to earn the greatest reward• Utility learning• Q-learning
K-Nearest Neighbor
Neural Network > Machine Learning ?• Consider hidden relationships between features!
Recurrent Neural Network (RNN)
Benefits• Deal with sequential information• Perform the same task for every element of a sequence• Has memory• Can be unrolled like a chain
Application• Language Modeling and Generating Text• Machine Translation• Speech Recognition• Generating Image Descriptions
Training• Backpropagation Through Time (BPTT)
Variations• Bidirectional RNNs• Deep (Bidirectional) RNNs• LSTM networks
Long Short Term Memory Network(LSTM)
Memory Problem of RNN• Sometimes we need more context• RNN is unable to connect the information further in the past
Benefits of LSTM• Can learn long-term dependencies
Difference between RNN & LSTM• RNN: single layer (tanh)• LSTM: four interactive layers
Cell state• The conveyor belt
Gates (3 in total for LSTM)• A way that let information through• E.g. A sigmoid neural net layer & a pointwise multiplication operation
Optional Math – Sigmoid function
Step 1: Forget Gate Layer• Decide what info to throw away• Look at h[t-1] and x[t] and output a number 0~1 to decide how much cell state to keep C[t-1]• E.g. When see a new subject, we want to forget the gender of the old subject
Step 2: Input Gate Layer• Decide what info to add• A sigmoid: decide which value to update• A tanh layer: create a new candidate value C~[t]• E.g. add a new gender of the new subject
Step 3: Combine step 1 & 2• Combine step 1 & 2• Multiply the old state by f[t]: to forget the things• Add i[t] * C~[t] : to add new candidate value (scaled)
Step 4: Filter/output the Cell state• Decide what to output• sigmoid: decide which part to output• tanh: push the value to be between -1 ~ 1• Multiply them to only output the part we decided to• E.g. output a info related to a Verb• E.g. output whether the subject it singular or plural
Step 4: Filter/output the Cell state• Decide what to output
Variants on LSTM (1)• Peephole: let the gate layer look at the cell state (entire/ partial)
Variants on LSTM (2)• Coupled forgot and input gates: Not deciding separately f[t] * C[t-1] + (1-f[t]) * C~[t]
Variants on LSTM (3)• Gated Recurrent Unit (GRU): combine the forget and input layer into a single “update gate” merge the cell state and the hidden state simpler and popular
RNN / LSTM Effectiveness
Multiple types of RNN use cases
Turing-Complete• Running a fixed program with certain inputs and some internal variables (can simulatearbitrary programs)• Andrej Karpathy (Ph.D. @ Stanford):
Non-sequential data• Though the data is not in form of sequences, we can still use RNN by processit sequentially.
Some cool RNN/LSTM applications• http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Great references• [1] RNN: http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/?subscribe=success#blog_subscription-2• [2] LSTM: http://colah.github.io/posts/2015-08-Understanding-LSTMs/• [3] RNN Effectiveness: http://karpathy.github.io/2015/05/21/rnn-effectiveness/• [4] Backpropagation: http://cs231n.github.io/optimization-2/#backprop• [5] ML categories: http://enhancedatascience.com/2017/07/19/machine-learning-explained-supervised-learning-unsupervised-learning-and-reinforcement-learning/

Recommended

PDF
Recurrent Neural Networks. Part 1: Theory
PPTX
PDF
Long Short Term Memory
 
PDF
Backpropagation in RNN and LSTM
PDF
LSTM Tutorial
PDF
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
PPTX
RNN-LSTM.pptx
PDF
LSTM
PDF
Rnn and lstm
PDF
Recurrent neural networks rnn
PDF
Introduction to Recurrent Neural Network
PPTX
Introduction to CNN
PDF
Recurrent Neural Networks, LSTM and GRU
 
PPTX
Deep Learning - RNN and CNN
PPTX
Long Short Term Memory LSTM
PDF
LSTM Basics
PPTX
Recurrent neural network
PPTX
04 Multi-layer Feedforward Networks
PPTX
Activation functions
PPTX
Convolutional neural network
PDF
Deep Learning: Recurrent Neural Network (Chapter 10)
PPTX
Recurrent Neural Network
PPT
Perceptron
PPTX
Feedforward neural network
PPTX
PDF
RNN and its applications
PDF
Introduction to Recurrent Neural Network
 
PPS
Neural Networks
PPTX
recurrent_neural_networks_april_2020.pptx
PPTX
RNN and LSTM model description and working advantages and disadvantages

More Related Content

PDF
Recurrent Neural Networks. Part 1: Theory
PPTX
PDF
Long Short Term Memory
 
PDF
Backpropagation in RNN and LSTM
PDF
LSTM Tutorial
PDF
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
PPTX
RNN-LSTM.pptx
PDF
LSTM
Recurrent Neural Networks. Part 1: Theory
Long Short Term Memory
 
Backpropagation in RNN and LSTM
LSTM Tutorial
Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorf...
RNN-LSTM.pptx
LSTM

What's hot

PDF
Rnn and lstm
PDF
Recurrent neural networks rnn
PDF
Introduction to Recurrent Neural Network
PPTX
Introduction to CNN
PDF
Recurrent Neural Networks, LSTM and GRU
 
PPTX
Deep Learning - RNN and CNN
PPTX
Long Short Term Memory LSTM
PDF
LSTM Basics
PPTX
Recurrent neural network
PPTX
04 Multi-layer Feedforward Networks
PPTX
Activation functions
PPTX
Convolutional neural network
PDF
Deep Learning: Recurrent Neural Network (Chapter 10)
PPTX
Recurrent Neural Network
PPT
Perceptron
PPTX
Feedforward neural network
PPTX
PDF
RNN and its applications
PDF
Introduction to Recurrent Neural Network
 
PPS
Neural Networks
Rnn and lstm
Recurrent neural networks rnn
Introduction to Recurrent Neural Network
Introduction to CNN
Recurrent Neural Networks, LSTM and GRU
 
Deep Learning - RNN and CNN
Long Short Term Memory LSTM
LSTM Basics
Recurrent neural network
04 Multi-layer Feedforward Networks
Activation functions
Convolutional neural network
Deep Learning: Recurrent Neural Network (Chapter 10)
Recurrent Neural Network
Perceptron
Feedforward neural network
RNN and its applications
Introduction to Recurrent Neural Network
 
Neural Networks

Similar to RNN & LSTM: Neural Network for Sequential Data

PPTX
recurrent_neural_networks_april_2020.pptx
PPTX
RNN and LSTM model description and working advantages and disadvantages
PDF
rnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
PPTX
DEEP LEARNING -Recurrent Neural Networks
PDF
Recurrent Neural Networks (D2L8 Insight@DCU Machine Learning Workshop 2017)
PDF
Video Analysis with Recurrent Neural Networks (Master Computer Vision Barcelo...
PPTX
10.0 SequenceModeling-merged-compressed_edited.pptx
PDF
Recurrent Neural Networks I (D2L2 Deep Learning for Speech and Language UPC 2...
PDF
Recurrent and Recursive Nets (part 2)
PPT
14889574 dl ml RNN Deeplearning MMMm.ppt
PDF
Recurrent Neural Networks A Deep Dive in 2025.pdf
PDF
Recurrent Neural Networks (D2L2 2017 UPC Deep Learning for Computer Vision)
PDF
Deep Learning for Computer Vision: Recurrent Neural Networks (UPC 2016)
PDF
Sequence Modelling with Deep Learning
PDF
Recurrent Neural Networks RNN - Xavier Giro - UPC TelecomBCN Barcelona 2020
PDF
Recurrent Neural Networks
PDF
Concepts of Temporal CNN, Recurrent Neural Network, Attention
PPT
Recurrent neural network power point presentation
PPT
Recurrent neural network power point presentation
PPTX
Introduction to deep learning
recurrent_neural_networks_april_2020.pptx
RNN and LSTM model description and working advantages and disadvantages
rnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
DEEP LEARNING -Recurrent Neural Networks
Recurrent Neural Networks (D2L8 Insight@DCU Machine Learning Workshop 2017)
Video Analysis with Recurrent Neural Networks (Master Computer Vision Barcelo...
10.0 SequenceModeling-merged-compressed_edited.pptx
Recurrent Neural Networks I (D2L2 Deep Learning for Speech and Language UPC 2...
Recurrent and Recursive Nets (part 2)
14889574 dl ml RNN Deeplearning MMMm.ppt
Recurrent Neural Networks A Deep Dive in 2025.pdf
Recurrent Neural Networks (D2L2 2017 UPC Deep Learning for Computer Vision)
Deep Learning for Computer Vision: Recurrent Neural Networks (UPC 2016)
Sequence Modelling with Deep Learning
Recurrent Neural Networks RNN - Xavier Giro - UPC TelecomBCN Barcelona 2020
Recurrent Neural Networks
Concepts of Temporal CNN, Recurrent Neural Network, Attention
Recurrent neural network power point presentation
Recurrent neural network power point presentation
Introduction to deep learning

Recently uploaded

PPTX
SAC 28 Finals Data Insights of Startup Investing 2025 1119.pptx
PPT
Transportation Management Plan (Effectiveness Study).ppt
PDF
Glass Facade Treatments and designs in architecture
PDF
Social Issues in Pakistan.pdf about social issues in pakistan
PPT
fgfdgdgdgdfgerst4t4tqwweqe3eawdsfsdsdsdd
PPTX
ammonia process.pptx Amisha Group 7.pptxAmisha Group 7.pptxAmisha Group 7.pptx
PDF
2= https___10-0-0-0-1.ph_.pdf10.0.0.0.1,
PPT
Acids and alkalis ppt.ppt SVfhasvas jhsvdhsvduisa
PPT
atomic structure2.ppt qwgdwgdwidgqwuidg wdhwgdwgwydg qwgdwydgwydg87
PPTX
Engineering Project Proposal by Slidesgo.pptx
PPTX
Introduction to Data Warehouse and Schema.pptx
PDF
MariaDB Monitoring for Enhancing Performance, Availability, and Security
PPTX
Chapter 4, Blocksmbjkhnl;m',\;lkjhcgjvbkln;m.pptx
PDF
The Gemini Advantage: A Strategic Overview of Google’s Multimodal AI Ecosystem
PPTX
MC25104 - Data structures and algorithms using python Python_OOP_Theory_and_P...
PDF
Interpretability and Explainability Module 4.pdf
PPTX
Correlation-Regression analysis -16.11.25.pptx
PDF
Steven Imrisek | Quantitative UX Research Portfolio: Two Digital Health Case ...
PPTX
MULTIPLE REGRESSION AND CORRELATION OF IMPORTS, THE NATIONAL INCOME AND RELAT...
PPTX
spam detacting system.pptx for mca students
SAC 28 Finals Data Insights of Startup Investing 2025 1119.pptx
Transportation Management Plan (Effectiveness Study).ppt
Glass Facade Treatments and designs in architecture
Social Issues in Pakistan.pdf about social issues in pakistan
fgfdgdgdgdfgerst4t4tqwweqe3eawdsfsdsdsdd
ammonia process.pptx Amisha Group 7.pptxAmisha Group 7.pptxAmisha Group 7.pptx
2= https___10-0-0-0-1.ph_.pdf10.0.0.0.1,
Acids and alkalis ppt.ppt SVfhasvas jhsvdhsvduisa
atomic structure2.ppt qwgdwgdwidgqwuidg wdhwgdwgwydg qwgdwydgwydg87
Engineering Project Proposal by Slidesgo.pptx
Introduction to Data Warehouse and Schema.pptx
MariaDB Monitoring for Enhancing Performance, Availability, and Security
Chapter 4, Blocksmbjkhnl;m',\;lkjhcgjvbkln;m.pptx
The Gemini Advantage: A Strategic Overview of Google’s Multimodal AI Ecosystem
MC25104 - Data structures and algorithms using python Python_OOP_Theory_and_P...
Interpretability and Explainability Module 4.pdf
Correlation-Regression analysis -16.11.25.pptx
Steven Imrisek | Quantitative UX Research Portfolio: Two Digital Health Case ...
MULTIPLE REGRESSION AND CORRELATION OF IMPORTS, THE NATIONAL INCOME AND RELAT...
spam detacting system.pptx for mca students

RNN & LSTM: Neural Network for Sequential Data

  • 1.
    RNN & LSTMNeuralNetwork for Sequential Data- Jeff Hu -
  • 2.
    Machine Learning Categories•Supervised• Unsupervised• Reinforcement Learning
  • 3.
    Supervised Machine Learning•Training Set: Inputs + Outputs• Learn a link between the inputs and the outputs• Linear and logistic regression• Support vector machine• K-nearest neighbors (k-NN)• Naive Bayes• Neural network• Gradient boosting• Classification trees and random forest
  • 4.
    Unsupervised Machine Learning•Training Set: Inputs• Cluster the inputs• K-means• Hierarchical clustering• Mixture models• PCA• ICA• Auto-encoder
  • 5.
    Reinforcement Learning• TrainingSet: N/A• Find the best way to earn the greatest reward• Utility learning• Q-learning
  • 6.
  • 7.
    Neural Network >Machine Learning ?• Consider hidden relationships between features!
  • 8.
  • 10.
    Benefits• Deal withsequential information• Perform the same task for every element of a sequence• Has memory• Can be unrolled like a chain
  • 11.
    Application• Language Modelingand Generating Text• Machine Translation• Speech Recognition• Generating Image Descriptions
  • 12.
  • 13.
    Variations• Bidirectional RNNs•Deep (Bidirectional) RNNs• LSTM networks
  • 14.
    Long Short TermMemory Network(LSTM)
  • 16.
    Memory Problem ofRNN• Sometimes we need more context• RNN is unable to connect the information further in the past
  • 17.
    Benefits of LSTM•Can learn long-term dependencies
  • 18.
    Difference between RNN& LSTM• RNN: single layer (tanh)• LSTM: four interactive layers
  • 19.
    Cell state• Theconveyor belt
  • 20.
    Gates (3 intotal for LSTM)• A way that let information through• E.g. A sigmoid neural net layer & a pointwise multiplication operation
  • 21.
    Optional Math –Sigmoid function
  • 22.
    Step 1: ForgetGate Layer• Decide what info to throw away• Look at h[t-1] and x[t] and output a number 0~1 to decide how much cell state to keep C[t-1]• E.g. When see a new subject, we want to forget the gender of the old subject
  • 23.
    Step 2: InputGate Layer• Decide what info to add• A sigmoid: decide which value to update• A tanh layer: create a new candidate value C~[t]• E.g. add a new gender of the new subject
  • 24.
    Step 3: Combinestep 1 & 2• Combine step 1 & 2• Multiply the old state by f[t]: to forget the things• Add i[t] * C~[t] : to add new candidate value (scaled)
  • 25.
    Step 4: Filter/outputthe Cell state• Decide what to output• sigmoid: decide which part to output• tanh: push the value to be between -1 ~ 1• Multiply them to only output the part we decided to• E.g. output a info related to a Verb• E.g. output whether the subject it singular or plural
  • 26.
    Step 4: Filter/outputthe Cell state• Decide what to output
  • 27.
    Variants on LSTM(1)• Peephole: let the gate layer look at the cell state (entire/ partial)
  • 28.
    Variants on LSTM(2)• Coupled forgot and input gates: Not deciding separately f[t] * C[t-1] + (1-f[t]) * C~[t]
  • 29.
    Variants on LSTM(3)• Gated Recurrent Unit (GRU): combine the forget and input layer into a single “update gate” merge the cell state and the hidden state simpler and popular
  • 30.
    RNN / LSTMEffectiveness
  • 31.
    Multiple types ofRNN use cases
  • 32.
    Turing-Complete• Running afixed program with certain inputs and some internal variables (can simulatearbitrary programs)• Andrej Karpathy (Ph.D. @ Stanford):
  • 33.
    Non-sequential data• Thoughthe data is not in form of sequences, we can still use RNN by processit sequentially.
  • 34.
    Some cool RNN/LSTMapplications• http://karpathy.github.io/2015/05/21/rnn-effectiveness/
  • 35.
    Great references• [1]RNN: http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/?subscribe=success#blog_subscription-2• [2] LSTM: http://colah.github.io/posts/2015-08-Understanding-LSTMs/• [3] RNN Effectiveness: http://karpathy.github.io/2015/05/21/rnn-effectiveness/• [4] Backpropagation: http://cs231n.github.io/optimization-2/#backprop• [5] ML categories: http://enhancedatascience.com/2017/07/19/machine-learning-explained-supervised-learning-unsupervised-learning-and-reinforcement-learning/
[8]ページ先頭
©2009-2025 Movatter.jp