Search results
Results from the WOW.Com Content Network
Long short-term memory (LSTM) [1] is a type of recurrent neural network (RNN) aimed at mitigating the vanishing gradient problem [2] commonly encountered by traditional RNNs. Its relative insensitivity to gap length is its advantage over other RNNs, hidden Markov models , and other sequence learning methods.
That is, LSTM can learn tasks that require memories of events that happened thousands or even millions of discrete time steps earlier. Problem-specific LSTM-like topologies can be evolved. [56] LSTM works even given long delays between significant events and can handle signals that mix low and high-frequency components.
Hochreiter developed the long short-term memory (LSTM) neural network architecture in his diploma thesis in 1991 leading to the main publication in 1997. [3] [4] LSTM overcomes the problem of numerical instability in training recurrent neural networks (RNNs) that prevents them from learning from long sequences (vanishing or exploding gradient).
For example, in multiplicative units, such as the forget gate of LSTM, the bias can be initialized to 1 to allow good gradient signal through the gate. [2] For neurons with ReLU activation, one can initialize the bias to a small positive value like 0.1, so that the gradient is likely nonzero at initialization, avoiding the dying ReLU problem.
The problem was initially investigated by Sharkey and Sharkey (1995), [33] Robins (1993) [35] and Ratcliff (1990), [2] and French (1999). [10] Kaushik et al. (2021) [ 34 ] reintroduced the problem in the context of modern neural networks and proposed a solution.
The standard LSTM architecture was introduced in 2000 by Felix Gers, Schmidhuber, and Fred Cummins. [20] Today's "vanilla LSTM" using backpropagation through time was published with his student Alex Graves in 2005, [21] [22] and its connectionist temporal classification (CTC) training algorithm [23] in 2006. CTC was applied to end-to-end speech ...
Connectionist temporal classification (CTC) is a type of neural network output and associated scoring function, for training recurrent neural networks (RNNs) such as LSTM networks to tackle sequence problems where the timing is variable.
Other approaches include solving it as a constrained linear programming problem, [27] making each expert choose the top-k queries it wants (instead of each query choosing the top-k experts for it), [28] using reinforcement learning to train the routing algorithm (since picking an expert is a discrete action, like in RL), [29] etc.