Search results
Results from the WOW.Com Content Network
In machine learning, backpropagation [1] is a gradient estimation method commonly used for training a neural network to compute its parameter updates. It is an efficient application of the chain rule to neural networks.
Main page; Contents; Current events; Random article; About Wikipedia; Contact us; Help; Learn to edit; Community portal; Recent changes; Upload file
The terminology "back-propagating errors" was actually introduced in 1962 by Rosenblatt, [24] but he did not know how to implement this, although Henry J. Kelley had a continuous precursor of backpropagation in 1960 in the context of control theory. [40] In 1970, Seppo Linnainmaa published the modern form of backpropagation in his Master's ...
Backpropagation was first described in 1986, with stochastic gradient descent being used to efficiently optimize parameters across neural networks with multiple hidden layers. Soon after, another improvement was developed: mini-batch gradient descent, where small batches of data are substituted for single samples.
Backpropagation through time (BPTT) is a gradient-based technique for training certain types of recurrent neural networks, such as Elman networks. The algorithm was independently derived by numerous researchers.
In 1970, Seppo Linnainmaa published the modern form of backpropagation in his master thesis (1970). [23] [24] [13] G.M. Ostrovski et al. republished it in 1971. [25] [26] Paul Werbos applied backpropagation to neural networks in 1982 [7] [27] (his 1974 PhD thesis, reprinted in a 1994 book, [28] did not yet describe the algorithm [26]).
However, it was not the backpropagation algorithm, and he did not have a general method for training multiple layers. In 1965, Alexey Grigorevich Ivakhnenko and Valentin Lapa published Group Method of Data Handling. It was one of the first deep learning methods, used to train an eight-layer neural net in 1971. [14] [15] [16]
Rprop, short for resilient backpropagation, is a learning heuristic for supervised learning in feedforward artificial neural networks. This is a first-order optimization algorithm. This algorithm was created by Martin Riedmiller and Heinrich Braun in 1992. [1]