Comparison of Some Optimizers in Long Short-Term Memory Networks with an Application to Tigris River Water Imports

The Long Short-Term Memory (LSTM) RNN architecture was developed to overcome the limitations of traditional RNNs in recognizing long-term dependencies in sequential input. LSTM networks and other neural network topologies rely heavily on optimizers for training. Every iteration of a gradient-based algorithm attempts to approach the minimizer/maximizer cost function by using the gradient's objective function information. Moreover, a comparative analysis of a number of optimizers is presented in this work, including FTRL, Nadam, Adagrad, Adadelta, SGD, RMSprop, and Adagrad. By directly affecting the objective function's convergence, the study emphasizes the crucial role optimizers play in improving gradient descent efficiency. This work fills a gap in the literature on choosing the best optimizers for time-series data by concentrating on water import statistics for the Tigris River. The results offer insightful information about how to choose optimizers wisely to reduce time complexity and increase accuracy. The results of the study show that these optimizers' performance is generally comparable when compared using the root mean square error (RMSE) criterion based on water data. However, Nadam produces RMSE values at 500 epochs that are lower than those of the other methods.