Stochastic Dynamic Programming (SDP) is a powerful approach applicable to nonconvex and stochastic stagewise problems. We investigate the impact of the formulation of the subproblems and of the choice of optimization method used to solve them on the overall performance of SDP in the context of hydropower reservoir management. We report numerical results on real systems and compare a number of state-of-the-art solvers. In one set of tests, subproblems feature nonlinear hydropower production constraints, while in a second set, the latter are approximated with linear constraints. Our results show that, when linear hydropower constraints are acceptable during policy computation, a Sequential Linear Programming (SLP) approach requires the lowest number of function evaluations while being the most robust in terms of number of subproblems solved successfully. On the other hand, IPOPT exhibits the best performance during the simulation phase in the presence of nonlinear hydropower constraints, where SLP is less reliable. A combination of SLP for policy computation using linear hydropower constraints with IPOPT in the simulation phase using nonlinear hydropower constraints results in an improvement of the average annual cumulative cost reduces the total run time by a factor of about 3.3 compared to IPOPT alone using nonlinear hydropower constraints during both phases.