Pressure on ancillary reserves in power systems has significantly mounted due to the recent generalized increase of the fraction of (highly fluctuating) wind and solar energy sources in grid generation mixes. Dedicated energy storage devices have seen their role reaffirmed as potentially low carbon print, if expensive tools, for smoothing the resulting generation/demand imbalances. However, a hitherto under utilized, relatively inexpensive energy storage alternative is that formed by the tiny energy wells of electric origin attached to millions of individual customer electric thermal loads. A hierarchical mean field games approach is proposed for shaping their collective load, whereby the top level sets system optimal mean aggregate temperature target trajectories. In turn based on a local state and a mean field dependent cost function, each individual load develops a decentralized local control law such that the aggregate load can meet the set targets. This control law is to be followed only as long as local comfort and safety constraints are secured, thus guaranteeing acceptability by customers. The corresponding mathematical theory is developed and numerical results are reported.