The front wheel bearings of my ES350 were the same way, with the same effect on the speedo drive. 5 minutes, a small hammer and a large flat blade screwdriver cured all. Work your way around it carefully, been done for years that way. The hub rings unmistakably when the bearing is "home". Everything stays put once the "cone nuts" are screwed in tight. NOT as nice or as smart as 2CVs socket method, and the consequences of a mis-stroke are likely a new bearing. But new bearings are a good thing, eh?
"Use it up, wear it out, make it do or do without." anon