You're forgetting everything about acceleration and gearing.
A smaller sprocket will allow you to go at a higher speed at lower revs, but the engine will have to work just as hard or harder. Acceleration will decrease based on engine power.
Try accelerating a bicycle to 25mph using your leg power (a fixed amount of HP and Torque) going through the gears sequentially. Now try to accelerate to 25mph using only gears 2, 4, and 6. Your acceleration (rate of increasing speed) will be slower due to the limitation of your biological hp and torque (power) factor.
In an engine with infinite power, yes a smaller sprocket will increase acceleration. When you hit a fixed wall in the total equation, something is compromised.
If what you said was absolutely true, we all would be running 6 tooth front sprockets and 6 tooth rear sprockets, because it would be faster, instead of 18t front and 45t rear. We need the gearing to compensate for power losses, both at certain RPMs and totally as a whole on the peak ranges.
As you said, you're not a gear-head.