@Nigel:
Enjoyed your post and you have put your observations and questions on RE very well. I will try to give another perspective as to why RE is the way it is. To do that, we will go back in time to 50's and 60's for some time.
IT IS THE ECONOMYRE (India) was established within a few years after the British left India. At that time, the Indian industry was minimal, the country was economically depleted, technology and infrastructure hardly existed, and trade with other countries was negligible. Under those circumstances, RE production was started. Their first and foremost objective was to produce enough motorcycles to supply Indian army to protect long borders of India with China and Pakistan.
Economics played a big role both in the survival of RE and for what it is today.
It was cheaper than other bikes such as BSA, Triumph, and Norton. Had it been an expensive bike, it may have gone the same path as other British bikes did - it would have died too with Japanese invasion. Even though the British factory could make more powerful REs but it would require better engineering, metallurgy - basically more money to build it, which would not have been conducive for Indian market at that time. As RE (UK) closed its shop, RE (India) kept producing the least expensive, simplest bike for years to come.
JAPANESE INVASION OF INDIAN MARKET - RE IN DANGERThen Yamaha of Japan started to get a few bikes to India. It did not fare well because of tight import regulations. I think they eventually closed that venture. It took many years after that that Indian markets started to open to foreign joint ventures. With that the production of Japanese bikes started to climb very rapidly to meet huge demand for bikes.
Within a decade of Japanese invasion of Indian market, RE (India) was facing same fate what had happened to RE (UK). But it survived due to take over by Eicher. Also, there are die-hard fans of RE in India, it has a national status symbol for its association with Indian army that would have kept it alive.
So throughout those decades, RE has come close to being gone a few times. With that history, and environment and little purchasing power of your customers, there is no way to add to the price of your product. The 350 cc 20 HP RE was the king of road and certainly cheaper to make than any other bike with 40 HP and 100 mph speed. It did its job pretty well for social and econimic situation of Indian market.
To answer your question - why didn't RE (UK) produce 40 hp bikes ? My guess is that they must have sensed that they are not going to survive and just put their eggs in the RE (India) basket.
NEW ERA OF RE - EFIIn the last several years, RE (India) did make small changes. More recenlty, it had no choice but to make the electronic fuel injection,UCE 2009 bikes to meet European environment regulations. They spent a lot of money (from their perspective and capacity) to bring these new EFI models.
SURVIVAL OF FITTEST - CAN'T RUSH A TURTLERE reminds me of an animal, the turtle, which is considered a play mate of powerful (40HP
) and huge but now extinct dinosaurs. It has survived millions of years UNCHANGED. The turtle is not fast (nor is our beloved RE that tops at 80 mph), nor it is very strong (my G5 may be maxing at 27 HP). One feature of turtle is its hard shell - a protection from predators. RE had a protective 'shell' in India - it was the closed Indian market that protected it from outside attack for many years. May be turtle is not very appetizing - nor is an RE appealing to for a speedster, so they left it alone.
Survival is not of the fastest, nor of the biggest, nor of the strongest - it is of the Fittest, the one who fits within the immediate environment around it.
FUTURE - FURRY or McDEEBS or FIREBALLIt is very tempting to wish for a RE - Furry, Mcdeeb or even Fireball in future. Most likely they will be small scale individuals or small party modifications. RE (India) is unlikely to be persuaded to make them - simply because they can hardly meet the demand for current models of RE in India. Their exports make up only a small fraction of total output.
It will certainly be nice to have an RE model that can compete with a Bonneville or Norton commando - even if it is a limited edition. It will certainly cost more, but there will be a few buyers for that too.