> The final approach I've encountered is where research problems come from an external body.

For a good helping of the government research organizations, they hire academics and practitioners as Program Managers (PMs) who gather the experts in the field to come up with research programs, agendas, and seedling programs. Of course, the PMs have to sell areas to their higher ups too, but it does have some degree of dynamism and alignment with academia. The amount of resources that could be brought to bear on a problem is impressive.

I do wonder about this whole notion of “success” of research. Hundreds of years ago, wealth patrons sponsor the “researchers” of their times. Creative people found problems that interested them which sometimes led to unexpected results and new areas. I don’t think serendipity should be discounted.

> You can probably imagine my astonishment when I realised that in some communities there is a widely accepted list of "the next problems to work upon".

It would be nice if there was some curated list of problems to work on, where the items on the list are obtained and sorted through some consensus mechanism. I know HN is secretly obsessed with To-do lists, so perhaps this "Super To-do list" is something someone finds interesting to build.

Related question: where do startup ideas come from?

How is that even a problem? Ideas bubble into my head much faster than I have the time or ability to work on them. Granted, most of them are crap, but that's because most of them are outside of my field of expertise.

As for usefulness, I seem to remember that it was the father of category theory who on his deathbed said, "I can die happy, knowing that my research will never have any practical applications". Now it's the basis for most type systems.

Regarding "scouting mode", one time proven trick is assembling a large stable of grad students to do the heavy lifting and tackle the boring parts of research, while one spends most time scouting. The downside is senior researchers tend to devolve into managers. One inspiring thing about the likes of Feynman is it seems they were able to keep tackling problems and doing a lot of the heavy lifting themselves, even late in their careers.

I’ve worked in industry supporting UK govt research by helping to define research programmes. This might involve doing horizon scanning to identify tech that could address a customer requirement and then defining risk reduction tasks that would raise the technology readiness level (TRL). Industry could then bid to execute these risk reduction tasks, which would then involve a mix of engineers and academics as required.

My rule: is your research idea likely to bring money to the table in 10 years or less? Don't research it! There is enough of this kind of research going on already.

When a mommy DARPA Grant and daddy Defense Contractor love each other very much...