
See David Hounshell's Science and Corporate Strategy https://www.worldcat.org/title/science-and-corporate-strateg...
Richard Feynman's commentary on being burned out and playing with ideas of spinning plates, which culminated in his Nobel Prize award, comes to mind. Recently featured on HN (and something of a perennial favourite): https://news.ycombinator.com/item?id=26931359
https://www.asc.ohio-state.edu/kilcup.1/262/feynman.html?rep...
You'll find similar observations regarding corporate research and development. See David Hounshell's work on R&D at DuPont: Science and Corporate Strategy: Du Pont R&D 1902-1980, originally published in the 1980s.
https://www.worldcat.org/title/science-and-corporate-strateg...
Similar stories exist for AT&T's Bell Labs, Xerox PARC, IBM Research, and even Ford Motor Company's research division. In the case of the latter, Henry Ford (in)famously gave his engineers significant discretion, but somewhat-less-than-optimal equipment, the latter apparently a spur to creativity in order to overcome limitations.
Geoffrey West has discussed this regarding the Santa Fe Institute, I believe in the Q&A of this presentation: https://youtube.com/watch?v=w-8sbSPf4ko (At 1:09:00)
West quotes the late Max Perutz's guidelines for organizing research, in full:
Impishly, whenever he was asked whether there are simple guidelines along which to organise research so that it will be highly creative, he would say: no politics, no committees, no reports, no referees, no interviews; just gifted, highly motivated people picked by a few men of good judgment. Certainly not the way research is usually run in our fuzzy democracy but, from a man of great gifts and of extremely good judgment, such a reply is not elitist. It is simply to be expected, for Max had practised it and shown that this recipe is right for those who, in science, want to beat the world by getting the best in the world to beat a path to their door.
http://www.theguardian.com/news/2002/feb/07/guardianobituari...
As an anti-creative environment, the most effective obstruction is to ensure that everything is not only consequential, but a path to failure. This tends to emerge in oppressive and excessively bureaucratic environments. John Cleese outlines the general parameters toward the end of this presentation: https://youtube.com/watch?v=Pb5oIIPO62g
The start-up world seems to me to increasingly exemplify the high-consequence, no-win environment.
The Unbound Prometheus --- a history of eastern europe technology and its social impacts.
From The American System to Mass Production --- Hounshell, a history of the development of the assembly line, and in particular the forerunners to Ford's mythologized mass production line (which, in many elements, surpassed it, and demonstrate that it was not particularly unique except in marketing)
America By Design --- Noble is a very circumspect, controversial historian of technology.
The Protestant Ethic and the Spirit of Capitalism --- classic on the controversial idea that protestantism was responsible for capitalism and the resulting technology
Does Technology Drive History -- collection of essays by the founders of the field of science, technology, and society.
Civilizing the Machine --- technology's interaction with American values and how those developed concurrently.
Major Problems in the History of American Technology --- a collection of original documents and essays interpreting them in a historical basis, ed. by a founder of the field.
The Tentacles of Progress --- how technology lends itself to imperialism, and furthers exploitation, even when other nations fund the development of infrastructure in developing nations.
The Machine in the Garden --- possibly the single most important book on this list, the one that turned my life upside down and which I think about most regularly. It posits that America's idyll of a tech-free natural scene is actually a balance of technology and nature, and is a artificial nature propped up by machine, and demonstrates this history of this tension in American life from Shakespeare to Jefferson to Thoreau to the modern day. It completely turned my conception of the perfect life upsidedown.
Digital Apollo --- a history of the computing history of the Apollo program, especially the tensions in the software development sides between good management and perfect programs.
Science and Corporate Strategy, DuPont R&D 1902-1982 --- if you're interested in the development of big chem, or in the history of companies having R&D departments, this is the book for you. The DuPonts were very well-educated, many being MIT alums, and the chemical company they built was innovative in many different ways --- this book is a interesting discussion of the society and the technological pressures that drove and still drive innovation.
"The Memorandum as a Management Genre" by JoAnne Yates <http://www.ismlab.usf.edu/dcom/Ch6_YatesMemoMgtCommQtly1989....> (PDF)
Organizing America by Charles Perrow is Yet Another Book Exploring the Origins of The Corporation: <https://press.princeton.edu/books/ebook/9781400825080/organi...> <https://archive.org/details/organizingameric0000perr>
The New Books Network is a great resource generally for academic books and author interviews. Search on "history of corporation" yields: <https://newbooksnetwork.com/search?+q=history+of+corporation>. And I see a number of future timesinks, starting with Counter Cola: <https://newbooksnetwork.com/counter-cola>
David Hounshell's Science and Corporate Strategy is one of several books about major industrial R&D labs. His covers Du Pont. <https://archive.org/details/sciencecorporate0008houn>
AT&T's Bell Labs is another notable topic (Jon Gertner, The Idea Factory, though I've not read it). I believe there are likely books on Xerox, Kodak, and IBM. Rand Corporation hosts its own very extensive publications library: <https://www.rand.org/pubs.html>
I'll add Mickelthwaite's book to my own rather imposing pile.
What if any is your materials background?
Are you familiar with David Hounshell's Science and Corporate Strategy <https://www.worldcat.org/title/science-and-corporate-strateg...>?
The nature, questions, and value of research are one element of a larger set of questions I've been exploring.
I've recently run across David Hounshell's work on R&D at DuPont: Science and Corporate Strategy: Du Pont R&D 1902-1980, originally published in the 1980s.
https://www.worldcat.org/title/science-and-corporate-strateg...
For a ... short-ish ... synopsis, covering main themes, see Hounshell's article: "Measuring the Return on Investment in R&D: Voices from the Past, Visions of the Future"
https://www.ncbi.nlm.nih.gov/books/NBK45334/?report=classic
Upshot: R&D is a risk-based activity, and it can have high payoffs, but also high costs. Much of Hounshell's book is about management's attempts to both organise its research and research teams for optimal results, and to manage the costs and risks associated with it. There was a period in which the labs were phenomenally productive, including a roughly decade-long span from the 1920s through 1930s where virtually the entire modern plastics industry (Graduate fans take note) was invented. But also flops (a fake leather product produced in the 1960s notably).
This fits in with notions I've been noodling at of risks in economics -- an element of virtually all economic activities, but with distinctly different characteristics and scope. R&D tends more toward the dice-rolling variety (you may win), whereas some activities gamble with catastrophic and systemic loss. Some fields appear to be little but raw risk balancing (much of finance, real estate, and of course, insurance) whilst others provide a greater opportunity to directly influence, reduce, adopt, or mitigate risks (engineering, generally).
There's also the risk, or uncertainty, involved in attempting new activities. The recent HN submission "The Drugs Won't Work" https://medium.com/@belledejour_uk/the-drugs-won-t-work-659c... (https://news.ycombinator.com/item?id=20789000) talks of the problems of new drugs discovery, effectively a search through a 10^200 node space for potential beneficial compounds, with very few heuristics in reducing or guiding the search (Lipinski's Rule of Five being one: https://en.wikipedia.org/wiki/Lipinski%27s_rule_of_five).
All research (and much coding) is effectively a search though a large possible solution space, subject to constraints and domain characteristics, for possible useful elements or combinations. The problem of huge search spaces means that alternatives to brute-force exploration are quickly necessary.
Back to Du Pont: the chemical search space is also large (pharmacology is effectively a branch of that), and low-hanging fruit was found early with small, simple, and readily-attained compounds: the law of diminishing returns. Worse was the discovery, often much later, that along with useful features came harmful ones -- the law of unintended consequences being another suprisingly general principles that arose from a specific discipline (sociology) but is applicable in nearly all others.
As for management: by Hounshell's account, management largely turned its researchers loose and told them to have fun.
There's the further challenge that the discovery of keys rarely coincides with the knowledge of the lock in which it fits. The phrase "a solution in search of a problem" was first applied (by Theodore Maiman: https://en.wikipedia.org/wiki/Theodore_Maiman) to a technology of which there are almost certainly numerous instances of in your immediate vicinity: lasers.
Early applications were thought to be in the application and delivery of power (Arthur C. Clarke makes a passing reference in 2001 A Space Odyssey to blasting the Lunar monolith with one), but it was the characteristic of lasers as a tightly coherent transmissive blank slate onto which information could be encoded which proved its greatest use.
(This makes me suspect that materials such as graphene -- a uniform, monoatomic plane, might have its greatest application as information storage medium rather than physical materials, much as doped silicon has proved useful in semiconductors.)
Another part of my general study has been in thinking how technology works by considering what technological mechanisms fundamentally exist, and how those interact. My list presently numbers nine (materials, fuels, power transmission & transformation, process knowledge, structural knowledge, networks, systems, information, hygiene -- fuller descriptions elsewhere, still in process), which may or may not prove ultimately accurate. But has been somewhat useful in thinking through numerous problems and applications. Each mechanism has specific characteristics and limitations.
Upshot: I don't think "technology" is a bottomless bag of tricks. It's definitely useful, and well get more from it, but it also has costs, including complexity (itself a form of risk, which is to say, of debt), but there are bounds. R&D is exploring a space with constraints, in which both benefits and risks may be found. Treating the matter probabalistically and as one to be approached in on a risk-based approach may prove generally useful.