
The FBI Venona project (https://en.wikipedia.org/wiki/Venona_project) detected a significant number of Soviet spies operating in the US during the "McCarthy period." After the fall of the Berlin Wall a number of documents were recovered from the East Germany that substantiated the FBI work. Senator (D) Daniel Patrick Moynihan wrote about this in detail in "Secrecy" https://www.amazon.com/Secrecy-Experience-Daniel-Patrick-Moy... (See also https://en.wikipedia.org/wiki/Moynihan_Commission_on_Governm... for context)
* A highly valuable asset within government is information and thus workers are incentivized to hoard it and protect its value, including by classifying it and not sharing it (i.e., keeping secrets).
* Secrecy can lead to extraordinarily bad results. Instead of being exposed to open debate, the secret information or idea is restricted to a few insiders who lack diverse perspectives, can easily form an echo chamber, or who may be politically pressured or desirous to support the conclusion.
[0] Unusual career, if you don't know of him: Senator, PhD. and published sociologist, Ambassador to India and the UN, White House advisor on urban affairs. With his deep skills and experience in both research and government, few are more qualified to talk about these things.
