What do the Apache Log4j Vulnerability, Security Community Outreach Efforts, Cognitive Infrastructure, Resilience, Anti-Fragility, John Boyd and Dune have in Common? The December 2021 OODA Network Member Meeting
To help members optimize opportunities and reduce risk, OODA hosts a monthly video call to discuss items of common interest to our membership. These highly collaborative sessions are always a great way for our members to meet and interact with each other while talking about topics like global risks, emerging technologies, cybersecurity, and current or future events impacting their organizations. We also use these sessions to help better focus our research and better understand member needs.
To encourage openness of discussion, these sessions take place with Chatham House rules, where participants are free to use the information in the meeting but are asked not to directly quote or identify other participants (we also keep privacy in mind when preparing summaries of these sessions, like the one that follows).
The December call was held on Friday, December 17th, and began with a discussion of the upcoming OODA Salon scheduled for December 20th, which is now available online: Digital Innovation and Threats, A discussion with Deputy Director of CIA for Innovation Jennifer Ewban.
Topics for discussion on the December Monthly call were:
- Apache Log4j Vulnerability
- In the Trenches
- IT Asset Inventory
- Dusting off the Vulnerability Playbook
- Timeline of impact and mitigation
- F5/Web Application Firewall (WAF) Investments
- Rules Matter/Firewall Review
- Secondary/Tertiary Vendors
- Lof4j declaration to nation-states
- Free Software Dependency
- Deep Logs Still a Concern
- Origin Story
- China’s Ministry of State Security (MSS)?
- What is the Next Log4j?
- iPhone/Telsa Rename Vulnerability
- Cybersecurity: Communication Style, Public Relations, Outreach Efforts, and Workforce Skills
- Bearers of Bad News
- Communication/Workforce issues in Cybersecurity
- Distribution of Security Knowledge and Support Materials
- Pro bono CISO activities
- Information Sharing – how is experimentation operationalized?
- Cyber XPrize Model
- The OODA Almanac 2022
- Cybersecurity reckoning for Web3 and cryptocurrency projects
- Contested metaverse
- Cyber as a great power war
- Cognitive Infrastructure
- Further tracking of “autonomy”
- Opportunities for advantages that are diminishing over time
- Unknown unknowns/”Surprise”
- Simultaneous Crisis Moments/Worst Case Scenario
- How to manage crisis/integrate intelligence/identify opportunities
- Resilience and Anti-fragility across domains
- John Boyd and Dune
December Monthly Call – Discussion
The Log4j vulnerability was the lead topic for discussion on the call. OODA CEO Matt Devost led with some general impressions to guide the conversation:
“Log4j is pretty systemic as vulnerabilities go, given how deeply embedded it is on so many different types of devices. I think it’s kind of untreated territory, at least in the history of my career, and it’s something that’s going to have an incredibly long dwell time inside folks’ networks as vendors are slow to upgrade, et cetera. I am also very interested in the geopolitical angle here. And I posted a note on Twitter thinking: are there some vulnerabilities like this that are bad enough, systemic enough, that the U.S. should make some sort of pre-declaratory statement that nation-state using the vulnerabilities would be considered some sort of active war or active conflict – almost like declaring time out during your street hockey game so nobody gets hit by a car?”
Is there ever a class of vulnerability that is so widespread and impactful that the US would declare any nation-state use of it as an act of war?
Question I'm going to raise on today's OODA Network meeting. #OODA
— Matt Devost (@MattDevost) December 17, 2021
“So I’m very interested to see what develops with regards to active exploitation. And we’re already seeing threat actors in things happening in the wild, but by way of a kind of targeted nation-state activity that has pretty drastic or has a potential for drastic consequences or even the development of exploit tools that we’ve seen based on other state campaigns, eventually migrating to cybercriminals and the like, and have an even more substantial impact in the form of ransomware, et cetera. Is there a vulnerability? – is this “it”? – that qualifies as needing that statement regarding the “don’t use this: let’s all take a time out. Let’s get the core elements of our infrastructure prepared because we are all sharing.”
“With regards to mitigation: I think the single best source that I’m seeing right now is the CISA page. We did an article about that on the site. They’ve set up a GitHub repository around vendor responses and vendor patching. So I think that’s probably the greatest single source of information. There are a lot of deep links there to other sites and vendors. We would like to hear from some of the folks that are in the trenches if you have any input on how this is impacting your organization?”
Apache Log4j Vulnerability
A general membership discussion of the impact of and member’s direct experience with the Log4j vulnerability ensued. Following is a high-level summary of the discussion.
In the Trenches: Members spoke to their experiences assessing and mitigating the impact of the vulnerability. Questions clients have been asking and members have been attending to include:
- What about the inside of the data center? We are still attending to that issue.
- The administrative side has been more straightforward: What about vendors? What are our vendor colleagues doing?
- On vendor responsiveness: “We are getting responses from them about their approaches and, just gotta hope that they are doing the right thing. And I’m confident that they are from a product perspective.”
IT Asset Inventory: A member spoke to his organization’s recent efforts: “What we’re looking to do is to make sure that our inventories, the work that we’ve been doing over the last five to six years about improving inventory across the board – open source, hardware other kinds of components – we’ve invested a lot in the last five to six years is to improve that. And that’s paying dividends on our ability to understand the environment and understand it relatively quickly”
Dusting off the Vulnerability Playbook: “We have certainly seen similar things like this occur in the past or some of the chip vulnerabilities that have come out. There have been some framework vulnerabilities like STRUTS and others, that have come out in the past. Unfortunately, we’ve had to dust that playbook off a few times.”
Timeline of impact and mitigation: A few members spoke to previous professional experiences with vulnerabilities when impact and mitigation assessments “took months.” There was a broad consensus on the call that member organizations had a handle on the Log4j vulnerability through the work they put into it over the most recent weekend.
F5/Web Application Firewall (WAF) Investments: “The other bit is making sure that we’re seeing what kind of connectivity is coming up in out of the firm. So some investments that we’ve made in verification tech were interesting. Our AWS and F5/Web application firewall (WAF) investments that we’ve made – we make some use of it. We’re making a good bit of use of both of those technologies.
Rules Matter/Firewall Review: A member shared that “now and then the old firewalls we’re still showing some interesting things. In general, we don’t allow a whole lot of outbound traffic whether it’s from our cloud or data centers and that kind of rigor about rules has been helpful. And we literally just finished an almost year-long firewall review view and that’s actually been helpful because we’ve cleaned up some legacy rules that have been around for a while. So all of the investments that we’ve made in the blocking and tackling of inventory and dependency management and getting some of these capabilities in place has been helpful for us.”
Secondary/Tertiary Vendors: Concern was voiced that you may be able to work directly with and validate the response of primary vendors, but there are vendor relationships further down the line that are difficult to review.
Consensus of the OODA Network of Experts was that such a declaration would never be made, but we had a great discussion.
— Matt Devost (@MattDevost) December 17, 2021
Lof4j declaration to nation-states: There was a consensus on the call that a “Log4j declaration” (or a statement distributed to nation-states during an equivalent vulnerability incident) was effectively impossible: too many players; Nation-states aren’t the only actors out there; There are plenty of non-nation state actors that nation-states could use as agents; and one member noted: “It’s too ubiquitous and there are too many possible bad guys and who knows: we may be using this exploit, or we may have our already. I think it’s a nice concept, but I’m very dubious…color me very skeptical.”
Free Software Dependency: Log4j was free software maintained by three people which goes to the fact that the most widely deployed software is also is usually the least supported. The surprising thing has been the scale and scope of the vulnerability and the fact that there was no commercial support for it. But, in the end, It is exploitability that is the problem. Vendors should stop using unvetted freely available libraries.
Deep Logs Still a Concern: A member offered this word of caution: “With the volume passing of logs between different environments you may have everything you think you are forwarding to outward-facing infrastructure remediated – and the logs have then passed to a storage environment which may be vulnerable and execute. And then if it has the external ability to reach up back to the net, which may beacon up to the attacker from deep within your network. And you may not even realize that that has occurred if you don’t have a proper inspection and logging on those systems as well. So we’ve had some really interesting challenges as this has played out. A big part of the problem is there is no way for most vulnerability management processes to even keep track of the intel. We have this incredible rotten wood underlying a lot of these key areas where we have one bug and there is blood in the water and everyone knows you can keep digging into that rock to get more exploitable usable vulnerabilities. This is the future of every one of the exploits we care about. It doesn’t matter what we introduced, whether the Java was intended to be scriptable or not. The adversary is going to bootstrap a scriptable language off of any type of logic gate they can build into the exploitation environment. And that is insane. We are not ready to deal with that.”
Origin Story: The origin of the vulnerability discovery and the exploit development broke last Thursday night as a Minecraft exploit announced by the Alibaba Cloud Security Team.
China’s Ministry of State Security (MSS)? The group discussed this scenario proffered by a member: “I am still baffled that the Ministry of State Security let this one out the door and I’m still not quite sure if they understood the full scope of it or that the Alibaba team even understood the implications. In theory, all vulnerabilities are supposed to be passed through review before they are disclosed [by the MSS]. Now, whether that happened here or not, we don’t know. Is this a nascent development of their own vet because of the extent of their own vulnerability of their infrastructure? I can’t believe that that is true given how useful this capability would’ve been.”
What is the Next Log4j? Smart infrastructure plus increasing complexity equals a massive amount of new code in the U.S. – which is not being tested went it is deployed. As a result, we will face this type of threat on a continuous basis for the next decade.
iPhone/Telsa Rename Vulnerability: From the monthly call chat string: “Per screenshots shared online, changing the device name of an iPhone or Tesla to a special exploit string was enough to trigger a ping from Apple or Tesla servers, indicating that the server at the other end was vulnerable to Log4Shell.”
The Security Community: Communication Style, Public Relations, Outreach Efforts, and Workforce Skills
The conversation transitioned to cultural issues within the security community. Issues that were considered by the group include:
Bearers of Bad News: How much is it a learned behavior not to sow fear, uncertainty, and doubt because we have been taught not to make extreme predictions or communicate the worst-case scenarios?
Communication/Workforce issues in Cybersecurity: As a discipline, the public health medical community has to consider how they communicate things for clarity and accessibility. We as a discipline are still not good at that. We also have a workforce development challenge. Do we need the cyber security public relations role formalized? It is currently not listed in any of the taxonomies of the workforce of the future. Universities and the government are trying to not only figure out what positions need to exist but how to teach them. This skillset is not represented.
Distribution of Security Knowledge and Support Materials: From a healthcare analogy perspective, we have programs that are designed to represent the underrepresented, make sure they get access to healthcare, et cetera. We don’t have the equivalent with regard to security talent. Where is the list of the top 20 open source libraries and how do we dispatch some help to make sure they are secure? A member noted that “we even talked about providing labor to review the code samples that are used in computer science textbooks. Because if you review the code that is offered up as you are learning computer science, none of it is really done from a dev sec ops perspective. You are basically teaching insecure coding practices. So what if you spent the level of effort required in order to improve those resources, so the next generation of computer coders are a little bit more attuned to some of the security requirements or the way that code is exploited? So there are lots of different models, but they all have resources associated with them. Does this elevate the concern enough that we have some of these major corporations, that will need to be the front runners, willing to develop or willing to spend those resources?”
Pro bono CISO activities: The group is not sure how the medical and legal ‘pro bono’ models fit for the security discipline but agreed but that it is a dialogue that should be pursued by the membership on future calls.
Information Sharing – how is experimentation operationalized? The Defense Science Board and the FSI SAC – Financial Services Information Sharing and Analysis Center – were mentioned by members. The concern is access for organizations below the cyber poverty line. We need to innovate in that space in particular RE: access and training.
Cyber XPrize Model: What are things that are fundamental that everybody uses? Let’s go build one from scratch.
Fire Department/Safety Inspector Model: A member contributed via e-mail that small to medium size companies may need a different model – Fire Departments and Safety Inspection Agencies came to mind as models.
OODA Alamanac 2022
The OODA team reached out to the Network Members for feedback on the 2022 version of the OODA Almanac (for the 2021 version, see OODA Loop – The OODA Almanac – 2021 Edition). Topics under consideration for the Alamanac with contributions from the membership on the call include:
Cybersecurity reckoning for Web3 and cryptocurrency projects: There is a lot of publicity around what’s happening in the cryptocurrency NFT web3 space. But also we are seeing huge instances of online fraud and vulnerability in that space. The blockchain code may be secure if the consumer-facing GUI and web applications are not. Digital sovereignty discussions within this community were mentioned.
Contested metaverse: We talked a bit about metaverse in the Almanac last year, obviously with Facebook changing their name to meta, and all of these investments rolling in – there are going to be significant developments in that space. We see the emergence of the multiverse or omniverse, or maybe parallel metaverses emerging, but then also starting to think about what are the security and privacy implications of those environments. Non-specific threats (i.e school shootings), Misinformation, influence operations, Cybercrime, and virtual tradecraft in virtualized environments were also discussed. A whole new governance model and sovereignty issues (or new non-sovereign environments) were areas of interest as well.
Cyber as a great power war: We’re seeing increasing pressure on groups like NSO group, that we’re providing almost nation state-level capabilities to what we would consider tier two, tier three nation-states. Also, entities like Apple getting proactively involved – warning individuals around targeting, engaging in lawsuits against entities like NSO. So it makes us wonder now if the really sophisticated cyber engagement starts to be something that is more focused on China or Russia with an even greater emphasis on China.
Cognitive Infrastructure: Bob’s done a lot of great work on the site around the importance of developing and securing our cognitive infrastructure. But we see the emergence of a lot of these platforms that again, are under the control of foreign-based entities that have a significant impact in the cognitive infrastructure (see Bob’s formative thinking on the topic, part of a two-part series – OODA Loop – America’s Most Critical Infrastructure is also Our Most Neglected Infrastructure).
Further tracking of “autonomy”: Autonomous machine learning, autonomy and algorithms and systems, holistic systems like Tesla cars – what are the opportunities for advantage?
Opportunities for advantages that are diminishing over time: Building on insights from the computer chip supply chain war game, there are diminishing opportunities that exist to change certain variables that would improve or better our outlook against some of the worst scenarios that we identified, namely the development of robust supply chain initiatives. But also the development of domestic or allied chip manufacturing capabilities to reduce foreign independence on certain players, namely China. So we wanna also try and document in the Almanac: what are some of those opportunities for advantage? especially the ones that are diminishing over time, i.e. we would need to do them soon or we don’t get the advantage at all.
Unknown unknowns/”Surprise”: What happens when the links between exponential technologies start to go horizontal i.e. Black swans, risk and opportunity, with intersections and connectivity between technology innovations as a source of these new unknowns? A member clarified the challenge: “So it’s like strategic foresight at the far end, early warning closer, and then actual good situational awareness. Now we’re still trying to deal with situational awareness, you gain understanding, but we have to somehow bring them all in.”
Simultaneous Crisis Moments/Worst Case Scenarios: A member applied the conversation on advantage and situational awareness to geopolitical challenges: “NATO, Ukraine, Iran: I honestly don’t think we have properly connected structures on all of these crisis moments simultaneously. We silo them. I mean, how many folks have paid attention to the hit on the field distribution system in Iran. It was a one-day news event – but it has been a major cycle for that adversary and a major retaliatory cycle for that adversaries infrastructure and ops teams.”
How to manage crisis/integrate intelligence/identify opportunities: “Permission structures, speaking truth to power, talking about last mile and really dystopian things: there are all these societal reinforcements for not speaking up. It is also human nature to interpret the worst-case scenario as also the least probable scenario. “Unknown unknowns” – there is also the “failure of imagination” trope. How can your organization or how can we all think about permission structures or having systems in place for this last mile dystopian thinking and really breaking all of the usual incentives not to think about worst-case scenarios or communicate them within an organization? A member responded: “The only way to do it is to one integrate a full appreciation of what intelligence is into all levels of command for all types of enterprises.”
Resilience and Anti-fragility across domains: “What can be done so that the outcome matters as little as possible? It seems like complexity is one of those problems. It leads to lost resilience and that is kind of the overarching theme: how can you build more resilience across these topics? It is time to reduce complexity in as many systems and ways as we can.”
John Boyd and Dune
Amazing find by @chrizbot that he mentioned when I recorded an OODAcast session with him this week.
— Matt Devost (@MattDevost) December 17, 2021
Black Swans and Gray Rhinos
Now more than ever, organizations need to apply rigorous thought to business risks and opportunities. In doing so it is useful to understand the concepts embodied in the terms Black Swan and Gray Rhino. See: Potential Future Opportunities, Risks and Mitigation Strategies in the Age of Continuous Crisis
Cybersecurity Sensemaking: Strategic intelligence to inform your decisionmaking
The OODA leadership and analysts have decades of experience in understanding and mitigating cybersecurity threats and apply this real world practitioner knowledge in our research and reporting. This page on the site is a repository of the best of our actionable research as well as a news stream of our daily reporting on cybersecurity threats and mitigation measures. See: Cybersecurity Sensemaking
Corporate Sensemaking: Establishing an Intelligent Enterprise
OODA’s leadership and analysts have decades of direct experience helping organizations improve their ability to make sense of their current environment and assess the best courses of action for success going forward. This includes helping establish competitive intelligence and corporate intelligence capabilities. Our special series on the Intelligent Enterprise highlights research and reports that can accelerate any organization along its journey to optimized intelligence. See: Corporate Sensemaking
Artificial Intelligence Sensemaking: Take advantage of this megatrend for competitive advantage
This page serves as a dynamic resource for OODA Network members looking for Artificial Intelligence information to drive their decision-making process. This includes a special guide for executives seeking to make the most of AI in their enterprise. See: Artificial Intelligence Sensemaking
COVID-19 Sensemaking: What is next for businesses and governments
From the very beginning of the pandemic, we have focused on research on what may come next and what to do about it today. This section of the site captures the best of our reporting plus daily intelligence as well as pointers to reputable information from other sites. See OODA COVID-19 Sensemaking Page.
Space Sensemaking: What does your business need to know now
A dynamic resource for OODA Network members looking for insights into the current and future developments in Space, including a special executive’s guide to space. See: Space Sensemaking
Quantum Computing Sensemaking
OODA is one of the few independent research sources with experience in due diligence on quantum computing and quantum security companies and capabilities. Our practitioner’s lens on insights ensures our research is grounded in reality. See Quantum Computing Sensemaking.
The OODAcast Video and Podcast Series
In 2020, we launched the OODAcast video and podcast series designed to provide you with insightful analysis and intelligence to inform your decision-making process. We do this through a series of expert interviews and topical videos highlighting global technologies such as cybersecurity, AI, quantum computing along with discussions on global risk and opportunity issues. See: The OODAcast