Introduction: Mitigating Risks From Artificial Intelligence

This special report is a guide into some of the darker sides of AI deployments.

When fielded incorrectly, AI can do more harm to your business than good. Rogue AI can damage brands, invoke bad decisions and cause your organization to unintentionally fall out of compliance with regulations or potentially even break the law. It is important to remember that whenever we enter into a period or advanced innovation, adversaries also seek to exploit those innovations. In the case of AI, adversaries are already using this technology for nefarious ends. We are of course aware of the many dystopian predictions by those that fear an AI uprising. We believe the many serious people who sign onto letters expressing fear of out of control AI do so to underscore that citizens of free nations need to pay attention to this, and in general we agree. But with this post and our other OODA research and reporting we take the approach of offering real world solutions to mitigate risk, and consider this a better approach than just screaming about the sky falling.

This guide is part of a series of OODAloop member reports. Also see the OODA Loop Guide to AI and our research report on AI for Business Advantage.

OODA Loop Sponsor

When Artificial Intelligence Goes Wrong

AI has already proven itself as a hugely valuable technology. That is why it is here to stay and will keep evolving in service to just about every sector of the economy. We remain absolute believers in the importance of AI to our future.

Unfortunately, this acceleration of AI solutions causes many to overlook critically important cybersecurity and business risk considerations. And the very nature of these solutions is bringing new risks.

Here are key concerns:

• Self-Corruption: Algorithms that can teach themselves can corrupt themselves. Code that learns and modifies itself has been shown to introduce the risk of brand damage. Code that modifies itself can also cause harm to employees and introduce other risks to the firm.
• Hallucination: This term is used to capture the issues primarily with large language models like GPT which can be so creative they flat out lie. Care must be used to ensure decisions are not being based on results that are hallucinated.
• Inscrutability: Many Machine Learning algorithms, especially Deep Learning solutions that leverage back propagation, can add in so many new variables to their own models that no human can understand what they are doing. You can’t, for example, look at the deep learning algorithm that Google uses to identify cat photos on the internet and find the variables that detected the cats. We simply don’t know how the model was able to perform this work. This is not a big deal when it comes to identifying cats in pictures. But what about models that will be shaping your business strategy or that will be helping diagnose healthcare patients? When a solution is inscrutable like this it is hard to trust the results. This causes a lack of trust in outputs that result in sub-optimized solutions.
• Deceivability: Since most AI solutions assume the presence of trusted training data, they can be easily deceived (no AI today can recognize mistruths in data, doing so would require a large quantity of example data that does not have mistruths and programs focused on ensuring the new data does not have mistruths, requiring its own recursive expert system).
• New Attack Vectors: Bad actors can learn to manipulate AI in surprising new ways, including attacking training data, modifying inputs (especially external data dependencies), and even change algorithms.
• Data Protection: The large collections of data required for advanced security solutions are frequently stored with little to no security controls, resulting in larger risks to firms
• Bias: There have been numerous examples of bias being coded into AI systems. Additionally, the previously mentioned inscrutability of AI and ability of some AI solutions to modify themselves, means this problem can creep into a solution over time and with no warning. The term “algorithmic bias” describes the state of outcomes of some machine learning algorithms putting some groups at a disadvantage. The creators of these algorithms may now have meant for this bias or discrimination to be there, but the companies need to seek that out in results and take action when this is uncovered. The challenge is not a new one, bias has been around for years, but with AI the problem is being accelerated. And with newer deep learning approaches sometimes the problem is not detectable till the model has run for a while.
• Adversary Use of AI: A huge concern. Adversaries at all levels, including competing businesses, criminals, countries, nations are now able to tap into AI including commercially available SaaS solutions. This threatens to increase the ability of criminals to conduct social engineering attacks and increase the ability of nation’s to conduct misinformation/disinformation attacks. Nations can also use AI on the battlefield in new threatening ways.

Most of these challenges can be mitigated, and those that cannot might be better understood and dealt with in other creative ways.  Mitigating the problems starts with a clear understanding of how the technology has been malfunctioning or underperforming.

Examples of Artificial Intelligence Gone Wrong

Here are more specific examples of AI going wrong:

• ChatGPT Hallucinations: Almost every user we run into has a story about hallucinations from ChatGPT. This example of AI gone wrong has been seen by millions. At this stage in its development, many are learning the hard way about this issue and we expect the issue of hallucination to be widely known now. This issue can be engineered out with smart enterprise solutions.
• Facebook: Facebook’s many data woes are not caused by AI. Neither were the actions by Russian and other actors who used Facebook to sow discord before the 2016 election. The example of AI going wrong relates to how Facebook has tried for years now to leverage AI to mitigate these threats. And so far there is no evidence their AI savvy has helped mitigate these problems at all. Catching bad actors and malicious posting or malicious ad buys might be aided by automated tools but the approach has been to hire large teams of smart humans to read, review, correlate, and take action against bad actors on the Facebook platform.
• Smart Speakers: Research shows that many popular smart speakers which are enabled by AI are 30 percent less likely to understand non-native U.S. accents.
• Facial Recognition: Several AI enabled facial recognition systems are shown to  perform much worse on African American faces.
• Sentencing Guidelines: One of the most frequently cited examples of algorithmic bias is the system known as COMPAS (Correctional Offender Management Profiling for Alternative Systems). This system, exampled in detail in a ProPublica report,  is used for generating sentencing guidelines. It uses a basic form of machine learning. It is trained on data to deliver a scoring system that is then applied to new data. The system attempts to predict the likelihood of recidivism. And is one factor that can be used in recommending sentences for convicted offenders. It was discovered to predict that black defendants will have a higher risk of recidivism than warranted, and ti also predicted lower rates for white defendants than warranted. The results of this study have been debated and argued, but one thing is clear, since there is a lack of transparency over the algorithms, everyone has reason to doubt what the firm that runs the code says.
• Amazon Resume System: Other examples of algorithmic bias involve the potential for unfair treatment in how companies assess and hire job candidates. Many firms now use algorithms to score job applicants in ways that may well be biased, even though this is not intentional, it is bad. One of the most famous cases of this involved a resume database screening tool put in place by Amazon in 2017. Over time the Machine Learning system became biased against women. The program had to be terminated because it was a misogynist.
• Financial Rating: Algorithmic bias has also been found in financial programs and has caused problems with credit scoring. Right now complex neural networks are being leveraged in credit scoring. If you get turned down for a loan, it might be that no one will be able to explain why.
• Generative AI Bias: The current state of generative AI, especially around images, has been shown to have bias, including more sexualized images when creating pictures of women and fewer racially diverse images when creating pictures of people. At this time the engineering solutions to mitigate these issues are not working out, but we believe progress will be made soon.
• Media Bias: Many major news organizations, search providers and major social media sites have been accused of “burying” news that does not fit their worldview. This is always denied by the organizations accused of this, but the fact that inscrutable algorithms are at play are cause for concern.
• Google, Bing and Every Other Search System: It is a sad fact that AI has not kept up with human needs for search and discovery, forcing people to focus hard on how to be creative and work very hard crafting complex search strategies to find the right information. We consider this a failure of current AI, especially for missions and business functions that matter.

A key lesson from these and many other major issues is that businesses should understand the new risks that come with AI solutions. Planning in advance can mitigate the occurrence of problems, and, when they occur, can speed the response to the incident.

Artificial Intelligence Has Compliance Issues

Real world experience with AI solutions has brought to light the many risks articulated above. And business leaders have seen enough that many companies are slowing down their AI initiatives till they can come to grips with the risks. A chief concern is the inability to ensure security and compliance, which is complicated by the explainability of complex AI. The issues of fairness and ethics also inject cause for management scrutiny. Mistakes here can do more than damage brands, they can hurt society. They can also cause firms to run afoul of the regulatory environment.

Regulations impacting AI deployments include:

• The Civil Rights Acts of 1964 and 1991
• The Americans with Disabilities Act
• The Genetic Information Nondiscrimination Act
• The Health Insurance Portability and Accountability Act
• Computer Fraud and Abuse Act (CFAA)
• Electronic Communications Privacy Act
• The Copyright Act
• Child Pornography and Prevention Act of 1996
• The Family Educational Rights and Privacy Act (FERPA)
• The Fair Housing Act
• Federal Reserve SR 11-7
• The EU Greater Data Privacy Regulation (GDPR)
• New York Cybersecurity Regulations
• Workplace AI recruitment selection regulations in New York, Illinois and Maryland
• Individual US State consumer privacy and breach notification laws
• Interpretive Guidance and Regulations of the Security and Exchange Commission (SEC) and National Credit Union Administration (NCUA) and Regulations of The Federal Financial Institution Examination Council (FFIEC)

The US and EU have issued AI ethics guidance and so has the Department of Defense.  While these do not have the force of law, they should be considered by any serving or supporting militaries.

Even if you could get your AI into compliance with these many regulations, it still does not mean you are secure from the AI making poor decisions. And these regulations do little to nothing to prevent a hostile adversary from attacking your AI solutions and will not prepare you to respond to an AI security event.

This means AI solutions require security that goes far beyond traditional IT needs. This is not just about protecting data and is not just about compliance.

Fear of AI May Be Unfounded, But Fear Can Impact Decisions

Automation has long had an impact on employment. Since the dawn of the industrial age there has been fear that increasing automation would cause job displacement, but history has shown consistent innovation around technology produced more jobs than the automation of work displaced. The optimist will always point to this history as they project the future of AI on the workforce. However, AI is different than other technologies. It may change the needs for routine work so fast that the job creation in the economy cannot keep up. Our advice is for business leaders and employees both to prepare themselves for more rapid job displacement through automation. Assume that there will be less demand for more routine jobs and more demand for positions that require things AI cannot do (those that require human empathy, creativity, rapid comprehensive understanding (intuition)).

Employers should consider helping retrain workers now so that their greatest talents can be tapped into as the routine parts of their work are automated. This will help all to embrace the power of AI tools with less fear of the future.

The Age of Geopolitical AI Is Upon Us

We’ve seen adoption of AI in business processes, but we’ll start seeing the emergence of AI in geopolitics and also greater geopolitical influence on AI. Increasingly, machine learning is being used to support geopolitical analysis as sources of data have grown to the point where human analytical analysis at scale is no longer achievable. Given the controversy surrounding Google’s departure from USG programs like Project Maven, this an area that is ripe for disruption and innovation from emerging companies. Hopefully, government acquisition programs can keep up.

We are also entering into an era of increased geopolitical influence on the algorithms that drive business processes, especially those in the investment sector.  Don’t think geopolitics can influence algorithms, take a close look to how the stock market reacted to the arrest of a Huawei Technologies Co Ltd’s chief financial officer, Meng Wanzhou or the economic impact of trade and IP enforcement issues between China and the US.

AI is also in use in the war in Ukraine, with Russia investing in AI solutions to enhance their use of misinformation and disinformation and Ukraine making use of AI to improve defense in several operational ways. AI in geopolitical conflict is here to stay.

Conclusions and Recommendations

We review things that go wrong to inform your business strategy. We capture more insights on how to do that in our follow-on report titled Artificial Intelligence for Business Advantage.  That is a good next step in ensuring your business is approaching AI with risk mitigation in mind.

Looking for a primer on what executives need to know about real AI and ML? See A Decision-Maker’s Guide to Artificial Intelligence.

Related Reading:

Black Swans and Gray Rhinos

Now more than ever, organizations need to apply rigorous thought to business risks and opportunities. In doing so it is useful to understand the concepts embodied in the terms Black Swan and Gray Rhino. See: Potential Future Opportunities, Risks and Mitigation Strategies in the Age of Continuous Crisis

Explore OODA Research and Analysis

Use OODA Loop to improve your decision making in any competitive endeavor. Explore OODA Loop

Decision Intelligence

The greatest determinant of your success will be the quality of your decisions. We examine frameworks for understanding and reducing risk while enabling opportunities. Topics include Black Swans, Gray Rhinos, Foresight, Strategy, Stratigames, Business Intelligence and Intelligent Enterprises. Leadership in the modern age is also a key topic in this domain. Explore Decision Intelligence

Disruptive/Exponential Technology

We track the rapidly changing world of technology with a focus on what leaders need to know to improve decision-making. The future of tech is being created now and we provide insights that enable optimized action based on the future of tech. We provide deep insights into Artificial Intelligence, Machine Learning, Cloud Computing, Quantum Computing, Security Technology, Space Technology. Explore Disruptive/Exponential Tech

Security and Resiliency

Security and resiliency topics include geopolitical and cyber risk, cyber conflict, cyber diplomacy, cybersecurity, nation state conflict, non-nation state conflict, global health, international crime, supply chain and terrorism. Explore Security and Resiliency

Community

The OODA community includes a broad group of decision-makers, analysts, entrepreneurs, government leaders, and tech creators. Interact with and learn from your peers via online monthly meetings, OODA Salons, the OODAcast, in-person conferences, and Slack forum. Explore The OODA Community.