ArchiveDisruptive TechnologyOODA Original

Opportunities for Advantage: Natural Language Processing – Meta AI Builds a Huge GPT-3 Model and Makes it Available for Free

OODA CTO Bob Gourley recently provided a discussion of the potential impacts and use cases of improved natural language processing (NLP), in which he highlighted the major developments in computer language understanding in a way that can help enterprise and government leaders better prepare to take action on these incredible new capabilities. Major improvements in the ability of computers to understand what humans write say and search are being made commercially available. These improvements are significant and will end up changing just about every industry in the world. But at this point, they are getting little notice outside a narrow segment of experts.

Background

Major developments of interest to this ‘expert’ class have been reviewed here at OODA Loop:

The Current AI Innovation Hype Cycle: Large Language Models, OpenAI’s GPT-3 and DeepMind’s RETRO:  For better or for worse, Large Language Models (LLMs) – used for natural language processing by commercial AI Platform-as-a-Service (PaaS) subscription offerings – have become one of the first “big data” applied technologies to become a crossover hit in the AI marketplace:  “Large language models—powerful programs that can generate paragraphs of text and mimic human conversation—have become one of the hottest trends in AI in the last couple of years. But they have deep flaws, parroting misinformation, prejudice, and toxic language.” (3)

From a big data perspective, LLMs are gigantic datasets or data models. In the world of AI, LLM’s are huge neural networks that increase in size based on the number of parameters included in the model and are used by neural networks for training. Neural network parameters are values constantly refined while training an AI model, resulting in AI-based predictions. The more parameters, the more the data training results in structured information (organized around the parameters of the LLM) – enhancing the accuracy of the predictions generated by the model.

In April of 2020, the bleeding edge of innovation in this space was the Facebook chatbot Blender, made open source by Facebook with 9.4 billion parameters and an innovative structure for training on 1.5 billion publicly available Reddit conversations  – with additional conversational language datasets for conversations that contained some kind of emotion;  information-dense conversations; and conversations between people with distinct personas.  Blender’s 9.4 billion parameters dwarfed Google’s Meena (released in January 2020) by almost 4X.  (1)

OpenAI, a San Francisco-based research and deployment company, released GPT-3 in June of 2020  – and the results were instantly compelling: Natural language processing (NLP) with a seeming mastery of language that generated sensible sentences and was able to converse with humans via chatbots.  By 2021, the MIT Technology Review was proclaiming OpenAI’s GPT-3 a top 10 breakthrough technology, “a big step toward AI that can understand and interact with the human world.”

Open-Source Natural Language Processing: EleutherAI’s GPT-J:  Initially, access to OpenAI’s GPT-3 was a selective process complete with a waiting list.  It has since been commercialized in collaboration with Microsoft.   In response, EleutherAI – a self-described “grassroots collective of researchers working to open-source AI research”  launched GPT-J in July 2020 as a quest to replicate the OpenAI GPT collection of models. The goal is to “break the OpenAI-Microsoft monopoly” through broadening availability and the collective intelligence of open-source development of a competing class of GPT models.

GPT is an acronym for “generative pre-trained transformer.” The first paper on the” GPT of a language model was written by Alec Radford and colleagues, and published in a preprint on OpenAI’s website on June 11, 2018.  It showed how a generative model of language is able to acquire world knowledge and process long-range dependencies by pre-training on a diverse corpus with long stretches of contiguous text. (4)

Meta AI is now in the GPT-3 model game  – with the release of a massive proprietary GPT-3 model which the company has made available for free to researchers.

To continue reading please consider joining as either a subscriber or full member to support our continued research and analysis. For more on benefits of membership see below.

Want more insight? Log in for the full report

Already a member?  Sign in to your account.

Become A Member

OODA Loop provides actionable intelligence, analysis, and insight on global security, technology, and business issues. Our members are global leaders, technologists, and intelligence and security professionals looking to inform their decision making process to understand and navigate global risks and opportunities.

You can chose to be an OODA Loop Subscriber or an OODA Network Member. Subscribers get access to all site content, while Members get all site content plus additional Member benefits such as participation in our Monthly meetings, exclusive OODA Unlocked Discounts, discounted training and conference attendance, job opportunities, our Weekly Research Report, and other great benefits. Join Here.

Related Reading:

Explore OODA Research and Analysis

Use OODA Loop to improve your decision making in any competitive endeavor. Explore OODA Loop

Decision Intelligence

The greatest determinant of your success will be the quality of your decisions. We examine frameworks for understanding and reducing risk while enabling opportunities. Topics include Black Swans, Gray Rhinos, Foresight, Strategy, Stratigames, Business Intelligence and Intelligent Enterprises. Leadership in the modern age is also a key topic in this domain. Explore Decision Intelligence

Disruptive/Exponential Technology

We track the rapidly changing world of technology with a focus on what leaders need to know to improve decision-making. The future of tech is being created now and we provide insights that enable optimized action based on the future of tech. We provide deep insights into Artificial Intelligence, Machine Learning, Cloud Computing, Quantum Computing, Security Technology, Space Technology. Explore Disruptive/Exponential Tech

Security and Resiliency

Security and resiliency topics include geopolitical and cyber risk, cyber conflict, cyber diplomacy, cybersecurity, nation state conflict, non-nation state conflict, global health, international crime, supply chain and terrorism. Explore Security and Resiliency

Community

The OODA community includes a broad group of decision-makers, analysts, entrepreneurs, government leaders and tech creators. Interact with and learn from your peers via online monthly meetings, OODA Salons, the OODAcast, in-person conferences and an online forum. For the most sensitive discussions interact with executive leaders via a closed Wickr channel. The community also has access to a member only video library. Explore The OODA Community

Daniel Pereira

Daniel Pereira

Daniel Pereira is research director at OODA. He is a foresight strategist, creative technologist, and an information communication technology (ICT) and digital media researcher with 20+ years of experience directing public/private partnerships and strategic innovation initiatives.