Open-Source Natural Language Processing: EleutherAI’s GPT-J
OpenAI, a San Francisco-based research and deployment company, released GPT-3 in June of 2020 – and the results were instantly compelling: Natural language processing (NLP) with a seeming mastery of language that generated sensible sentences and was able to converse with humans via chatbots. By 2021, the MIT Technology Review was proclaiming OpenAI’s GPT-3 a top 10 breakthrough technology and “a big step toward AI that can understand and interact with the human world.”
Initially, access to GPT-3 was a selective process complete with a waiting list. It has since commercialized in collaboration with Microsoft. In response, EleutherAI – a self-described “grassroots collective of researchers working to open-source AI research” launched GPT-J in July 2020 as a quest to replicate the OpenAI GPT collection of models. The goal is to “break the OpenAI-Microsoft monopoly” through broadening availability and the collective intelligence of open-source development of a competing class of GPT models.
GPT is an acronym for “generative pre-trained transformer.” The first paper on the” GPT of a language model was written by Alec Radford and colleagues, and published in a preprint on OpenAI’s website on June 11, 2018. It showed how a generative model of language is able to acquire world knowledge and process long-range dependencies by pre-training on a diverse corpus with long stretches of contiguous text. (1)
Want more insight? Log in for the full report
Already a member? Sign in to your account.
OODA Loop provides actionable intelligence, analysis, and insight on global security, technology, and business issues. Our members are global leaders, technologists, and intelligence and security professionals looking to inform their decision making process to understand and navigate global risks and opportunities.
You can chose to be an OODA Loop Subscriber or an OODA Network Member. Subscribers get access to all site content, while Members get all site content plus additional Member benefits such as participation in our Monthly meetings, exclusive OODA Unlocked Discounts, discounted training and conference attendance, job opportunities, our Weekly Research Report, and other great benefits. Join Here.
For more information please click here. Thanks!
Already a member? Sign in to your account.
Black Swans and Gray Rhinos
Now more than ever, organizations need to apply rigorous thought to business risks and opportunities. In doing so it is useful to understand the concepts embodied in the terms Black Swan and Gray Rhino. See: Potential Future Opportunities, Risks and Mitigation Strategies in the Age of Continuous Crisis
Explore OODA Research and Analysis
Use OODA Loop to improve your decision making in any competitive endeavor. Explore OODA Loop
The greatest determinant of your success will be the quality of your decisions. We examine frameworks for understanding and reducing risk while enabling opportunities. Topics include Black Swans, Gray Rhinos, Foresight, Strategy, Stratigames, Business Intelligence and Intelligent Enterprises. Leadership in the modern age is also a key topic in this domain. Explore Decision Intelligence
We track the rapidly changing world of technology with a focus on what leaders need to know to improve decision-making. The future of tech is being created now and we provide insights that enable optimized action based on the future of tech. We provide deep insights into Artificial Intelligence, Machine Learning, Cloud Computing, Quantum Computing, Security Technology, Space Technology. Explore Disruptive/Exponential Tech
Security and Resiliency
Security and resiliency topics include geopolitical and cyber risk, cyber conflict, cyber diplomacy, cybersecurity, nation state conflict, non-nation state conflict, global health, international crime, supply chain and terrorism. Explore Security and Resiliency
The OODA community includes a broad group of decision-makers, analysts, entrepreneurs, government leaders and tech creators. Interact with and learn from your peers via online monthly meetings, OODA Salons, the OODAcast, in-person conferences and an online forum. For the most sensitive discussions interact with executive leaders via a closed Wickr channel. The community also has access to a member only video library. Explore The OODA Community