Christian Terwiesch, A professor at the University of Pennsylvania’s Wharton school, one of the most prestigious US business schools, decided to test out how well GPT technologies like ChatGPT could do on one of his key course tests. To his surprise, ChatGPT did better than many of his students!
He published a paper based on his research here:
Would Chat GPT3 Get a Wharton MBA?
From the paper summary:
OpenAI’s Chat GPT3 has shown a remarkable ability to automate some of the skills of highly compensated knowledge workers in general and specifically the knowledge workers in the jobs held by MBA graduates including analysts, managers, and consultants. Chat GPT3 has demonstrated the capability of performing professional tasks such as writing software code and preparing legal documents. The purpose of this paper is to document how Chat GPT3 performed on the final exam of a typical MBA core course, Operations Management. Exam questions were uploaded as used in a final exam setting and then graded.
The “academic performance” of Chat GPT3 can be summarized as follows. First, it does an amazing job at basic operations management and process analysis questions including those that are based on case studies. Not only are the answers correct, but the explanations are excellent. Second, Chat GPT3 at times makes surprising mistakes in relatively simple calculations at the level of 6th grade Math. These mistakes can be massive in magnitude. Third, the present version of Chat GPT is not capable of handling more advanced process analysis questions, even when they are based on fairly standard templates. This includes process flows with multiple products and problems with stochastic effects such as demand variability. Finally, ChatGPT3 is remarkably good at modifying its answers in response to human hints. In other words, in the instances where it initially failed to match the problem with the right solution method, Chat GPT3 was able to correct itself after receiving an appropriate hint from a human expert. Considering this performance, Chat GPT3 would have received a B to B- grade on the exam.
This has important implications for business school education, including the need for exam policies, curriculum design focusing on collaboration between human and AI, opportunities to simulate real world decision making processes, the need to teach creative problem solving, improved teaching productivity, and more.
We see this as just one more datapoint that we are all in for some Jagged Transitions for 2023.
