ChatGPT Can Pass the Bar Exam. So What?

We’ve all read articles that Gen AI will soon replace doctors, lawyers, accountants, and a host of other credentialed professions. Indeed, ChatGPT has passed or nearly passed all three components of the U.S. Medical Licensing Exam, passed the Uniform Bar Exam with a score approaching the 90th percentile, and passed the CPA exam (on the second try).

Does this mean advances in generative AI can usurp any knowledge-based profession? Are credentials even worth pursuing, given the ability of large language models to perform the core tasks of these professionals? Will future occupations simply be interacting with an AI-powered bot that can resolve your concerns in a fraction of the time (and cost)?

While the concerns have merit, the reality is more complicated. Despite the attention-grabbing headlines, ChatGPT’s passing of these professional assessments won’t immediately threaten the livelihood of most of our credentialed professionals because acing a test is not the same as being a practicing professional. In fact, there are three core dynamics at play that point towards credentials as becoming more valuable. First, an accelerated decrease in the value of the “back office.”  Second, continuing the credential as an “AI-unassailable” public trust provision. Finally, an increase in the sophistication of the assessment of professionals to determine the ability to meet the bar of “doing things only humans can do.”

”Back office” work is most vulnerable

Multiple industries have already successfully deployed AI to rapidly improve back office operations. This most basic form of AI, machine learning, is  effective here as models trained at replicating highly repetitive work with modified inputs. So a system that writes a contract for a specific use case after being trained on thousands of other contracts for similar/identical legal issues will likely produce reliable results requiring minimal review from a senior attorney. This has the potential to save a tremendous amount of time and money in the production of legal work.  

But does this reduce the requirement for lawyers to be involved in the equation? Perhaps at the margins, but certainly not for most of the process around collecting, interpreting, and engaging with individuals about the meaning and impact of the legal documents produced. We’ve already seen lawyers sanctioned for submitting a ChatGPT-generated legal brief that included fictitious citations. 

Similarly, before the advent of spreadsheet technology, bookkeepers painstakingly produced much of what now takes minutes. But do we look at financial numbers more or less as a function of that change? Are the financial analysts and credentialed professionals who sign off on those documents more or less central to our way of operating in today’s world? I think few who work in any modern enterprise would argue “less.” 

The biggest threat from AI will be for the many individuals whose jobs involve skilled but highly repetitive tasks that are particularly vulnerable to replacement by AI-driven technology. 

Digital spreadsheet tools didn’t replace certified accountants, but they did remove legions of bookkeepers whose work was now automated. Digital document discovery tools didn’t remove the need for lawyers; they eliminated the armies of paralegals and support staff previously needed to cross-reference and manage discovery requests. 

AI will have a similar and amplified effect. Basic knowledge-based tasks will continue to get automated, driving opportunities either to those roles at the high end of knowledge work OR to those functions that are only partially knowledge-based and also require significant hands-on engagement (think plumbing or electrical - these are knowledge-based and certified, but I think few would argue that Gen AI will soon replace your plumber!) 

To understand why I believe professional credentials will become more valuable, let’s first take a look at what they are. 

Professional credentials provide a basis for public trust, and will continue to do so

We tend to think of these in terms of a certification that workers have a certain amount of knowledge (and the news about Chat GPT suggests that this is the common way of evaluating credentials). And certainly, this is part of what the credential says - the rigorous exams that professionals are required to take certify that they have at least the base level knowledge required to provide help in the area where they are hired. 

But is this all? Of course not - which is why many professions require copious amounts of additional training (years in a role, apprenticeship style programs, residencies, etc) as well as continuing education to ensure that they can actually deliver on the promise of their profession. This is because professional credentials really act as certificates of public trust. 

We all want to know when we go to our doctor, nurse, lawyer, accountant, or see CFA or CFP behind someone’s name that we are dealing with someone who knows what they are doing and is regulated in some way. Putting your personal, health, financial, or business interests in the hands of a credentialed professional is, and will continue to be a big leap of trust - and credentials exist to ensure that trust is warranted.

Was the bridge you drove over yesterday certified by a credentialed engineer? Was the last medical professional you saw licensed and certified? I thought so. 

Would you have it any other way? I thought not. 

That said, with generative AI starting to pass professional assessments, we need to take a closer look at the credentialing process that separates what humans can do from AI.

Expecting more from assessments in an AI-powered age

Much has been written about knowledge work that is “safe” from Gen AI, but it seems to generally boil down to three categories - roles that involve psychomotor skills, interpersonal skills, and creativity. Most credentialed professionals use some combination of all of these categories to be truly successful in their roles. While large language models can pass an exam, they can’t act as certified intermediaries in communicating a medical diagnosis with empathy, in understanding ambiguities in the application of law, or in creative problem-solving. 

In other words, would anyone want to be represented by Chat GPT in a court of law? Replace their hip? No. But as credentialed roles become even more important, so, too, do the pathways to ensure that these roles are, in fact, delivering on what we need of them, particularly in the new AI-powered world. 

In doing so, we need to remember how recognized professional credentials communicate public trust. The baseline that these credentials convey – that the professionals with licenses have acquired the requisite knowledge to practice in their respective industries – is no longer enough. 

Licensure tests will need to evolve to assess more than what AI can demonstrate and go beyond the demonstration of meeting a certain level of knowledge to demonstration of complex, successful application of that knowledge in real-life situations. 

Many state and national testing boards are moving in this direction. Tests from across the spectrum (Nursing, Bar, CFA, CPA - the list goes on) have been evolving to make their exams more representative of the types of challenges practitioners will face. Kaplan as an organization has leaned into those changes and has invested significantly in both systems and technology designed to help train and assess professional judgment in real-world situations (e.g., our products in medical simulation). 

Professionals (and the public) have an opportunity to push for even more changes in the future. This means testing what lies in the gap between passing today’s medical boards or the bar exam and being a good practitioner – exercising judgment and discretion, building trust, thinking creatively, understanding nuanced situations in problem-solving, and communicating with empathy. 

Many tools exist that can be leveraged to improve what we currently do, including:

• Simulation programs that assess both knowledge and application - many exams now contain scenario-based questions, but technology gives us the ability to immerse individuals in real-world scenarios to assess readiness (see medical simulations with iHP)

• Credentials earned in combination with work training or apprenticeship programs, so we know they are linked (a good example are our programs offered to employees of partner organizations)

• Job-based training leading to advanced credentials and certificates

• Credentials are offered immediately after college courses (such as through our school partnership programs), which allows for core education to be more tightly linked to certifications  

In sum, a new generation of assessments should combine rigor with measuring a vital set of human skills. That will give individuals opportunities to avoid shrinking sectors of the economy that are facing obsolescence because of AI. And they will help credentials maintain their real-world relevance by ensuring that they measure professional judgment and, in doing so, maintain public trust.

Steve Marietti is chief commercial officer for Kaplan North America, overseeing Kaplan's exam prep, professional licensing, and industry-focused continuing education programs. Reach out to Steve to share your feedback on this article or discuss how Kaplan can help your company or institution.