Ken Ono feared AI. Now he trains it

Physics Today recently published an interesting and thought-provoking Q&A with eminent mathematician Ken Ono, who recently took a leave of absence from his tenure-track position at the University of Virgina to join Axiom Math, an company developing AI tools to verify mathematical proofs. From the article’s opening paragraphs:

“Am I doing a better service to science by writing 20 more papers that only people in my field will read or by participating in this movement?” Ken Ono asked himself last year. At the time, Ono was an endowed chair at the University of Virginia. In the following months, he’d take a leave of absence from the university, move to Silicon Valley, and become the founding mathematician at an AI startup, Axiom Math.

Axiom is one of several companies focused on developing AI tools to verify mathematical proofs. In the future, similar tools could be applied to check AI-generated computer code, a potentially lucrative service now that AI tools can write lines of code faster than any human can debug them. Startups Harmonic and Math Inc as well as Google’s DeepMind are targeting similar AI-powered verification tools.

Axiom has attracted investor attention, reaching a valuation of $1.6 billion earlier this year. (Ono says that he has shares in the company and that he receives a salary that is similar to his former compensation at Virginia.) His interests, he says, are in helping researchers use AI to think in more-creative ways and across more disciplines.

News flash: ChatGPT sometimes asserts false statements are true (Part 2)

I feel like I’ve done my good deed for the day by uncovering another instance when ChatGPT claimed a “fact” from the secondary mathematics curriculum that is simply incorrect. This one’s a doozy: it claimed that the binomial coefficent $\displaystyle {23 \choose 6} = 100,947$ is equal to $\displaystyle {22 \choose 5} = 26,334$ . (The first identity in the first line below is correct; the second one is not.)

Lesson: ChatGPT is a nice tool but you get what you pay for.

OpenAI Constructs New (and Correct) Mathematical Proof

In previous posts, I’ve commented about how AI can give startingly good and startingly bad results. This one definitely qualifies as startingly good. From the Wall Street Journal:

“If you are a mathematician,” one of the world’s leading mathematicians recently wrote, “you may want to make sure you are sitting down before reading further.”

And you’ll definitely need to sit down if you’re not a mathematician.

Because a famous math problem that stumped humans for the better part of a century has finally been toppled—by AI.

The Wall Street Journal article above does a very good job of summarizing this amazing development. For more information, please see OpenAI’s press release and references therein.

News flash: ChatGPT sometimes asserts false statements are true (Part 1)

I feel like I’ve done my good deed for the day by uncovering an instance when ChatGPT claimed a “fact” from the secondary mathematics curriculum that is simply incorrect.

While using ChatGPT to do some brainstorming on a research project, I received the following response (as part of a bigger response):

Sadly, whenever ChatGPT asserts a fact without documentation, it behooves the user to double-check the “fact.” In this case, there is a pretty blatant sign error.

As taught in high school AP calculus, the Taylor series expansion for $\ln(1+y)$ is

$\ln(1+y) = \displaystyle \sum_{q=1}^\infty \frac{(-1)^{q-1}}{q} y^q$ ,

and so

$-ln(1-y) = \displaystyle -\sum_{q=1}^\infty \frac{(-1)^{q-1}}{q} (-1)^q y^q$

$= \displaystyle\sum_{q=1}^\infty \frac{(-1)^{2q}}{q} y^q$

$= \displaystyle\sum_{q=1}^\infty \frac{y^q}{q}$

Said another way,

$\displaystyle \sum_{q=1}^{A-1} \frac{y^q}{q} + \sum_{q=A}^\infty \frac{y^q}{q} = -\ln(1-y)$

$\displaystyle \sum_{q=A}^\infty \frac{y^q}{q} = -\ln(1-y) - \sum_{q=1}^{A-1} \frac{y^q}{q}$

$\displaystyle \sum_{r=0}^\infty \frac{y^{r+A}}{r+A} = -\ln(1-y) - \sum_{q=1}^{A-1} \frac{y^q}{q}$

$\displaystyle \sum_{r=0}^\infty \frac{y^{r}}{r+A} = \frac{1}{y^A} \left[ -\ln(1-y) - \sum_{q=1}^{A-1} \frac{y^q}{q} \right]$

So that’s the correct answer. Notice that there is a minus sign in front of the sum on the right-hand side, not a plus sign.

So I asked ChatGPT to double-check. Here’s the first part of the response.

Lesson: You get what you pay for.

AI and Proving Theorems

Paul Erdős famously said that mathematicians are machines that turn coffee into theorems. A couple of recent articles in the Wall Street Journal revealed the current state-of-the-art for AI to do the same.

In July, Ben Cohen published the article The High-Schoolers Who Just Beat the World’s Smartest AI Models. The focus of the article was the 2025 International Mathematical Olympiad, the pinnacle of the calendar for high school mathematics competitions. In the United States, the pathway to the IMO is first excellng at a sequence of increasing difficult exams: the AMC->12 (or possibly AMC->10), then the American Invitational Mathematics Exam (AIME), and then USA Mathematical Olympiad (USAMO) or USA Junior Mathematical Olympiad (USJAMO). The top USAMO and USJAMO participants then get invited to a special camp from which the participants in that year’s IMO are selected.

My personal story: back in high school, my score on the AMC->12 (then called the AHSME) qualified me for the AIME my sophomore and junior years, where my run in the competition ended with a resounding thud. My senior year, I caught lightning in a bottle and somehow qualified for the USAMO; I’m not sure what the cut-off is these days, but back then only 150 or so high school students qualified for the USAMO each year. My excitement at qualifying for the USAMO gave way to utter humiliation after I actually attempted the exam (to say that I “took” the exam is probably a misuse of the work “took”.) All this to say: I never came close to sniffing the IMO. From the Wall Street Journal article:

The famously grueling IMO exam is held over two days and gives students three increasingly difficult problems a day and more than four hours to solve them. The questions span algebra, geometry, number theory and combinatorics—and you can forget about answering them if you’re not a math whiz. You’ll give your brain a workout just trying to understand them.

Because those problems are both complex and unconventional, the annual math test has become a useful benchmark for measuring AI progress from one year to the next. In this age of rapid development, the leading research labs dreamed of a day their systems would be powerful enough to meet the standard for an IMO gold medal, which became the AI equivalent of a four-minute mile.

But nobody knew when they would reach that milestone or if they ever would—until now.

The unthinkable occurred earlier this month when an AI model from Google DeepMind earned a gold-medal score at IMO by perfectly solving five of the six problems. In another dramatic twist, OpenAI also claimed gold despite not participating in the official event. The companies described their feats as giant leaps toward the future—even if they’re not quite there yet.

In fact, the most remarkable part of this memorable event is that 26 students got higher scores on the IMO exam than the AI systems.

A second article by Ben Cohen, The Math Legend Who Just Left Academic — for an AI Startup Run by a 24-Year-Old, might be a precursor of things to come. One of the two starts of the article is number theorist Dr. Ken Ono. From the article:

In recent years, Ono began tracking AI’s remarkable progress as it rapidly improved. He was intrigued, though not intimidated. AI was astonishing at cognitive tasks and solving problems it had already seen, but it struggled with the creative elements of his field, which require intuition and abstract thinking.

That creativity is so fundamental to pure mathematics that Ono figured his job would be safe for decades.

But last spring, he was one of 30 mathematicians invited to curate research-level problems as a test of the AI models. He left the symposium profoundly shaken by what he’d seen.

“The lead I had on the models was shrinking,” he said. “And in areas of mathematics that were not in my wheelhouse, I felt like the models were already blowing me away.”

For months afterward, Ono felt like he was grieving his identity. He didn’t know what to do next, knowing that AI models would only get smarter.

“Then I had an epiphany,” he said. “I realized what the models were offering was a different way of doing math.”

Dr. Ono is now taking an extended leave from the University of Virginia to join a new AI startup company called Axiom. From Tech Funding News:

Led by Carina Hong, Axiom Math is developing an AI system that not only solves complex math problems but also generates new mathematical knowledge by proposing conjectures: mathematical statements that have yet to be proven.

The model produces rigorous, step-by-step proofs that can be independently verified using proof assistants such as Lean and Coq. This approach aims to transform English-language math from textbooks and research papers into code, enabling the AI to create and validate new problems that push the boundaries of existing knowledge…

Currently, Axiom is working on models that can discover and solve new math problems. The researchers also hope to apply their work in areas like finance, aircraft design, chip design, and quantitative trading.

Beyond pure mathematics, Axiom’s AI tool is being tested for practical applications in fields requiring rigorous computational precision, including finance, aircraft and chip design, and quantitative trading.

Time will tell if the intersection of AI with mathematics can generate a profitable company. What I don’t doubt is that the previously unthinkable — original mathematical work by AI — will eventually happen, given enough time.

ChatGPT and mathematics

I recently read the delightful blog post ChatGPT Is Not Ready to Teach Geometry (Yet), with the wonderful sub-headline “The viral chatbot is often wrong, but never in doubt. Educators need to tread carefully.” Many thanks to the article AI Bot ChatGPT Needs Some Help With Math Assignments in the Wall Street Journal for directing me to this post. Both of these articles are cited at length below; I recommend both.

In case you’ve been on the moon for the past few months, much digital ink has been spilled in the past few months about how ChatGPT will affect education. From the blog post:

On HITC, Ellissa Bain described how ChatGPT could “Write Your Papers in Seconds,” explaining that this is “great for students” and “examiners will never know.” It’s not at all great for student learning, but it can definitely fool examiners; a team at the New YorkTimes Upshot confirmed that a small group of experts—including a teacher, a professor, a tutor, and a famous children’s author—often failed to distinguish between ChatGPT’s output and the writing of actual fourth and eighth graders in response to essay prompts from the National Assessment of Education Progress. In The Atlantic, teacher Daniel Herman pronounced “The end of high-school English,” explaining that “what GPT can produce right now is better than the large majority of [student] writing.” ChatGPT could compare Hamlet to Beloved, illustrate Buddhist ideas using a pet hamster, and write a convincing essay describing how its (fabricated) experience volunteering at a pet shelter had prepared it for success at Stanford. The bot’s ability to fabricate an admissions essay raises the question of whether colleges that no longer require standardized tests will need to drop the essay requirement as well.

Unfortunately, as Paul von Hipple, an associate professor at the University of Texas, discovered, ChatGPT isn’t so great at the Pythagorean theorem. From the blog post:

What are some common misunderstandings of Pythagoras’ theorem?

ChatGPT responded with four examples, which I’ll take just slightly out of order. The first three were howlers:

Some common misunderstandings about Pythagoras’ theorem include:

Believing that it only works for right triangles. While Pythagoras’ theorem is most commonly used to find the length of a missing side in a right triangle, it can also be used to determine whether a triangle is a right triangle. If the lengths of the sides of a triangle satisfy the formula of Pythagoras’ theorem (i.e., if the square of the length of the hypotenuse is equal to the sum of the squares of the other two sides), then the triangle is a right triangle.

Believing that it only works for right triangles with a 90 degree angle: Pythagoras’ theorem is specifically designed for right triangles, which have one 90 degree angle. However, it is possible for a triangle to have two acute angles (angles less than 90 degrees) and still be a right triangle.

Believing that it only works for right triangles with legs that are perpendicular to each other: In a right triangle, the sides that are not the hypotenuse are called the legs. While it is true that the legs of a right triangle are typically perpendicular to each other, this is not a requirement for the theorem to be applicable. As long as a triangle has one 90 degree angle, Pythagoras’ theorem can be used to find the length of a missing side.

It gets better. Dr. von Hipple then asked the following:

Suppose the hypotenuse of a right triangle is 7.1 miles long, and one of the legs is 3.85 miles long. How long is the other leg?

ChatGPT returned a wrong answer: 5.93 miles. (See the blog post for more on this error.)

Dr. von Hipple then, with a simple typo, inadvertently asked ChatGPT to solve a triangle that can’t be solved:

I wondered if it would recognize a right triangle if I described it indirectly. So I started my next question:

Suppose a triangle has three sides called A, B, and C. A is 7 inches long and B is 7 inches long. The angle between A and C is 45 degrees, and so is the angle between A and B. What is the length of side C?

This was a typo; the 45-degree angle was placed between the wrong two sides. Nevertheless ChatGPT gave an answer:

Since the angle between A and B is 45 degrees, and the angle between A and C is also 45 degrees, the triangle is an isosceles right triangle, where A and B are the legs and C is the hypotenuse….

Dr. von Hipple’s conclusion:

This doesn’t make sense. If A and B are the legs of a right triangle, the angle between them can’t be 45 degrees; it has to be 90. ChatGPT went ahead and calculated the length of C using Pythagoras’ theorem, but it had revealed something important: it didn’t have a coherent internal representation of the triangle that we were talking about. It couldn’t visualize the triangle as you or I can, and it didn’t have any equivalent way to catch errors in verbal descriptions of visual objects.

In short, ChatGPT doesn’t really “get” basic geometry. It can crank out reams of text that use geometric terminology, but it literally doesn’t know what it is talking about. It doesn’t have an internal representation of geometric shapes, and it occasionally makes basic calculation errors…

What is ChatGPT doing? It is bloviating, filling the screen with text that is fluent, persuasive, and sometimes accurate—but it isn’t reliable at all. ChatGPT is often wrong but never in doubt.

The Wall Street Journal article cited above provided some more howlers. Here are a couple:

So what to make of all this? I like this conclusion from the Wall Street Journal:

Another reason that math instructors are less fussed by this innovation it that they have been here before. The field was upended for the first time decades ago with the general availability of computers and calculators.

Whereas English teachers are only now worrying about computers doing their students’ homework, math teachers have long wrestled with making sure students were actually learning and not just using a calculator. It’s why students have to show their work and take tests on paper.

The broader lesson is that AI, computers and calculators aren’t simply a shortcut. Math tools require math knowledge. A calculator can’t do calculus unless you know what you’re trying to solve. If you don’t know any math, Excel is just a tool for formatting tables with a lot of extra buttons.

Eventually, artificial intelligence will probably get to the point where its mathematics answers are not only confident but correct. A pure large language model might not be up for the job, but the technology will improve. The next generation of AI could combine the language skills of ChatGPT with the math skills of Wolfram Alpha.

In general, however, AI, like calculators and computers, will likely ultimately be most useful for those who already know a field well: They know the questions to ask, how to identify the shortcomings and what to do with the answer. A tool, in other words, for those who know the most math, not the least.