Previous discussion: Don Knuth plays with ChatGPT - May 20, 2023, 626 comments, 927 points https://news.ycombinator.com/item?id=36012360

For question 3, ChatGPT 5 Pro gave better answer:

> It isn’t “wrong.” Wolfram defines Binomial[n,m] at negative integers by a symmetric limiting rule that enforces Binomial[n,m] = Binomial[n,n−m]. With n = −1, m = −1 this forces Binomial[−1,−1] = Binomial[−1,0] = 1. The gamma-formula has poles at nonpositive integers, so values there depend on which limit you adopt. Wolfram chooses the symmetry-preserving limit; it breaks Pascal’s identity at a few points but keeps symmetry. If you want the convention that preserves Pascal’s rule and makes all cases with both arguments negative zero, use PascalBinomial[−1,−1] = 0. Wolfram added this explicitly to support that alternative definition.

Of course this particular question might have been in the training set.

Honestly 2.5 years feel like infinity when it comes to AI development. I'm using ChatGPT very regularly, and while it's far from perfect, recently it gave obviously wrong answers very rarely. Can't say anything about ChatGPT 5, I feel like in my conversations with AI, I've reached my limit, so I'd hardly notice AI getting smarter, because it's already smart enough for my questions.

Here is gpt 5 thinking posting all 20 questions verbatim. Appreciate I might get better results one question at a time.

https://chatgpt.com/share/6897a21b-25c0-8011-a10a-85850870da...

Pretty interesting - some contamination, some better answers, and it failed to write a sentence with all 5-letter-words. I’d have expected it to pass this one!

Simple example: “Every night, dreams swirl swiftly.

I'll never get over the fact that the grad student didn't even bother to use gpt-4, so this was using gpt 3.5 or something.

The Haj novel by Leon Uris, by the way, is a disturbing Zionist book that depicts Arabs as backward, violent, and incapable of progress without outside control, and that justifies the taking of Palestinian land by Jewish settlers.

I was reading yesterday about a Buddhist concept (albeit quite popular in the west) called Begginer's Mind. I think this post represents it perfectly.

We are presented with a first reaction to chatgpt, we must never forget how incredible this technology is, and not become accustomed to it.

Donald knuth approached several of the questions from the absence of knowledge, asking questions as basic as "12. Write a sentence that contains only 5-letter words.", and being amazed not only by correct answers, but incorrect answers parsed effectively and with semantic understanding.

It would be great to have an update from Knuth. There is no other Knuth.

> I myself shall certainly continue to leave such research to others, and to devote my time to developing concepts that are authentic and trustworthy. And I hope you do the same.

In a way taming these stochastic beasts into reliable and trustworthy software components is more like (quantitative) social science than computer science.

2023 was a crazy and exciting year for AI research. LLMs have come a long way, but clearly still have a long way to go. They should do much better on most of these questions.

The discussion at the end also reminded me of how a lot of us took Gary Marcus' prose more seriously at the time before many of his short-term predictions started failing spectacularly.

Knuth's ChatGPT Experiment Insights https://gemini.google.com/share/3768c883b67c

Here, person with a 300 IQ who is blind, can you count the letters in this written word? "Well no, bec-" Ahaha, dumbass!

> Silly jokes told with mirth bring mirthful grins.

Anyone have an idea how this happened? Supposed to be a sentence of only 5 letter words.

It's sad that we've made the internet so disorganized and crammed with advertising and crap that we now need tools to find actual information and summarize it for us.

[flagged]