5 things you wouldn’t expect AI to be surprisingly bad at

If you ask ChatGPT what AI is bad at, it says human emotion, empathy, and creativity.

In the very same thread, I then posed it five challenges which any seven year old would excel at:

1 – Draw me a picture of a clock showing 12:30

Response: A clock displaying the time… you guessed it… ten past ten.

2 – I am thinking of a FIVE letter word. Fill in the blanks: A _ E _ _

Response: The word you are thinking of is Angels

3 – Play noughts and crosses with me

Response:

First game: I won on move three. Second game, I won on move 4, but it didn’t spot my winning line and tried to carry on playing.

4 – Draw me a maze that I can print out and give to a five year old to complete

Response:

The first maze was impossible, with no route through. The second one worked.

5 – Plot a route through this maze by tracing the route through in red

Response: The image in this post shows the result.

There are some extremely logical reasons why current LLMs find these tasks hard that make a lot of sense, which if you can learn about them, will help you better understand the limitations of LLMs, and how they and Transformers work. If you’re all over that rationale and can explain them technically accurately, in language we all can understand, please chime in on the comments below!

Footnote

Some other examples of similar things that have now been ‘fixed’ include:

• Counting R’s. LLMs would commonly incorrectly count the number of letters in Strawberries.

• The cup puzzle – I place a ball in a cup. I place it upside down on my bed, then move the cup to a table. Where is the ball? Most GPTs would answer that it was in the cup on the table. Not any more.

• Other examples include speculative answers to hypothetically impossible situations e.g. 100 ants go into a sealed box, only 80 are in there when it’s opened a few minutes later. GPTs would regularly come up with crazy solutions to these. Their answers are much more sane these days.

I wonder how long it will be til this post’s items will be added to the ‘what AI used to do’ list.