Failing to reason with LLMs (ARC AGI kaggle update with Llama3)

Lightning talk at PyDataLondon 2024 August, I spoke on the Kaggle ARC AGI competition and how I've pushed on with using Program as Thought and ideas behind Self Consistency and CRITIC to enhance the model-writing capability.
https://www.meetup.com/pydata-london-meetup/events/302434648/

Transcript

1. Abstractly reasoning – failing with an LLM (next steps for ARC AGI) PyDataLondon 2024-08 lightning talk @IanOzsvald – ianozsvald.com
2. Can LLMs reason? ARC AGI Abstract JSON “initial → target” Tried “don’t code, just reason” Llama3 70B pretty smart Llama3 8B writes code pretty well, sometimes Abstraction & Reasoning Challenge By [ian]@ianozsvald[.com] Ian Ozsvald
3. 30% solutions pretty good! By [ian]@ianozsvald[.com] Ian Ozsvald It counts! Comments! Reasonable numpy! Correct substitution!
4. Convincing weirdness By [ian]@ianozsvald[.com] Ian Ozsvald
5. Big issue – it gets stuck on the same ideas Get LLM to read lots of failed model outputs, summarise, then maybe I could ask it to make new strategies? Notes → NotANumber.email newsletter Next steps By [ian]@ianozsvald[.com] Ian Ozsvald