Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Failing to reason with LLMs (ARC AGI kaggle up...

Avatar for ianozsvald ianozsvald
August 06, 2024

Failing to reason with LLMs (ARC AGI kaggle update with Llama3)

Lightning talk at PyDataLondon 2024 August, I spoke on the Kaggle ARC AGI competition and how I've pushed on with using Program as Thought and ideas behind Self Consistency and CRITIC to enhance the model-writing capability.
https://www.meetup.com/pydata-london-meetup/events/302434648/

Avatar for ianozsvald

ianozsvald

August 06, 2024
Tweet

More Decks by ianozsvald

Other Decks in Technology

Transcript

  1. Abstractly reasoning – failing with an LLM (next steps for

    ARC AGI) PyDataLondon 2024-08 lightning talk @IanOzsvald – ianozsvald.com
  2. Can LLMs reason? ARC AGI Abstract JSON “initial → target”

    Tried “don’t code, just reason” Llama3 70B pretty smart Llama3 8B writes code pretty well, sometimes Abstraction & Reasoning Challenge By [ian]@ianozsvald[.com] Ian Ozsvald
  3. 30% solutions pretty good! By [ian]@ianozsvald[.com] Ian Ozsvald It counts!

    Comments! Reasonable numpy! Correct substitution!
  4. Big issue – it gets stuck on the same ideas

    Get LLM to read lots of failed model outputs, summarise, then maybe I could ask it to make new strategies? Notes → NotANumber.email newsletter Next steps By [ian]@ianozsvald[.com] Ian Ozsvald