RBS meets LLMs - Type inference using LLM

Transcript

1. RBS meets LLMs
   Leaner Technologies, Inc.
   黒曜
   (@kokuyouwind)
   ~ Type inference using LLM ~
   

   View Slide

2. Talk: Japanese
   Slides: English (+ Japanese)
   

   View Slide

3. $ whoami
   黒曜 / @kokuyouwind
   Work at: Leaner Technologies, Inc.
   Ruby Sponsor
   Day 1: Sponsor Talk (Done)
   Day 2: Leaner Drinkup
   

   View Slide

4. The idea is covered by Matz's Keynote,
   but this is the first time you've heard of it!
   Matz
   のKeynote
   で出た話と発想が被っていますが、
   初めて聞いたことにしてください！
   

   View Slide

5. Large Language Model (LLM)
   Machine learning models trained on large amounts of text data.
   ChatGPT Bard Bing Chat
   ⼤量のテキストデータを使ってトレーニングされた機械学習モデル
   ⼤規模⾔語モデル
   

   View Slide

6. Example: ChatGPT
   

   View Slide

7. Can we do something fun with this?
   🤔
   なにか⾯⽩いことに使えないかな？
   

   View Slide

8. Type Inference
   user = User.new
   # `user` is an User type variable
   name = user.name
   # `name` is a String type variable(?)
   1
   2
   3
   4
   5
   Humans sometimes infer type from word meanings.
   ⼈間は単語の意味から型を推測することがある
   

   View Slide

9. With ChatGPT,
   can we infer the type
   from the meaning of a word?
   ChatGPT
   を使えば、単語の意味から
   型を推測できるのでは？
   

   View Slide

10. Let's see some cases!
    Tool Settings
    model: ChatGPT(gpt-3.5-turbo)
    Temperature: 0
    GUI: BetterChatGPT
    

    View Slide

11. Case 1: User Builder
    user = UserBuilder.new.name('kokuyouwind').build
    # We infer types as follows:
    # * UserBuilder#name returns an UserBuilder
    # * UserBuilder#build returns an User
    1
    2
    3
    4
    5
    

    View Slide

12. Case 1: User Builder
    

    View Slide

13. Case 1: User Builder
    

    View Slide

14. (There is extra output, but)
    Perfect!!!
    👏
    

    View Slide

15. Case 2: Company Repository
    company = CompanyRepository.new.find(1).name
    # We infer types as follows:
    # * CompanyRepository#find returns a Company
    # * Company#name returns a String
    # Note.
    # This expression has the same syntactic form with:
    # user = UserBuilder.new.name('kokuyouwind').build
    1
    2
    3
    4
    5
    6
    7
    8
    9
    

    View Slide

16. Case 2: Company Repository
    

    View Slide

17. (There is extra output, but)
    Perfect, again!!!
    👏
    

    View Slide

18. Problems
    RBS sometimes broken (
    たまにRBS
    の構⽂がおかしい)
    Extra Outputs (
    不要な出⼒がある)
    Only RBS output needed (RBS
    だけ出⼒させたい)
    Single RBS (RBS
    がひとまとめになっている)
    Separate User's RBS to user.rbs, and so on
    (User
    のRBS
    はuser.rbs
    に、などクラスごとに分割したい)
    

    View Slide

19. Improve: FewShot
    Demonstrate type inference for UserBuilder,
    then ask about the CompanyRepository case.
    UserBuilder
    の模範解答を⼊⼒してから、CompanyRepository
    について推論させる
    Demo Input
    Main Question
    Output
    模範解答
    メインの問い合わせ
    出⼒
    

    View Slide

20. Improve: FewShot
    Demo Input Main Question
    模範解答を⼊⼒ メインの問い合わせ
    

    View Slide

21. View Slide

22. Really Perfect!!!
    👏
    

    View Slide

23. Please talk to me at parties, etc.
    ぜひドリンクアップなどで話しかけてください！
    I've tried to infer types for metaprogramming,
    refine existing RBS, etc.,
    but I don't have time to talks. So,
    メタプログラミングコードに対する型の推測や既存RBS
    の詳細化なども試しましたが
    今回は話す時間が⾜りないので、
    

    View Slide

24. Future Works
    CommandLine Tool (
    コマンドラインツール化)
    Fine tuning for type inference (
    型推測⽤のファインチューニング)
    Agent-style autonomous drive (
    エージェントスタイルの⾃律駆動)
    Refer to the necessary files (
    必要なファイルの参照)
    Correction with type check results (
    型チェックを元にした修正)
    Run with LLMs locally (
    ローカルで動くLLM
    を使った動作)
    Alpaca.cpp, ChatRWKV, or else
    

    View Slide