Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Generative A.I. + Law - A Year in Review

Generative A.I. + Law - A Year in Review

Generative A.I. + Law - A Year in Review - Including GPT Passes the Bar & Eleven Days that Defined the Year in (Legal) A.I.

Talk Delivered at The Legal Tech Fund Summit - TLTF Miami (2023)

December 4, 2023

Daniel Martin Katz

December 06, 2023
Tweet

More Decks by Daniel Martin Katz

Other Decks in Technology

Transcript

  1. G E N E R AT I V E A . I . + L AW
    @ computational
    professor daniel martin katz danielmartinkatz.com
    Illinois tech - chicago kent law 273Ventures.com
    A Y E A R I N R E V I E W I N C LU D I N G G P T PA S S E S T H E B A R


    & E L E V E N DAY S T H AT D E F I N E D T H E Y E A R I N ( L E G A L ) A I
    https://bit.ly/3yxhY2e

    View full-size slide

  2. NOVEMBER 30, 2022

    View full-size slide

  3. I THINK THIS DAY WILL
    GO DOWN AS A VERY
    IMPORTANT DAY IN
    THE HISTORY OF
    TECHNOLOGY

    View full-size slide

  4. NOVEMBER 30, 2022
    IS THE DAY THAT CHATGPT
    WAS FIRST RELEASED
    BECAUSE

    View full-size slide

  5. THIS WAS THE
    BEGINNING OF THE
    BROADER PUBLIC’S
    AWARENESS …

    View full-size slide

  6. OF WHAT WAS
    ALREADY TRUE …

    View full-size slide

  7. WHICH IS


    THAT IN THE
    BACKGROUND

    View full-size slide

  8. THERE HAVE BEEN
    FUNDAMENTAL
    IMPROVEMENTS
    OCCURRING WITHIN
    THE WORLD OF
    NATURAL LANGUAGE
    PROCESSING (NLP)

    View full-size slide

  9. I PERSONALLY HAVE
    BEEN WORKING IN THIS
    AREA FOR A WHILE …

    View full-size slide

  10. THINKING ABOUT
    HOW TO USE
    LANGUAGE MODELS
    HERE IN THE


    LEGAL SECTOR

    View full-size slide

  11. BUT IT HAS BEEN HARD
    TO COMMUNICATE TO
    MANY FOLKS IN THE
    LEGAL AND RELATED
    SECTORS …

    View full-size slide

  12. ABOUT THE
    NATURE OF THE
    LANDSCAPE SHIFT
    IN THE FIELD …

    View full-size slide

  13. MAYBE IT IS


    ROBOLAWYER


    FATIGUE …?

    View full-size slide

  14. MAYBE IT IS THE
    ABSENCE OF A CLEAR
    DEMONSTRATION
    PROJECT …

    View full-size slide

  15. SO NOW WE ARE JUST


    12 MONTHS INTO
    ARGUABLY THE MOST
    SUCCESSFUL PRODUCT
    LAUNCH IN HISTORY …

    View full-size slide

  16. SO TO KICK THINGS OFF
    HERE AT THE TLTF SUMMIT

    View full-size slide

  17. TODAY —


    I THOUGHT I WOULD
    OFFER SOME OPENING
    COMMENTS ON THIS
    MAGICAL YEAR …

    View full-size slide

  18. STARTING WITH AN
    OVERVIEW …

    View full-size slide

  19. AND THEN


    TRANSITIONING TO
    A ‘YEAR IN REVIEW’

    View full-size slide

  20. A ‘YEAR IN REVIEW’
    THROUGH THE LENS OF
    ELEVEN DATES THAT
    HELPED DEFINED THE
    PRESENT AND FUTURE
    FOR OUR FIELD…

    View full-size slide

  21. BUT FIRST —


    HOW DID WE GET HERE?

    View full-size slide

  22. ALL OF THIS
    GENERATIVE A.I.
    DID NOT COME OUT
    OF NOWHERE …

    View full-size slide

  23. MACHINES ARE
    INCREASINGLY
    IMPROVING IN
    THEIR ABILITY TO
    PROCESS AND
    ‘UNDERSTAND’
    LANGUAGE

    View full-size slide

  24. INITIAL PERFORMANCE AND HUMAN PERFORMANCE ARE
    NORMALIZED TO -1 AND 0 RESPECTIVELY (KIELA ET AL., 2021).

    View full-size slide

  25. AND LEADING
    MODELS ARE RAPIDLY
    GROWING IN SCALE

    View full-size slide

  26. SIZE OF LARGE LANGUAGE MODELS IN
    BILLIONS OF PARAMETERS

    View full-size slide

  27. THE IMPLICATION IS
    THAT PREVIOUSLY
    UNTOUCHABLE
    PROBLEMS WILL
    BEGIN TO COME
    WITHIN REACH …

    View full-size slide

  28. THIS IS MOSTLY
    A STORY OF
    BRUTE FORCE
    COMPUTING …

    View full-size slide

  29. Moore’s law
    !

    View full-size slide

  30. Kryder’s law
    !

    View full-size slide

  31. AND THAT
    BRUTE FORCE
    GETS APPLIED
    TO LANGUAGE
    AND PATTERNS
    IN LANGUAGE

    View full-size slide

  32. WHAT IS GPT ?


    WHAT IS CHATGPT ?


    WHAT IS GPT-4 ?

    View full-size slide

  33. GPT = GENERATIVE


    PRE-TRAINED


    TRANSFORMER
    (WE CAN TRY TO UNPACK ALL OF THIS IN A MINUTE)

    View full-size slide

  34. GPT =
    IT IS A


    LARGE LANGUAGE MODEL (LLM)


    BUT FAR FROM THE ONLY ONE
    GENERATIVE


    PRE-TRAINED


    TRANSFORMER

    View full-size slide

  35. XAVIER AMATRIAIN, TRANSFORMER MODELS: AN INTRODUCTION AND
    CATALOG, ARXIV:2302.07730 (2023) HTTPS://ARXIV.ORG/ABS/2302.07730
    THE LEADING LLM MODELS

    View full-size slide

  36. BETTER
    HARDWARE
    PARALLEL


    COMPUTING
    ATTENTION


    MECHANISM
    THE PATH TO MEGASCALE


    LARGE LANGUAGE MODELS


    HAS BEEN DRIVEN BY A MIXTURE OF

    View full-size slide

  37. THE SAME TECH THAT HAS
    POWERED HIGHLY IMMERSIVE
    GAMING HAS ALSO PUSHED
    SCIENCE FORWARD …
    https://www.thegamer.com/pc-games-best-intense-graphics/

    View full-size slide

  38. APRIL 5, 2023
    NOT COMPARED DIRECTLY WITH THE H100 BUT THIS SHOWS
    YOU THE NATURE OF THE COMPETITION TAKING PLACE
    https://www.cnbc.com/2023/04/05/google-
    reveals-its-newest-ai-supercomputer-
    claims-it-beats-nvidia-.html

    View full-size slide

  39. WOULD TAKE
    300 YEARS TO
    TRAIN EVEN
    GPT-3 ON A
    SINGLE GPU
    BUT WE
    TRANSFORMER


    ARCHITECTURE
    ALLOWS FOR
    SIGNIFICANT
    PARALLELIZATION

    View full-size slide

  40. TRAINED MODEL ON
    A BILLION PAIRS OF
    WORDS IN JUST 3.5
    DAYS ON EIGHT
    NVIDIA GPUS …
    THIS APPROACH
    HIGHLIGHTED A
    PATH TO THE
    PRESENT WITH
    EVER LARGER
    FUTURE
    https://blogs.nvidia.com/blog/2022/03/25/
    what-is-a-transformer-model/

    View full-size slide

  41. NOW PARALLEL COMPUTING IS
    BEING DONE ON A MEGASCALE

    View full-size slide

  42. SOME INPUTS TO GPT
    COMMON CRAWL
    WEBTEXT2 BOOKS1/2 WIKIPEDIA
    OPENAI LIKELY DID SOME SIGNIFICANT
    CLEANING / PREPROCESSING OF THESE
    SOURCES PRIOR TO TRAINING
    GPT-3 IS 175B PARAMETERS


    TRAINED WITH 499B TOKENS

    View full-size slide

  43. THE POINT HERE IS
    TO MAKE CLEAR
    THAT THERE ARE
    MANY STEPS AND
    MOVING PARTS …

    View full-size slide

  44. THERE ARE OTHER PLAYERS
    BESIDES OPENAI AND THEY
    WILL LIKELY PURSUE OTHER
    STEPS / MOVING PARTS …
    AND MANY MORE …

    View full-size slide

  45. NOW WHY IS ALL OF
    THIS INTERESTING HERE
    IN THE WORLD OF LAW?

    View full-size slide

  46. LANGUAGE IS THE


    ‘COIN OF THE REALM’
    HERE IN THE


    LEGAL WORLD

    View full-size slide

  47. VIRTUALLY ALL ROADS IN
    LAW LEAD TO A DOCUMENT
    (CONSUMPTION, PRODUCTION OR BOTH)

    View full-size slide

  48. IF MACHINES CAN
    ‘UNDERSTAND’ NOT ONLY
    GENERAL LANGUAGE BUT
    COMPLEX ‘LEGALESE’ …

    View full-size slide

  49. THEN THEY MIGHT BE
    ABLE TO REALLY HELP
    OUR FIELD …

    View full-size slide

  50. BUT HOW WELL CAN
    LEADING MODELS
    PERFORM ON COMPLEX
    PROFESSIONAL TASKS ?

    View full-size slide

  51. RIGHT
    BEFORE
    THE
    HOLIDAYS
    ROUGHLY DEC 22, 2022

    View full-size slide

  52. I CALLED MY FRIEND
    AND FREQUENT
    COLLABORATOR
    MIKE BOMMARITO

    View full-size slide

  53. AND SAID LET’S
    EVALUATE CHATGPT ON
    THE MULTIPLE CHOICE
    PART OF THE BAR EXAM

    View full-size slide

  54. AND IT DID
    PRETTY WELL …

    View full-size slide

  55. V1.01 - December 29, 2022 V2.01 - January 3, 2023

    View full-size slide

  56. LITTLE DID WE
    KNOW AT THE
    TIME THAT THIS
    PAPER PUT US
    ON A PATH …

    View full-size slide

  57. TO BE INVOLVED
    IN THE LAUNCH
    OF GPT-4*
    *OBVIOUSLY WE ARE JUST PLAYING A
    VERY SMALL ROLE IN THE BIG PICTURE
    (BUT WE WILL TAKE IT!)

    View full-size slide

  58. GPT-4 BAR EXAM
    https://openai.com/research/gpt-4
    March 14, 2023

    View full-size slide

  59. https://www.youtube.com/watch?v=4pGG79VdNmU

    View full-size slide

  60. CHATGPT BAR
    EXAM PASSES
    Paper Now Available on SSRN!
    March 15, 2023 - Version 1.01
    https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233

    View full-size slide

  61. AND IT CAN DO
    THE ESSAYS AS
    WELL …

    View full-size slide

  62. (GPT-3.0 was released in 2020)

    View full-size slide

  63. (GPT-3.0 was released in 2020)
    This ain’t passing any bar exam

    View full-size slide

  64. OBVIOUSLY THE BAR
    EXAM IS *NOT* THE
    ‘BE ALL END ALL’ BUT
    IT IS SOME MEASURE
    OF CAPABILITY

    View full-size slide

  65. BUT THE BROADER QUESTION
    IS HOW TO INDUSTRIALIZE
    THESE RAW CAPABILITIES
    WITHIN SPECIFIC LEGAL
    WORKSTREAMS / USE CASES

    View full-size slide

  66. OVER THE PAST YEAR …

    View full-size slide

  67. MANY (LEGAL)
    ORGANIZATIONS HAVE
    HAD TO CONSIDER HOW
    TO INCORPORATE AND/
    OR METABOLIZE THESE
    DEVELOPMENTS

    View full-size slide

  68. FROM ‘OUT OF THE BOX’
    RAW CAPABILITIES TO
    INDUSTRIAL APPLICATIONS

    View full-size slide

  69. OUR BAR EXAM ANALYSIS
    IS REALLY AN EVALUATION
    OF THE ‘OUT OF THE BOX’
    CAPABILITIES OF GPT-4
    (ALSO CALLED A ‘ZERO-SHOT’
    ANALYSIS OF THE CAPABILITIES)

    View full-size slide

  70. IN OTHER WORDS, WE ARE
    SHOWING THE BARE MINIMUM
    OF THE CAPABILITIES HERE

    View full-size slide

  71. ‘OUT OF THE BOX’
    ENTER
    PROMPT
    RECEIVE
    ANSWER
    (ZERO-SHOT)

    View full-size slide

  72. ‘AD HOC PROMPT
    ENGINEERING’
    ENTER
    PROMPT
    RECEIVE
    ANSWER
    ’PROMPT ENGINEERING’ IS ABOUT
    TUNING / REFINING PROMPTS TO
    OBTAIN BETTER ANSWERS
    (HALF-SHOT)

    View full-size slide

  73. MUCH MORE CAN BE
    DONE WITH ADDITIONAL
    ENGINEERING LAYERS
    BUILT ON TOP OF THESE
    RAW CAPABILITIES

    View full-size slide

  74. HERE ARE TEN
    EXAMPLES OF LLMs
    APPLICATIONS /
    USE CASES IN THE
    LEGAL SECTOR

    View full-size slide

  75. LOOK AT THESE
    EXAMPLES OF
    THE CURRENT
    ‘ART OF THE
    POSSIBLE’

    View full-size slide

  76. BECAUSE SOME OF
    THE APPROACHES /
    FRAMEWORKS CAN
    BE RETROFITTED TO
    OTHER PROBLEMS

    View full-size slide

  77. (1)


    DRAFT A BRIEF
    HERE DAMIEN RIEHL FROM VLEX
    WALKS THROUGH HOW TO BUILD
    MORE OF A BRIEF THROUGH
    SEQUENTIAL PROMPTING …
    HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT-
    LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/

    View full-size slide

  78. HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT-
    LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/

    View full-size slide

  79. CONTRACT NEGOTIATION
    (2)
    (AUTOMATIC REDLINE AGAINST A PLAYBOOK)

    View full-size slide

  80. (3)


    REGULATORY MONITORING

    View full-size slide

  81. MANY (ALL) ORGANIZATIONS
    MUST MONITOR THE CHANGING
    REGULATORY ENVIRONMENT

    View full-size slide

  82. PARTICULARLY ON A GLOBAL BASIS —
    THE IDENTIFICATION AND ROUTING
    OF ALL OF THIS WORK IS A GOOD
    CANDIDATE FOR MACHINES …

    View full-size slide

  83. THINK OF AN ‘AIR TRAFFIC
    CONTROL SYSTEM’ BUT FOR
    REGULATORY MANAGEMENT

    View full-size slide

  84. 1. IDENTIFY NEW REGULATIONS


    2. IDENTIFY ANY IMPACTED INTERNAL POLICIES /
    PROCEDURES


    3. IDENTIFY INDIVIDUAL / TEAM RESPONSIBLE
    FOR COMPLIANCE WITH NEW REGULATIONS


    4. ISSUE TICKET TO THE INDIVIDUAL / TEAM TO
    UPDATE POLICY AND COMPLIANCE PLAN
    APPROXIMATE WORKFLOW FOR
    REGULATORY MONITORING**
    **NOTE: EXACT INTERNAL PROCESS MAY DIFFER BY ORGANIZATION

    View full-size slide

  85. (4)


    PATENT DRAFTING

    View full-size slide

  86. (5)


    LEGAL RESEARCH

    View full-size slide

  87. THOSE ARE JUST A FEW MORE OF
    MANY OTHER POTENTIAL AREAS
    WHERE THIS CLASS OF TECH
    MIGHT BE USEFUL …
    AND MANY MANY OTHER EXAMPLES
    6. E-DISCOVERY


    7. DEALS DATABASES


    8. LITIGATION MGMT


    9. BUSINESS OF LAW


    10. COMPLEX FINANCIAL DOCS

    View full-size slide

  88. GOOD NEWS YOU ARE
    NOT GOING TO HAVE TO
    IMAGINE ALL OF THE
    POSSIBILITIES

    View full-size slide

  89. BECAUSE MORE THAN


    30+ COMPANIES ARE GOING TO
    PRESENT OVER THE NEXT FEW DAYS

    View full-size slide

  90. STARTUP SHOWCASE COMPANIES

    View full-size slide

  91. SCALE STAGE COMPANIES

    View full-size slide

  92. BUT TO SET THE STAGE
    FOR THE DAYS TO COME …

    View full-size slide

  93. ELEVEN DAYS THAT HELPED
    DEFINE THE PRESENT AND
    FUTURE FOR GEN (LEGAL) A.I.
    THE YEAR IN REVIEW

    View full-size slide

  94. FOR EACH OF THESE
    ELEVEN DATES, I WILL
    OFFER THOUGHTS ON
    THE IMPLICATIONS OF
    THESE DATES TO THE
    FUTURE OF (LEGAL) AI

    View full-size slide

  95. NOVEMBER 30, 2022
    TWO DATES HIGHLIGHTED SO FAR
    #1 OF 11 IMPLICATION
    GPT / LLMS GO
    MAINSTREAM

    View full-size slide

  96. IMPLICATION
    MARCH 15, 2023
    #2 OF 11
    GPT-4 IS A SERIOUS
    UPGRADE OVER
    PRIOR VERSIONS
    TWO DATES HIGHLIGHTED SO FAR

    View full-size slide

  97. MARCH 30, 2023
    THE RISE OF


    VERTICALIZED LLMS
    #3 OF 11

    View full-size slide

  98. FOLKS OFTEN ASK ME —


    WILL THERE BE A ‘LAW GPT’?
    PROBABLY SOON

    View full-size slide

  99. HERE IS PROBABLY THE
    CLOSEST EXAMPLE…
    USING ~300+ BILLION


    FINANCE TOKENS**

    View full-size slide

  100. https://www.bloomberg.com/company/press/
    bloomberggpt-50-billion-parameter-llm-tuned-
    fi
    nance/
    https://arxiv.org/pdf/2303.17564.pdf

    View full-size slide

  101. NOT ONLY COMPETITION AMONG
    FOUNDATIONAL MODELS
    IMPLICATION
    BUT ALSO VERTICALIZED GPTS
    (E.G. LAW GPTS)
    (GPT, CLAUDE , GEMINI / BARD, ETC.)

    View full-size slide

  102. APRIL 6, 2023
    INFO SEC, GEN AI POLICIES AND
    THE RISE PRIVATE INSTANCES
    #4 OF 11

    View full-size slide

  103. https://mashable.com/article/samsung-chatgpt-leak-details

    View full-size slide

  104. THE ROGUE ASSOCIATE
    (OR PARTNER) PASTING
    STUFF INTO GPT OR
    ANOTHER LLM MODEL
    LAW VERSION OF THE CONCERN

    View full-size slide

  105. FUD
    VS
    FEAR,


    UNCERTAINTY,


    & DOUBT
    FEAR OF
    MISSING
    OUT
    FOMO
    ORGANIZATIONS / INDIVIDUALS TOGGLE
    BETWEEN THESE TWO POINTS OF VIEW
    (SOMETIME WITHIN THE SAME CONVERSATION)

    View full-size slide

  106. ORGANIZATIONS LAUNCH PRIVATE
    INSTANCES OF GPT-4, ETC.

    View full-size slide

  107. https://legaltechtrends.substack.com/p/legal-tech-trends-newsletter-14
    ORGANIZATIONS LAUNCH PRIVATE
    INSTANCES OF GPT-4, ETC.

    View full-size slide

  108. FEAR / DOUBT IS ULTIMATELY
    GOING TO BE MANAGED AS THESE
    CAPABILITIES BECOME BAKED INTO
    ENTERPRISE GRADE OFFERINGS
    IMPLICATION

    View full-size slide

  109. MAY 27, 2023
    THE NEW YORK


    CHATGPT LAWYER
    #5 OF 11

    View full-size slide

  110. BECAUSE OF THE
    POTENTIAL FOR MODEL
    HALLUCINATION, ETC.

    View full-size slide

  111. IT IS IMPORTANT TO USE
    THESE TOOLS (AT LEAST FOR
    NOW) AS PART OF A HUMAN
    IN THE LOOP PROCESS

    View full-size slide

  112. ONE WAY TO DRAMATICALLY
    REDUCE THE LIKELIHOOD OF
    A HALLUCINATION IS TO
    ANCHOR OUTPUT AGAINST
    SOME ‘GROUND TRUTH’ …

    View full-size slide

  113. ONE SHOT
    ENHANCE THIS
    QUERY USING
    CONTEXTUALLY
    RELEVANT
    INFORMATION
    ORIGINAL
    QUERY
    OUTPUT*
    (RETRIEVAL AUGMENTATION)
    (COULD BE
    CONSTRUCTED ON VIA
    ‘PROMPT ENGINEERING’)
    ENTERED INTO A
    FOUNDATIONAL
    MODEL
    (VARIOUS TECHNIQUES ARE
    POSSIBLE SUCH AS USING
    EMBEDDINGS AND STORED
    IN A VECTOR DATABASE)
    ( QUERY &
    CONTEXTUAL
    INFO)
    REVISED


    QUERY
    **NOTE OUTPUT CAN BE
    FURTHER ENHANCED IN FEW
    SHOT EXAMPLES

    View full-size slide

  114. THERE IS A BROADER
    ENGINEERING ARC WHICH
    WE WILL SEE AS TOOLS /
    OFFERINGS MATURE

    View full-size slide

  115. LAYERS OF ENGINEERING ON TOP
    OF LLMS IS KEY TO TRULY UNLOCK
    COMPLEX VERTICALS LIKE LAW
    IMPLICATION

    View full-size slide

  116. JUNE 26, 2023
    CASETEXT &
    THOMSON REUTERS
    DEAL
    #6 OF 11

    View full-size slide

  117. https://techcrunch.com/2023/06/26/
    thomson-reuters-buys-casetext-an-ai-
    legal-tech-startup-for-650m-in-cash/

    View full-size slide

  118. EVERY STARTUP IN THIS ROOM
    OWES A DEBT OF GRATITUDE TO
    CASETEXT FOR SETTING A FAIRLY
    LARGE PUBLIC NUMBER

    View full-size slide

  119. MASS APPEAL
    PRODUCT
    FIN(LEGAL)TECH
    DOCUSIGN


    LEGALZOOM
    LIT FINANCE


    RISK MGMT
    BUT EVEN LARGER $$ ARE
    POSSIBLE FOR THIS SPACE

    View full-size slide

  120. MORE BROADLY, I THINK MANY
    LEGACY LEGALTECH PROVIDERS
    ARE GOING TO NEED HELP ON
    THEIR GEN AI TRANSITION

    View full-size slide

  121. EVENTUALLY SOME WILL GIVE
    UP ON THEIR HALF BAKED
    INTERNAL EFFORTS AND SEEK
    TO INNOVATE VIA ACQUISITION

    View full-size slide

  122. MORE M&A


    TRANSACTIONS ARE
    COMING IN 2024
    IMPLICATION

    View full-size slide

  123. JULY 18, 2023
    LLAMA 2 RELEASE
    #7 OF 11

    View full-size slide

  124. WILL OPEN SOURCE MODEL
    ULTIMATELY MATCH OR
    EXCEED PERFORMANCE OF
    LEADING MODELS ?

    View full-size slide

  125. THIS REMAINS TO BE SEEN …

    View full-size slide

  126. OPEN SOURCE LLMS ARE IN A
    SERIOUS HORSE RACE WITH
    OPENAI, ETC.
    IMPLICATION
    SIGNIFICANT IMPLICATIONS
    FOR INVESTMENT,
    ENGINEERING, ETC.

    View full-size slide

  127. SEPTEMBER 15, 2023
    GPT USE IN KNOWLEDGE WORK


    THE AGE OF CENTAUR CHESS
    #8 OF 11

    View full-size slide

  128. BIG PICTURE HERE IS A
    BRAND NEW STUDY WHICH
    HIGHLIGHTS BOTH THE
    POTENTIAL PRODUCTIVITY
    AND QUALITY GAINS IN
    PROFESSIONAL SERVICES

    View full-size slide

  129. “Our study conducted with Boston Consulting Group, a global management consulting
    fi
    rm,
    we examine the performance implications of AI on realistic, complex, and knowledge-
    intensive tasks. …
    [The] experiment involved 758 consultants … subjects were randomly assigned to one of
    three conditions: no AI access, GPT-4 AI access, or GPT-4 AI access with a prompt
    engineering overview.
    For each one of a set of 18 realistic consulting tasks within the frontier of AI capabilities,
    consultants using AI were signi
    fi
    cantly more productive (they completed 12.2% more tasks on
    average, and completed tasks 25.1% more quickly), and produced signi
    fi
    cantly higher quality
    results (more than 40% higher quality compared to a control group).”
    https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321

    View full-size slide

  130. CENTAUR CHESS
    HUMAN


    +


    MACHINE
    HUMAN
    OR
    MACHINE
    >

    View full-size slide

  131. AND REMEMBER THIS STUDY ONLY
    CONSIDERS GPT-4 WITH PROMPTING
    IMPLICATION
    GREATER PERFORMANCE WILL
    FOLLOW WHEN ENGINEERING
    ENHANCEMENTS ARE ADDED
    SIGNIFICANT QUALITY AND
    EFFICIENCY RESULTS

    View full-size slide

  132. NOVEMBER 6, 2023
    OPEN AI DEV DAY


    AGENTS, AGENTS, AGENTS
    #9 OF 11

    View full-size slide

  133. GENERATIVE AGENTS

    View full-size slide

  134. WE HAVE SEEN
    QUITE A BIT OF
    THIS SINCE THE
    LAUNCH OF
    CHATGPT

    View full-size slide

  135. WHAT IS THE PRODUCT VALUE ADD
    OVER BASELINE CAPABILITIES?
    TRUST BUT VERIFY
    THERE IS GOING TO BE
    SIGNIFICANT PRESSURE ON
    LEGAL TECH COMPANIES TO
    OVERSTATE THE ACTUAL
    CONTRIBUTION OF THEIR
    OFFERING ABOVE JUST AN
    LLM API CALL
    I SUGGEST YOU


    TRUST BUT VERIFY


    ALL CLAIMS

    View full-size slide

  136. IMPLICATION
    MORE INNOVATION


    MORE DISRUPTION


    MORE NEED FOR ADAPTATION
    AND THIN PRODUCTS


    BUILT ON GPT WILL LIKELY
    NEED TO PIVOT …

    View full-size slide

  137. NOVEMBER 17, 2023
    SUPPLY CHAIN RISK


    FOR THE WORLD OF GEN AI
    #10 OF 11

    View full-size slide

  138. NOVEMBER 17 2023
    https://www.nytimes.com/2023/11/17/technology/openai-sam-altman-ousted.html

    View full-size slide

  139. NOVEMBER 22, 2023
    https://www.nytimes.com/2023/11/22/technology/openai-sam-altman-returns.html

    View full-size slide

  140. THIS AMONG OTHER THINGS
    HIGHLIGHTS SUPPLY CHAIN
    RISK IN AI …

    View full-size slide

  141. BUT ALSO A MORE
    FUNDAMENTAL TRUTH …

    View full-size slide

  142. K-LLMS
    WHERE K IS SOME
    NUMBER GREATER
    THAN ONE
    THIS WAS ALWAYS GOING TO
    BE ABOUT MIXING AND
    MATCHING MODELS
    (FANCY WAY TO
    SAY MIX AND
    MATCH MODELS)

    View full-size slide

  143. IMPLICATION
    COMPANIES NEED TO
    BE ABLE TO ADAPT AS
    TECH LANDSCAPE
    RAPIDLY EVOLVES
    COMPANIES WITH A SINGLE
    SOURCE LLM STRATEGY ARE
    AT REAL RISK

    View full-size slide

  144. FINAL THOUGHTS

    View full-size slide

  145. ELEVEN DAYS THAT
    HELPED DEFINE THE
    PRESENT AND FUTURE
    FOR GEN (LEGAL) A.I.

    View full-size slide

  146. SO FAR I HAVE MENTIONED
    TEN OF THE DATES THAT
    WILL HELP DEFINE THE
    FUTURE FOR OUR FIELD…

    View full-size slide

  147. APRIL 6, 2023
    JUNE 26, 2023
    JULY 18, 2023
    NOVEMBER 6, 2023
    NOVEMBER 17, 2023
    MARCH 30, 2023
    SEPTEMBER 15, 2023
    MAY 27, 2023
    SAM ALTMAN FIRED
    OPEN AI DEV DAY
    LLAMA 2 RELEASE
    HBS STUDY RELEASE
    CASETEXT + TR DEAL
    NY CHATGPT LAWYER
    BLOOMBERG GPT RELEASE
    SAMSUNG CODE INCIDENT
    SUMMARY OF TEN KEY DATES FOR THE YEAR
    MARCH 15, 2023 GPT-4 RELEASE DAY
    NOVEMBER 30, 2022 CHATGPT RELEASE DAY

    View full-size slide

  148. HIGHLIGHTS AI SUPPLY CHAIN RISKS
    OPEN SOURCE MODELS MIGHT WIN
    BIG GAINS IN PROFESSIONAL SERVICES
    MORE M&A ACTIVITY TO FOLLOW
    NEED LAYERS ON TOP BASE MODELS
    BLOOMBERG GPT
    SAMSUNG INCIDENT
    GPT-4 RELEASE
    11.30.22 CHATGPT RELEASE GPT GOES MAINSTREAM
    3.15.23
    3.30.23 RISE OF VERTICALIZED LLMS
    GPT-4 IS A SERIOUS UPGRADE
    4.06.23 ENTERPRISE DEPLOYMENT NEEDED
    SAM ALTMAN FIRED
    OPEN AI DEV DAY
    LLAMA 2 RELEASE
    HBS STUDY RELEASE
    CASETEXT + TR DEAL
    NY CHATGPT LAWYER
    TEN DAYS AND THEIR IMPLICATIONS
    5.27.23
    6.26.23
    7.18.23
    9.15.23
    11.6.23
    11.17.23
    AGENTS - MORE INNOVATION COMING

    View full-size slide

  149. BUT I THINK YOU
    ALREADY KNOW THE
    ELEVENTH DAY THAT
    IS GOING TO DEFINE
    THE FUTURE …

    View full-size slide

  150. DECEMBER 4, 2023
    BECAUSE THAT DAY IS TODAY
    #11 OF 11

    View full-size slide

  151. WHAT AN AMAZING
    COLLECTION OF
    INDIVIDUALS AND
    ORGANIZATIONS …

    View full-size slide

  152. THIS IS INDEED
    GOING TO BE THE
    ‘ROOM WHERE IT
    HAPPENS’ …

    View full-size slide

  153. BECAUSE ALL OF YOU
    ARE GOING TO MAKE
    IT HAPPEN …

    View full-size slide

  154. NO DOUBT THIS HAS BEEN
    THE MOST EXCITING YEAR
    IN TECHNOLOGY IN A
    VERY LONG TIME

    View full-size slide

  155. HERE WE GO
    BUT THE BEST IS
    YET TO COME !
    !

    View full-size slide

  156. 273VENTURES.COM
    KELVIN.LEGAL

    View full-size slide

  157. https://www.linkedin.com/in/daniel-katz-3b001539/

    View full-size slide

  158. G E N E R AT I V E A . I . + L AW
    @ computational
    professor daniel martin katz danielmartinkatz.com
    Illinois tech - chicago kent law 273Ventures.com
    Y E A R I N R E V I E W I N C LU D I N G G P T PA S S E S T H E B A R


    & E L E V E N DAY S T H AT D E F I N E D T H E Y E A R I N ( L E G A L ) A I
    https://bit.ly/3yxhY2e

    View full-size slide