Generative A.I. + Law - Background, Applications and Use Cases Including GPT-4 Passes the Bar Exam

Professor Daniel Martin Katz - Generative A.I. + Law - Background, Applications and Use Cases Including GPT-4 Passes the Bar Exam - Topics include some History of NLP, The Path to Generative A.I. (including ChatGPT and GPT-4), Some Applications / Use Cases in the Legal Sector, GPT-4 Passes the Bar Exam, Legal A.I. Operational Issues and the Path Forward ... (Updated 06.14.23)

Transcript

1. G E N E R AT I V E A . I . + L AW @ computational professor daniel martin katz danielmartinkatz.com Illinois tech - chicago kent law 273Ventures.com B AC KG R O U N D, A P P L I CAT I O N S A N D U S E CA S E S I N C LU D I N G G P T- 4 PA S S E S T H E B A R E X A M https://bit.ly/3yxhY2e
2. NOVEMBER 30, 2022
3. I THINK THIS DAY WILL GO DOWN AS A VERY IMPORTANT DAY IN THE HISTORY OF TECHNOLOGY
4. NOVEMBER 30, 2022 IS THE DAY THAT CHATGPT WAS FIRST RELEASED BECAUSE
5. THIS WAS THE BEGINNING OF THE BROADER PUBLIC’S AWARENESS …
6. OF WHAT WAS ALREADY TRUE …
7. WHICH IS THAT IN THE BACKGROUND
8. THERE HAVE BEEN FUNDAMENTAL IMPROVEMENTS OCCURING WITHIN THE WORLD OF NATURAL LANGUAGE PROCESSING (NLP)
10. I PERSONALLY HAVE BEEN WORKING IN THIS AREA FOR A WHILE …
11. THINKING ABOUT HOW TO USE NLP MODELS IN THE LEGAL DOMAIN
13. https://sites.google.com/view/nllp/nllp-2019
14. https://law.mit.edu/pub/openedgar/release/1
15. https://aclanthology.org/2022.acl-long.297.pdf
16. https://speakerdeck.com/danielkatz/legal-nlp-breaking-the-legal-language-barrier
17. BUT IT HAS BEEN HARD TO COMMUNICATE TO MANY FOLKS IN THE LEGAL SECTOR …
18. ABOUT THE NATURE OF THE LANDSCAPE SHIFT IN THE FIELD …
19. MAYBE IT IS ROBOLAWYER FATIUGE …?
21. MAYBE IT IS THE ABSENCE OF A CLEAR DEMONSTRATION PROJECT …
23. SO NOW WE ARE JUST A FEW MONTHS INTO WHAT IS ARGUABLY THE MOST SUCCESSFUL PRODUCT LAUNCH IN HISTORY …
25. TODAY — I WOULD LIKE TO DISCUSS WHAT ALL OF THESE DEVELOPMENTS …
26. MIGHT MEAN FOR THE DELIVERY OF LEGAL SERVICES …
28. SO FOR MY OPENING STATEMENT PLEASE TAKE INTO ACCOUNT THE FOLLOWING FOUR CONSIDERATIONS…
30. < CONSIDERATION 1 > LANGUAGE IS THE ‘COIN OF THE REALM’ HERE IN THE WORLD OF LAW < CONSIDERATION 2 > THE MARCH OF LEGAL COMPLEXITY < CONSIDERATION 3 > MACHINES ARE INCREASINGLY IMPROVING IN LANGUAGE PROCESSING < CONSIDERATION 4 > LEGAL LANGUAGE != REGULAR LANGUAGE ?
32. < CONSIDERATION 1 >
33. LANGUAGE IS THE ‘COIN OF THE REALM’ HERE IN THE WORLD OF LAW
35. VIRTUALLY ALL ROADS IN LAW LEAD TO A DOCUMENT (CONSUMPTION, PRODUCTION OR BOTH)
37. LAWYERS, JUDGES AND REGULATORS ARE MASSIVE PRODUCERS OF TEXT
38. BRIEFS, MEMOS, STATUTES, OPINIONS, REGULATIONS, CONTRACTS, ETC.
39. ARE JUST SOME OF THE LEGAL WORK PRODUCT …
40. PRODUCED ON A DAILY BASIS ACROSS THE WORLD’S VARIOUS LEGAL SYSTEMS
42. NOT ONLY IS IT THE SHEER VOLUME OF TEXT BUT ALSO THESE MASSIVE VOLUMES OF TEXT ARE NOTORIOUSLY COMPLEX
44. < CONSIDERATION 2 >
45. THE MARCH OF LEGAL COMPLEXITY
46. W H E T H E R YO U A R E A L A R G E M U LT I N AT I O N A L CO R P O R AT I O N , A S M A L L B U S I N E S S O R A N I N D I V I D UA L C I T I Z E N …
47. L AW H A S A CO M P L E X I TY C H A L L E N G E …
48. IF YOU TAKE A STEP BACK
49. AND LOOK AT THE UNIT ECONOMICS OF LAW
50. THE REAL ANIMATING DRIVER IN THE DEMAND FOR UNITS OF LEGAL PRODUCTION
51. IS COMPLEXITY
52. SOCIAL, ECONOMIC AND POLITICAL COMPLEXITY
53. MANIFESTS ITSELF IN OUR DOMAIN AS LEGAL COMPLEXITY
54. WHILE IT IS SOMEWHAT HARD TO PRECISELY DEFINE …
56. VIRTUALLY EVERY WAY YOU MIGHT CONSIDER IT LEGAL COMPLEXITY HAS GROWN
57. DANIEL MARTIN KATZ, CORINNA COUPETTE, JANIS BECKEDORF & DIRK HARTUNG, COMPLEX SOCIETIES AND THE GROWTH OF THE LAW, 10 SCIENTIFIC REPORTS 18737 (2020) CORINNA COUPETTE, JANIS BECKEDORF, DIRK HARTUNG, MICHAEL BOMMARITO, & DANIEL MARTIN KATZ, MEASURING LAW OVER TIME: A NETWORK ANALYTICAL FRAMEWORK WITH AN APPLICATION TO STATUTES AND REGULATIONS IN THE UNITED STATES AND GERMANY, FRONT. PHYS. (2021 FORTHCOMING)
58. SIGNIFICANT GROWTH IN LAWS AND REGULATIONS
59. KEY TAKE AWAY - OVER PAST TWO DECADES ~50%+ MORE STATUTES ~200% MORE REGULATIONS
60. SO THE QUESTION IS HOW TO MATCH THAT COMPLEXITY WITH THE APPROPRIATE
61. COMPLEXITY MITIGATION TACTICS …
62. MIXTURE OF PEOPLE, PROCESS AND TECHNOLOGY
63. DESIGNING MORE SCALABLE SYSTEMS
64. http://prawfsblawg.blogs.com/prawfsblawg/2017/03/-complexity-mitigation-strategies- for-law-law-land-and-beyond-and-some-other-thoughts-on-had fi eld-su.html#more SOME THOUGHTS ON LEGAL COMPLEXITY MITIGATION STRATEGIES
66. < CONSIDERATION 3 >
67. MACHINES ARE INCREASINGLY IMPROVING IN THEIR ABILITY TO PROCESS AND ‘UNDERSTAND’ LANGUAGE
68. INITIAL PERFORMANCE AND HUMAN PERFORMANCE ARE NORMALIZED TO -1 AND 0 RESPECTIVELY (KIELA ET AL., 2021).
69. AND LEADING MODELS ARE RAPIDLY GROWING IN SCALE
70. SIZE OF LARGE LANGUAGE MODELS IN BILLIONS OF PARAMETERS
71. THE IMPLICATION IS THAT PREVIOUSLY UNTOUCHABLE PROBLEMS WILL BEGIN TO COME WITHIN REACH …
73. BECAUSE COMPLEXITY IS ARGUABLY SCALING FASTER THAN AVAILABLE RESOURCES …
74. WE ARGUABLY NEED A FORCE MULTIPLIER …
75. AND LANGUAGE IS CORE TO MOST (ALL) LEGAL TASKS
76. THE FACT THAT LANGUAGE TECHNOLOGY HAS BEEN MATERIALLY IMPROVING MIGHT BODE VERY WELL …
77. AS THE FORCE MULTIPLIER, PRODUCTIVITY ENHANCER …
78. THE FIELD ARGUABLY NEEDS …
79. TO SUPPORT THE QUANTITY, QUALITY, AND ACCESSIBILITY OF LEGAL SERVICES DEMANDED BY SOCIETY
81. < CONSIDERATION 4 >
82. SO WHAT ABOUT LEGAL LANGUAGE ?
83. LETS BE CLEAR — FOR AI / NLP TO MAKE A DEEP INCURSION INTO THIS FIELD …
84. IT NEEDS TO BE ABLE TO MORE FUNDAMENTALLY ENGAGE WITH LANGUAGE
85. AND NOT JUST ANY LANGUAGE …
86. BUT LEGAL LANGUAGE
87. THE PUBLIC HAS A DIM VIEW OF LEGAL LANGUAGE …
88. THEY DESCRIBE LEGAL DOCUMENTS AND ARGUMENTS USING TERMS SUCH AS ‘LEGALESE’ ‘LEGAL JARGON’ ‘LEGAL GOBBLEDYGOOK’
90. LEGAL ENGLISH != ENGLISH (AND THIS IS TRUE FOR MANY OTHER LANGUAGES)
91. INDEED MANY LAW STUDENTS CONSIDER THEIR INITIAL FORAY INTO THE FIELD …
92. AS AN EXERCISE IN ‘LEARNING A NEW LANGUAGE’
93. IN SUPPORT OF THIS TASK …
94. LAW FEATURES SPECIALIZED DICTIONARIES
95. TEXT BASED SUMMARIES
96. AND MANY OTHER RESOURCES DESIGNED TO SUPPORT …
97. THE DEVELOPMENT OF THE LINGUISTIC IMMERSION PROGRAM THAT WE CALL #LEGALEDUCATION
99. NLP MODELS HAVE BEEN DEPLOYED IN THE LEGAL SECTOR …
100. BUT THE QUALITY AND PERFORMANCE HAS BEEN MIXED …
101. FRANKLY *NO* LEGAL TECH SOLUTION BUILT TO DATE HAS BEEN ABLE TO REALLY WORK WELL WITH NUANCES OF LEGAL LANGUAGE …
102. BUT THAT MIGHT BE ABOUT TO CHANGE …
104. TODAY I WILL DIVIDE THE REMAINING PRESENTATION INTO FIVE PARTS
105. (1) THE (LONG) PATH TO GENERATIVE A.I. (2) WHAT IS GPT / CHATGPT / GPT-4 ? (3) BAR EXAM AS WINDOW INTO CAPABILITIES (4) LLMs IN THE DELIVERY OF LEGAL SERVICES (5) STRATEGIC CONSIDERATIONS AND THE ROAD AHEAD PRESENTATION IN FIVE PARTS
107. (1) THE (LONG) PATH TO GENERATIVE A.I.
108. IN ORDER TO UNDERSTAND WHERE THIS MIGHT ALL BE GOING …
109. IT IS USEFUL TO UNDERSTAND THE ARC THAT HAS BROUGHT US TO THIS MOMENT …
110. SO LETS GO BACK TO THE EARLY DAYS …
111. BUT BEFORE WE TALK ABOUT ANYTHING TOO FANCY …
112. I WOULD LIKE TO REMIND EVERYONE …
113. BEFORE THERE WERE COMPUTERS
114. HUMANS ARE THE ORIGINAL COMPUTERS
115. HUMANS ONCE DID ALL OF THE COMPUTING
116. OVER THE COURSE OF MANY YEARS WE WERE ABLE TO LEVERAGE ALTERNATIVE FORMS OF COMPUTATION …
118. THE FIELD OF ARTIFICIAL INTELLIGENCE IS ROUGHLY 75 YEARS OLD …
119. “BEFORE 1949, ‘COMPUTERS LACKED A KEY PREREQUISITE FOR INTELLIGENCE: THEY COULDN’T STORE COMMANDS, ONLY EXECUTE THEM …” http://sitn.hms.harvard.edu/ fl ash/2017/history-arti fi cial-intelligence/
120. SO WHAT IS ‘ARTIFICIAL INTELLIGENCE’ ?
123. BIG IDEA IN AI IS TO DEVELOP IN MACHINES SOME LEVEL OF SYNTHETIC (OR ARTIFICIAL) REPRESENTATION OF A PREVIOUSLY HUMAN CENTERED PROCESS
124. ARTIFICIAL INTELLIGENCE IS A BROAD FIELD
126. EVEN IN THE FOUNDATIONAL DAYS THERE WAS A VIEW ABOUT THE ROLE OF A ‘CONVERSATIONAL AGENT’ AS A MEANS TO EVALUATE THE QUALITY OF AN AI SYSTEM
127. WHICH IS TO SAY THAT LANGUAGE HAS ALWAYS BEEN A CORE TOPIC FOR THE FIELD OF A.I.
128. ALAN TURING IS ARGUABLY THE FATHER OF THE FIELD OF COMPUTER SCIENCE
129. AND THIS IDEA OF CONVERSATIONAL AGENT COMES FROM HIM …
130. NOTE THE MOVIE WAS MOSTLY ABOUT CRACKING ENIGMA BUT HERE IS WHERE IT GETS ITS TITLE THE TURING TEST
132. IT IS IMPORTANT TO REMEMBER WITH TOPICS SUCH AS ARTIFICIAL INTELLIGENCE AND NATURAL LANGUAGE PROCESSING …
133. THE FIELD OF A.I. HAS HAD MANY FALSE STARTS AND A.I. WINTERS
134. EVEN SOME OF THE WORLD’S LEADING EXPERTS HAVE GOTTEN THINGS WRONG …
135. “MACHINES WILL BE CAPABLE, WITHIN TWENTY YEARS, OF DOING ANY WORK THAT A MAN CAN DO.” – Herbert Simon in 1965
136. “MACHINES WILL BE CAPABLE, WITHIN TWENTY YEARS, OF DOING ANY WORK THAT A MAN CAN DO.” – Herbert Simon in 1965 did not pan out by 1985 but did turn out okay for him …
137. SO I MENTION THIS ONLY SO THAT WE GROUND OURSELVES A BIT …
139. A.I. BROADLY HAS MADE MAJOR PROGRESS ON MANY TOPICS …
141. MACHINE LEARNING
143. MACHINE LEARNING HAS BEEN USED TO UNDERTAKE VARIOUS FORMS OF PREDICTION IN LAW
144. EXAMPLE LITIGATION PREDICTION
145. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698 Katz DM, Bommarito MJ II, Blackman J (2017), A General Approach for Predicting the Behavior of the Supreme Court of the United States. PLoS ONE 12(4): e0174698.
146. http://www.sciencemag.org/news/ 2017/05/arti fi cial-intelligence-prevails- predicting-supreme-court-decisions Professor Katz noted that in the long term … “We believe the blend of experts, crowds, and algorithms is the secret sauce for the whole thing.” May 2nd 2017
147. https://www.wsj.com/articles/hidden- in-plain-sight-a-powerful-way-to- beat-the-market-1497367597 FT Big Read Feb 7, 2019
148. http://arxiv.org/abs/1508.05751 available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
150. LITIGATION FUNDING
154. “Lawyers say the real value in mediation and arbitration might in the future come from large-scale data analysis of arbitrators and mediators themselves, in an effort to predict outcomes and potentially affect the course of settlements … Matthew Saunders, partner at Ashurst, notes that data analytics “could be extended to predicting which way arbitrators or a mediator might go”. Such technology may yet be some way off. In mediation, “a skilled facilitator helps the parties to explore where common ground can be found as the basis for an amicable settlement,” says James Freeman, arbitration partner at Allen & Overy. “The mediation process”, he adds, “is inherently a human one”.
155. DATA —> VARIABLES MODELS (STATISTICAL) OUTPUTS WITH VALIDATION
156. MUCH MORE COULD BE SAID ON THE LEGAL PREDICTION SIDE OF THIS …
158. TODAY I AM MOSTLY GOING TO FOCUS ON GENERATIVE LANGUAGE MODELS …
159. A.I. IS A VERY BROAD FIELD
160. AND OF COURSE THERE ARE OTHER GENERATIVE MODELS MY SON AND I RECENTLY CREATED ‘BROCCOLI MAN’ GENERATIVE A.I. ART
162. HISTORICALLY LANGUAGE HAS ALWAYS LAGGED OTHER ADVANCES IN A.I. …
163. BECAUSE OF THE UNDERLYING STATISTICAL DEPENDENCIES - LANGUAGE IS A *HARD* PROBLEM …
164. YET MAJOR ADVANCES ARE FINALLY COMING TO MARKET …
166. HISTORICALLY NLP (LIKE THE REST OF A.I.) WAS RULES-BASED
167. RULES-BASED A.I. WAS ACTUALLY THE ONLY OPTION AT ONE POINT
168. Moore’s law !
169. Kryder’s law !
170. EVENTUALLY COMPUTER CHIPS WERE FAST ENOUGH AND DATA STORAGE WAS CHEAP ENOUGH …
171. THAT NLP COULD REALLY EMBRACE STATISTICAL MODELING …
172. NLP SLOWLY BECOMES A STATISTICAL FIELD IN THE 1990’S AND 2000’S
173. MARK LIBERMAN, "THE TREND TOWARDS STATISTICAL MODELS IN NATURAL LANGUAGE PROCESSING." NATURAL LANGUAGE AND SPEECH: SYMPOSIUM PROCEEDINGS BRUSSELS, SPRINGER, (1991)
174. BUT THE PRIMARY EFFORT HERE WAS ON SYNTAX BASED APPROACHES …
175. “There have been a series of clever approaches to backdoor into semantics* … (*while also being scalable) Semantic Methods (Fairly Difficult) Syntax Methods (Fairly Easy) Historically, Big Divide between Semantics and Syntax Quasi-Semantic Methods
177. SYNTAX
178. WORDS WORD FREQUENCY PART OF SPEECH FREQUENCY ETC.
179. CTRL + F IS EXACT ‘STRING’ MATCHING
180. REGULAR EXPRESSION (REGEX) RULES-BASED METHOD(S) THAT CAN BE USED TO LOOK FOR WORD PATTERNS AND RETURN RESULTS FOR THOSE PATTERNS
181. TF - IDF TERM FREQUENCY INVERSE DOCUMENT FREQUENCY EXPLOITS THE FREQUENCY OF WORDS IN DOCUMENTS IN ORDER TO PROFILE THEM
183. SEMANTICS
184. SEMANTICS IS ABOUT THE MEANING OF INDIVIDUAL WORDS SEMANTICS IS RELATIONSHIP BETWEEN WORDS THAT INTERACT TO PRODUCE HIGHER ORDER MEANING
185. LOTS OF THE TRAINING FOR LAWYERS IS ACTUALLY ABOUT THE DEEP SEMANTIC INTERPRETATION OF LANGUAGE CONTRACTS STATUTES REGULATIONS JUDICIAL DECISIONS EXAMPLES —>
186. SEMANTICS IS HARD FOR MACHINES
188. BUT THE PAST DECADE THERE HAS BEGUN TO BE PROGRESS ON THIS FRONT …
189. WE HAVE SEEN THE EMERGENCE OF QUASI-SEMANTIC METHODS
190. QUASI-SEMANTIC METHODS WHICH CAN INCREASINGLY TAKE CONTEXT INTO ACCOUNT
191. WHAT WE SEE TODAY ARE FOLKS USING DEEP NEURAL NETWORKS TO WORK ON NLP PROBLEMS
192. GENERATIVE A.I. IN THE LINGUISTIC CONTEXT IS A MASH UP OF DEEP LEARNING + NLP
193. CLASSIC VS NEURAL NLP
195. BTW WE SEE THIS SAME PROGRESSION IN LEGAL NLP AS WELL …
196. HERE WE ANALYZE TRENDS WITHIN EVERY LEGAL NLP PAPER (MORE THAN 600+ PAPERS) WRITTEN OVER THE PAST DECADE
197. GROWTH IN NUMBER OF PAPERS AND LANGUAGES
198. THE NEURAL AGENDA HAS COME TO LEGAL NLP
199. ALMOST NO TEXT GENERATION OR MACHINE SUMMARIZATION (BUT NOW THAT IS VERY LIKELY TO CHANGE)
201. (2) WHAT IS GPT ? WHAT IS CHATGPT ? WHAT IS GPT-4 ?
202. GPT = GENERATIVE PRE-TRAINED TRANSFORMER (WE CAN TRY TO UNPACK ALL OF THIS IN A MINUTE)
203. GPT = IT IS A LARGE LANGUAGE MODEL (LLM) BUT FAR FROM THE ONLY ONE GENERATIVE PRE-TRAINED TRANSFORMER
204. TODAY I WILL MOSTLY LIMIT MY COMMENTS TO GPT BUT JUST WANTED TO FLAG THERE ARE OTHER MODELS OUT THERE
205. XAVIER AMATRIAIN, TRANSFORMER MODELS: AN INTRODUCTION AND CATALOG, ARXIV:2302.07730 (2023) HTTPS://ARXIV.ORG/ABS/2302.07730 THE LEADING LLM MODELS
206. THE LEADING LLM MODELS XAVIER AMATRIAIN, TRANSFORMER MODELS: AN INTRODUCTION AND CATALOG, ARXIV:2302.07730 (2023) HTTPS://ARXIV.ORG/ABS/2302.07730
208. THE OPEN AI GPT MODELS COMBINE SEVERAL IDEAS TOGETHER FROM THE VARIOUS PAPERS IN THIS PROGRESSION …
209. Word2Vec (2013)
212. ELMO (2018)
213. BERT (2019)
221. BTW IT SHOULD BE NOTED THAT …
222. CHATGPT / GPT-3.5 / GPT-4 ARE PROPRIETARY MODELS SO WE DO NOT KNOW FOR CERTAIN ALL OF THE DEEP DETAILS
224. WE WILL NOT GET INTO THE SUPER TECHNICAL DETAILS TODAY …
225. FOR A VERY APPROACHABLE TREATMENT GO HERE https://writings.stephenwolfram.com/2023/02/ what-is-chatgpt-doing-and-why-does-it-work/
226. I MET STEPHEN WOLFRAM’S ROBOT PRESENCE IN HOUSTON IN 2011 HE APPEARED AT A COMPUTATIONAL LAW CONFERENCE VIA ROBOT (A REAL BOSS MOVE) STEPHEN AND YOURS TRULY TALKING ABOUT RULE 90 IT WAS A ‘BIG BANG’ MOMENT
228. BETTER HARDWARE PARALLEL COMPUTING ATTENTION MECHANISM THE PATH TO MEGASCALE LARGE LANGUAGE MODELS HAS BEEN DRIVEN BY A MIXTURE OF
229. THE SAME TECH THAT HAS POWERED HIGHLY IMMERSIVE GAMING HAS ALSO PUSHED SCIENCE FORWARD … https://www.thegamer.com/pc-games-best-intense-graphics/
230. APRIL 5, 2023 NOT COMPARED DIRECTLY WITH THE H100 BUT THIS SHOWS YOU THE NATURE OF THE COMPETITION TAKING PLACE https://www.cnbc.com/2023/04/05/google- reveals-its-newest-ai-supercomputer- claims-it-beats-nvidia-.html
231. WOULD TAKE 300 YEARS TO TRAIN EVEN GPT-3 ON A SINGLE GPU BUT WE TRANSFORMER ARCHITECTURE ALLOWS FOR SIGNIFICANT PARALLELIZATION
232. TRAINED MODEL ON A BILLION PAIRS OF WORDS IN JUST 3.5 DAYS ON EIGHT NVIDIA GPUS … THIS APPROACH HIGHLIGHTED A PATH TO THE PRESENT WITH EVER LARGER FUTURE https://blogs.nvidia.com/blog/2022/03/25/ what-is-a-transformer-model/
233. NOW PARALLEL COMPUTING IS BEING DONE ON A MEGASCALE
235. GPT-3 (RELEASED IN 2020) IS THIRD GENERATION OF THE GPT FAMILY
236. GPT-3 (RELEASED IN 2020) IS THIRD GENERATION OF THE GPT FAMILY GPT-3.5 IS AN INTERMEDIATE IMPROVEMENT ON ORIGINAL GPT-3
237. GPT-3 (RELEASED IN 2020) IS THIRD GENERATION OF THE GPT FAMILY GPT-4 IS THE MOST RECENT RELEASE AND IT IS LIKELY TO BE A ‘FAMILY OF MODELS’ GPT-3.5 IS AN INTERMEDIATE IMPROVEMENT ON ORIGINAL GPT-3
238. JUST LIKE GPT-3.5 IS NOW A FAMILY OF MODELS
239. SOME INPUTS TO GPT COMMON CRAWL WEBTEXT2 BOOKS1/2 WIKIPEDIA OPENAI LIKELY DID SOME SIGNIFICANT CLEANING / PREPROCESSING OF THESE SOURCES PRIOR TO TRAINING GPT-3 IS 175B PARAMETERS TRAINED WITH 499B TOKENS
240. TRANSFORMER ARCHITECTURE NEURAL NETWORK MODEL SCALE REINFORCEMENT LEARNING PRETRAINING JUST SOME KEY TERMINOLOGY YOU SHOULD LEARN CONTEXT WINDOW ATTENTION MECHANISM MODEL TUNING GRADIENT DESCENT
242. TAKE ORIGINAL LLM GPT-3 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent- Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 CHATGPT IS AN OFFSHOOT OF GPT-3
243. COMBINE SPECIFIC TRAINING ON INSTRUCTIONS TAKE ORIGINAL LLM GPT-3 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent- Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 CHATGPT IS AN OFFSHOOT OF GPT-3
244. FURTHER TUNE ON INSTRUCTIONS COMBINE SPECIFIC TRAINING ON INSTRUCTIONS TAKE ORIGINAL LLM GPT-3 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent- Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 CHATGPT IS AN OFFSHOOT OF GPT-3
245. FURTHER TUNE ON INSTRUCTIONS SUPPORT WITH SPECIFIC REINFORCEMENT LEARNING FEEDBACK LOOP COMBINE SPECIFIC TRAINING ON INSTRUCTIONS TAKE ORIGINAL LLM GPT-3 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent- Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 CHATGPT IS AN OFFSHOOT OF GPT-3
246. THE GPT-4 TECHNICAL REPORT OFFERS SOME BUT NOT MOST OF THE KEY DETAILS
247. RLHF WAS CLEARLY PART OF THE BREW HERE BUT GENERALLY DOES NOT SEEM TO BE MOVING THE NEEDLE ALL THAT MUCH AT THIS POINT
248. SO THE POINT HERE IS TO MAKE CLEAR THAT THERE ARE MANY STEPS AND MOVING PARTS …
249. THERE ARE OTHER PLAYERS BESIDES OPENAI AND THEY WILL LIKELY PURSUE OTHER STEPS / MOVING PARTS … AND MANY MORE …
251. (3) BAR EXAM AS WINDOW INTO INCREASED CAPABILITIES
252. AGAIN WE HAD BEEN TRYING TO DISCUSS / HIGHLIGHT THE INCREASING CAPABILITIES OF LANGUAGE MODELS FOR SOME TIME
253. WE THOUGHT THAT SOME SORT OF A PUBLIC DEMONSTRATION WOULD HELP MAKE THE POINT …
254. REMEMBER THIS IS A TASK THAT MANY WOULD THINK IS IMPOSSIBLE (LAST YEAR I WOULD HAVE SAID IT WOULD NOT OCCUR FOR MANY YEARS)
255. SOME TASKS LAWYERS REGULARLY UNDERTAKE ARE ACTUALLY WAY *EASIER* THAN THE BAR EXAM (AND OTHERS ARE HARDER) LET’S BE CLEAR …
256. TO HAVE A CHANCE ON THE BAR EXAM …
257. EXAMINEE MUST POSSESS A THRESHOLD AMOUNT OF LEGAL KNOWLEDGE AND READING COMPREHENSION SKILLS AND SEMANTIC AND SYNTACTIC COMMAND OF THE ENGLISH LANGUAGE
259. RIGHT BEFORE THE HOLIDAYS ROUGHLY DEC 22, 2022
260. I CALLED MY FRIEND AND FREQUENT COLLABORATOR MIKE BOMMARITO
261. AND SAID LET’S EVALUATE GPT-3.5 (TEXT-DAVINCI-003) ON THE MULTIPLE CHOICE PART OF THE BAR EXAM
262. AND IT DID PRETTY WELL …
263. V1.01 - December 29, 2022 V2.01 - January 3, 2023
266. LITTLE DID WE KNOW AT THE TIME THAT THIS PAPER PUT US ON A PATH …
267. TO BE INVOLVED IN THE LAUNCH OF GPT-4* *OBVIOUSLY WE ARE JUST PLAYING A VERY SMALL ROLE IN THE BIG PICTURE (BUT WE WILL TAKE IT!)
268. GPT-4 BAR EXAM https://openai.com/research/gpt-4 March 14, 2023
269. CHATGPT BAR EXAM PASSES Paper Now Available on SSRN! March 15, 2023 - Version 1.01 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233
272. https://www.law.com/legaltechnews/ 2023/03/17/how-gpt-4-mastered-the- entire-bar-exam-and-why-that-matters/ abajournal.com/web/article/latest- version-of-chatgpt-aces-the-bar-exam- with-score-in-90th-percentile
273. https://www.lawnext.com/ 2023/03/gpt-takes-the-bar-exam- again-this-time-it-score-among- top-10-of-test-takers.html
274. https://www.youtube.com/watch?v=4pGG79VdNmU
276. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233
277. < THE UBE > UNIFORM BAR EXAM
280. < THE MBE >
281. MULTISTATE BAR EXAM (MBE) SUBJECTS TESTED TORTS CONTRACTS EVIDENCE REAL PROPERTY CIVIL PROCEDURE CONSTITUTIONAL LAW CRIMINAL LAW AND PROCEDURE
286. < THE MEE >
287. MULTISTATE ESSAY EXAMINATION (MEE) SUBJECTS TESTED TORTS EVIDENCE CONTRACTS FAMILY LAW REAL PROPERTY CONFLICT OF LAWS TRUSTS AND ESTATES CONSTITUTIONAL LAW BUSINESS ASSOCIATIONS FEDERAL CIVIL PROCEDURE UNIFORM COMMERCIAL CODE CRIMINAL LAW AND PROCEDURE
288. MULTISTATE ESSAY EXAMINATION (MEE) https://github.com/mjbommar/gpt4-passes-the-bar/blob/main/data/MEE.md
289. ‘REPRESENTATIVE GOOD ANSWERS’ https://mdcourts.gov/sites/default/ fi les/import/ble/ examanswers/2022/202207uberepgoodanswers.pdf SEVERAL STATE BARS RELEASE REPRESENTATIVE GOOD ANSWERS THESE ARE VERY HELPFUL FOR EVALUATION PURPOSES AS THEY ARE ACTUAL ANSWERS WHICH ARE ABOVE MERELY PASSING ANSWERS (BUT NOT NECESSARILY PERFECT)
291. (GPT-3.0 was released in 2020)
292. (GPT-3.0 was released in 2020) This ain’t passing any bar exam
295. < THE MPT >
296. MULTISTATE PERFORMANCE TEST (MPT) SKILLS TESTED EXAMINEE MUST COMPLETE A PRACTICAL LAWYERING TASK SUCH AS LEGAL ANALYSIS, FACT ANALYSIS, PROBLEM SOLVING, ORGANIZATION AND MANAGEMENT OF INFORMATION, AND CLIENT COMMUNICATION
297. MULTISTATE PERFORMANCE EXAM (MPT) 10-15 PAGES OF MATERIALS THE FILE = THE FACTS THE LIBRARY = THE LAW ~5000 TOKEN INPUTS (ACCESS HERE) BIT.LY/40F3FQ2
300. OVERALL PERFORMANCE
301. SOME KEY SHORTCOMINGS OF GPT-4 ON THE BAR EXAM FAILED TO PROPERLY CALCULATE THE DISTRIBUTION OF ASSETS FROM A TESTAMENTARY TRUST WHICH HAS BEEN DEEMED TO BE INVALID PROVIDED AN INCORRECT ANSWER ON A CIVIL PROCEDURE QUESTION REGARDING DIVERSITY JURISDICTION AFTER THE JOINDER OF A NECESSARY PARTY PROVIDED IMPROPER ANALYSIS ON A REAL PROPERTY (REAL ESTATE) SUBQUESTION REGARDING BOTH THE PROPER DESIGNATION OF A FUTURE INTEREST AND THE APPLICATION OF THE RULE AGAINST PERPETUITIES
304. https://github.com/mjbommar/gpt4-passes-the-bar ACCESS THE FULL OUTPUT AND OTHER SUPPORTING MATERIALS HERE
306. OUR BAR EXAM ANALYSIS IS WHAT IS CALLED A ‘ZERO SHOT’ ANALYSIS
307. ZERO SHOT ANALYSIS REFLECTS THE FLOOR AND NOT THE CEILING OF CURRENT CAPABILITIES
309. ZERO SHOT ENTER PROMPT RECEIVE ANSWER ’PROMPT ENGINEERING’ IS ABOUT TUNING / REFINING PROMPTS TO OBTAIN BETTER ANSWERS
311. THIS ZERO SHOT AND NOT EVEN THE LATEST MODEL (I.E. NOT GPT-4)
312. *NOT* A VERY SOPHISTICATED TAKE ON THIS SITUATION BUT REFLECTIVE OF THE MODAL PERSPECTIVE OF LAWYERS / LAW PROFS NOTE: ALL COMMERICAL RESEARCH TOOLS ALREADY HAVE ‘A.I.’ IN THEM
314. ONE WAY TO DRAMATICALLY REDUCE THE LIKELIHOOD OF A HALLUCINATION IS TO MOVE OUT OF THE ‘ZERO SHOT’ PARADIGM
315. AND ANCHOR INITIAL OUTPUT AGAINST SOME ‘GROUND TRUTH’ …
316. ONE SHOT GET RESULT QUERY THAT RESULT AGAINST SOMETHING ELSE REFINE RESULT ENTER PROMPT OUTPUT FINAL ANSWER (RETRIEVAL AUGMENTATION) (IF NEEDED) EXAMPLES
318. THE GENERALIZATION OF ALL OF THIS IS AN ORCHESTRATION LAYER TO BRING MULTIPLE STREAMS / TECHNOLOGIES TOGETHER
319. FEW SHOT (CHAIN OF THOUGHT PROMPTING)
320. HERE DAMIEN RIEHL FROM FASTCASE WILL WALK YOU THROUGH HOW TO BUILD MORE OF A BRIEF THROUGH SEQUENTIAL PROMPTING … HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT- LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/
321. HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT- LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/
322. FEW SHOT (LANGCHAIN / AUTOGPT PROMPTING)
323. FEW SHOT (LEGAL DOMAIN) (LANGCHAIN / AUTOGPT STYLE ORCHESTRATION LAYER) https://docs.kelvin.legal/docs/examples/due-dilligence/ https://docs.kelvin.legal/docs/examples/litigation-automation/
325. THERE ARE LOTS OF OPPORTUNITIES OUTSIDE THE ZERO-SHOT CONTEXT
326. OR EVEN OTHER PLUGINS … FOR EXAMPLE WOLFRAM COULD HELP SOLVE FOR ISSUES WITH QUANTITATIVE REASONING
327. https://arxiv.org/pdf/2303.17651.pdf https://selfre fi ne.info/ https://github.com/madaan/self-re fi ne
328. GENERATIVE AGENTS
331. (4) LLMs IN THE DELIVERY OF LEGAL SERVICES
332. IN GENERAL …
333. DRAFTING + EDITING COMMUNICATIONS EMAILS REPORTS PRESENTATIONS MARKETING MATERIALS
334. DRAFTING + EDITING LEGAL DOCUMENTS CONTRACTS BRIEFS MEMOS INTERROGATORIES DEMAND LETTERS
335. CONDUCTING LEGAL RESEARCH EXTERNAL POINTS OF LAW INTERNAL KNOWLEDGE MGMT
336. SUMMARIZING LARGE BODIES OF TEXTUAL MATERIAL SUMMARIZING LEGAL DOCUMENTS
337. AND REMEMBER WE CAN COMBINE OUTPUTS WITH OTHER MACHINE LEARNING MODELS, ETC.
338. OR EVEN OTHER PLUGINS … FOR EXAMPLE WOLFRAM COULD HELP SOLVE FOR ISSUES WITH QUANTITATIVE REASONING
340. BUT LETS LOOK AT SEVERAL CONCRETE EXAMPLES …
341. (1) DRAFT A MNDA
344. (2) DRAFT A REPLY BRIEF HERE DAMIEN RIEHL FROM FASTCASE WILL WALK YOU THROUGH HOW TO BUILD MORE OF A BRIEF THROUGH SEQUENTIAL PROMPTING … HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT- LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/
346. (3) DRAFT REPORT OR PRESENTATION
347. https://youtu.be/S7xTBa93TX8
349. DUE DILIGENCE / M&A SUPPORT (4)
351. https://docs.kelvin.legal/docs/examples/due-dilligence/
352. https://docs.kelvin.legal/docs/examples/due-dilligence/ DUE DILIGENCE /M&A SUPPORT
353. https://docs.kelvin.legal/docs/examples/due-dilligence/ DUE DILIGENCE /M&A SUPPORT
354. https://docs.kelvin.legal/docs/examples/due-dilligence/
356. LITIGATION SUPPORT & TRIAGE (5)
359. (6) BUILD A LEGAL KNOWLEDGE GRAPH https://www.ibm.com/topics/knowledge-graph
360. (6) BUILD A LEGAL KNOWLEDGE GRAPH HTTPS://TAX-GRAPH.273VENTURES.COM/
361. (6) BUILD A LEGAL KNOWLEDGE GRAPH
362. (6) BUILD A LEGAL KNOWLEDGE GRAPH
364. (7) LANGUAGE MODELS ARE ZERO-SHOT TAGGERS
365. https://www.linkedin.com/posts/daniel- katz-3b001539_feedbackfriday-legaltech- legaldata-activity-7045070829428113409-fIZz
367. https://www.law.com/ legaltechnews/2023/03/20/sali- harnesses-most-advanced-gpt- models-to-help-legal-industry- speak-the-same-data-language/ March 23, 2023 Michael Bommarito - Gentleman Farmer
368. https://sali-search.kelvin.legal
370. (8) REGULATORY MONITORING
371. https://www.linkedin.com/posts/ bommarito_summarize-everything-published-in- the-federal-activity-7048621468422725632-CjuO April 3, 2023
373. (9) SUBSTANTIVE LEGAL ANALYSIS (COPYRIGHT)
374. HTTPS://WWW.YOUTUBE.COM/WATCH?V=NQZCRHR8YPU&T=11S
376. HERE ARE JUST A FEW MORE OF MANY OTHER POTENTIAL AREAS WHERE THIS CLASS OF TECH MIGHT BE USEFUL … E-DISCOVERY LEGAL BILLING DEALS DATABASES CONTRACT DRAFTING AND MANY MANY OTHER EXAMPLES
377. HERE A JUST A COUPLE OF RESOURCES THAT HAVE BEEN MADE AVAILABLE (THERE WILL BE MANY MORE) https://ssrn.com/abstract=4404017
378. https://thebrainyacts.beehiiv.com/
380. NOW A QUICK WORD OF CAUTION
381. BECAUSE OF THE POTENTIAL FOR MODEL HALLUCINATION, ETC.
382. IT IS IMPORTANT TO USE THESE TOOLS (AT LEAST FOR NOW) AS PART OF A HUMAN IN THE LOOP PROCESS
383. IN OTHER WORDS, FOR MOST USE CASES HUMANS SHOULD ALWAYS BE PART OF THE ‘RETRIEVAL AUGMENTATION’ LAYER
384. AND OF COURSE OR MACHINES CAN AID IN RETRIEVAL AUGMENTATION / QUALITY ASSURANCE
385. CENTAUR CHESS HUMAN + MACHINE HUMAN OR MACHINE >
387. (5) STRATEGIC CONSIDERATIONS AND THE ROAD AHEAD
388. FUD VS FEAR, UNCERTAINTY, & DOUBT FEAR OF MISSING OUT FOMO
389. FUD FEAR, UNCERTAINTY, & DOUBT HALLUCINATIONS CONFIDENTIALITY INFORMATION SECURITY
390. AMAZING CAPABILITIES KEEPING UP WITH THE MARKET NOT WANTING TO LOOK OUT OF STEP TO CLIENTS FEAR OF MISSING OUT FOMO
391. FUD VS FEAR, UNCERTAINTY, & DOUBT FEAR OF MISSING OUT FOMO ORGANIZATIONS / INDIVIDUALS TOGGLE BETWEEN THESE TWO POINTS OF VIEW (SOMETIME WITHIN THE SAME CONVERSATION)
393. SEVERAL EARLY TOOLS IN THE MARKET …
395. MAY 4, 2023 https://www.lexisnexis.com/en- us/products/lexis-plus-ai.page
396. MAY 23, 2023 https://www.lawnext.com/2023/05/ thomson-reuters-previews-its-plans-for- generative-ai-announces-integration-with- microsoft-365-copilot.html
399. OBVIOUSLY WE HAVE A COMPANY AND WE WORKED WITH CASETEXT ON THE BAR PAPER — SO PERHAPS WE ARE NOT 100% NEUTRAL … SHANG GAO PABLO ARREDONDO
400. BUT OF COURSE THESE WILL NOT BE THE LAST TOOLS AND IT REMAINS TO BE SEEN HOW IT WILL ALL SHAKE OUT …
401. I THINK IT IS CLEAR THAT THIS IS GOING TO BE A LONG RACE …
403. WHAT IS THE DATA AND TECHNOLOGY STRATEGY THAT ORGANIZATIONS CAN EMPLOY IN LIGHT OF THESE CHANGE IN THE TECHNOLOGY LANDSCAPE ?
404. BUILD VS ASSEMBLE VS BUY BUILD IS LIKELY OFF THE TABLE FOR THE SHORT TO MEDIUM TERM ASSEMBLE HOWEVER IS A VIABLE OPTION SO CAREFUL PROCUREMENT IS GOING TO BE THE STRATEGIC PATH
405. DATA STRATEGY HOW TO COLLECT / REGULARIZE DATA FOR USE INSIDE THESE SYSTEMS OR AS A LAYER ON TOP OF THESE SYSTEMS
407. SEVERAL OTHER STRATEGIC CONSIDERATIONS WHAT IS THE ROLE OF DOMAIN SPECIFIC TRAINING? HOW COULD I CUSTOMIZE THESE MODELS FOR USE WITHIN MY OWN ORGANIZATION ? WHICH MODELS / LLMS DO I LEVERAGE ? HOW DO I CHOOSE ? CAN I MIX AND MATCH ? WHAT IS THE DIFFERENTIAL IMPACT OF THESE MODELS BY LEGAL ORGANIZATION ? HOW DO I THINK ABOUT QUESTIONS OF PRIVACY AND INFORMATION SECURITY ? COPYRIGHT?
408. WHAT IS THE DIFFERENTIAL IMPACT OF THESE MODELS BY LEGAL ORGANIZATION ? (1)
409. I THINK THAT THERE ARE GOING TO BE DIFFERENTIAL IMPACTS BY TYPE OF LEGAL ORGANIZATION
410. MANAGERS SHOULD REORGANIZE JOBS SO ORGANIZATION CAN FURTHER CONCENTRATE TASKS WHICH CAN BE ENABLED BY A.I. MICRO LEVEL
411. LOOK TO PLACES WHERE THERE IS A CONCENTRATION OF LOW TO MEDIUM COMPLEXITY TASKS MACRO LEVEL ALSP, LPO, LEGAL OPS, SPECIALIZE LAW FIRMS, ETC.
412. TRAINING FOLKS TO USE THESE TOOLS TO THE MAXIMUM EXTENT POSSIBLE HUMAN CAPITAL (THIS WILL BE AN IMPORANT LIMITATION ON ORGANIZATIONAL SUCCESS)
414. HOW SHOULD I THINK ABOUT QUESTIONS OF PRIVACY AND INFORMATION SECURITY ? COPYRIGHT? (2)
415. SHOULD / CAN I PUT CONFIDENTIAL CLIENT DATA IN THESE SYSTEMS ? MANY FOLKS HAVE ASKED ME —
416. PROCEED WITH CAUTION — I WOULD TELL YOU TO AUDIT PRECISELY WHAT IS HAPPENING AND WHERE IT IS HAPPENING SHOULD / CAN I PUT CONFIDENTIAL CLIENT DATA IN THESE SYSTEMS ? (CLIENT CONSENT + AUDITED SAFEGUARDS) MANY FOLKS HAVE ASKED ME —
417. THE ROGUE ASSOCIATE (OR PARTNER) PASTING STUFF INTO GPT OR ANOTHER LLM MODEL ONE VERSION OF THE CONCERN
418. https://mashable.com/article/samsung-chatgpt-leak-details
419. WE HAVE BEEN WORKING WITH A FEW FIRMS ON HOW TO NAVIGATE THE THORNY INFO SEC AND LEGAL ETHICS ISSUES
420. IF NOT, DOES VENDOR HAVE SOC2 / ISO2700 ? HOW MANY OTHER PARTIES ARE IN THE FLOW OF YOUR INFORMATION ? CONTRACTUAL REPRESENTATIONS FROM THOSE 3RD PARTIES? IS THE SOLUTION AVAILABLE ON PREM ? SOME VENDOR QUESTIONS WHAT CONTROLS DOES THE VENDOR APPLY TO THE TRANSITING OF YOUR DATA ?
421. LEVERAGING THESE TOOLS ON INTERNAL DATA IS REALLY THE HOLY GRAIL …. INTERNAL DATA WE ARE GOING TO GET THERE BUT IT REQUIRES A REAL THOUGHT OUT APPROACH HERE
422. WHAT IS GOING TO HAPPEN WITH COPYRIGHT ? MANY FOLKS HAVE ASKED ME —
423. WHAT IS GOING TO HAPPEN WITH COPYRIGHT ? I THINK THIS IS GOING TO LEAD TO A MAJOR DECISION IN COPYRIGHT AND/OR ACTION BY CONGRESS, ETC. MANY FOLKS HAVE ASKED ME —
424. https://arxiv.org/pdf/2303.15715.pdf THIS IS A VERY GOOD PAPER ON THE TOPIC
426. WHAT IS THE ROLE OF DOMAIN SPECIFIC TRAINING? (3)
427. WILL A GENERAL MODEL BEAT A DOMAIN SPECIFIC MODEL ? SURE THAT IS POSSIBLE / LIKELY — IF THE GENERAL MODEL IS LARGE AND DOMAIN MODEL IS SMALL
428. AGAIN HERE IS AN EXAMPLE WITH ACTUAL RESULTS ATTACHED TO IT …
429. https://www.bloomberg.com/company/press/ bloomberggpt-50-billion-parameter-llm-tuned- fi nance/ https://arxiv.org/pdf/2303.17564.pdf
430. https://twitter.com/rasbt/status/1642880757566676992
431. SO HOW COULD THIS BE IMPROVED EVEN FURTHER ?
432. SOURCES OF LLM MODEL IMPROVEMENT USE DIFFERENT TRAINING DATA NEURAL NET ACCESS / IMPLEMENTATION INSTRUCTION MODULE FINE TUNING RETRIEVAL AUGMENTATION (RAG) PROMPT ENGINEERING CHAIN OF THOUGHT PROMPTING RLHF AGENT KNOWLEDGE GRAPH TRAVERSAL MODEL CREATOR MODEL USER (REINFORCEMENT LEARNING FROM HUMAN FEEDBACK)
433. SO WILL THERE BE A ‘LAW GPT’? PROBABLY SOON IT IS AN OPEN QUESTION WILL IT BE BETTER THAN GPT-4 ON LAW TASKS?
435. HOW COULD I CUSTOMIZE THESE MODELS FOR USE WITHIN MY OWN ORGANIZATION ? (4)
436. HOW IS ALL OF YOUR DATA GOING TO GET INSIDE OF THESE SYSTEMS … ?
437. WE ARE LIKELY ON A PATH TO INTERNAL GPT STYLE OFFERINGS
438. FOLKS ARE EVENTUALLY GOING BUILD / TRAIN / TUNE THEIR OWN GPT STYLE MODELS…
439. TO DO SO FOLKS ARE GOING TO NEED TO STRUCTURE / PREPROCESS THEIR DATA
440. https://nationalmagazine.ca/en-ca/articles/legal-market/legal-tech/2023/chatting-with-gpt
441. CONNECTING THE THREE LAYERS https://nationalmagazine.ca/en-ca/articles/ legal-market/legal-tech/2023/chatting-with-gpt
443. WHICH MODELS / LLMS DO I LEVERAGE ? HOW DO I CHOOSE ? CAN I MIX AND MATCH ? (5)
444. SO WE HAVE SEEN QUITE A BIT OF THIS SINCE THE LAUNCH OF CHATGPT NOW
445. THIS IS NOT INHERENTLY BAD BUT I THINK ORGANIZATIONS NEED TO THINK ABOUT THEIR PRECISE STRATEGY HERE NOW
446. SOURCES OF LLM MODEL IMPROVEMENT USE DIFFERENT TRAINING DATA NEURAL NET ACCESS / IMPLEMENTATION (ALPACA, ETC.) INSTRUCTION MODULE FINE TUNING RETRIEVAL AUGMENTATION (RAG) PROMPT ENGINEERING CHAIN OF THOUGHT PROMPTING (LANGCHAIN, ETC.) RLHF (E.G. BLOOMBERG GPT) (DOLLY, ETC.) (PLUGINS / UI / GROUNDING) MODEL CREATOR MODEL USER (VARIOUS FORMS) (VARIOUS FORMS) (REINFORCEMENT LEARNING FROM HUMAN FEEDBACK) (AGENTS BUILD / TRAVERSE GRAPHS) (MANY APPROACHES) AGENT KNOWLEDGE GRAPH TRAVERSAL
447. REMEMBER YOU ARE GOING TO HAVE DIRECT ACCESS …
448. SO WHAT IS THE VENDOR VALUE ADD OVER BASE LLM MODEL? TRUST BUT VERIFY THERE IS GOING TO BE SIGNIFICANT PRESSURE ON VENDORS TO OVERSTATE THE ACTUAL CONTRIBUTION OF THEIR OFFERING ABOVE JUST AN LLM API CALL I SUGGEST YOU TRUST BUT VERIFY ALL CLAIMS
449. CAN YOU DETERMINE WHETHER A VENDOR IS SIMPLY SELLING YOU A UI/ UX WRAPPER ON TOP OF AN LLM ? AGAIN THIS MIGHT BE OKAY BUT DO NOT PAY AN EXCESSIVE PREMIUM …
450. IT WOULD BE A BIG MISTAKE TO GO ALL IN ON ANY VENDOR OR SPECIFIC LLM AT THIS POINT PLACE A SMALL TO MEDIUM BET
451. AS THE YEAR PROGRESSES, I BELIEVE THIS WILL *NOT* JUST BE A GPT STORY …
452. IT IS A HIGHLY COMPETITVE LANDSCAPE AND OTHER FOLKS WILL BE ENTERING THE MARKET WITH ADDITIONAL OFFERINGS
453. THERE ARE OTHER PLAYERS BESIDES OPENAI AND THEY WILL LIKELY PURSUE OTHER STEPS / MOVING PARTS … AND MANY MORE …
454. REMEMBER THAT GOOGLE ACTUALLY INVENTED MUCH OF THE TECH THAT BROUGHT YOU GPT https://www.forbes.com/sites/richardnieva/ 2023/02/08/google-openai-chatgpt-microsoft- bing-ai/ https://www.nytimes.com/2023/01/20/ technology/google-chatgpt-arti fi cial- intelligence.html
455. CAN YOU BOTH RAPIDLY AND RIGOROUSLY EVALUATE THE QUALITY OF MODELS OR FIGURE OUT HOW TO MIX AND MATCH THEM … ? (ENSEMBLE LEARNING ANYONE?)
457. FINAL THOUGHTS
458. THIS IS SOMETHING OF A DIFFERENT FLAVOR FROM THE AVERAGE TECHNOLOGICAL INNOVATION …
460. NOTE THIS IS A CHATGPT PAPER NOT EVEN CONSIDERING POTENTIAL ADDITIONAL GAINS IN GPT-4 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4375283 “TIME TAKEN DECREASES BY 0.8 SDS AND OUTPUT QUALITY RISES BY 0.4 SDS.”
461. OVER 50% PRODUCTIVITY GAINS FROM GPT-3.5 VERSION OF COPILOT (USED FOR PROGRAMMING) https://arxiv.org/pdf/2302.06590 COPILOT X RELEASED MARCH 23RD 2023 https://github.blog/2023-03-22-github-copilot- x-the-ai-powered-developer-experience/ FEBRUARY 13 2023
462. https://arxiv.org/abs/2303.10130 “OUR FINDINGS REVEAL THAT AROUND 80% OF THE U.S. WORKFORCE COULD HAVE AT LEAST 10% OF THEIR WORK TASKS AFFECTED BY THE INTRODUCTION OF LLMS, WHILE APPROXIMATELY 19% OF WORKERS MAY SEE AT LEAST 50% OF THEIR TASKS IMPACTED. WE DO NOT MAKE PREDICTIONS ABOUT THE DEVELOPMENT OR ADOPTION TIMELINE OF SUCH LLMS”
464. TASKS VS JOBS WHAT IS BEING MADE POSSIBLE? WE RACE WITH THE MACHINE LABOR MARKET IMPLICATIONS (IT IS COMPLICATED) (IN A WORLD WHERE TEXT GENERATION IS FAR CHEAPER)
466. OVERALL, THERE SEEMS TO BE NON-LINEAR PERFORMANCE PERHAPS AKIN TO BROADER LANGUAGE ACQUISTION ?
467. IS THERE IS SOMETHING MORE FUNDAMENTAL GOING ON EMERGENT BEHAVIOR ?
468. CUT+PASTE SAFETY FIRST THIS IS ALL KINDA BACK TO THE FUTURE FOR ME
469. https://research.google/pubs/pub52065/
470. https://futurism.com/gpt-4-sparks-of-agi https://www.youtube.com/watch?v=qbIk7-JPB2c
471. THIS IS AN ACTIVE DEBATE IN THE ACADEMIC LITERATURE I WOULD SAY THE JURY IS STILL OUT
472. AND WHILE THERE ARE LIMITS AND THINGS DO NOT GROW EXPONENTIALLY FOREVER
473. I THINK THIS IS GOING TO BE THE MOST EXCITING YEAR IN TECHNOLOGY IN A VERY LONG TIME !
477. 273VENTURES.COM KELVIN.LEGAL
478. https://www.linkedin.com/in/daniel-katz-3b001539/
479. @COMPUTATIONAL
480. G E N E R AT I V E A . I . + L AW @ computational professor daniel martin katz danielmartinkatz.com Illinois tech - chicago kent law 273Ventures.com B AC KG R O U N D, A P P L I CAT I O N S A N D U S E CA S E S I N C LU D I N G G P T- 4 PA S S E S T H E B A R E X A M https://bit.ly/3yxhY2e