Share
Embed
Transcript
11 slides
-
എܠ--.ධՁͷඞཁੑ σʔλ४උ Ϟσϧ܇࿅ʢֶशʣ ςετσʔλͰධՁ ਫ਼ɾ࠶ݱͳͲࢉग़ ίʔυͰ࠶ݱՄೳ ϓϩϯϓτઃܭ --.Ͱਪ ࠾ ίʔυͰ࠶ݱՄೳʁ
ػցֶशϞσϧͷධՁ --.ͷධՁ ධՁ͖ͭ͢ͷϙΠϯτ ✓ͲͷϞσϧ͕ߴੑೳ͔ͩͬͨ ✓ͳͥͦͷ݁ʹࢸͬͨͷ͔ ✓खॱͱաఔ͕ͤΔ͔ w $IBU(15Ͱճࢼͯ͠ʮ͍͍ײͩ͡ͳʯ w ʮ(15͕ݡ͍ʯͱ͍͏ӟ͚ͩͰϞσϧબ w ͨ·ͨ·ޭͨ͠ϓϩϯϓτͰʮ༏लʯͱஅ -
6SZV 4 &WBMVBUJOH-BSHF-BOHVBHF.PEFMTGPS*6$/3FE-JTU4QFDJFT*OGPSNBUJPOBS9JW w *6$/ઈ໓ةዧछධՁͷࣄྫ w ੜଟ༷ੑอશͷͰ--.ͷ׆༻͕ظ͞Ε͍ͯΔ͕ɺ ઐతஅʹ͓͚Δ৴པੑʹ͕ٙΔɻ w
ʢݱߦͷ--.ʹڞ௨ͨ͠ʣͭͷॏେͳ՝ w ࣝͱਪͷΪϟοϓ ˠࣄ࣮͍ͬͯΔ͕ɺͦΕΛԠ༻ͨ͠அࠔ w ࡏ͢ΔόΠΞε ˠಈʢਓؾछʣʹڧ͘ɺແಈʹऑ͍ എܠ--.ධՁͷඞཁੑ https://arxiv.org/abs/2510.02830 ٬؍త͔ͭݫີͳධՁϑϨʔϜϫʔΫ͕ෆՄܽ ਖ਼ղͷဃ ྨֶతࣝ อશঢ়گͷਪ 94.9% 27.2% -
w Φʔϓϯιʔεಁ໌ੑͷߴ͍࣮ w ҆શੑࢤ҆શੑͱ৴པੑΛ࠷ॏཁࢹ w ࠶ݱੑ࠶ݱՄೳͳՊֶతݕূ w ॊೈੑͱ֦ுੑ0QFO"* (PPHMF "OUISPQJD
Y"* ϩʔΧϧڥʢ0MMBNBʣɺଟ༷ͳϞσϧΛ ϕϯμʔϩοΫΠϯͳ͠ͰධՁɻ ӳࠃ"*҆શݚڀॴ͕ओಋ ධՁϑϨʔϜϫʔΫʮ*OTQFDU"*ʯ https://inspect.aisi.org.uk/ ++"MMBJSF 34UVEJPઃऀ ͕ ϓϩδΣΫτΛϦʔυ -
ධՁͷϞδϡʔϧԽ5BTL %BUBTFU 4PMWFS 4DPSFS ධՁϩδοΫΛίʔυͱͯ͠ମܥతʹཧɺ࠶ར༻͕ՄೳͱͳΔ 5BTL࣮ݧܭը %BUBTFUೖྗσʔλ 4PMWFSճઓུ 4DPSFSධՁج४ ධՁʹ༻͢Δೖྗσʔλͱ
ਖ਼ղϥϕϧͷηοτ ϓϩϯϓτΤϯδχΞϦϯάͳͲɺ Ϟσϧ͔ΒճΛҾ͖ग़ͨ͢Ίͷઓུ ධՁશମͷϫʔΫϑϩʔΛఆٛ Ϟσϧͷग़ྗΛਖ਼ղͱൺֱ͠ɺ είΞΛࢉग़͢ΔͨΊͷධՁج४ Task( dataset=..., solver=chain(...), scorer=..., ) -
*OTQFDU"*ʹΑΔ*6$/ධՁλεΫͷ࣮ 6SZV ͷͭͷλεΫͷద༻ྫ λεΫ త ༻ͨ͠4PMWFS4DPSFSͷྫ ྨֶతྨ ϨουϦετΧςΰϦධՁ ཧత
ڴҖͷಛఆ ਖ਼͍͠ྨ܈Λબͤ͞Δ ͭͷΧςΰϦ͔ΒͭΛಛఆ ࠃ໊ͷϦετΛੜ ͷڴҖΧςΰϦ͔ΒෳΛબ https://github.com/uribo/iucn-redlist-evals chain(), optimize_choices()*, system_message(), multiple_choice_with_cache()*, taxon_partial_scorer()* system_message(), generate(), match() system_message(), generate(), geo_distribution_scorer()* system_message(), generate(), threat_assessment_scorer()* -
*OTQFDU"*ʹΑΔ*6$/ධՁλεΫͷ࣮ 6SZV ͷͭͷλεΫͷద༻ྫ *OQVU 5BSHFU Aquila chrysaetos https://github.com/uribo/iucn-redlist-evals
1IPUP3PDLZ $$#:IUUQTDSFBUJWFDPNNPOTPSHMJDFOTFTCZ WJB8JLJNFEJB$PNNPOT B LC $IPJDFT A. Animalia (Kingdom) > Chordata (Phylum) > Aves (Class) > Accipitriformes (Order) > Pandionidae (Family), B. … (Kingdom) > … (Phylum) > … (Class) > … (Order) > Accipitridae (Family), C. … (Kingdom) > … (Phylum) > … (Class) > … (Order) > Cathartidae (Family), D. … (Kingdom) > … (Phylum) > … (Class) > … (Order) > Sagittariidae (Family)”, E. … (Kingdom) > … (Phylum) > … (Class) > … (Order) > Elanidae (Family)" "OTXFS B &WBMVBUF Correct EX, EW, CR, EN, VU, NT, LC, DD NT Incorrect Montenegro; Italy; France; Albania etc., Country list Montenegro; France; Iraq etc., Partial Agriculture & aquaculture; Pollution; Energy production & mining; Transportation & service corridors etc. Threats list None Incorrect 5BTL ʢΠψϫγʣ ܽམɺ -
3൛͋ΔϤʂWJUBMTύοέʔδ --.ͱͷରFMMNFSύοέʔδΛհͯ͠ߦ͏ https://vitals.tidyverse.org/ library(vitals) library(ellmer) simple_qa <- tibble::tibble( input =
c("日本の初代総理大臣は誰か", "Posit(旧RStudio)のチーフサイエンティストは誰か"), target = c("伊藤博文", "Hadley Wickham") ) tsk <- Task$new( dataset = simple_qa, solver = generate(chat_ollama(model = "gpt-oss:20b")), scorer = model_graded_fact() ) tsk$eval() tsk$score() 5BTL࣮ݧܭը %BUBTFUೖྗσʔλ 4PMWFSճઓུ 4DPSFSධՁج४ ਪϞσϧͷࢦఆ -
ධՁͷίʔυԽ ධՁϓϩηεΛίʔυͱͯ͠هड़ɾཧ͢Δ ͭͷϝϦοτ ✓৽ϞσϧͰͷଈ࠲ͳ࠶ݕূ ✓ධՁͷಁ໌ੑͱՄೳੑ ✓ίϛϡχςΟͰͷڞ༗ɾվળ w Ͳͷج४Ͱఆ͔ͨ͠໌֬ w ݁Ռͷࠜڌ͕Մೳ
w ϩάͱͯ͠ه͞ΕΔ ධՁͷಁ໌ੑ w ධՁίʔυΛ(JU)VCͰެ։ w ϕϯνϚʔΫͱͯ͠ػೳ ίϛϡχςΟͰͷར༻