Foundation Models frameworkで画像分析

WWDC26 Recap - Japan-\(region).swift
https://luma.com/x230s9i1

デモの動画はこちら
https://www.youtube.com/shorts/bgyOazjfArc

Transcript

1. Ϧϣ΢ @RyoDeveloper Foundation Models framework Ͱը૾෼ੳ WWDC26 Recap - Japan-\(region).swift
2. Ϧϣ΢ • @RyoDeveloper • גࣜձࣾLuup iOS Engineer
3. Foundation Models framework Apple Intelligenceͷج൫ͱͳΔେن໛ݴޠϞσϧΛ Swift API͔Βར༻Ͱ͖ΔϑϨʔϜϫʔΫ • ϓϥΠόγʔͷ֬อ • ΦϑϥΠϯ࣮ߦ΋Մೳ • OSʹ૊Έࠐ·Ε͍ͯΔ • ແྉͰར༻Մೳ
4. WWDC26ͰͷFoundation Models frameworkͷΞοϓσʔτ • ը૾ೖྗ • Evaluations framework • ίΞϑϨʔϜϫʔΫͷΦʔϓϯιʔεԽ • ৽͍͠ΦϯσόΠεϞσϧ • Private Cloud Compute • αʔυύʔςΟϞσϧରԠ
6. LUUPΞϓϦͰͷ ׆༻ ※PoCͱ࣮ͯ͠૷ͨ͠Πϝʔδ Ͱ͢
7. Foundation Models frameworkͷը૾ೖྗ let session = LanguageModelSession() let response = try await session.respond { "͜ͷը૾ʹ͍ࣸͬͯΔं྆ʹ๨Ε෺͕ͳ͍͔ڭ͑ͯɻ" Attachment(image) } let content = response.content ͜ͷं྆ͷલΧΰʹ͸ࠇ͍ϦϡοΫ͕ೖ͍ͬͯ·͢ɻ
8. Guided Generation @Generable struct ForgottenItemPresence { @Guide(description: "ं྆ʹ๨Ε෺͕͍ࣸͬͯΔ͔") let hasForgottenItem: Bool } let session = LanguageModelSession() let response = try await session.respond( generating: ForgottenItemPresence.self ) { "͜ͷը૾ʹ͍ࣸͬͯΔं྆ʹ๨Ε෺͕ͳ͍͔ڭ͑ͯɻ" Attachment(image) } let hasForgottenItem = response.content.hasForgottenItem true
9. Evaluations framework ΞϓϦͷΠϯςϦδΣϯτػೳͷ඼࣭Λଌఆ͢Δ SwiftϑϨʔϜϫʔΫ • Swift Testingͱ౷߹͠ɺϓϩϯϓ τΛௐ੔͠ͳ͕Βਫ਼౓ΛଌఆͰ ͖Δ
10. ධՁ var dataset: ArrayLoader<ModelSample<Bool>> { ArrayLoader(samples: [ ModelSample( prompt: Prompt { "͜ͷը૾ʹ͍ࣸͬͯΔं྆ʹ๨Ε෺͕ͳ͍͔ڭ͑ͯɻ" Attachment(imageURL: withLuggageURL) }, expected: true ), ModelSample( prompt: Prompt { "͜ͷը૾ʹ͍ࣸͬͯΔं྆ʹ๨Ε෺͕ͳ͍͔ڭ͑ͯɻ" Attachment(imageURL: withoutLuggageURL) }, expected: false ), ]) }
12. Foundation Models frameworkͷ஫ҙ఺ • σόΠεɾ஍Ҭ౳ͷ੍໿ • AI͕ੜ੒͢Δग़ྗʹ͸ޡΓؚ͕·ΕΔՄೳੑ • ༻్ʹ߹ΘͤͯଞͷϑϨʔϜϫʔΫͱͷ࢖͍෼͚΋ݕ౼
13. ·ͱΊ • Foundation Models frameworkͷը૾ೖྗ͸؆୯ʹ૊ΈࠐΊΔ • Evaluations frameworkͰਫ਼౓ΛܧଓతʹධՁ͠ɺ඼࣭Λ୲อ • ಛੑΛཧղͯ͠ɺదͨ͠৔໘Ͱ׆༻͢Δ