Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Core ML版Stable DiffusionをiOSで快適に動かす

shu223
October 17, 2023

Core ML版Stable DiffusionをiOSで快適に動かす

「Mobile勉強会 Wantedly × チームラボ #11」での発表資料です。
https://wantedly.connpass.com/event/297601/

記事: Core ML版Stable DiffusionをiOSで快適に動かす

shu223

October 17, 2023
Tweet

More Decks by shu223

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ • అ मҰ • @shu223 (GitHub, Qiita, Zenn, note,

    , YouTube, Podcast, etc...) • ॻ੶ʢ঎ۀग़൛4࡭ɺݸਓग़൛ଟ਺ @BOOTHʣ:
  2. Core ML Stable Diffusion ͱ͸ Core ML ϑΥʔϚοτʹม׵͞Εͨ Stable Diffusion

    ͷϞσϧ • ैདྷͷϞσϧΛAppleϋʔυ΢ΣΞͰಈ͔͢ => CPUͷΈར ༻ • Core MLϞσϧ => CPU, GPU, Neural EngineΛར༻ → Appleͷϋʔυ΢ΣΞΛ࠷େݶ׆͔ͤΔͷ͕Core MLϞσϧ 1 1 ৄࡉͳ࠷దԽͷղઆɿ Deploying Transformers on the Apple Neural Engine - Apple Machine Learning Research
  3. Core ML Stable Diffusion ͷϦϙδτϦ https://github.com/apple/ml-stable-diffusion • Appleۘ੡ • Ϟσϧม׵ίʔυɺmacOS/iOSͰಈ͔ͨ͢ΊͷSwiftίʔυ౳

    ΛؚΉ • ެࣜϒϩάهࣄɿ Stable Diffusion with Core ML on Apple Silicon - Apple Machine Learning Research
  4. CLI͔Βಈ͔͢ SwiftͰॻ͔ΕͨCLIαϯϓϧ StableDiffusionSampleΛར༻ swift run StableDiffusionSample \ "a photo of

    an astronaut riding a horse on mars" \ --resource-path </path/to/models> --seed 93 \ --output-path </path/to/output/image> • --resource-path • ίϯύΠϧࡁΈCore MLϞσϧϑΝΠϧ .mlmodelc ͕ ೖͬͨσΟϨΫτϦͷύεΛࢦఆ
  5. ࣗ࡞ΞϓϦʹ૊ΈࠐΉ 2/2 import StableDiffusion ... let pipeline = try StableDiffusionPipeline(resourcesAt:

    resourceURL) pipeline.loadResources() let image = try pipeline.generateImages(prompt: prompt, seed: seed).first
  6. reduceMemory Λ༗ޮʹ͢Δ ModelInfo.swift ͷ reduceMemory ϓϩύςΟΛܾఆ͢Δ ॲཧΛॻ͖׵͑Δ 3 var reduceMemory:

    Bool { // Enable on iOS devices, except when using quantization if runningOnMac { return false } if isXL { return !deviceHas8GBOrMore } return !(quantized && deviceHas6GBOrMore) } 3 ݱߦ࣮૷Ͱ͸iPhone 15 Proͷ৔߹͸ඞͣແޮԽ͞Εͯ͠·͏
  7. prewarmResources loadResources ޙ͙͢ʹ unloadResources Λߦ͏ func prewarmResources() throws { try

    loadResources() unloadResources() } Both .mlpackage and .mlmodelc models are compiled upon first load. ʢதུʣ .mlmodelc files do cache this compiled asset and non-first load times are reduced to just a few seconds.
  8. ը૾ੜ੒ͷॲཧ࣌ؒ Got images: [Optional(<CGImage 0x12810d510> (IP) <<CGColorSpace 0x283e04780> (kCGColorSpaceDeviceRGB)> width

    = 768, height = 768, bpc = 8, bpp = 24, row bytes = 2304 kCGImageAlphaNone | 0 (default byte order) | kCGImagePixelFormatPacked is mask? No, has masking color? No, has soft mask? No, has matte? No, should interpolate? Yes)] in 83.99049496650696 1ຕ͋ͨΓ 83.99 ඵ ىಈ࣌ؒ΋ґવͱͯ͠ 5෼ ͔͔͍ͬͯΔ
  9. ը૾ੜ੒࣌ؒΛ୹͘͢Δํ๏ αΠζͷখ͍͞Ϟσϧʹมߋ͢Δ The Neural Engine is capable of accelerating models

    with low-bit palettization: 1, 2, 4, 6 or 8 bits. ʢχϡʔϥϧɾΤϯδϯ͸ɺ௿ ϏοτͷύϨλΠζͰϞσϧΛߴ଎ԽͰ͖Δʣ compressed weights are faster to fetch from memory ... ʢѹॖ͞ Εͨ΢ΣΠτ͸ϝϞϦ͔ΒͷϑΣον͕ߴ଎ʹͳΓɺʣ 5 5 apple/ml-stable-diffusion: Stable Diffusion with Core ML on Apple Silicon ͷREADMEΑΓ
  10. ݱࡏͷ࢖༻ϞσϧΛௐ΂Δ 1/3 let loader = PipelineLoader(model: iosModel()) func iosModel() ->

    ModelInfo { guard deviceSupportsQuantization else { return ModelInfo.v21Base } if deviceHas6GBOrMore { return ModelInfo.xlmbpChunked } return ModelInfo.v21Palettized } → iPhone 15 ProͰ͸ ModelInfo.xlmbpChunked Λར༻ 6 6 deviceSupportsQuantization ͕ true ͔ͭ deviceHas6GBOrMore ͕ true
  11. ͍ܰSDϞσϧΛબͿϙΠϯτ • SDXLͰ͸ͳ͍΋ͷɾɾɾXL͸࠷৽ͷߴը࣭൛Ͱ͋ΓɺσΧ ͍ • palletized ͱ͍͍ͭͯΔ΋ͷɾɾɾѹॖ͞Ε͍ͯΔ͜ͱΛࣔ ͢ • 16bit

    -> 6-bit • SDόʔδϣϯͷҧ͍ʹΑΔαΠζͷҧ͍͸͋·Γͳ͍ → "apple/coreml-stable-diffusion-2-1-base-palettized"
  12. ֤छύϑΥʔϚϯε • ىಈ࣌ؒ 7ɿ100ඵલޙʢ300%ߴ଎Խʣ • ࢖༻ϝϞϦྔͷϐʔΫ஋ 8ɿ໿850MB • ը૾ੜ੒ॲཧ࣌ؒɿ10ඵҎԼʢ800%ߴ଎Խʣ 8

    αΠζͷখ͍͞Ϟσϧͷ࢖༻ʹ൐͍ɺreduceMemory ͸࠶ͼແޮԽͨ͠ 7 ΞϓϦͷىಈ࣌ؒશମͰ͸ͳ͘ɺύΠϓϥΠϯͷϩʔυ࣌ؒΛܭଌ
  13. ͞ΒͳΔىಈ࣌ؒͷ୹ॖ ϞσϧͷϩʔυʢMLModelͷॳظԽʣ͕90%Ҏ্Λ઎ΊΔ time to load unet: 57.41793203353882 time to load

    textEncoder: 30.462964057922363 time to load decoder: 3.1042929887771606 time to load encoder: 2.146523952484131 Pipeline resources loaded in 106.28662097454071
  14. ·ͱΊ Stable DiffusionͷΑ͏ͳΫισΧCore MLϞσϧΛiOSͰಈ͔͢ • ΩϟογϡػߏΛར༻ͯ͠౎౓ϩʔυ & Ξϯϩʔυ • →

    ϝϞϦ࢖༻ྔΛ཈͑Δ • ϞσϧαΠζΛখ͘͢͞Δ • → ىಈ࣌ؒɾϝϞϦ࢖༻ྔɾը૾ੜ੒࣌ؒ͢΂ͯվળ • Hugging Face୳͢/coremltoolsͰࣗલѹॖ/Mixed-Bit Palettization