AIを活用したレシート読み取り機能の開発から得られた実践知 / AI Receipt Scan Practice

iOSDC Japan 2025の登壇資料です。
https://fortee.jp/iosdc-japan-2025/proposal/40681d36-4abd-4e54-84c4-448fdea48009

Transcript

1. "*Λ׆༻ͨ͠ϨγʔτಡΈऔΓػೳͷ
   ։ൃ͔ΒಘΒΕ࣮ͨફ஌  
   J04%$
   +BQBO
   
   !SPDLOBNF
2. 㲔 Smart, Secure, Mobile Super powered Banking and Living.  --.Λ׆༻ͨ͠ػೳΛ։ൃͨ͜͠ͱ͕͋Δਓ🙋
3. X
4.  എܠ "*ϨγʔτಡΈऔΓ #·ͱΊ "*ࢧग़τʔΫ
5.  എܠ "*ϨγʔτಡΈऔΓ #·ͱΊ "*ࢧग़τʔΫ
6. ࠓ೔࿩͢͜ͱ
7.  ࠓ೔࿩͢͜ͱ  "*ϨγʔτಡΈऔΓػೳͷ֓ཁ
    Ϩγʔτ0$3
    'PVOEBUJPO
   .PEFMTͷՄೳੑ
8.  4NBSU#BOL
   *OD
   
   .PCJMF
   "QQMJDBUJPO
   %FWFMPQFS ϩΫωϜ !@SPDLOBNF !SPDLOBNF ࣗݾ঺հ
9.  
   "*ϨγʔτಡΈऔΓػೳͷ֓ཁ
17.  
    "*ϨγʔτಡΈऔΓػೳͷ֓ཁ όοΫΤϯυଆͰͷ࣮૷ʹ͍ͭͯৄࡉ͸ͪ͜Β IUUQTTQFBLFSEFDLDPNNP[OJPOGSPNHSPVOEFEUFDIDIPJDFTUPNBHJDBMVYBDBTFTUVEZPGBJ SFDFJQUTDBOOJOH !NP[OJPO
18.  
    Ϩγʔτ0$3
24. IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOBWGPVOEBUJPOTFUUJOHVQBDBQUVSFTFTTJPO
25. let session = AVCaptureSession() func configureSession() { session.beginConfiguration() defer { session.commitConfiguration() } setupInput() setupOutput() } func setupInput() { … } func setupOutput() { … }
26. func configureSession() { … setupInput() setupOutput() } func setupInput() { let device = AVCaptureDevice.default( .builtInWideAngleCamera, for: .video, position: .back )! let deviceInput = try! AVCaptureDeviceInput(device: device) if session.canAddInput(deviceInput) { session.addInput(deviceInput) } } func setupOutput() { … }
27. func configureSession() { … setupInput() setupOutput() } func setupInput() { … } func setupOutput() { let videoDataOutput = AVCaptureVideoDataOutput() videoDataOutput.setSampleBufferDelegate( self, queue: sampleBufferQueue ) if session.canAddOutput(videoDataOutput) { session.addOutput(videoDataOutput) } videoDataOutput.connection(with: .video)?.isEnabled = true }
28. // AVCaptureVideoDataOutputSampleBufferDelegate method func captureOutput( _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection ) { // TODO: ϏσΦϑϨʔϜͷϋϯυϦϯά }
29.  
    Ϩγʔτ0$3
    
    4FUVQ
    DBNFSB 0$3ͷਫ਼౓ΛߴΊΔͨΊʹ )JHI
    3FTPMVUJPO 'PDVT
30.  
    Ϩγʔτ0$3
    
    4FUVQ
    DBNFSB 0$3ͷਫ਼౓ΛߴΊΔͨΊʹ )JHI
    3FTPMVUJPO 'PDVT
31. func configureSession() { … if session.canSetSessionPreset(.hd4K3840x2160) { session.sessionPreset = .hd4K3840x2160 } … }
32.  
    Ϩγʔτ0$3
    
    4FUVQ
    DBNFSB 0$3ͷਫ਼౓ΛߴΊΔͨΊʹ )JHI
    3FTPMVUJPO 'PDVT
33.  
    Ϩγʔτ0$3
    
    4FUVQ
    DBNFSB Ϩγʔτ͸Χϝϥͱൺֱత ۙڑ཭ͰࡱӨ͞ΕΔɻ
    ͦͷલఏΛ
    "7$BQUVSF%FWJDF
    ΁͋Β͔ ͡Ί఻͓͑ͯ͘ͱϑΥʔΧ ε͕߹͍΍͘͢ͳΔɻ ϑΥʔΧεΛ߹ΘͤΔ
34. func setupInput() { … if device.isAutoFocusRangeRestrictionSupported { device.autoFocusRangeRestriction = .near } … }
35.  
    Ϩγʔτ0$3
    
    4FUVQ
    DBNFSB ࠷খϑΥʔΧεڑ཭ͱ͸ɺ઱໌ͳϑΥʔ ΧεΛ߹ΘͤΔ͜ͱ͕Ͱ͖ΔɺϨϯζ͔ Βඃࣸମ·Ͱͷ࠷΋͍ۙڑ཭ͷ͜ͱɻ
    ඃࣸମ͕Χϝϥͷ࠷খϑΥʔΧεڑ཭Α Γ΋͍ۙ৔߹ɺϑΥʔΧεΛ߹ΘͤΒΕ ͣʹը૾͕΅΍͚ͯ͠·͏ɻ ࠷খϑΥʔΧεڑ཭Λߟྀ͢Δ
36.  
    Ϩγʔτ0$3
    
    4FUVQ
    DBNFSB ඞͣ࠷খϑΥʔΧεڑ཭Ҏ্཭͞ͳ͍ ͱϨγʔτ͕ΧϝϥϓϨϏϡʔʹऩ·Β ͳ͍Α͏ʹζʔϜΛద༻͢Δɻ
    88%$
    ͷηογϣϯ
    8IBU`T
    OFX
    JO
    DBNFSB
    DBQUVSF
    ʹͯ঺հ͞Ε͍ͯΔํ ๏ɻ ڑ཭͕։͘Α͏ʹζʔϜΛௐ੔
37. func setupInput() { … let minimumSubjectDistance = minimumSubjectDistance( fieldOfView: device.activeFormat.videoFieldOfView, minimumReceiptWidth: 60, // ૝ఆ͞ΕΔ࠷খͷϨγʔτ୹ล (mm) previewFillPercentage: 0.8 // ϓϨϏϡʔͷ෯ʹର͢Δඃࣸମͷׂ߹ ) } func minimumSubjectDistance( fieldOfView: Float, minimumReceiptWidth: Float, previewFillPercentage: Float ) -> Float { let radians = degreesToRadians(fieldOfView / 2.0) let filledSize = minimumReceiptWidth / previewFillPercentage return filledSize / tan(radians) }
38. func setupInput() { … let deviceMinimumFocusDistance = Float(device.minimumFocusDistance) if minimumSubjectDistance < deviceMinimumFocusDistance { let zoomFactor = deviceMinimumFocusDistance / minimumSubjectDistance device.videoZoomFactor = CGFloat(zoomFactor) } }
39. #FGPSF "GUFS
42.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU จࣈྻ͕هࡌ͞Εۣͨܗঢ়ͷΦϒ δΣΫτΛϨγʔτͱͯ͠ݕग़ɻ
    ͦͷ্ͰɺϢʔβʔͷखϒϨ͕ൃੜ ͠ͳ͍ϕετͳλΠϛϯάΛݕ஌͠ ͯ0$3Λ࣮ߦ͍ͨ͠ɻ Ϩγʔτݕ஌ϩδοΫ
    
43.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU ݕग़ҐஔͱαΠζ͕ۙࣅ͠ଓ͚ͨ৔߹ʹϨγʔτΛݕ஌ͨ͠ͱΈͳͯ͠ 0$3΁Ҡߦ͢Δ͜ͱʹɻ Ϩγʔτݕ஌ϩδοΫ
    
44. IUUQTTQFBLFSEFDLDPNOBLBNVVVJPTEDKBQBO TMJEF J04%$
    ʰγʔϜϨεͳମݧΛ࣮ݱ͢Δຊਓ֬ೝϑϩʔͷߏஙʱ !OBLBNVVV
47.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU "QQMFͷίϯϐϡʔλʔϏδϣϯ ϑϨʔϜϫʔΫɻ
    ػցֶशΛ༻͍ͯը૾ɾө૾ͷղ ੳΛΞϓϦ্Ͱ࣮ߦͰ͖Δɻ 7JTJPO
    GSBNFXPSL IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOWJTJPO
48.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU ը૾͔ΒจࣈΛؚΉۣܗͷྖҬΛ ݕग़͢ΔϦΫΤετɻ
    ·ͣ͸͜ΕͰը૾͔ΒϨγʔτͬ Ά͍ۣܗΛݕग़͢Δɻ %FUFDU%PDVNFOU4FHNFOUBUJPO3FRVFTU IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOWJTJPOEFUFDUEPDVNFOUTFHNFOUBUJPOSFRVFTU
49.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU ͋Β͔͡Ίಛఆ͞ΕۣͨܗΦϒ δΣΫτͷಈ͖Λɺෳ਺ͷը૾΍ ϏσΦϑϨʔϜʹΘͨͬͯ௥੻͢ Δը૾ղੳϦΫΤετɻ 5SBDL3FDUBOHMF3FRVFTU IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOWJTJPOUSBDLSFDUBOHMFSFRVFTU
50. actor ReceiptDetector { func processVideoFrame(sampleBuffer: CMSampleBuffer) async { await startToTrackReceiptRect(sampleBuffer: sampleBuffer) } private func startToTrackReceiptRect(sampleBuffer: CMSampleBuffer) async { do { let request = DetectDocumentSegmentationRequest() let detectedDocumentObservation = try await request.perform( on: sampleBuffer, orientation: .right ) } catch { … } } }
51. actor ReceiptDetector { private var trackRectangleRequest: TrackRectangleRequest? … private func startToTrackReceiptRect(sampleBuffer: CMSampleBuffer) async { … guard let detectedDocumentObservation, detectedDocumentObservation.confidence > 0.5 else { return } let trackingRequest = TrackRectangleRequest( detectedRectangle: detectedDocumentObservation, .revision1 ) trackingRequest.trackingLevel = .accurate self.trackRectangleRequest = trackingRequest }
52. actor ReceiptDetector { private var trackRectangleRequest: TrackRectangleRequest? func processVideoFrame(sampleBuffer: CMSampleBuffer) async { guard let trackRectangleRequest else { await startToTrackReceiptRect(sampleBuffer: sampleBuffer) return } await trackReceiptRect( trackRectangleRequest: trackRectangleRequest, sampleBuffer: sampleBuffer ) } private func trackReceiptRect(…) async { … } }
53. actor ReceiptDetector { … private func trackReceiptRect( trackRectangleRequest: TrackRectangleRequest, sampleBuffer: CMSampleBuffer ) async { do { let rectangleObservation = try await trackRectangleRequest.perform( on: sampleBuffer, orientation: .right ) // ✅ ݕग़ҐஔͱαΠζ͕ۙࣅ͠ଓ͚͍ͯΔ͔Ͳ͏͔Λ൑ఆ͢Δ } catch { … } } }
54.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU ຖϑϨʔϜը૾ॲཧΛ࣮ߦ͢Δ ͱෛՙ͕ߴ͍ɻ
    ը૾ॲཧΛ࣮ߦதͷ৔߹͸ૣظ Ϧλʔϯ͠ɺಉ࣌ʹෳ਺ͷը૾ ॲཧ͕૸Βͳ͍Α͏ʹ͢Δɻ ύϑΥʔϚϯεվળ5JQT
    
55. actor ReceiptDetector { private var isDetectingReceipt = false … func processVideoFrame(sampleBuffer: CMSampleBuffer) async { if isDetectingReceipt { return } isDetectingReceipt = true defer { isDetectingReceipt = false } … } … }
56.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU Ϩγʔτݕ஌ʹ͓͍ͯ͸ը૾͸ ͦ͜·Ͱߴղ૾౓Ͱ͋Δඞཁ͸ ͳ͍ɻ
    খ͘͞ϦαΠζͨ͠ը૾ʹର͠ ۣͯܗݕग़ͱ௥੻ͷϦΫΤετ Λ࣮ߦ͢Δɻ ύϑΥʔϚϯεվળ5JQT
    
57. guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } let originalImage = CIImage(cvImageBuffer: pixelBuffer) let originalSize = originalImage.extent.size let originalShortSide = min(originalSize.width, originalSize.height) let targetShortSide: CGFloat = 512 let resizeScale = min(1.0, targetShortSide / originalShortSide) let resizedImage = originalImage.transformed( by: CGAffineTransform(scaleX: resizeScale, y: resizeScale) ) …
60.  
    Ϩγʔτ0$3
    
    3FDPHOJ[F
    5FYU ը૾಺ͷςΩετΛೝࣝ͢ΔϦΫ Τετɻ 3FDPHOJ[F5FYU3FRVFTU IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPOWJTJPOSFDPHOJ[FUFYUSFRVFTU
61. final class ReceiptTextRecognizer { func recognizeText(sampleBuffer: CMSampleBuffer) async throws -> [String] { var request = RecognizeTextRequest(.revision3) request.recognitionLanguages = [ Locale.Language(identifier: "ja-JP"), Locale.Language(identifier: "en-US") ] request.recognitionLevel = .accurate let result = try await request.perform( on: sampleBuffer, orientation: .right ) } }
62.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU ςΩετೝࣝ͸͔ͳΓॲཧ͕ॏ ͍ɻ
    Ͱ͖Δ͚ͩෛՙΛখ͘͢͞Δͨ Ίʹ
    SFHJPO0G*OUFSFTU
    ͰςΩε τೝࣝͷର৅ྖҬΛݶఆ͢Δɻ ύϑΥʔϚϯεվળ5JQT
63. func recognizeText( sampleBuffer: CMSampleBuffer, receiptRect: NormalizedRect ) async throws -> [String] { var request = RecognizeTextRequest(.revision3) let expandedRect = expandNormalizedRect(receiptRect, percent: 0.05) request.regionOfInterest = expandedRect … } private func expandNormalizedRect( _ rect: NormalizedRect, percent: CGFloat ) -> NormalizedRect { // ✅ ۣܗΛ෯ɾߴ͞ͷׂ߹ʢpercentʣ͚࢛ͩํʹ֦ு͢Δ }
64.  
    Ϩγʔτ0$3
    
    3FDPHOJ[F
    5FYU ࣮ߦ݁Ռͱͯ͠ʮը૾಺ͰςΩε τͱ൑ఆ͞Ε֤ͨྖҬʯΛද͢
    3FDPHOJ[FE5FYU0CTFSWBUJPO
    ͷ഑ ྻ͕ฦΔɻ
    ֤ྖҬʹ͸ը૾಺ʹ͓͚Δ࠲ඪ͕ ׂΓৼΒΕ͍ͯΔɻ 3FDPHOJ[F5FYU3FRVFTU3FTVMU
65. { "string": "ίΧɾίʔϥ ϠΧϯϊϜΪνϟ", "bounding_box": { "x": 0.11, "y": 0.70, "width": 0.38, "height": 0.01 } }, { "string": "650ml", "bounding_box": { "x": 0.49, "y": 0.70, "width": 0.14, "height": 0.01 } }, …
66. { "string": "ίΧɾίʔϥ ϠΧϯϊϜΪνϟ", "bounding_box": { "x": 0.11, "y": 0.70, "width": 0.38, "height": 0.01 } }, { "string": "650ml", "bounding_box": { "x": 0.49, "y": 0.70, "width": 0.14, "height": 0.01 } }, …
67.  
    Ϩγʔτ0$3
    
    3FDPHOJ[F
    5FYU Ϩγʔτͷ্͔ΒԼʹฒͿςΩετ ͱಉ͡ॱংͰߦͣͭվߦ۠੾ΓͰ ੔ܗ͢Δɻ
    গͳ͍τʔΫϯ਺Ͱ֤จࣈྻͷ࠲ඪ ৘ใΛ఻͑Δ͜ͱ͕Ͱ͖Δɻ --.͕ղऍ͠΍͍͢ܗʹલॲཧ
68. [ "ηϒϯ-ΠϨϒϯ", "඼઒౦ޒ൓ా1ஸ໨ళ", "౦ژ౎඼઒۠౦ޒ൓ా1-12-2", "ి࿩ɿ03-3440-5055 Ϩδˌ1", "ࣄۀऀొ࿥൪߸5010701018426", "2024೥09݄13೔ʢۚʣ09:22 ੹210", "ྖऩॻ", "ίΧɾίʔϥ ϠΧϯϊϜΪνϠ 650ml ˎ138", "஋Ҿֹ -20”, "ʢ঎඼୅ۚ ¥138ʣ", … ]
69.  
    Ϩγʔτ0$3
    
    3FDPHOJ[F
    5FYU ༨ஊ
    88%$
    Ͱൃද͞Εͨ
    3FDPHOJ[F%PDVNFOUT3FRVFTU IUUQTEFWFMPQFSBQQMFDPNWJEFPTQMBZXXED
70. PostalAddress( fullAddress: "౦ژ౎඼઒۠౦ޒ൓ా 1 － 12 － 2", street: "౦ޒ൓ా1 － 12 － 2", city: "඼઒۠", state: "౦ژ౎" )
71. PhoneNumber( phoneNumber: "03-3440-5055", label: "Work" )
72. CalendarEvent( allDay: false, startDate: "2024-09-13 09:22:00 +0000", startTimeZone: nil, endDate: nil, endTimeZone: nil )
73. MoneyAmount( currency: jpy, amount: 138 ) MoneyAmount( currency: jpy, amount: 127 )
74. MoneyAmount( currency: jpy, amount: 138 ) MoneyAmount( currency: jpy, amount: 127 )
75.  
    Ϩγʔτ0$3
    
    %FUFDU
    3FDFJQU w ॅॴ΍ి࿩൪߸ʹ͍ͭͯ͸͔ͳΓਫ਼౓ߴ͘நग़ͯ͘͠Εͦ͏
    w ֹۚͷநग़ʹ͍ͭͯ͸Ϝϥ͕͋ΔͨΊ׬શʹ৴པ͢Δ͜ͱ͸೉ͦ͠ ͏ 3FDPHOJ[F%PDVNFOUT3FRVFTU΁ͷධՁ
78.  
    'PVOEBUJPO
    .PEFMTͷՄೳੑ
79.  
    'PVOEBUJPO
    .PEFMTͷՄೳੑ "QQMF
    *OUFMMJHFODF
    ͷݴޠϞσ ϧΛ
    4XJGU
    ͔Βܕ҆શʹѻ͑Δ ϑϨʔϜϫʔΫͰɺཁ໿ɾந ग़ɾϦϥΠτ౳ΛΦϯσόΠε ͔ͭϓϥΠόγʔ༏ઌͰ࣮ߦ͠ ·͢ɻ 'PVOEBUJPO
    .PEFMTͱ͸ IUUQTEFWFMPQFSBQQMFDPNEPDVNFOUBUJPO'PVOEBUJPO.PEFMT
80. 'PVOEBUJPO
    .PEFMTͰ
    Ϩγʔτ৘ใͷߏ଄Խ͸Ͱ͖Δͷ͔ʜʁ
81. @Generable struct ReceiptInferenceResponse { @Guide(description: "Merchant/store name.") let shopName: String? @Guide(description: "Purchase (transaction) date normalized to ISO 8601 `YYYY-MM-DD`.") let date: String? @Guide(description: "Grand total amount paid in JPY (tax- inclusive), as a non-negative integer.") let amount: Int? @Guide(description: "Category for this receipt.") let category: Category? }
82. final class ReceiptStructureInferrer { private let session: LanguageModelSession init() { self.session = LanguageModelSession( instructions: Instructions { """ … """ } ) } }
83. Instructions { """ You are an information extractor for retail receipts. Carefully read through the input text in Japanese, scanned from a receipt. Line breaks correspond to each line of the receipt from top to bottom. Extract receipt information and return it in the provided schema format Each category definition for a receipt is following: - groceries: foodstuff, groceries - diningOut: payments related to restaurants, bars, and other dining expenses - householdSupplies: daily necessities, excluding food items … """ }
84. final class ReceiptStructureInferrer { … func infer(receiptTextLines: [String]) async throws -> ReceiptInferenceResponse { let prompt = Prompt { … } let response = try await session.respond( to: prompt, generating: ReceiptInferenceResponse.self, options: GenerationOptions(temperature: 0.1) ) return response.content } }
85. Prompt { """ ## Task Extract fields from the following OCR lines of a Japanese retail receipt. ## Input \(receiptTextLines.joined(separator: "\n")) ## Requirements - The output format must be valid. - Ensure that the output is complete and not truncated. - If any information is missing or unclear, use "nil" for that field. """ }
86. ಈ͔ͯ͠ΈΔ
    📸
90.  
    'PVOEBUJPO
    .PEFMTͷՄೳੑ w ళ໊ɾ೔෇ͷநग़͸݁ߏ͏·͍͘͘
    w ֹۚͷநग़ɺΧςΰϦͷਪ࿦पΓͰগ͠՝୊͋Γ
    w ͜Ε͕ແྉ͔ͭMPDBM׬݁Ͱମݧͱͯ͠ఏڙͰ͖Δͷ͸͍͢͝ 'PVOEBUJPO
    .PEFMTͷධՁ
91.  ·ͱΊ
92.  ·ͱΊ w "*ϨγʔτಡΈऔΓػೳ
    📸
    w --.ͷ
    lߏ଄Խz
    ͱ͍͏ڧΈΛ׆͔ͯ҆͠Ձʹߴ͍ਫ਼౓Λ࣮ݱͨ͠
    w (BSCBHF
    JO
    HBSCBHF
    PVU
    🔥
    w ΠϯϓοτΛ͍͔ʹ៉ྷʹ͢Δ͔͕--.ͷਪ࿦ͷਫ਼౓ʹ௚݁͢Δ
    w 'PVOEBUJPO
    .PEFMT
    🧠
    w -PDBM
    --.ͷϙςϯγϟϧΛײͤͨ͡͞
93. ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ