Conquering PDFs: document understanding beyond plain text

NLP and data science could be so easy if all of our data came as clean and plain text. But in practice, a lot of it is hidden away in PDFs, Word documents, scans and other formats that have been a nightmare to work with. In this talk, I'll present a new and modular approach for building robust document understanding systems, using state-of-the-art models and the awesome Python ecosystem. I'll show you how you can go from PDFs to structured data and even build fully custom information extraction pipelines for your specific use case.

For the practical examples, I'll be using spaCy, and the new Docling library and layout analysis models. I'll also cover Optical Character Recognition (OCR) for image-based text, how to convert tabular data to pandas DataFrames, and strategies for creating training and evaluation data for information extraction tasks like text classification and entity recognition using PDFs and other documents as inputs.

Blog post: https://explosion.ai/blog/pdfs-nlp-structured-data

Transcript

1. Document understanding beyond plain text Ines Montani Explosion
2. Open-source library for industrial-strength natural language processing spacy.io downloads
3. Open-source library for industrial-strength natural language processing spacy.io ChatGPT can write spaCy code! downloads
4. Modern scriptable annotation tool for machine learning developers prodigy.ai companies users
5. Modern scriptable annotation tool for machine learning developers prodigy.ai Alex Smith Developer Kim Miller Analyst GPT-4 API companies users
6. Documents
7. Businesses want electronic copies that map 1:1 to paper documents. Documents
8. Businesses want electronic copies that map 1:1 to paper documents. Documents
9. Documents
10. “I have the data in a PDF.” “I have the data on my computer.” Documents
11. explosion.ai/blog/human-in-the-loop-distillation At their core, many NLP systems consist of flat classifications. You can shove them into a single prompt, or you can decompose them into smaller pieces. Many classification tasks are straightforward to solve nowadays – but they become vastly more complicated if one model needs to do them all at once.
12. docling-project.github.io/docling Open-source library and models for document processing Docling Technical Report (Auer et al., 2024)
13. docling-project.github.io/docling unified structured format DoclingDocument Open-source library and models for document processing Docling Technical Report (Auer et al., 2024)
14. docling-project.github.io/docling unified structured format DoclingDocument Open-source library and models for document processing Docling Technical Report (Auer et al., 2024) just like spaCy’s Doc object
15. github.com/explosion/spacy-layout
16. github.com/explosion/spacy-layout
17. github.com/explosion/spacy-layout process and create a spaCy Doc
18. github.com/explosion/spacy-layout text-based contents process and create a spaCy Doc
19. github.com/explosion/spacy-layout text-based contents document layout process and create a spaCy Doc
20. github.com/explosion/spacy-layout text-based contents document layout process and create a spaCy Doc
21. github.com/explosion/spacy-layout text-based contents layout sections document layout process and create a spaCy Doc
22. github.com/explosion/spacy-layout text-based contents layout sections document layout content, tokens, o sets process and create a spaCy Doc
23. github.com/explosion/spacy-layout text-based contents layout sections section type document layout content, tokens, o sets process and create a spaCy Doc
24. github.com/explosion/spacy-layout text-based contents layout sections section type document layout bounding box content, tokens, o sets process and create a spaCy Doc
25. github.com/explosion/spacy-layout PDF
26. github.com/explosion/spacy-layout transformer-based English pipeline PDF
27. github.com/explosion/spacy-layout transformer-based English pipeline PDF Doc object processing
28. github.com/explosion/spacy-layout transformer-based English pipeline PDF apply pipeline to Doc Doc object NLP pipeline Doc object processing
29. github.com/explosion/spacy-layout Doc object named entities, part-of-speech tags, dependencies, … transformer-based English pipeline PDF apply pipeline to Doc Doc object NLP pipeline Doc object processing
31. find layout span containing an entity
32. find layout span containing an entity Apple Inc. ORG
33. find layout span containing an entity Apple Inc. ORG bounding box
34. find layout span containing an entity Apple Inc. ORG bounding box closest heading
35. find layout span containing an entity Apple Inc. ORG bounding box closest heading Company summary SECTION_HEADER
36. github.com/explosion/spacy-layout
37. bounding box github.com/explosion/spacy-layout
38. bounding box github.com/explosion/spacy-layout pandas.DataFrame
39. TableFormer: Table Structure Understanding with Transformers (Nassar et al., 2022) bounding box github.com/explosion/spacy-layout pandas.DataFrame
40. TableFormer: Table Structure Understanding with Transformers (Nassar et al., 2022) bounding box github.com/explosion/spacy-layout pandas.DataFrame customize table representation in text Table with columns: Names, Amount
42. layout analysis natural language processing
43. explosion.ai/blog/pdfs-nlp-structured-data
44. explosion.ai/blog/pdfs-nlp-structured-data Doc object processing PDF
45. explosion.ai/blog/pdfs-nlp-structured-data Doc object processing PDF Doc object NLP pipeline
46. explosion.ai/blog/pdfs-nlp-structured-data Doc object processing PDF Doc object NLP pipeline annotation
47. explosion.ai/blog/pdfs-nlp-structured-data Doc object processing PDF Doc object NLP pipeline annotation transfer learning
48. explosion.ai/blog/pdfs-nlp-structured-data Doc object processing PDF Doc object NLP pipeline annotation transfer learning Examine the role of layout features Does it actually matter for the task? What can we abstract away to generalize better?
49. prodigy.ai/docs/plugins $ prodigy pdf.spans.manual papers blank:en ./pdfs -- label EVENT,PLACE -- focus text,list_item
50. prodigy.ai/docs/plugins $ prodigy pdf.spans.manual papers blank:en ./pdfs -- label EVENT,PLACE -- focus text,list_item recipe
51. prodigy.ai/docs/plugins $ prodigy pdf.spans.manual papers blank:en ./pdfs -- label EVENT,PLACE -- focus text,list_item recipe input data
52. prodigy.ai/docs/plugins $ prodigy pdf.spans.manual papers blank:en ./pdfs -- label EVENT,PLACE -- focus text,list_item recipe input data sections
53. prodigy.ai/docs/plugins $ prodigy pdf.spans.manual papers blank:en ./pdfs -- label EVENT,PLACE -- focus text,list_item recipe input data sections
54. prodigy.ai/docs/plugins $ prodigy pdf.spans.manual papers blank:en ./pdfs -- label EVENT,PLACE -- focus text,list_item recipe input data sections saved to dataset
55. $ prodigy train ./models -- ner papers - - eval-split 0.3
56. $ prodigy train ./models -- ner papers - - eval-split 0.3 output
57. $ prodigy train ./models -- ner papers - - eval-split 0.3 output dataset
58. $ prodigy train ./models -- ner papers - - eval-split 0.3 output dataset evaluation %
59. $ prodigy train ./models -- ner papers - - eval-split 0.3 NLP model output dataset evaluation %
60. $ prodigy train ./models -- ner papers - - eval-split 0.3 NLP model output dataset apply PDF processing evaluation %
61. $ prodigy train ./models -- ner papers - - eval-split 0.3 NLP model output dataset apply PDF processing apply model evaluation %
62. $ prodigy train ./models -- ner papers - - eval-split 0.3 NLP model output dataset apply PDF processing apply model process documents at scale ship & deploy evaluation %
64. Work from a unified structured format and get your data out of PDFs as early as you can. PDFs are a bad source of truth
65. Combine document processing with NLP components you can develop independently. Modularity is your superpower Work from a unified structured format and get your data out of PDFs as early as you can. PDFs are a bad source of truth
66. Combine document processing with NLP components you can develop independently. Modularity is your superpower Layout analysis models are steadily getting better, faster and smaller! This is only the beginning Work from a unified structured format and get your data out of PDFs as early as you can. PDFs are a bad source of truth
67. explosion.ai/blog/pdfs-nlp-structured-data Combine document processing with NLP components you can develop independently. Modularity is your superpower Layout analysis models are steadily getting better, faster and smaller! This is only the beginning Work from a unified structured format and get your data out of PDFs as early as you can. PDFs are a bad source of truth
68. Explosion spaCy Prodigy Mastodon Bluesky explosion.ai spacy.io prodigy.ai @ines@sigmoid.social @inesmontani.bsky.social LinkedIn