Going beyond simple word-list creation using CasualConc @ American Association of Corpus Linguistics Conference 2018

Transcript

1. Going beyond simple word-list creation using CasualConc Yasuhiro IMAO Osaka University, Japan AACL 2018 at Georgia State University, Atlanta GA
2. The Purpose Introducing CasualConc The REAL purpose Making you want to buy a Mac!
3. A few observations Through attending presentations / reading papers Hugely depend on Methods of analysis the tools one uses Use of statistics the access to the resources specialized application programing skills someone who can write scripts
4. To advance the field more easy-to-use and accessible tools are necessary
5. Current Situation AntConc and Antxxx The gold standard? WordSmith Tools / Monoconc Pro and other more small specialized application
6. Problems with AntConc IT IS A “WINDOWS” APPLICATION We need an alternative for Mac users!
7. Limitation of AntConc (which could affect analyses) Limited file handling Very limited information on word/n-gram lists Simple keyword analyses (LL, χ2 , total frequency-based) Not much going beyond basic analyses (separate apps required) (according to the AntConc website)
8. Introducing CasualConc
9. A little bit of background I started developing a concordancer around 2005 I released the first, limited version around 2008 KWIC, Word/n-gram Lists, Collocation It is a Mac native app!
11. Before going into more detail
12. Small Scale Corpus Research Building your own specialized corpus Possibly adding annotations (POS, syntactic, etc.) Which tools to use?
13. A suggestion (not the answer) Use my app!
14. A suggestion (not the answer) I have developed few companion apps CasualTranscriber (transcription helper) CasualTextractor (text extractor/editor) CasualTagger (tagging helper) CasualPConc (parallel concordancer)
15. CasualTagger
16. CasualTagger KWIC search + short-cut tag insertion Batch Processing Tagging (TreeTagger, STF CoreNLP, MeCab) Tokenizing (macOS built-in tagger) Sentence Splitting (macOS built-in tagger, STF CoreNLP) Misspelling detection (non-dictionary words)
18. CasualTagger KWIC search + short-cut tag insertion Batch Processing Tagging (TreeTagger, STF CoreNLP, MeCab) Tokenizing (macOS built-in tagger) Sentence Splitting (macOS built-in tagger, STF CoreNLP) Misspelling detection (non-dictionary words) compound-2-word variation detection
20. Finally, the main point!
21. Today’s highlights Corpus file management - going beyond loading files More informative word/n-gram lists Individual file word/n-gram lists 2 More keyword extraction functions - going beyond LL/χ Visualizing frequency data - incl. multivariate analyses Some other niche functions
22. Corpus File Management
23. Corpus File Management Accept Drag & Drop!
24. Where were those files I want to use? Of course, you can use Spotlight to search them, but… If the application remembers where they are…
25. Corpus File Management
26. Corpus File Management
27. More Informative Word/n-gram List
28. Proportion, # of Files, % of Files
29. Proportion, # of Files, % of Files
30. 3-grams / 4-grams comparison
31. p-frames
32. Lemma / Spelling variations/ multi-word
33. Lemma / Spelling Variation
34. Multi-word Recognition
35. Multi Filters
36. Search Word List
37. Individual File Word/n-gram Lists
38. Individual File/Corpus Word/n-gram Lists
39. More keyword extraction functions
40. Sample ICNALE - Writing Written learner English corpus College students in Asian countries/regions JPN, CHN, HKG, IDN, KOR, PAK, PHI, SIN, THA, TWN Two topics Ave. 220-230 words
41. Simple Keyword Extraction (LL)
42. Multiple Keyword Statistics
43. Key-2-gram Extraction
44. Key-3-gram Extraction
45. Going Beyond Simple Keyness
46. Multiple Keyness Comparisons
47. TF-IDF (term-frequency / inverse document frequency)
48. TF-IDF (term-frequency / inverse document frequency)
49. Mann-Whitney-U (using multiple files)
50. Random Forest (1000 words)
51. Visualizing Frequency Data
52. CasualConc integrates
53. Word Cloud
54. Let’s go back to keyword analysis
55. Simple Keyword Extraction (LL)
56. Keyword analysis is done How can you check the validity? Let’s see how well they separate the groups Use the LL result above 6.63 (p < .01)
57. Principle Component Analysis (PCA)
58. Cluster Analysis
59. What about different ability levels?
60. Principle Component Analysis (PCA)
61. Cluster Analysis
62. Heat Map
63. By the way, do you check raw data? Not a lot of people really look at the data… Let’s check the use of “I think”
64. Box Plot
65. Histogram
66. What about a word that are too common to be looked at?
67. Box Plot
68. Histogram
69. Some other misc features…
70. Relative Position Plot (paragraph) therefore / however Japanese Non-Japanese ENS
71. Relative Position Polot (sentence) therefore / however Japanese Non-Japanese ENS
72. Vocab Profiler
73. Tokenizer
74. Tokenizer
75. Proper Name Extraction
76. The Way Forward Utilizing Stanford CoreNLP - dependency information
77. Grammatical Relationship Search