A map of 256 tags of StackOverflow (at BigDive wrap-up)

BigDive (data science, visualization and scalable programming workshop): http://www.bigdive.eu/
Project (a map of programming language, techniques, devices, concepts and issues): https://github.com/stared/tag-graph-map-of-stackexchange/wiki

Transcript

1. A map of 256 tags (of Stack Overflow) Piotr Migdał BigDive wrap-up, 1-26 Oct 2012, Torino, Italy https://github.com/stared/tag-graph-map-of-stackexchange/wiki http://www.bigdive.eu/
4. SDUVLQJ RV[ ZLQGRZVSKRQH DSL WH[W [FRGH G\QDPLF UHVW GDWHWLPH QRGHMV VHVVLRQ OLQTWRVTO P\VTO TXHU\ DUFKLWHFWXUH [PO LSDG MDYDHH UXE\RQUDLOV IODVK IRUPV RSHQFY KWPO PDWODE PXOWLWKUHDGLQJ LPDJH KRPHZRUN FVV IOH[ VWULQJ S\WKRQ PHPRU\PDQDJHPHQW ZFI LQWHUQHWH[SORUHU GDWDELQGLQJ ZLQGRZV VVO IDFHERRN GHSOR\PHQW FDNHSKS PVDFFHVV FRPSLOHU GHVLJQSDWWHUQV VLOYHUOLJKW VRUWLQJ TW DQGURLGOD\RXW JXL QHW OLVW F HYHQWV DUUD\V YHUVLRQFRQWURO EODFNEHUU\ DVSQHWPYF VHFXULW\ OLVWYLHZ PRGUHZULWH EXWWRQ YLHZ YLVXDOVWXGLR FRRNLHV JRRJOH VWRUHGSURFHGXUHV QHWEHDQV VSULQJ WZLWWHU WHVWLQJ ZHEVHUYLFHV LPDJHSURFHVVLQJ GHVLJQ XVHULQWHUIDFH SOXJLQV ORRSV ZHEDSSOLFDWLRQV VLOYHUOLJKW VTOVHUYHU FDQYDV ZSI ZLQGRZV VFDOD SKRQHJDS DFWLRQVFULSW GHOSKL VTOVHUYHU GOO XQLWWHVWLQJ U WHPSODWHV JHQHULFV VSULQJPYF ERRVW HPDLO PDWK DQGURLG UHGLUHFW KLEHUQDWH DVVHPEO\ [VOW ZLQDSL MDYDVFULSW F PDYHQ FRFRDWRXFK DMD[ KWPO XEXQWX MTXHU\XL OD\RXW SRVW GLY EDVK JRRJOHFKURPH YED SRVWJUHVTO FRGHLJQLWHU GUXSDO YEQHW JRRJOHDSSHQJLQH JZW PYF VZLQJ SGI XLYLHZ MTXHU\DMD[ FRUHGDWD ILUHIR[ UXE\RQUDLOV HQFRGLQJ VHDUFK YDULDEOHV YLP FRFRD DSDFKH XQL[ KDVNHOO XUO OLQX[ WDEOH LRV VRFNHWV VHOHFW UHJH[ JLW QHWZRUNLQJ VHUYLFH RSHQJO FVV HQFU\SWLRQ LSKRQHVGN SHUO YLVXDOVWXGLR MSD MVRQ ZRUGSUHVV LSKRQH PHPRU\ HPDFV LLV [SDWK VKHOO LLV F [DPO F OLQT RRS JUDSKLFV WVTO ORJJLQJ MTXHU\SOXJLQV UXE\ MTXHU\ H[FHSWLRQ MRLQ DOJRULWKP DVSQHWPYF KWDFFHVV SRLQWHUV GRP HFOLSVH GDWDEDVHGHVLJQ JRRJOHPDSV ZLQIRUPV ]HQGIUDPHZRUN LIUDPH DFWLYHUHFRUG MTXHU\PRELOH YLGHR SRZHUVKHOO EURZVHU IXQFWLRQ VTOLWH ILOH PYYP GDWDVWUXFWXUHV ELQGLQJ DQW REMHFWLYHF PDJHQWR XLWDEOHYLHZ FXUO ILOHXSORDG IDFHERRNJUDSKDSL JFF MDYDVFULSWHYHQWV LQKHULWDQFH DVSQHW VTOVHUYHU VKDUHSRLQW H[FHO HQWLW\IUDPHZRUN DSSOLFDWLRQ DXWKHQWLFDWLRQ DQLPDWLRQ SKS DFWLRQVFULSW SHUIRUPDQFH QKLEHUQDWH VRDS YLVXDOVWXGLR GHEXJJLQJ MVI KWWS PRQJRGE FODVV REMHFW LRV XSGDWH DVSQHWPYF GDWDEDVH MVS RUDFOH VTO GDWH VHULDOL]DWLRQ SKS GDWD H[WMV [FRGH VYQ YDOLGDWLRQ JULGYLHZ UHIOHFWLRQ WRPFDW MDYD PRELOH RSWLPL]DWLRQ HQWLW\IUDPHZRUN JUDLOV GMDQJR YLVXDOF FDFKLQJ DXGLR VHUYOHWV
5. P(tag1 and tag2) P(tag1)P(tag2) 4 an edge between two tags when
8. 99.8% [binding] 99.7% [data-binding] 99.7% [activerecord] 99.7% [jpa] 99.6% [uiview] 99.6% [uitableview] 99.6% [core-data] 99.6% [reflection] 99.5% [xaml] 99.5% [join] ... 95.6% [google] 95.3% [design] 95.2% [homework] 95.1% [algorithm] 94.6% [architecture] 94.6% [linux] 94.4% [networking] 94.0% [application] 94.0% [math] 89.9% [ubuntu] 1754 [visual-c++] 1608 [scala] 1530 [compiler] 1513 [reflection] 1440 [generics] 1392 [winapi] 1380 [gcc] 1359 [haskell] 1351 [optimization] 1347 [git] ... 308 [service] 306 [android] 305 [variables] 300 [data] 293 [div] 283 [homework] 259 [wordpress] 248 [button] 240 [iphone-sdk-4.0] 182 [application] 338 [scala] 285 [version-control] 255 [unit-testing] 234 [asp.net-mvc] 212 [.net] 196 [reflection] 193 [generics] 191 [vim] 186 [haskell] 184 [git] ... 23 [loops] 22 [dynamic] 22 [div] 22 [service] 21 [google] 21 [data] 20 [opencv] 20 [facebook] 15 [button] 8 [application] Tag health (open questions) Average reputation of the poster Typical reputation of the poster As healthy as a horse Not the bad either... 1337-free zone A lot of strong guys Not that many newbies n00b questions, mostly Out of 256 most popular tags at Stack Overflow 0.6% of all tags cover 90% of content
9. 2.99 [post] 2.98 [dynamic] 2.97 [text] 2.97 [data] 2.95 [uiview] 2.90 [table] 2.89 [view] 2.87 [div] 2.86 [button] 2.86 [object] ... 1.72 [xslt] 1.63 [powershell] 1.62 [emacs] 1.61 [matlab] 1.60 [blackberry] 1.60 [git] 1.54 [haskell] 1.48 [vim] 1.43 [magento] 1.39 [r] 1014 [data-binding] 992 [binding] 980 [jquery-ajax] 974 [mvvm] 967 [xslt] 954 [jpa] 936 [homework] 934 [hibernate] 934 [inheritance] 930 [nhibernate] ... 511 [pdf] 507 [shell] 503 [drupal] 489 [iphone-sdk-4.0] 487 [browser] 486 [application] 473 [unix] 471 [blackberry] 470 [emacs] 424 [vim] Average number of additional tags 0.90 [ios5] 0.87 [jquery-mobile] 0.87 [phonegap] 0.85 [ios] 0.84 [node.js] 0.84 [facebook-graph-api] 0.84 [android-layout] 0.83 [asp.net-mvc-3] 0.82 [css3] 0.80 [mongodb] ... 0.59 [.net] 0.59 [architecture] 0.58 [design] 0.58 [svn] 0.58 [visual-studio] 0.57 [sharepoint] 0.57 [sql-server-2005] 0.54 [version-control] 0.54 [linq-to-sql] 0.51 [visual-studio-2008] Typical body length (in characters) Freshness (0 to 1) Almost Meta Tags? Lonely tags... Up to the point Well, I need to explain... New stuff, wee! Somewhat outdated http://bit.ly/map-of-so
10. And Twitter? • Yeah, I wanted to map-reduce a bit... • And I have tried...
11. GDPQLWVWUXH WZLWIDNWD SLOLKVDWX UHYLV PLWVXGRPRH IDNWDQ\D PREVWHUZRUOG PDLVIROORZHUV HSLFWZHHWV FHULWDFLQWD QRDEXVRDQLPDO SDZSDZW\ HVFRVDGHMRYHQHV QLFRMLQVHL RQHRNURFN VWRSPHURNRN WDQ\DNHNDPX LKDWHTXRWHV PHJXBJDPH SHUVLE RPJWKDWVDWHHQ ELJIROORZ FRORSOBPVJ SL[LY WUJ RKWHHQTXRWHV VXWDGRUD VWDUGULYHU SHSDWDK YDOHQWLQHZLVK SODQHWDUV LYHWHQRSODQHWDUV WDXUXV RSJR\D OH\VLQGH E]Y JR\D SLVFHV KDODPDGULG J DULHV UHDOPDGULG VHFUHWVWRU\ ]RGLDFIDFWV OLEUD VLJQRVIRGDV FRSDDFE WZLWWHURII SHL[HV WZH[LW OH RSHUDFLRQJR\D H\HVZLGHVKXW WZLWLWLRQ QDWLRQV     KDSS\ELUWKGD\DVWRQ L P F V DVWRQVELUWKGD\ VDOH S MOV XVD IDYRULWPX VD\DEHUWDQ\D KRXURXBDQLPH   SXPDV MDQ IE LUDQ HJ\SW WFRW [E EDKPDQ OFY S IHE QHZV DWWKHHQGRIWKHGD\ DOJHULD IUDVHVIDFHWZLW LUDQHOHFWLRQ JRRJOH EDKUDLQ WF WZLWWHU \HPHQ IDFHERRN WDKULU LSKRQH SKRWR QRZSOD\LQJ PXVLF YDOHQWLQH QS ZLQ WYDVDKL DQGURLG YDOHQWLQHVGD\ LSRG MRE MREV QKN TXRWH LWXQHV ORYH VD JLYHDZD\ YLGHR EXVLQHVV WZHHWP\MREV IUHH WUDYHO EORJ LSDG IXMLWY KHDOWK ODVWIP SKRWRJUDSK\ FHOWLFV VXE DVVXPHSHODQ]D QZ VWULNHIRUFH RQHRIP\IROORZHUV VPK ZWI RRPI ORO VXEWZHHW SUHJXQWDVVLQUHVSXHVWD IDWR IRUHYHUDORQH EEE PHQWLRQNH QDVFDU HVTXHQWD VDODKJDXO WHDPFHOWLFV EEE LWDXFDUGUDFKD JDURWDQREEE QED IXL IDLO SDUWLX  WKDWLVDOO HXQXQFD UDQGRP FXPDQQDQ\D VRDOERZERZ KHDW ODNHUV EUDVLO IPO WHQVR IROORZ II VRXJRIROORZ MIE IUDVHVGHHVFROD DXWRIROORZ QI RSHQIROORZ IROORZEDFN QRZIROORZLQJ KEX RKD\R IXFN VRXQGFORXG YH]HV IROORZGDLERV\X  XS ELHEHUIDFW IROORZPHMS W LIROORZEDFN UHWZHHW FRPPXQLW\ WZLWEDFNU UW SRUQ WHDPEUHH]\ WIE DGD\ LQVWDQWIROORZEDFN WHDPIROORZEDFN VH[ SLFNRQH LQVWDQWIROORZ WHDPDXWRIROORZ GRQJKDHGD\V VKRXWRXW PXVWIROORZ JUDPP\IRUULUL QLFRYLGHR IROORZPH  JUDPP\V ELHEHUIDFWV IDFW QHYHUVD\QHYHUG QHYHUVD\QHYHU JRRGOXFNEUXQR WKDWVORYH JUDPP\V MXVWVD\LQ \RXZLOOQHYHUFDWFKPH Q\IZ JUDPP\IRUPM IROORZPHG RWOLYH FDQG\KHDUWUHMHFWV GUHDPELJ GHDG QVQG PXVHZLOOEHYLFWRULRXV EDIWDV JDJDJUDPP\V QRZZDWFKLQJ PLOH\ZRUOGZLGHWRXU FKHDSYGD\JLIWV SHUU\JUDPP\V GRL URGDMD QHYHUVD\QHYHUVXQGD\ IROORZXVG LPDJLQH \RXNQRZLORYH\RXZKHQ RIIOLQH UHMHFWHGFDQG\KHDUWV GD\VRINLVVHV \RXFDQWKROODLI ODWLQDPHULFDQHHGVVZLIW KDYHXHYHU SDUDPRUHJUDPP\V JUDPP\ JUDPP\OLYH ELHEHU WZLWSLFW MXVWVD\LQJ VQO JUDPP\IRUELHEHU UHDOWDON EDIWD ERUQWKLVZD\
12. an edge between two tags when P(tag1|tag2) 0.5% P(tag2|tag1) 0.5% or
14. Thank you all! • It was awesome! Proof? • I wanted to read 1.5 books and finish 1.5 papers... • And what’s next? • Make it interactive! (e.g. Gephi to D3.js) http://www.bigdive.eu/
15. And BTW... (a slideshow injection)
16. Offtopicarium ...or just google it 4-6 Jan 2013 Warsaw, Poland An unconference for ~30. 20min on any topic. phd students, programmer, startupers , social activists, any other geeks http://offtopicarium.wikidot.com/ (almost) no fee... ...but bring a sleeping bag
17. SDUVLQJ RV[ ZLQGRZVSKRQH DSL WH[W [FRGH G\QDPLF UHVW GDWHWLPH QRGHMV VHVVLRQ OLQTWRVTO P\VTO TXHU\ DUFKLWHFWXUH [PO LSDG MDYDHH UXE\RQUDLOV IODVK IRUPV RSHQFY KWPO PDWODE PXOWLWKUHDGLQJ LPDJH KRPHZRUN FVV IOH[ VWULQJ S\WKRQ PHPRU\PDQDJHPHQW ZFI LQWHUQHWH[SORUHU GDWDELQGLQJ ZLQGRZV VVO IDFHERRN GHSOR\PHQW FDNHSKS PVDFFHVV FRPSLOHU GHVLJQSDWWHUQV VLOYHUOLJKW VRUWLQJ TW DQGURLGOD\RXW JXL QHW OLVW F HYHQWV DUUD\V YHUVLRQFRQWURO EODFNEHUU\ DVSQHWPYF VHFXULW\ OLVWYLHZ PRGUHZULWH EXWWRQ YLHZ YLVXDOVWXGLR FRRNLHV JRRJOH VWRUHGSURFHGXUHV QHWEHDQV VSULQJ WZLWWHU WHVWLQJ ZHEVHUYLFHV LPDJHSURFHVVLQJ GHVLJQ XVHULQWHUIDFH SOXJLQV ORRSV ZHEDSSOLFDWLRQV VLOYHUOLJKW VTOVHUYHU FDQYDV ZSI ZLQGRZV VFDOD SKRQHJDS DFWLRQVFULSW GHOSKL VTOVHUYHU GOO XQLWWHVWLQJ U WHPSODWHV JHQHULFV VSULQJPYF ERRVW HPDLO PDWK DQGURLG UHGLUHFW KLEHUQDWH DVVHPEO\ [VOW ZLQDSL MDYDVFULSW F PDYHQ FRFRDWRXFK DMD[ KWPO XEXQWX MTXHU\XL OD\RXW SRVW GLY EDVK JRRJOHFKURPH YED SRVWJUHVTO FRGHLJQLWHU GUXSDO YEQHW JRRJOHDSSHQJLQH JZW PYF VZLQJ SGI XLYLHZ MTXHU\DMD[ FRUHGDWD ILUHIR[ UXE\RQUDLOV HQFRGLQJ VHDUFK YDULDEOHV YLP FRFRD DSDFKH XQL[ KDVNHOO XUO OLQX[ WDEOH LRV VRFNHWV VHOHFW UHJH[ JLW QHWZRUNLQJ VHUYLFH RSHQJO FVV HQFU\SWLRQ LSKRQHVGN SHUO YLVXDOVWXGLR MSD MVRQ ZRUGSUHVV LSKRQH PHPRU\ HPDFV LLV [SDWK VKHOO LLV F [DPO F OLQT RRS JUDSKLFV WVTO ORJJLQJ MTXHU\SOXJLQV UXE\ MTXHU\ H[FHSWLRQ MRLQ DOJRULWKP DVSQHWPYF KWDFFHVV SRLQWHUV GRP HFOLSVH GDWDEDVHGHVLJQ JRRJOHPDSV ZLQIRUPV ]HQGIUDPHZRUN LIUDPH DFWLYHUHFRUG MTXHU\PRELOH YLGHR SRZHUVKHOO EURZVHU IXQFWLRQ VTOLWH ILOH PYYP GDWDVWUXFWXUHV ELQGLQJ DQW REMHFWLYHF PDJHQWR XLWDEOHYLHZ FXUO ILOHXSORDG IDFHERRNJUDSKDSL JFF MDYDVFULSWHYHQWV LQKHULWDQFH DVSQHW VTOVHUYHU VKDUHSRLQW H[FHO HQWLW\IUDPHZRUN DSSOLFDWLRQ DXWKHQWLFDWLRQ DQLPDWLRQ SKS DFWLRQVFULSW SHUIRUPDQFH QKLEHUQDWH VRDS YLVXDOVWXGLR GHEXJJLQJ MVI KWWS PRQJRGE FODVV REMHFW LRV XSGDWH DVSQHWPYF GDWDEDVH MVS RUDFOH VTO GDWH VHULDOL]DWLRQ SKS GDWD H[WMV [FRGH VYQ YDOLGDWLRQ JULGYLHZ UHIOHFWLRQ WRPFDW MDYD PRELOH RSWLPL]DWLRQ HQWLW\IUDPHZRUN JUDLOV GMDQJR YLVXDOF FDFKLQJ DXGLR VHUYOHWV https://github.com/stared/tag-graph-map-of-stackexchange/wiki