BoT2013 海量資料時代的網路分析

Transcript

1. 海量資料時代的網路分析 BoT2013
   ୋʞ̨֣ᝄਜ
   Botnet
   ਈ಻ၾԣطҦஔ޼ীึ ॽख͍ Allen Own allenown@devco.re
2. Who Am I ॽख͍ (Allen Own) allenown@devco.re DEVCORE Ꮦ˃੒ဧ
   ੂБڗ CHROOT ϓࡰ HITCON ̨ᝄᎡ܄ϋึਓᐼ̜ NISRA ༟τྠඟ௴፬ɛ ༟τҦঐږ޷ᆤᘩᒄϋڿࠏ
3. ආБִ݁ఊЗeΆุഃသீ಻༊ਖ਼ࣩ ΂Άุeኪஔʿִ݁ఊЗᑺࢪʿᚥਪ ༟ৃτΌӻ୕޼Ӻʿක೯ ၣ༩τΌ஝ྌܔໄ Ꭱ܄ҸᏘ˓جeᎡ܄৛ᔳ ҸᏘ೻όeܝژ೻όක೯ၾ޼Ӻ EC-Council Certified Ethical Hacker Computer Hacking Forensic Investigator
4. 2013/07/19 (五) ~ 20 (六) 中央研究院 ⼈人⽂文社會科學館
6. What is Big Data?
10. Big Data Big data[1][2] is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, [3] search, sharing, transfer, analysis,[4] and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions."[5][6][7]
11. How Big Data fights back against APTs and Malware? http://www.seculert.com/blog/2013/05/how-big- data-fights-back-against-apts-and-malware.html http://info.umbrella.com/infographic-using-big-data- for-malware-protection.html
12. What is Big Data?
13. What is Big Data? BIG
14. How about... 9TB?
15. How about... 9TB?
16. Internet Census 2012 http://internetcensus2012.bitbucket.org/ While playing around with the Nmap Scripting Engine (NSE) we discovered an amazing number of open embedded devices on the Internet. Many of them are based on Linux and allow login to standard BusyBox with empty or default credentials. We used these devices to build a distributed port scanner to scan all IPv4 addresses. These scans include service probes for the most common ports, ICMP ping, reverse DNS and SYN scans. We analyzed some of the data to get an estimation of the IP address usage.
17. Internet Census 2012 http://internetcensus2012.bitbucket.org/download/ internet_census_2012.torrent Decompressing all data results in 9TB of raw logfiles, but this code can also be used to recompress the data into gzip files. The gziped dataset should be ~1.5TB.
20. Hilbert Browser http://internetcensus2012.bitbucket.org/hilbert/ index.html
21. We prepared 26TB. Special thanks to GD.
23. We spent 2 months to decompress parts of zpaq files.
25. How to use?
26. How to use? ANS: Use your force.
28. grep -e "Apache/2.2.3" *
29. grep -e "Apache/2.2.3" * 1MBJOUFYU
    3VMFT
30. But... it took 15 minutes... (one single file)
31. Search Faster!
33. elasticsearch http://www.elasticsearch.org/ flexible and powerful PQFO
    TPVSDF, distributed real- time search and analytics engine for the cloud. Case Study: Fog Creek, Stack Overflow, SoundCloud, StumbleUpon, Github, foursquare, Wordpress, salesforce
35. You Know, for Search.
36. I'm too lazy to code.
38. logstash http://logstash.net logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. *U
    JT
    GVMMZ
    GSFF
    BOE
    GVMMZ
    PQFO
    TPVSDF
    
41. It works!
42. I need stronger UI/UX!
44. Kibana http://kibana.org Kibana is an PQFO
    TPVSDF (MIT License), browser based interface to Logstash and ElasticSearch. Once you have those in place, Kibana is a breeze to install and configure (really, I swear). And as you'll see below, none too hard to operate. Check out the screenshots for an idea of what Kibana is all about.
50. LIVE DEMO
51. "OBMZTJT
    PG
    *OUFSOFU
    $FOTVT
     4FBSDI
    LOPXO
    WVMOFSBCMF
    TFSWFST
    
    BQQMJDBUJPOT "OBMZ[F
    *1
    BDUJWJUJFT %FUFDU
    CPUOFU
    BDUJWJUJFT 'JOE
    TQFDJGJD
    EPNBJO
    IPTUT
52. 2
    
    "
53. 5IBOLT 翁浩正 Allen Own allenown@devco.re