Introduction of Web scraping for PHP users

slides for Japanese PHP Conference 2013
http://phpcon.php.gr.jp/w/2013/#program

Transcript

1. Introduction of Web scraping for PHP users
2. 15 years ago
3. 15 years ago 　 PHP 3.0.4 release, includes get_meta_tags().
4. It works.. <?php get_meta_tags("http://example.com/"); array(1) { 'viewport' => string(35) "width=device-width, initial-scale=1" }
5. It works, sometimes! <?php get_meta_tags("http://www.discogs.com/"); PHP Warning: get_meta_tags(http://www.disc ogs.com/): failed to open stream: HTTP request failed! 500 Client Refused
6. You are file_get_contents() fanboy
7. <?php get_meta_tags("data://text/html,". file_get_contents( "http://www.discogs.com/", false, stream_context_create( ["http" => ["header" => "User-Agent: Mozilla/4.0"] ] ) ) );
8. It works, too $php -d user_agent="Mozilla/4.0" \ -r 'get_meta_tags("http://www.discogs.com/");'
9. You would FEEL there are some problem.
10. "do separation of concerns! GET HTML & parse HTML."
11. "do separation of concerns! GET HTML & parse HTML." Doubt! Doubt! Doubt!
12. Handling Request & Handling Response
13. HTTP Request
14. Sorry, today I don't talk about HTTP Request side. (no time talk about HttpClient, Spider, crawler in 15 minutes)
15. HTTP ReSPONSE
16. HTTP ReSPONSE HeaDERS - BODY - Not only HTML ;-)
17. SCRAP SCRAPING FROM RESPONSE!
18. There are some fact you should take a act
19. There are some fact you should take a act • ConTENT ENCODING • ChaRSET ENCODING • NORMALIZE HTML • EXTRACTING FROM HTML • SOLVE CONTEXT
20. CONTENT ENCODING
21. CONTENT ENCODING TODAY, WE ALREADY ACCEPTED IT.
22. CONTENT ENCODING • gzip • deflate • compress • identity I recommend using good Response handlers before struggling.
23. CONTENT ENCODING • gzip • deflate I recommend using good Response handlers before struggling. pear/HTTP_Request2 zendframework/zend-http guzzle/guzzle
24. CHARSET ENCODING
25. CHARSET ENCODING mb_convert_encoding(“UTF-8”, “auto”, $html)
26. CHARSET ENCODING mb_convert_encoding(“UTF-8”, “auto”, $html) This is not best way. You had already got hint in Response Headers & html's meta nodes.
27. CHARSET ENCODING <?php header("Content-Type: text/html; charset=Shift-JIS"); ?> ①②③④⑤ But, Don't forget, Most of Japanese PHP users do LIE.
28. CHARSET ENCODING diggin/diggin-http-charset diggin/guzzle-plugin-AutoCharsetEncodingPlugi I hope my component will help you.
29. NORMALIZE HTML
30. NORMALIZE HTML
31. NORMALIZE HTML before parse as HTML, you can fix it.
32. NORMALIZE HTML before parse as HTML, you can fix it. • php-ext/tidy, HTMLParser • other beautifiers • manually :-(
33. EXTRACTING FROM HTML
34. EXTRACTING FROM HTML Yes, there are several way in PHP. • PCRE / String Functions • dom • SimpleXML • php-ext/html_parse
35. EXTRACTING FROM HTML Mostly, boredom for entire HTML. • PCRE / String Functions • dom • SimpleXML • php-ext/html_parse
36. EXTRACTING FROM HTML DOM is a API FOR HTML & XML • PCRE / String Functions • dom • SimpleXML • php-ext/html_parse
37. EXTRACTING FROM HTML Xpath is your friend • PCRE / String Functions • dom • SimpleXML • php-ext/html_parse
38. EXTRACTING FROM HTML Remember PHP's Feature, DOMXPath:: registerPhpFunctions • PCRE / String Functions • dom • SimpleXML • php-ext/html_parse
39. Solve Context
40. Solve Context You will need solve context from got response • Filtering extracted result for Domain.
41. Solve Context Don't reinvent the wheel • Resolve relative URI / RFC-3986 - pear/Net_URL2, zendframework/zend-uri supports it
42. Solve Context Don't reinvent the wheel • “Databases” that helps you - wedata - OSS's repositories (not only PHP)
43. LIBRALIES • behat/mink • goutte, symfony/browserKit • zendframework/zend-dom • diggin-scraper • simple_html_dom • phpQuery • fluentDOM • php-jsonpointer • beberlei/phpricot
44. Today, web is under control by JavaScript
45. JAVASCRIPT We need "REAL" BROWSER for AUTOMATION • Selenium • PhantomJS / CasperJS • SlimerJS
46. You have a chance to survive with php.
47. Move forward, PHP. • HTML5 • HTTP 2.0 / SPDY • Browser binding ardemiranda/WebKitGtk • concurrent programing, asynchronous • collective intelligence • NLP natural language processing
48. Deeper and Deeper • elazar/web-scraping-with-php https://github.com/elazar/web-scraping-with-php • Accessing Web Resources with PHP http://joind.in/3386 • Spidering Hacks http://www.oreilly.co.jp/books/4873111870/ • fuba: exthtml https://fuba.jottit.com/exthtml • kitamomonga http://d.hatena.ne.jp/kitamomonga/ • The Architecture of Open Source Applications selenium - https://github.com/m-takagi/aosa-ja
49. Thanks