yapc_eu_2012

Perl programming in Yandex.Direct

Transcript

1. Intro Deployment Testing DBTools Plack::* Bonus Yandex.Direct: our successful anti-modern Perl project Elena Bolshakova, Oleg Komarov 2012-08-20 1/36
2. Intro Deployment Testing DBTools Plack::* Bonus http://yandex.ru 2/36
3. Intro Deployment Testing DBTools Plack::* Bonus http://yandex.com 3/36
4. Intro Deployment Testing DBTools Plack::* Bonus There are ads on yandex 4/36
5. Intro Deployment Testing DBTools Plack::* Bonus http://direct.yandex.com 5/36
6. Intro Deployment Testing DBTools Plack::* Bonus Presentation available at https://speakerdeck.com/u/komarov/p/yapc-eu-2012 http://bit.ly/N853eJ 6/36
7. Intro Deployment Testing DBTools Plack::* Bonus Team and codebase since 2003 14 perl programmers 310 KSLOC Perl 7/36
8. Intro Deployment Testing DBTools Plack::* Bonus Production Web interfaces + APIs interfaces for clients, sales managers, agencies etc (> 10 user roles + complicate hierarchy) > 12 mln. transactions/day DB: app. 41 × 109 records Perl: 35 servers of 5 types DB: 50 servers Ubuntu + deb-packages app. 150 our own debs 8/36
9. Intro Deployment Testing DBTools Plack::* Bonus keeping packages ready easy cases: debosh, instant deb packaging Files do deploy (perl scripts, crontabs etc.) + 5 lines of metadata + debosh = .deb ready to upload and install See on github: https://github.com/komarov/debosh For complicated cases: custom script for every package And what’s your way? 9/36
10. Intro Deployment Testing DBTools Plack::* Bonus Keeping packages in order What version of X do we have on DB servers? When version N of X was installed on dev server? dpkg-monitor: script for gathering all that data + web-interface It seems to us to be a good practice And what’s your way? 10/36
11. Intro Deployment Testing DBTools Plack::* Bonus Keeping packages in order 11/36
12. Intro Deployment Testing DBTools Plack::* Bonus Keeping DB in order: DB migrations about 5 schema or data migrations/week special files under ’deploy’ directory 12/36
13. Intro Deployment Testing DBTools Plack::* Bonus Keeping DB in order: DB migrations /project /deploy /2012_07_06_add_client_limit.migr # schema migration /2012_07_10_convert_favorites.pl # data migration ... /lib /t ... It seems to us to be a good practice And what’s your way? 13/36
14. Intro Deployment Testing DBTools Plack::* Bonus Keeping DB in order: DB migration example 2012_07_06_add_client_limit.migr: { type => ’sql’, heavy => 1, when => ’before’, time_estimate => ’3 min’, sql => ’alter table clients add column some_limit int’, } And what’s your way? 14/36
15. Intro Deployment Testing DBTools Plack::* Bonus Release testing continuous delivery compliance with the Sarbanes-Oxley Act JIRA as bugtracker Thanks CPAN for JIRA::Client branches in svn 15/36
16. Intro Deployment Testing DBTools Plack::* Bonus Unit tests 1 After every commit to mainline (+ personal notifications on fail) What to test: perl server-side js perl + DB (special DB for unit-tests, no mock) It seems to us to be a good practice http, tcp interaction Thanks CPAN for Test::HTTP::Server, AnyEvent::Socket 16/36
17. Intro Deployment Testing DBTools Plack::* Bonus Unit tests 2: non-functional Non-functional tests: encoding of source code i18n (documentation|test) coverage (perl|TT) syntax deprecated subs crontabs extra or missing js & TT (used but absent, present but unused) ... – whatever we can imagine It seems to us to be a good practice 17/36
18. Intro Deployment Testing DBTools Plack::* Bonus DBTools Our inhouse module (MySQL-specific) to handle db configs connects and reconnects boilerplate sql construction Also it is replication-aware supports transactions makes unit-tests easier 18/36
19. Intro Deployment Testing DBTools Plack::* Bonus DBTools: easy things do_insert_into_table(DB, "names", {id => 2, name => ’bb’} ); get_one_field_sql(DB, "SELECT count(*) FROM names"); get_one_column_sql(DB, "SELECT name FROM names"); get_all_sql(DB, "SELECT id, name FROM names"); 19/36
20. Intro Deployment Testing DBTools Plack::* Bonus DBTools: easy things my $id; do_in_transaction { $id = do_insert_into_table(DB, "names", {name => ’gg’} ); do_update_table(DB, "totals", {count__dont_quote => ’count + 1’}, where => {table_name => ’names’} ); }; # do smth with $id 20/36
21. Intro Deployment Testing DBTools Plack::* Bonus DBTools: sometimes you know better get_hashes_hash_sql(DB, [ "SELECT STRAIGHT_JOIN t1.id, t1.name, t2.status, t3.date FROM t1 JOIN t2 ON t1.id = t2.id JOIN t3 ON t2.some_id = t3.id", WHERE => { id => [1, 2, ..., 1000], ’t3.is_active’ => 1 } ]); 21/36
22. Intro Deployment Testing DBTools Plack::* Bonus DBTools no orm. why? full control over executed sql no FOREIGN KEYS in db tables have big rows, SELECT * is bad why not SQL::Abstract? you still need to manipulate with dbh and sth you can’t construct complex JOINs, can you? 22/36
23. Intro Deployment Testing DBTools Plack::* Bonus DBTools: one more example UPDATE names SET name = CASE id WHEN 1 THEN ’aa’ WHEN 2 THEN ’bb’ ... END WHERE id IN (1, 2, ...) 23/36
24. Intro Deployment Testing DBTools Plack::* Bonus DBTools: one more example Construct you CASE with sql_case(DB, "id", {1 => ’a’, 2 => ’b’, ..., }) Pretty simple, right? 24/36
25. Intro Deployment Testing DBTools Plack::* Bonus DBTools: one more example Construct you CASE with sql_case(DB, "id", {1 => ’a’, 2 => ’b’, ..., }) Pretty simple, right? When ”...” hides thousands, the performance might surprise you. 24/36
26. Intro Deployment Testing DBTools Plack::* Bonus DBTools: one more example Too many numeric keys? Construct binary tree of CASEs! IF(id < 50, IF(id < 25, CASE..., CASE...), IF(id < 75, CASE..., CASE...) ) 25/36
27. Intro Deployment Testing DBTools Plack::* Bonus Plack We love Plack 26/36
28. Intro Deployment Testing DBTools Plack::* Bonus Plack::Middleware::* ACL Auth Captcha MemLimit Reqid Usertrack CmdLogs IPREG DevTools 27/36
29. Intro Deployment Testing DBTools Plack::* Bonus P::M::DevTools use Aspect (); sub call { my ($self, $env) = @_; my @devtools_sections = qw/sql http/; for my $section (@devtools_sections) { $env->{"devtools.$section"} = []; } my @hooks; ... 28/36
30. Intro Deployment Testing DBTools Plack::* Bonus P::M::DevTools ... push @hooks, Aspect::before { my (undef, $url) = $_->args; push @{$env->{’devtools.http’}}, $url; } Aspect::call qr/^LWP::UserAgent::(?:get|post)$/; push @hooks, Aspect::around { # DBI->trace } Aspect::call qr/::exec_sql$/; ... 29/36
31. Intro Deployment Testing DBTools Plack::* Bonus P::M::DevTools ... push @hooks, Aspect::before { my ($response_type, $response) = $_->args; my $devtools = { map {$_ => $env->{"devtools.$_"}} @devtools_sections }; ... # $response->{_devtools} = $devtools; } Aspect::call qr/::respond$/; return $self->app->($env); } 30/36
32. Intro Deployment Testing DBTools Plack::* Bonus Plack::ResponseHelper A very thin layer that abstracts Plack’s specifics. Comes with P::RH::Attachment, P::RH::JSON, P::RH::Redirect, P::RH::Text DRY, give it a name and write a helper! See on CPAN: https://metacpan.org/module/Plack::ResponseHelper 31/36
33. Intro Deployment Testing DBTools Plack::* Bonus Plack::ResponseHelper Your ”hello world” app.psgi use Plack::ResponseHelper text => ’Text’; sub { respond text => ’Hello world!’; } 32/36
34. Intro Deployment Testing DBTools Plack::* Bonus Plack::ResponseHelper Your real app use Plack::ResponseHelper csv => [’Attachment’, {content_type => ’text/csv; charset=utf-8’}], json => ’JSON’, redirect => ’Redirect’, template => [’Template’, {get_tt_object => sub {...}}], page404 => [’Redirect’, {default_location => ’/404.html’}]; sub { # routing, etc respond $controller->($env); } 33/36
35. Intro Deployment Testing DBTools Plack::* Bonus Thanks! Any questions? Drop us a line: lena-san@yandex-team.ru oregano@yandex-team.ru https://speakerdeck.com/u/komarov/p/yapc-eu-2012 http://bit.ly/N853eJ 34/36
36. Intro Deployment Testing DBTools Plack::* Bonus XMLRPC::Transport::HTTP::Plack sub { my $env = shift; my $r = Plack::Request->new($env); my $cmd = router($r); my $multiform = $r->parameters; XMLRPC::Transport::HTTP::Plack ->dispatch_to($path, $module) ->handler($r, $cmd, $multiform); } See on CPAN: https://metacpan.org/module/XMLRPC::Transport::HTTP::Plack 35/36
37. Intro Deployment Testing DBTools Plack::* Bonus Server-side JS See on github: https://github.com/ezhi/javascript-v8 transparent manipulation with objects and functions memoizing of already converted objects See on github: http://bem.github.com/bem-method/html/all.en.html 36/36