Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Measuring everything in Real-Time
Search
Bastian Hofmann
August 25, 2012
Programming
4
720
Measuring everything in Real-Time
Talk from Froscon
Bastian Hofmann
August 25, 2012
Tweet
Share
More Decks by Bastian Hofmann
See All by Bastian Hofmann
Monitoring in Kubernetes with Prometheus and Grafana
bastianhofmann
0
290
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
86
Highly available cross-region deployments with Kubernetes
bastianhofmann
1
130
From source to Kubernetes in 30 minutes
bastianhofmann
0
130
Introduction to Kubernetes
bastianhofmann
1
140
CI/CD with Kubernetes
bastianhofmann
0
160
Creating a fast Kubernetes Development Workflow
bastianhofmann
1
250
Deploying your first Micro-Service application to Kubernetes
bastianhofmann
2
170
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
180
Other Decks in Programming
See All in Programming
Outline View in SwiftUI
1024jp
1
340
CSC509 Lecture 13
javiergs
PRO
0
110
Pinia Colada が実現するスマートな非同期処理
naokihaba
4
230
React CompilerとFine Grained Reactivityと宣言的UIのこれから / The next chapter of declarative UI
ssssota
1
100
TypeScriptでライブラリとの依存を限定的にする方法
tutinoko
3
710
とにかくAWS GameDay!AWSは世界の共通言語! / Anyway, AWS GameDay! AWS is the world's lingua franca!
seike460
PRO
1
910
Modular Monolith Monorepo ~シンプルさを保ちながらmonorepoのメリットを最大化する~
yuisakamoto
5
330
Jakarta EE meets AI
ivargrimstad
0
660
距離関数を極める! / SESSIONS 2024
gam0022
0
290
Figma Dev Modeで変わる!Flutterの開発体験
watanave
0
150
Better Code Design in PHP
afilina
PRO
0
130
Quine, Polyglot, 良いコード
qnighy
4
650
Featured
See All Featured
KATA
mclloyd
29
14k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
25
1.8k
Become a Pro
speakerdeck
PRO
25
5k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
232
17k
Imperfection Machines: The Place of Print at Facebook
scottboms
265
13k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
665
120k
Transcript
Measuring everything in Real-Time @BastianHofmann
Logging
Measuring
Testing
many roads
None
None
None
None
None
None
None
None
Questions? Ask!
http://speakerdeck.com/u/bastianhofmann
Logging
For when something goes wrong
server error log access log debug logs slow query log
...
Error Logs
$ tail -f error.log $ grep
ErrorLog /var/logs/apache/error.log
php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log
callable set_exception_handler( callable $exception_handler );
callable set_error_handler( callable $error_handler [, int $error_types = E_ALL |
E_STRICT ] );
Log additional info
Error code
None
HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html
None
Access Logs
192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"
200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog
/var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat
http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value =
"" ] )
LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{server_ip}n ..." custom nginx?
Debug Logs
https://github.com/Seldaek/monolog/
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel
$log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');
Handlers • Stream • Mail • FirePHP • ChromePHP •
Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new
Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );
Log in a structured way
JSON http://www.ietf.org/rfc/rfc4627.txt
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new
Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);
http://getcomposer.org/ http://packagist.org/
Logs from other services
web server db memcached db http service user request log
log log log log
Correlation / Tracing ID
web server db memcached db http service create unique trace_id
for request user request trace_id trace_id trace_id trace_id log log log log log
X-Trace-Id: bbr8ehb984tbab894
None
None
None
Aggregate the logs in a central place
Make them easily full- text searchable
Make them aggregate- able
Always Log to file
Seriously...
Always Log to file
AND REMEMBER
Always Log to file
Directly to a database
webserver webserver webserver DB
Disadvantages
Database is down?
Database is slow?
Database is full?
Better solutions?
graylog2 http://graylog2.org/
Full text search
Structured Messages
Metrics & Alarms
Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver
{ "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace
here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }
Disadvantages
Graylog/elasticsearch is down?
Graylog/elasticsearch is full?
Packet loss
Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF
None
Don't influence your application by logging
logstash http://logstash.net/
Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash
logstash logstash logstash GELF
Very rich plugin system
input filter output
file { type => "error" path => [ "/var/logs/php/*.log" ]
add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {
type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
output { amqp { host => "amqp.host" exchange_type => "fanout"
name => "logs" } }
input { amqp { host => 'amqp.host' port => '5672'
exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }
output { gelf { host => '127.0.0.1' } }
Measuring
For knowing what happens
Munin http://munin-monitoring.org/
Ganglia http://ganglia.sourceforge.net/
None
Graphite http://graphite.wikidot.com/
webserver webserver webserver graphite UDP
webserver webserver webserver graphite UDP statsd
StatsD https://github.com/etsy/statsd/
webserver webserver webserver statsd statsd statsd graphite aggregated UPD message
webserver webserver webserver statsd statsd statsd graphite aggregated UPD message
statsd
https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');
None
Technical Metrics
Business Metrics
Define your KPIs
Testing
For improving your app
Existing features
New features
A/B
BUY IT BUY IT $userId % 2 === 0 $userId
= 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
BUY IT BUY IT save random decision in session $userId
= 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
Save when experiment is shown
None
Save when a goal is reached
None
Conversion
Define your goals!
Disadvantages
Bad conversion during test
Only works for temporary tests
Inflexible: changing of tests not supported
There are better methods with better results
Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/
BUY IT BUY IT random decision $userId = 24 save
that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1
BUY IT BUY IT 2 90% users best option rest
other options
Integrate it into your architecture
None
Be careful
https://github.com/bashofmann/ Try it out
Did you like this talk? h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann
[email protected]
http://programm.froscon.de/2012/ events/916.html