Manage cloud server with open source tools

Transcript

1. #JCConf .BOBHF
   4FSWFST
   0O
   5IF
   $MPVE
   XJUI
   0QFO4PVSDF
   5PPMT 何永琳 @ Fliptop Director of Engineering ⼯工頭
2. 陾玑 • 捀➊랂䧮⦛꨽銳湋䱽ծ庠ꆀ
   • 剣➊랂䭷垥꨽銳湋䱽ծ庠ꆀ
   • 剣➊랂䊨Ⱘ〳⟄䍲⸔䧮⦛
   • NFUSJDT
   CZ
   DPEBIBMF
   • 33%
   5PPMT
3. 锞铇㻜涸㔐瘸鸏⦐㉏겗
4. ⡹♧㣔漗䎙⦐㼭儘 ref: http://xkcd.com/386/
5. ⡹♧㣔漗䎙⦐㼭儘 ref: http://xkcd.com/386/
6. ⡹♧㣔漗䎙⦐㼭儘 ref: https://jawbone.com/up
7. ⡹♧㣔漗䎙⦐㼭儘 ref: https://jawbone.com/up
8. 植㻜䖃䖃嫲⡹䟝⫹涸婫ꃏ
9. 铇㻜涸꬗㼩荈䧮僽佖㊥涸Ꟛ㨥
10. 㥶卓䧮⦛㼩荈魨涸钢濼鿪剤铐䊴 齡䧮⦛㻨涸鮿넓ヤ
11. 4PGUXBSF
    -JGFDZDMF
12. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ
    • ♳涰㣔涸麋遤
    • 顺俱䏨⪓⟧
    • 㢪鿈剪⹢ 40"
    .JDSP4FSWJDF ㉏겗
    • 䳕倝炽넒
13. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ
    • ♳涰㣔涸麋遤
    • 顺俱䏨⪓⟧
    • 㢪鿈剪⹢㉏겗
    • 䳕倝炽넒 • 5ISPVHIQVU
    • 3FTQPOTF
    UJNF
    • XPSTU
    DBTF
    35
    
14. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ
    • ♳涰㣔涸麋遤
    • 顺俱䏨⪓⟧
    • 㢪鿈剪⹢㉏겗
    • 䳕倝炽넒 • .FNPSZ
    -FBL
    • $BDIF
    6TBHF )JU .JTT
    • +7.
    )FBQ
    TJ[F
    
15. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ
    • ♳涰㣔涸麋遤
    • 顺俱䏨⪓⟧
    • 㢪鿈剪⹢㉏겗
    • 䳕倝炽넒 • QFSGPSNBODF
16. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ
    • ♳涰㣔涸麋遤
    • 顺俱䏨⪓⟧
    • 㢪鿈剪⹢㉏겗
    • 䳕倝炽넒 • "WBJMBCJMJUZ
    • 3FMJBCJMJUZ
    • 3FTQPOTF
    5JNF
    • %PXOUJNF
    • .BJOUFOBODF
    4DIFEVMF
17. 斊麌♳涸朜屣 • 㣐ꆀ崨ꆀ弈Ⰶ
    • ♳涰㣔涸麋遤
    • 顺俱䏨⪓⟧
    • 㢪鿈剪⹢㉏겗
    • 䳕倝炽넒 • 鐱⠮倝炽넒꨽宠
    • 倝涸禹窠⿮侷
18. .FUSJDT
    #Z
    $PEBIBMF Mind The Gap!
19. (BVHFT • 鎹ꏖ殹♴涸⧩ Metrics.newGauge(QueueManager.class, "pending-jobs", 
 new Gauge<Integer>() { @Override public Integer value() { return queue.size(); } });
20. $PVOUFS private final Counter pendingJobs = Metrics.newCounter(QueueManager.class, "pending-jobs"); public void addJob(Job job) { pendingJobs.inc(); queue.offer(job); } public Job takeJob() { pendingJobs.dec(); return queue.take(); }
21. .FUFS • 鎹ꏖ㶸》涸鸟桧 private final Meter requests = Metrics.newMeter(RequestHandler.class, "requests", “requests", TimeUnit.SECONDS); public void handleRequest(Request request, Response response) { requests.mark(); // etc }
22. )JTUPHSBNT • 鎹ꏖ顺俱涸窠鎙ⴔ區ծⴔ⡑ minimum maximum mean standard deviation 75 percentile 95 percentile 98 percentile 99 percentile 99.9 percentile
23. )JTUPHSBNT private final Timer responses = Metrics.newTimer(
 RequestHandler.class, "responses", TimeUnit.MILLISECONDS, TimeUnit.SECONDS); public String handleRequest(Request request, Response response) { final TimerContext context = responses.time(); try { // etc; return "OK"; } finally { context.stop(); } }
24. )JTUPHSBNT
    暶殯⸆腋
25. )JTUPHSBNT
    暶殯⸆腋 Data Points: 1,000 req/sec * 60 * 60 * 24 = 691,200,000 or 5 GB
26. )JTUPHSBNT
    暶殯⸆腋 • 䱰垺䖃❀ⴔꗻⰊ涸✲⟝⨊俷 ref: http://codahale.com/codeconf-2011-04-09-metrics-metrics-everywhere.pdf
27. 剤✫鸏❉须俲䖕䧮⦛銴䙦랃欽ヤ
28. +.9
    JOUFSGBDF
29. KTPO
    JOUFSGBDF "jvm" : { "vm" : { "name" : "Java HotSpot(TM) 64-Bit Server VM", "version" : "1.6.0_45-b06" }, "memory" : { "totalInit" : 8.5761344E7, "totalUsed" : 8.4700376E7, "totalMax" : 1.087111168E9, "totalCommitted" : 1.16129792E8, "heapInit" : 6.1447488E7, "heapUsed" : 2.876592E7, "heapMax" : 9.50796288E8, "heapCommitted" : 5.9834368E7, "heap_usage" : 0.030254556483922664, "non_heap_usage" : 0.4103368905874399, "memory_pool_usages" : { "Code Cache" : 0.058741251627604164, "Eden Space" : 0.0375141103764524, "Perm Gen" : 0.6161499953851467, "Survivor Space" : 0.003552734375, "Tenured Gen" : 0.028688572989019088 }
30. KTPO
    JOUFSGBDF "daemon_thread_count" : 17, "thread_count" : 74, "current_time" : 1415698443730, "uptime" : 20471620, "fd_usage" : 0.0013427734375, "thread-states" : { "timed_waiting" : 0.7837837837837838, "blocked" : 0.013513513513513514, "runnable" : 0.16216216216216217, "waiting" : 0.04054054054054054, "new" : 0.0, "terminated" : 0.0 }, "garbage-collectors" : { "MarkSweepCompact" : { "runs" : 5, "time" : 946 }, "Copy" : { "runs" : 268889, "time" : 498572 } } "application.info" : { "version" : { "type" : "gauge", "value" : "1.2.0-20140319-SNAPSHOT" } },
31. (BVHF "net.sf.ehcache.Cache.http-service-cache" : { "accuracy" : { "type" : "gauge", "value" : "None" }, "eviction-count" : { "type" : "gauge", "value" : 2521997 }, "in-memory-hits" : { "type" : "gauge", "value" : 1213691 }, "in-memory-misses" : { "type" : "gauge", "value" : 3531312 }, "in-memory-objects" : { "type" : "gauge", "value" : 3587 },
32. .FUFS "org.eclipse.jetty.servlet.ServletContextHandler" : { "2xx-responses" : { "type" : "meter", "event_type" : "responses", "unit" : "seconds", "count" : 733665, "mean" : 2.014399884583874, "m1" : 0.18145603566649257, "m5" : 0.6958598626391137, "m15" : 0.9097181383462438 }, "4xx-responses" : { "type" : "meter", "event_type" : "responses", "unit" : "seconds", "count" : 2006, "mean" : 0.00550780829135732, "m1" : 3.0263603123472702E-58, "m5" : 4.125713901938956E-14, "m15" : 4.134678963142411E-7 },
33. )JTUPHSBN "com.fliptop.spendscore.backend.craweler" : { "requests" : { "type" : "timer", "duration" : { "unit" : "milliseconds", "min" : 0.076781, "max" : 43874.994542, "mean" : 406.840418123641, "std_dev" : 583.3339808219433, "median" : 554.004911, "p75" : 603.6283882500001, "p95" : 708.6854972499999, "p98" : 849.4521964999997, "p99" : 1024.8766623300019, "p999" : 2524.4020420530032 }, "rate" : { "unit" : "seconds", "count" : 150290, "mean" : 0.41264477142585376, "m1" : 0.026800109705232115, "m5" : 0.09772776121229207, "m15" : 0.13448567778710924 } }
34. 殥㕮
35. 䬘陪㜡 ref:http://www.nagios.com/products/nagiosxi/screenshots
36. ⚮
    -PBE
    #BMBODFS ref:http://www.nagios.com/products/nagiosxi/screenshots public class MyMongoDBHealthCheck{ public MyMongoDBHealthCheck(MongoDB db) { this.db = db; super(“mongodb”); } @Override public Result check() { try { // run query return Result.healthy(); } catch (Exception e) { return Result.unhealthy("Cannot connect to Mongodb”); } } } HealthChecks.register(new MarketoHealthCheck(marketo));
37. ⚮
    -PBE
    #BMBODFS ref:http://www.nagios.com/products/nagiosxi/screenshots HealthChecks.register(new MarketoHealthCheck(marketo)); // in pre-processing filter Map<String, Result> health = HealthChecks.runHealthChecks() // proceed only if all health checks return OK.
 // return exception if any check fails.
38. 2"