Upgrade to Pro — share decks privately, control downloads, hide ads and more …

出了问题不要靠猜

Avatar for LI Daobing LI Daobing
September 14, 2013
Programming
4.1k
40

 出了问题不要靠猜

Avatar for LI Daobing

LI Daobing

September 14, 2013

More Decks by LI Daobing

See All by LI Daobing
How to attack TLS in PQC decade, part I
Avatar for LI Daobing lidaobing
0
49
HTTP协议相关的若干安全问题
Avatar for LI Daobing lidaobing
9
1.2k
Debian & Packaging
Avatar for LI Daobing lidaobing
1
580
Java 质量保障
Avatar for LI Daobing lidaobing
3
310
OAuth: How and Why?
Avatar for LI Daobing lidaobing
1
160
从 Struts 迁移到 Spring MVC，以及为什么？
Avatar for LI Daobing lidaobing
2
670
glusterfs 文件系统
Avatar for LI Daobing lidaobing
2
210
如何学习 Shell
Avatar for LI Daobing lidaobing
3
340

Other Decks in Programming

See All in Programming
Observability in Practice：Grafana 與 Edge Device SRE 的那些事
Avatar for Blueswen blueswen
0
170
JJUG CCC 2026 Spring: JSpecify で実現する Kotlin フレンドリーな Java API 設計
Avatar for ternbusty ternbusty
1
180
軽量Java基盤の設計 DIコンテナに頼らない、長期保守と1秒起動の実現 JJUG CCC 2026 Spring
Avatar for Masaaki Takahashi macha64
0
540
「AIで開発し、AIを届ける」をEvalでつなぐ 〜AIネイティブに始めるプロダクト開発の実践〜 / Connecting "Develop with AI, deliver AI" with Eval
Avatar for r-kagaya rkaga
4
5.3k
Lemonade + Foundry Toolkit でお手軽アプリ開発
Avatar for 瀬尾佳隆 seosoft
1
360
The NotImplementedError Problem in Ruby
Avatar for Koichi ITO koic
1
830
[2026年度第1回ORセミナー] 計画最適化ベンチャーと競技プログラミング人材
Avatar for terry-u16 terryu16
0
270
気づいたらRubyで100作品 ー クリエイティブコーディングが生活の一部になるまで / 100 Ruby Sketches Later: How Creative Coding Became Part of My Life
Avatar for chobishiba chobishiba
3
590
Webフレームワークの ベンチマークについて
Avatar for Yusuke Wada yusukebe
0
170
Hunting Vulnerabilities in Symfony with LLMs
Avatar for Vincent Amstoutz vinceamstoutz
0
550
メソッドのジェネリクスでGoの夢は広がるか？ / Kyoto.go #65
Avatar for utagawa kiki utgwkk
3
820
AIだと陥りがちなJakarta EE最新技術への移行時の落とし穴と解決策
Avatar for tnagao7 tnagao7
0
110

Featured

See All Featured
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
331
21k
Lightning talk: Run Django tests with GitHub Actions
Avatar for Sarah Abderemane sabderemane
0
200
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.7k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.5k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
170
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.2k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
55
3.4k
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
150
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
356
21k
Designing Experiences People Love
Avatar for Jonathan Moore moore
143
24k
Agile Actions for Facilitating Distributed Teams - ADO2019
Avatar for Mark Kilby mkilby
0
210
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
76
5.2k

Transcript

  1. ग़ྃ问题ෆཁᯪᘃ LI Daobing <[email protected]> ࣣڇӠଘ储 2013-09-14 Saturday, September 14, 13

  2. • ཥಓฌ <[email protected]> • Debian Developer • 2004 ೥䇖࢝઀৮ Debian

    • qterm, ibus, scim ౳软݅แಘ维护ਓ员 • 䇖ݯ爱޷ऀ • தจ维جඦՊલ؅ཧ员 • ࢀ༩维护 iso-codes, translationproject.org • python-lunardate, capistrano-scm-jenkins ࡞ऀ • manpages-zh, douban-ruby, iptux 现೚维护ਓ员 • զ对䇖ݯత؃๏ • ࢀ༩Ұ࿽项໨ɼ贡ݙզత时间ੋҼ为认ಉଞతཧ೦ɼࣕෆੋ૝վ଄ଞతཧ೦ • Github: http://github.com/lidaobing https://speakerdeck.com/lidaobing Saturday, September 14, 13

  3. ໨录 • ဓҰ࿽ Bug 讲ى • զ؃౸తཚ৅ • զਪᣑతղႊํҊ Saturday,

    September 14, 13

  4. Ұ࿽حոత Bug • ᠓盘త্传෰务 • Windows Լ IE, Chrome 浏览ثԼਖ਼ৗ

    • Linux Լ Firefox, Chrome 浏览ثԼਖ਼ৗ • Windows Լత Firefox ෆਖ਼ৗ, ্传ࣦ 败ɻ Saturday, September 14, 13

  5. ೗ԿఆҐBug Saturday, September 14, 13

  6. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) Saturday, September 14, 13

  7. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) Saturday, September

    14, 13

  8. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ

    HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” Saturday, September 14, 13

  9. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ

    HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” • ༻ telnet 发ૹෆಉత请ٻདྷ验证结论 Saturday, September 14, 13

  10. ೗ԿఆҐBug Saturday, September 14, 13

  11. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug Saturday, September 14,

    13

  12. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug • ௨஌෰务୺䇖发ਓ员ਚշम䐾 Saturday,

    September 14, 13

  13. ޷Bug/ᆀBug Saturday, September 14, 13

  14. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug Saturday, September 14, 13

  15. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug Saturday,

    September 14, 13

  16. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug •

    对ဋᆀ Bug, ଞత࠷େ༻处बੋಜଅ㟬补 ॆ೔ࢤ Saturday, September 14, 13

  17. 㠥Ұ࿽حոతBug • ဓެ࢘访问᠓᜾༗12sԆ迟 • ௚઀ ping 㠳应኷շ • ဓଖଞ᠓络དྷݯ访问຅༗问题 •

    监߇຅༗问题, ໵຅༗٬户དྷ๊ԇࠑࣄ Saturday, September 14, 13

  18. ઌᘃᘃ؃ Saturday, September 14, 13

  19. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ Saturday, September 14,

    13

  20. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ •

    ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ Saturday, September 14, 13

  21. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ •

    ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ • ෆ؅ɼ൓ਖ਼຅ਓ๊ԇ这ࣄ • 99% త༻户ձ۰౸问题时ෆձ报ࠂ Saturday, September 14, 13

  22. ෼ੳ Saturday, September 14, 13

  23. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK •

    ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ Saturday, September 14, 13

  24. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK •

    ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ • 结论: 问题᪑࿏༝/线࿏Ꮰ䎔ɼ䉯实ੋ෰务୺త问题 Saturday, September 14, 13

  25. ෼ੳ Saturday, September 14, 13

  26. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 •

    尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 Saturday, September 14, 13

  27. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 •

    尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 • 结论: Bug ༩ timestamp ૬䎔ɼୠနવᏠ๏ ղ释为ॄ㜮ଖଞ᠓络຅问题 Saturday, September 14, 13

  28. ղႊํҊA Saturday, September 14, 13

  29. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ

    Saturday, September 14, 13

  30. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ

    • ٙ问နવଘࡏɼ为ॄ㜮෰务ثෆ㠳应๭ ࠣ timestamp แ, ༗时ީຢೳਖ਼ৗ㠳应 Saturday, September 14, 13

  31. ෼ੳ Saturday, September 14, 13

  32. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ •

    䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ Saturday, September 14, 13

  33. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ •

    䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ • ؃དྷፙ౸ਅ实తݪҼྃ, ༗຅༗ߋ޷஍ղႊํҊʁ Saturday, September 14, 13

  34. ղႊํҊB • nginx ߴ൛ຊቮ经ࢧ࣋ keep-alive • ঋ级 nginx, 启༻ keep-alive,

    ߱௿୺ޱ઎༻ • 䎔闭୺ޱॏ༻ɼኂՃ㖘୺ޱ਺త监߇࿨ 报ܯ Saturday, September 14, 13

  35. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 Saturday, September 14, 13

  36. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 Saturday, September 14,

    13

  37. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 Saturday, September 14, 13

  38. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 Saturday, September 14, 13

  39. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼

    cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 • 对ဋಹࠣෆқ࠶现త Bug, ब损ࣦྃҰ࣍म 䐾తصձ Saturday, September 14, 13

  40. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 Saturday, September 14, 13

  41. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... Saturday, September 14,

    13

  42. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... • ਵศᘃҰ࿽ݪҼɼվ䫆۟୅码ɼવ޳ ॏ৽测试

    Saturday, September 14, 13

  43. զॴ؃౸తཚ৅ • ᠍গ൓ল Saturday, September 14, 13

  44. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ Saturday, September 14,

    13

  45. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug Saturday, September 14, 13

  46. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ Saturday, September 14, 13

  47. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug,

    ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ • ຅༗௨过补ॆ೔ࢤ౳खஈདྷ߱௿೔޳ఆҐBugత难 ౓ Saturday, September 14, 13

  48. զॴ؃౸తཚ৅ • ւԸ๏则: 㑌Ұى严ॏࣄނతഎ޳ɼඞવ ༗29࣍轻ඍࣄނ࿨300ىະ਱ઌஹҎٴ 1000ىࣄނ隐ױ Saturday, September 14, 13

  49. զ们ਪᣑతํ๏ • ग़ྃ问题ෆཁᯪᘃ • 浏览ث • ෰务୺೔ࢤ • ፛แ޻۩ Saturday,

    September 14, 13

  50. 浏览ث؃ॄ㜮ʁ Saturday, September 14, 13

  51. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? Saturday, September 14, 13

  52. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ Saturday, September 14,

    13

  53. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie Saturday, September 14, 13

  54. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ Saturday, September 14, 13

  55. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL,

    ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ • Request-Id Saturday, September 14, 13

  56. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId Saturday, September 14, 13

  57. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header

    తํࣜ 发ૹ౸٬户୺ Saturday, September 14, 13

  58. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header

    తํࣜ 发ૹ౸٬户୺ • ՄҎࡏ nginx 层໘实现҃ऀࡏ业务逻辑层 ໘实现 Saturday, September 14, 13

  59. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ Saturday, September 14, 13

  60. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 Saturday, September 14, 13

  61. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id Saturday, September 14, 13

  62. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) Saturday, September 14, 13

  63. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id,

    Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) • 结Ռੋॄ㜮ʁ Saturday, September 14, 13

  64. ෰务୺೔ࢤ؃ॄ㜮ʁ • 对ଖଞ෰务త请ٻ • 邮݅ɼ୹৴ɼଖଞ෰务... • 记录请ٻ详৘࿨໣时 Saturday, September 14,

    13

  65. ୠੋ经ৗ຅༗଍䭧త೔ࢤ Saturday, September 14, 13

  66. wireshark tcpdump • ፛แ޻۩ • wireshark: ༗ք໘ • tcpdump: sudo

    tcpdump -n -s 4096 -w 1.log port 80 Saturday, September 14, 13

  67. ଖଞ޻۩ • strace/dtruss: ܥ统调༻᪑᪨޻۩ • lsof: ྻग़จ݅ଧ䇖৘ႎ • valgrind: 查಺ଘᔔ࿐

    • ltrace: 查询库调༻ Saturday, September 14, 13

  68. 总结 • զ们ਪਸ௨过Ұ䝅ܥ统తํ๏དྷ෼ੳ问题ɼ寻ፙ 问题తࠜݯ • զ们൓对୞ᯪ试错๏དྷղႊ问题 • ೳ࠶现త Bug ੋ޷

    Bug, ೗Ռෆೳ࠶现ɼ໵ཁ፤౸ 对应త᠓络请ٻ࿨೔ࢤ • ೗Ռ这࣍ղႊෆྃ Bug, ಹ㜮बվળ㟬త೔ࢤɼ䉯 อԼ࣍ Bug ग़现త时ީೳղႊଞ Saturday, September 14, 13

  69. զ们团队 Saturday, September 14, 13

  70. Q&A Saturday, September 14, 13