出了问题不要靠猜

Transcript

1. ग़ྃ问题ෆཁᯪᘃ LI Daobing <lidaobing@gmail.com> ࣣڇӠଘ储 2013-09-14 Saturday, September 14, 13
2. • ཥಓฌ <lidaobing@gmail.com> • Debian Developer • 2004 ೥䇖࢝઀৮ Debian • qterm, ibus, scim ౳软݅แಘ维护ਓ员 • 䇖ݯ爱޷ऀ • தจ维جඦՊલ؅ཧ员 • ࢀ༩维护 iso-codes, translationproject.org • python-lunardate, capistrano-scm-jenkins ࡞ऀ • manpages-zh, douban-ruby, iptux 现೚维护ਓ员 • զ对䇖ݯత؃๏ • ࢀ༩Ұ࿽项໨ɼ贡ݙզత时间ੋҼ为认ಉଞతཧ೦ɼࣕෆੋ૝վ଄ଞతཧ೦ • Github: http://github.com/lidaobing https://speakerdeck.com/lidaobing Saturday, September 14, 13
3. ໨录 • ဓҰ࿽ Bug 讲ى • զ؃౸తཚ৅ • զਪᣑతղႊํҊ Saturday, September 14, 13
4. Ұ࿽حոత Bug • ᠓盘త্传෰务 • Windows Լ IE, Chrome 浏览ثԼਖ਼ৗ • Linux Լ Firefox, Chrome 浏览ثԼਖ਼ৗ • Windows Լత Firefox ෆਖ਼ৗ, ্传ࣦ 败ɻ Saturday, September 14, 13
5. ೗ԿఆҐBug Saturday, September 14, 13
6. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) Saturday, September 14, 13
7. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) Saturday, September 14, 13
8. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” Saturday, September 14, 13
9. ೗ԿఆҐBug • टઌ查؃浏览ثత߇੍୆ɼ观࡯ੋ൱༗错误(౎຅༗) • ༻ Wireshark ፊ获请ٻแɼ发现ࣦ败తҊྫ෰务୺ฦ ճ412(௨ৗ为200) • 对ൺ HTTP Requestɼ发现ࣦ败Ҋྫత HTTP 头༗ “Content-length”ɼࣕଖଞ৘ႎԼ为 “Content- Length” • ༻ telnet 发ૹෆಉత请ٻདྷ验证结论 Saturday, September 14, 13
10. ೗ԿఆҐBug Saturday, September 14, 13
11. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug Saturday, September 14, 13
12. ೗ԿఆҐBug • 䉯认ݪҼੋ෰务୺对 Content-Length త େখࣸහײɼ༩标।ෆූɼ଄੒ྃ这࿽ Bug • ௨஌෰务୺䇖发ਓ员ਚշम䐾 Saturday, September 14, 13
13. ޷Bug/ᆀBug Saturday, September 14, 13
14. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug Saturday, September 14, 13
15. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug Saturday, September 14, 13
16. ޷Bug/ᆀBug • ೳ࠶现త Bug बੋ޷ Bug • ೗Ռෆೳ࠶现ɼೳ፤౸䇗ৗ栈҃ऀ详细 ೔ࢤత໵ੋ޷Bug • 对ဋᆀ Bug, ଞత࠷େ༻处बੋಜଅ㟬补 ॆ೔ࢤ Saturday, September 14, 13
17. 㠥Ұ࿽حոతBug • ဓެ࢘访问᠓᜾༗12sԆ迟 • ௚઀ ping 㠳应኷շ • ဓଖଞ᠓络དྷݯ访问຅༗问题 • 监߇຅༗问题, ໵຅༗٬户དྷ๊ԇࠑࣄ Saturday, September 14, 13
18. ઌᘃᘃ؃ Saturday, September 14, 13
19. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ Saturday, September 14, 13
20. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ • ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ Saturday, September 14, 13
21. ઌᘃᘃ؃ • ެ࢘࿏༝ग़问题ྃʁ • ॏ启ҰԼ࿏༝试试؃ʁ • ୠ为ॄ㜮୞༗访问զ们ࣗݾ᠓᜾ձग़ࣄʁ • ਓ඼问题ʁ • ၏ఔং员还ੋ།෺Ұ఺ൺ较޷ • ෆ؅ɼ൓ਖ਼຅ਓ๊ԇ这ࣄ • 99% త༻户ձ۰౸问题时ෆձ报ࠂ Saturday, September 14, 13
22. ෼ੳ Saturday, September 14, 13
23. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK • ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ Saturday, September 14, 13
24. ෼ੳ • tcpdump፛แ • ٬户୺: 发ྃ6࿽ SYN แɼ౸࠷޳Ұ࿽࠽Ꮕ౸ SYN/ACK • ෰务୺: 䉯实Ꮕ౸ྃ6࿽ SYN แ, ࣕ׌䉯实લ໘5 ࿽ SYN แ౎຅༗ճ䐾 • ဓଖଞ᠓络访问ɼ䉯实ࡏट࿽ SYN แ೭޳ฦճ • 结论: 问题᪑࿏༝/线࿏Ꮰ䎔ɼ䉯实ੋ෰务୺త问题 Saturday, September 14, 13
25. ෼ੳ Saturday, September 14, 13
26. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 • 尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 Saturday, September 14, 13
27. ෼ੳ • ࠶ॏ৽෼ੳ tcpdump త记录ɼ发现લ5࿽ SYN แ带ྃ timestamp, ୈ6࿽຅带 • 尝试䎔ᎃ٬户୺ TCP timestamp, ॠ间㠳应 • 尝试䎔ᎃ෰务୺ TCP timestamp, ॠ间㠳应 • 结论: Bug ༩ timestamp ૬䎔ɼୠနવᏠ๏ ղ释为ॄ㜮ଖଞ᠓络຅问题 Saturday, September 14, 13
28. ղႊํҊA Saturday, September 14, 13
29. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ Saturday, September 14, 13
30. ղႊํҊA • 䎔ᎃ෰务୺త TCP timestamp • TCP timestamp ձӨ㠳传输଎౓ɼୠӨ 㠳ෆେ • ٙ问နવଘࡏɼ为ॄ㜮෰务ثෆ㠳应๭ ࠣ timestamp แ, ༗时ީຢೳਖ਼ৗ㠳应 Saturday, September 14, 13
31. ෼ੳ Saturday, September 14, 13
32. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ • 䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ Saturday, September 14, 13
33. ෼ੳ • ؃಺֩ݯ码 (CTO 亲ࣗૢ౛) • ೖޱصث༗େྔత൓޲୅ཧɼ୺ޱ经ৗෆ䭧༻ɼॴҎ䇖 启ྃTIME_WAIT ୺ޱॏ༻ • 䇖启ྃ TIME_WAIT ୺ޱॏ༻޳ɼ෰务୺ཁٻಉҰ࿽IPత SYN แ timestamp ඞ须ੋ顺ংత • 办ެ᠓络ੋ಺᠓, ኂ׌对զ们׭᠓访问኷频ൟɼ导கނো 发ੜ • ؃དྷፙ౸ਅ实తݪҼྃ, ༗຅༗ߋ޷஍ղႊํҊʁ Saturday, September 14, 13
34. ղႊํҊB • nginx ߴ൛ຊቮ经ࢧ࣋ keep-alive • ঋ级 nginx, 启༻ keep-alive, ߱௿୺ޱ઎༻ • 䎔闭୺ޱॏ༻ɼኂՃ㖘୺ޱ਺త监߇࿨ 报ܯ Saturday, September 14, 13
35. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 Saturday, September 14, 13
36. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 Saturday, September 14, 13
37. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼ cookie, ਗ਼ cache, ॏ৽ొ录 Saturday, September 14, 13
38. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼ cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 Saturday, September 14, 13
39. զॴ؃౸తཚ৅ • ༻寻ፙ workaround ୅ସղႊ问题 • 换࿽浏览ث㠧 • ॏ启, ਗ਼ cookie, ਗ਼ cache, ॏ৽ొ录 • 现场ඃഁᆀಘׯׯ净净 • 对ဋಹࠣෆқ࠶现త Bug, ब损ࣦྃҰ࣍म 䐾తصձ Saturday, September 14, 13
40. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 Saturday, September 14, 13
41. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... Saturday, September 14, 13
42. զॴ؃౸తཚ৅ • 尝试༻试错๏ղႊॴ༗问题 • ঋ级ґ赖แ, ঋ级ᎎ݅, ... • ਵศᘃҰ࿽ݪҼɼվ䫆۟୅码ɼવ޳ ॏ৽测试 Saturday, September 14, 13
43. զॴ؃౸తཚ৅ • ᠍গ൓ল Saturday, September 14, 13
44. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ Saturday, September 14, 13
45. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug, ༰қ产ੜճ归Bug Saturday, September 14, 13
46. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug, ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ Saturday, September 14, 13
47. զॴ؃౸తཚ৅ • ᠍গ൓ল • 寻ፙ౸ workaround ศ认为问题ղႊྃ • ຅༗༻测试ݻԽ Bug, ༰қ产ੜճ归Bug • Ұ࿽ Bug Մೳࡏଟ处ग़现ɼ຅༗尝试፺ࡧଖଞ༗ Bug త஍ํ • ຅༗௨过补ॆ೔ࢤ౳खஈདྷ߱௿೔޳ఆҐBugత难 ౓ Saturday, September 14, 13
48. զॴ؃౸తཚ৅ • ւԸ๏则: 㑌Ұى严ॏࣄނతഎ޳ɼඞવ ༗29࣍轻ඍࣄނ࿨300ىະ਱ઌஹҎٴ 1000ىࣄނ隐ױ Saturday, September 14, 13
49. զ们ਪᣑతํ๏ • ग़ྃ问题ෆཁᯪᘃ • 浏览ث • ෰务୺೔ࢤ • ፛แ޻۩ Saturday, September 14, 13
50. 浏览ث؃ॄ㜮ʁ Saturday, September 14, 13
51. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? Saturday, September 14, 13
52. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ Saturday, September 14, 13
53. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL, ํ๏, ࢀ਺, Accept, Cookie Saturday, September 14, 13
54. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL, ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ Saturday, September 14, 13
55. 浏览ث؃ॄ㜮ʁ • ੋ൱༗ JS 错误? • ᠓络请ٻੋ൱发ग़ʁ • 发ग़త请ٻੋ൱ਖ਼䉯ɿ URL, ํ๏, ࢀ਺, Accept, Cookie • ظ๬తฦճ值ੋॄ㜮ʁ • Request-Id Saturday, September 14, 13
56. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId Saturday, September 14, 13
57. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header తํࣜ 发ૹ౸٬户୺ Saturday, September 14, 13
58. Request-Id • 对㑌࣍请ٻ产ੜҰ࿽།ҰతId • 该 Id Ҏ HTTP Response Header తํࣜ 发ૹ౸٬户୺ • ՄҎࡏ nginx 层໘实现҃ऀࡏ业务逻辑层 ໘实现 Saturday, September 14, 13
59. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ Saturday, September 14, 13
60. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 Saturday, September 14, 13
61. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id, Request-Id Saturday, September 14, 13
62. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id, Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) Saturday, September 14, 13
63. ෰务୺೔ࢤ؃ॄ㜮ʁ • ࢛ཁૉ • 时间: 䇖࢝时间ɼ总໣时 • 谁: ༻户Id, Session-Id, Request-Id • ၏ॄ㜮: URL, ํ๏ɼXHR?, format, ࢀ਺(஫ҙ อ护ີ码) • 结Ռੋॄ㜮ʁ Saturday, September 14, 13
64. ෰务୺೔ࢤ؃ॄ㜮ʁ • 对ଖଞ෰务త请ٻ • 邮݅ɼ୹৴ɼଖଞ෰务... • 记录请ٻ详৘࿨໣时 Saturday, September 14, 13
65. ୠੋ经ৗ຅༗଍䭧త೔ࢤ Saturday, September 14, 13
66. wireshark tcpdump • ፛แ޻۩ • wireshark: ༗ք໘ • tcpdump: sudo tcpdump -n -s 4096 -w 1.log port 80 Saturday, September 14, 13
67. ଖଞ޻۩ • strace/dtruss: ܥ统调༻᪑᪨޻۩ • lsof: ྻग़จ݅ଧ䇖৘ႎ • valgrind: 查಺ଘᔔ࿐ • ltrace: 查询库调༻ Saturday, September 14, 13
68. 总结 • զ们ਪਸ௨过Ұ䝅ܥ统తํ๏དྷ෼ੳ问题ɼ寻ፙ 问题తࠜݯ • զ们൓对୞ᯪ试错๏དྷղႊ问题 • ೳ࠶现త Bug ੋ޷ Bug, ೗Ռෆೳ࠶现ɼ໵ཁ፤౸ 对应త᠓络请ٻ࿨೔ࢤ • ೗Ռ这࣍ղႊෆྃ Bug, ಹ㜮बվળ㟬త೔ࢤɼ䉯 อԼ࣍ Bug ग़现త时ީೳղႊଞ Saturday, September 14, 13
69. զ们团队 Saturday, September 14, 13
70. Q&A Saturday, September 14, 13