Lessons Learned from Five Years of Building Capture the Flag

SECCON 2017
Feb. 18, 2018

Transcript

1. Lessons Learned from Five Years of Building Capture the Flag Vito Genovese SECCON 2017 Feb. 18, 2018
2. Good Morning!
4. Capture the Flag "CTF"
5. DEF CON
6. 1996 first game
7. 2000 formalized how it was run
8. 2002-2004 ghettohackers
9. 2005-2008 Kenshoto
10. 2009-2012 ddtek
11. 2013-2017 Legitimate Business Syndicate
12. 2018-? Order of the Overflow
13. Two Distinct Formats
15. Jeopardy Style Scoreboard
16. Jeopardy Style Prompt
18. Jeopardy Style Solving
20. Jeopardy Style Points
22. Jeopardy Style DEF CON CTF Quals
23. Photo: robbje @ Eat Speep Pwn Repeat
24. asby From SHA2017 CTF
25. asby Get file
27. asby Identify Windows STDIO .exe
28. asby Determine goal
30. asby Guess the correct input
31. asby •Reverse engineer a Windows binary •Guess each character by hand •Write a program
32. asby Write program
36. asby Get solution
38. asby Get points
40. Jeopardy Style 1. Get challenge 2. Solve it 3. Get points
42. Attack-Defense •Reverse engineer •Patch flaws •Exploit others •Don't break it
43. ⚑ Attack-Defense PPP atmail scorebot Shellphish
44. Attack-Defense PPP atmail scorebot Shellphish deposit ⚑
45. Shellphish Attack-Defense PPP atmail scorebot steal ⚑
46. Shellphish Attack-Defense PPP atmail scorebot redeem ⚑
47. Shellphish Attack-Defense PPP atmail scorebot availability okay availability check
48. Shellphish Attack-Defense PPP atmail scorebot failed availability ☠ ☠can’t steal
49. Attack-Defense DEF CON CTF Finals
50. Rubix
51. Rubix
52. Rubix 54 Rubik's cube instructions …becomes shellcode
53. Lab RATs on Rubix Lab RATs posted a write-up: https://blog.rpis.ec/2017/08/defcon- finals-2017-introduction-rubix.html
54. Lab RATs on Rubix 1. write 9-bit to 8-bit netcat 2. analyze 9-bit strings in libc 3. symbolize libc 4. figure out how main() gets called
55. Lab RATs on Rubix Now the actual analysis starts…
56. Attack-Defense •How is it supposed to work? •How can we attack it? •How can we defend it?
57. Attack-Defense •Get points by capturing flags •Lose points by having flags captured •Lose lots of points by failing checks
58. Attack-Defense Complicated, frustrating, fun!
59. CTF Extremely ambitious
60. CTF •Running Smoothly •Fair Contest •Fun Challenges
61. Running Smoothly
62. Running Smoothly Starts early
63. Running Smoothly Who's on the team?
64. Legitimate Business Syndicate •Half 2005-2007 university team •Half 2012 coworkers
65. Legitimate Business Syndicate in 2006
66. Legitimate Business Syndicate •August 2012: ddtek steps down •December 2012: Gyno starts recruiting •February 2013: Proposal submitted •March 2013: Proposal accepted
67. Legitimate Business Syndicate •"Reverse engineers" 3/4 of the group •Different specialties •Radio: 2014, badger •Hardware: 2015, the year of single- board computers •Esoteric computing: 2017, cLEMENCy
68. Legitimate Business Syndicate 100% dependent on Selir's amazing infrastructure
69. Legitimate Business Syndicate I started for the database backed web application
70. Team Building People grow and change
71. Team Building Roles grow and change
72. Team Building •Who do you know? •Who do you trust? •Who do you like?
73. Communication “It's good.”
74. Communication async (chat) is great weekly meetings are great
76. Smooth Operation Support your team
77. Smooth Operation CTF software is software
78. Smooth Operation Automate testing and deployment
79. CTF •Running Smoothly •Fair Contest •Fun Challenges
80. Fair Contest
81. Fair Contest CTF is computer hacking
82. Fair Contest CTF is computer system
83. Fair Contest Hack the right thing the wrong way
84. Fair Contest Hack the wrong thing
85. Fair Contest Fix a thing the "wrong" way
86. Fair Contest Restrict players more
87. Qualifiers •Services on separate hosts •Multiple hosts in different locations •Connections get separate container •xinetd and runc •Limit system calls •seccomp
88. Finals More complex game More complex problems
89. Finals •Keep the game about reverse engineering •(Not OS administration)
90. Finals •2013: unprivileged team account, unprivileged service accounts •2014: understood "Superman defense" better
91. Superman Defense •Block opponent IPs •Prevent reading the flag
92. Cyber Grand Challenge US Defense Advanced Research Projects Agency (DARPA) project starting in 2014
93. Cyber Grand Challenge CTF for autonomous computers
94. Cyber Grand challenge Extremely formalized
95. Challenge Binaries •"CBs" •32-bit i386 •Special CGCEF executable format •Limited system calls •No retained state
96. Proof of Vulnerability •"PoVs" •32-bit i386 CGCEF •Demonstrate a vulnerability: •Register control •Memory disclosure •Run by scoring system
97. Offline Evaluation •Team interface gives out binaries •Team interface collects replacement CBs, PoVs •Runs availability checks and PoVs in isolation •Designed for reproducibility and audibility
98. Finals •2015: restrict system calls •2016: use CGC game format •2017: everything in limited emulator
99. Fair Contest Release scoring information
100. Fair Contest Think about accessibility
101. CTF •Running Smoothly •Fair Contest •Fun Challenges
102. Fun Challenges Break expectations
103. dosfun4u •Discover that it's a DOS binary •Debug and patch IDA Pro •Start actual reverse engineering
104. badger •MSP-430 on physical hardware •custom CDMA radio network
106. Consensus Evaluation •CGC's big attack-defense innovation •Everyone sees everyone else's patched binaries •Explosion in number of binaries that need reversing
107. 1000 cuts / crackme2000 Push teams into automated analysis Hundreds of binaries
108. Consensus Evaluation in 2016 Player asks about losing points Service being attacked, that's why "But we're using the same binariess as the winning team"
109. Consensus Evaluation in 2017 Rubix expected shellcode to work in availability checks Defenders would add checks to block "evil" or allow "good" shellcode Attackers would build new shellcode to pass checks "Felt like a multiplayer game against humans"
110. CTF •Running Smoothly •Fair Contest •Fun Challenges
111. CTF Still more to learn!
112. CTf More work ahead of us
113. CTF Opportunity to grow for more players
114. CTF Best way to learn is to do
115. Five years with the best group of people I've ever worked with
116. Five years building a contest for the friendliest and smartest community I know
117. Thanks for making it amazing!
119. vito@legitbs.net @vito_lbs