Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Security @ Scale 2015: Building Open Source Sof...

Javier Marcos
November 12, 2015

Security @ Scale 2015: Building Open Source Software for Security

Facebook's osquery is a Linux and OS X intrusion detection and response tool. The underlying infrastructure must test, build, and publish security software be secure by default. This discussion will show how the Facebook security team enabled Github contributors to safely submit C/C++/bash code to its CI and build server. The discussion will include Facebook's CI hardening process and the attack and vulnerability reports the team received through bug bounty targeting CI.

Javier Marcos

November 12, 2015
Tweet

More Decks by Javier Marcos

Other Decks in Technology

Transcript

  1. Goals Share our experiences building osquery security tool for host

    intrusion detection Provide examples of a hardened build process Present new ways of attacking and defending continuous integration
  2. “ So why open source? ” Accelerates Innovation We write

    better software We share our challenges “ I want every agent on my laptop pentested & code reviewed ” - Teddy Reed
  3. 5,319 of 284,000 followers 1 of 320 repos 496 of

    46,000 forks 2,420 of 91,000 commits
  4. commit 73a32b June 30, 2014 “Initial commit” today! Sep 7,

    2015 Release day! Oct 28, 2014 5000 followers Apr 1, 2015 First massive external-contributor feature A full year of open source activity Building a community around an open source security tool is challenging
  5. What is osquery? Explore your operating system using SQL Host

    visibility motivated by intrusion detection 100% OS API usage, no fork execve
  6. Why use SQL? WHERE uid != 0 [constraints] [concept] SELECT

    pid, name FROM processes [attributes]
  7. Why use SQL? WHERE uid != 0 [constraints] [concept] SELECT

    pid, name FROM processes [attributes] JOIN users ON processes.uid = users.uid [join]
  8. Why use SQL? WHERE uid != 0 SELECT pid, name,

    username FROM processes [attribute] JOIN users ON processes.uid = users.uid [join]
  9. Over 100 tables to join •processes •routes •shell_history •smbios_tables •suid_bin

    •system_controls •usb_devices •users •groups •rpm_packages •apt_sources •deb_packages •homebrew_packages •kernel_modules •memory_map •shared_memory •browser_plugins •startup_items •acpi_tables •arp_cache •crontab •file_events •kernel_info •listening_ports •logged_in_users •mounts •pci_devices
  10. osquery component diagram CMake Thrift C++11 Plugins Modules Extensions SQL

    Config Logging daemon interactive shell RocksDB LLVM
  11. osquery needs secure DevOps! Complicated and resource intensive C++ build

    231 C++11 sources: 5-9mins Over 10 supported platforms and distributions Performance, end to end, and regression testing Static and dynamic analysis Kernel extensions including unsafe stress tests OSX 10.9/10.10/10.11, Ubuntu 12/14, CentOS 6/7, RHEL 6/7, FreeBSD 10
  12. Review: Requirements Must build public code, and have public UI

    Must build C++11 with few dependencies Must support various OS X versions Must be trusted to produce packages automatically Must have “some” Internet access
  13. osquery needs secure DevOps! Originally used TravisCI then switched to

    Jenkins OMG the sky is falling our builds keep OOMing!
  14. osquery needs secure DevOps! https://github.com/facebook/osquery CLI signing bots “ok to

    test” Developer Javier Let the build begin! Code Review!
  15. as do most others… In Jenkins we trust Production Infrastructure

    Vendor Infrastructure 
 Corporate Infrastructure POPs
 Edges POPs
 Edges POPs
 Edges POPs
 Edges POPs
 Edges ISPs
 Firewall ISPs
 Firewall
  16. but not that much… In Jenkins we trust Vendor Infrastructure

    ISPs
 Firewall AWS https://jenkins.osquery.io SSH forward TCP/8080 “All Jenkins and build infrastructure kept in our corporate datacenter but isolated alongside our coffee shops” All metal are Mac Minis running ESX for virtualization licensing
  17. In Jenkins we trust Vendor Infrastructure ISPs
 Firewall Ubuntu 12

    Ubuntu 14 OSX 10.9 OSX 10.10 FreeBSD CentOS 6 CentOS 7 RHEL 6 RHEL 7 OSX 10.11 x5 x5 x2 x5 x2 x2 x5 x5 x2 x2 ESX virtual machines ESX ESX virtual machine
  18. Jenkins: Lockdown 101 Apache or nginx TLS proxy Web endpoint

    blacklisting (e.g., /script) Proper authentication and authorization Strict multi-factor authentication CSRF protections and automation bot awareness Jenkins API/action audit logs
  19. Jenkins: TLS proxy <VirtualHost *:80> ServerAdmin [email protected] ServerName jenkins.osquery.io Redirect

    / https://jenkins.osquery.io/ </VirtualHost> <VirtualHost *:443> ServerAdmin [email protected] ServerName jenkins.osquery.io SSLEngine on SSLCertificateFile /custom/osquery.io.crt SSLCertificateChainFile /custom/chain.crt SSLCertificateKeyFile /custom/osquery.io.key SSLCipherSuite HIGH:!aNULL:!MD5:!SSLv3:!SSLv2 SSLProtocol all -SSLv2 -SSLv3 DocumentRoot /var/www/html ProxyRequests Off ProxyPreserveHost On AllowEncodedSlashes NoDecode <Proxy *> Order deny,allow Allow from all </Proxy> ProxyPass /osquery ! ProxyPass / http://localhost:8888/ nocanon ProxyPassReverse / http://localhost:8888/ <VirtualHost *:443> Also see: https://github.com/hardening-io/chef-jenkins-hardening
  20. Jenkins: Web endpoint blacklisting <Location /api> Order Deny,Allow Deny from

    all </Location> <Location /restart> Order Deny,Allow Deny from all </Location> <Location /quietDown> Order Deny,Allow Deny from all </Location> <Location /user/jenkins> Order Deny,Allow Deny from all </Location> <Location /jnlpJars> Order Deny,Allow Deny from all </Location> <Location /script> Order Deny,Allow Deny from all </Location> Reduce risk to Jenkins admins: based on last 3 years of 0-days
  21. Jenkins: Proper authorization Github organization controls the API token Matrix-based

    ACLs with explicit admin list Any Github user may OAuth, but only has anon-equivalent access
  22. Jenkins: CSRF admins and bots Protect the specific set of

    admins using MFA Require attackers to 
 compromise admin’s 
 laptop, not just Prefer to enumerate settings/plugins from Jenkins, requires explicit egress to 
 https://jenkins-ci.org/
  23. Github: Pull Request hardening 1 2 3 Use Github API

    +
 bots to enforce
 license agreements Tag after code
 review
  24. Github: Pull Request hardening Do not allow PR to build

    without
 a code review by admins first This prevents clown-town build host DoSing,
 and attempts to get a trivial reverse shell
 (at least)
  25. 1

  26. Doomsday Scenario 1 Innocuous pull request 2 “ok to test”

    3 $ git commit --amend
 $ git push -f
  27. Doomsday Scenario 1 Innocuous pull request 2 “ok to test”

    3 $ git commit --amend
 $ git push -f 4 $ git commit --amend
 $ git push -f
  28. Doomsday Scenario 1 Innocuous pull request 2 “ok to test”

    3 $ git commit --amend
 $ git push -f 4 $ git commit --amend
 $ git push -f 5 Compromise:
 network, master, or 
 package build
  29. Doomsday Scenario 1 Innocuous pull request 2 “ok to test”

    3 $ git commit --amend
 $ git push -f 4 $ git commit --amend
 $ git push -f 5 Compromise:
 network, master, or 
 package build 6 osquery package
 contains injection
  30. Doomsday Scenario: Mitigations Unique build host credentials Inter-build host isolation

    Always-patched Jenkins: race between PR and patching Isolated package publishing infrastructure A compromised build infrastructure
 should not affect package integrity
  31. 1

  32. osquery on osquery “After hardening and isolating build infrastructure
 log

    the hell out of it….” Luckily, the tool we are obsessed with keeping safe
 can also help keep its own build infrastructure safe! …so meta!
  33. This should run in all build infra osquery on osquery

    { "options": { "disable_audit": "false", "audit_allow_config": "true" }, "schedule": { "process_events": { "query": "SELECT pid, path, cmdline, uid, gid, owner_uid, owner_gid, create_time, modify_time, parent, time, uptime FROM process_events", "interval": 60, "removed": false } } }
  34. osquery on osquery ELK - Elastic Search - Logstash -

    Kibana logstash-forwarder logstash-forwarder logstash-forwarder https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04 https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-centos-7 Guides: Pull Request
 Jenkins Package
 Jenkins AWS Proxy
  35. logstash configuration osquery on osquery 01-lumberjack-input.conf: input { lumberjack {

    port => 5000 type => "logs" ssl_certificate => “/path/to/ file.crt” ssl_key => “/path/tofile.key” codec => “json” } } 10-osquery.conf filter { if [type] == "osquery_json" { json { source => "message" } date { match => [ "unixTime", "UNIX" ] } }
  36. osquery on osquery Every move you make, every SSH you

    take…
 ….I’ll be watching you
  37. Conclusion Take extreme care of Jenkins Isolate & audit EVERYTHING

    If you are building tools with CI, try to use Github APIs