Responding @ Scale: osquery for Mass Incident Response and Detection

Kevin Thompson & Scott Roberts at BSidesDFW

Transcript

1. Responding @ Scale osquery for Mass Incident Detection & Response
2. Introductions
3. Kevin Thompson  Incident Responder @ Heroku  @bfist  I know more about WWE than all of you...
4. Heroku  "A company that runs apps for you"  Largest PaaS provider  Tens of thousands of Linux servers in production
5. Scott  DFIR Engineer @ GitHub  @sroberts  I know more about StarWars than all of you...
6. GitHub  "Making it easier to work together than alone"  Source code hosting & collaboration ( & )  Thousands of Linux servers & hundreds of Macs
7. We ripped off @Scale... But for a good reason...
8. A Story in Three Parts  osquery
9. A Story in Three Parts  osquery  osquery use
10. A Story in Three Parts  osquery  osquery use  osquery use at scale
11.  osquery
12. What is osquery? Expose system information via "SQL tables"  users, groups  installed packages, kernel modules  network communication, file system events Runs scheduled queries of tables
13. The People  Facebook  @marpaia  @theopolis  Along with 63 other people as of this writing...
14. What problems can osquery solve?  Single tool for finding system information  Easy to merge information from information sources  Results presented in multiple parsable format
15. How do you use osquery  Ad Hoc  Scheduled  Schedule a query  Collect the logs  Watch for changes
16. osqueryi  osquery's Run, Evaluate, Print, Loop  Useful for testing & one off checks Out of the Box Tools
17. osqueryi osqueryd  osquery's Daemon Tool  Useful for continuous detection  Schedules and runs pre-set queries and writes them to a logger  Takes some setup (we'll get to that) Out of the Box Tools
18. osqueryi osqueryd osqueryctl  osquery's System Control Tool  Turns osqueryd on, gets status, turns it off, etc  Takes care of system specific stuff (like LaunchAgents, which aren't fun) Out of the Box Tools
19.  osquery use
20. Getting started w/ osqueryi
21. Query Syntax SELECT columns FROM table WHERE modifier;
22. Basic Query select * from apt_sources;
23. Basic Syntax Breakdown
24. Basic Syntax Breakdown  Get all the data columns SELECT *
25. Basic Syntax Breakdown  Get all the data columns  From the Apt Sources (as in apt-get) Virtual Table SELECT * FROM apt_sources
26. Basic Query What it means? List all the information about all the sources where apt can download & install software select * from apt_sources;
27. Advanced Query SELECT name, path FROM kernel_extensions WHERE name NOT LIKE 'com.apple%';
28. Advanced Syntax Breakdown
29. Advanced Syntax Breakdown  Get the name & and path data SELECT name, path
30. Advanced Syntax Breakdown  Get the name & and path data  From the Kernel Extensions virtual Table SELECT name, path FROM kernel_extensions
31. Advanced Syntax Breakdown  Get the name & and path data  From the Kernel Extensions virtual Table  Where the name field doesn't start with SELECT name, path FROM kernel_extensions com.apple WHERE name NOT LIKE 'com.apple%'
32. Advanced Query What it means? Get the name & path about any loaded OSX Kernel Extensions that were not created by Apple SELECT name, path FROM kernel_extensions WHERE name NOT LIKE 'com.apple%';
33. Advanced Query
34. Joins This is sql syntax after all: What it means? Get the uid and name from any process with an open listening socket SELECT uid, name FROM listening_ports l, processes p WHERE l.pid=p.pid;
35. Real World Query Examples  Detects LoginWindow Persistence Mechanism  Detect RAT used by Hacking Team  Detect the Careto Malware LaunchDaemon select key, subkey, value from preferences where path = '/Library/Preferences/com.apple.loginwindow.plist'; select * from apps where bundle_identifier = 'com.ht.RCSMac' or bundle_package_type like 'OSAX'; select * from launchd where path like '%com.apple.launchport.plist';
36. Query Writing Philosophy Kevin: Interesting Haystacks Scott: Interesting Needles
37. Haystack Approach Write generalized queries that grab lots of potentially interesting information and sort it out later
38. Needle Approach Write very specific queries looking only for verified indications of compromise and act immediately
39. Special Capabilities  File Integrity Monitoring  Yara
40. File Integrity Monitoring  Specify directory paths and wildcards  Creates an inotify watcher  Publishes changes to file_events table
41. File Integrity Monitoring: Paths "file_paths": { "configuration": [ "/etc/%%" ], "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ], "ssh_keys": [ "/home/%/.ssh/authorized_keys" ] }
42. File Integrity Monitoring: Messages osqueryd[22331]: message=Added kernel listener to: /usr/bin/ osqueryd[22331]: message=Added kernel listener to: /usr/sbin/ osqueryd[22331]: message=Added kernel listener to: /bin/ osqueryd[22331]: message=Added kernel listener to: /sbin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/bin/ osqueryd[22331]: message=Added kernel listener to: /usr/local/sbin/ osqueryd[22331]: message=Added kernel listener to: /opt/bin/ osqueryd[22331]: message=Added kernel listener to: /opt/sbin/ osqueryd[22331]: message=Added kernel listener to: /etc/ osqueryd[22331]: message=Added kernel listener to: /home/
43. File Integrity Monitoring: Query "schedule": { "file_events": { "query": "select * from file_events;", "interval": 900, "removed":false } } }
44. File Integrity Monitoring: Output { "name": "file_events", "hostIdentifier": "ip-172-31-28-89", "calendarTime": "Tue Nov 3 19:53:38 2015 UTC", "unixTime": "1446580418", "columns": { "action": "CREATED", "category": "configuration", "md5": "d41d8cd98f00b204e9800998ecf8427e", "sha1": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "slot": "default", "target_path": "/etc/test", "time": "1446579678", "transaction_id": "0" }, "action": "added" }
45. Yara The pattern matching swiss knife for malware researchers  A pattern matching syntax for identifying malware  osquery can be configured to compare changed files to a set of Yara sigs  This is a beta feature at this time
46. Yara  Start by setting up the file paths you want to watch just as before "file_paths": { "binaries": [ "/usr/bin/%%", "/usr/sbin/%%", "/bin/%%", "/sbin/%%", "/usr/local/bin/%%", "/usr/local/sbin/%%", "/opt/bin/%%", "/opt/sbin/%%" ] }
47. Yara  Then tell osquery which files contain Yara sigs Note the key, "binaries" has to be one of the file paths "Yara": { "signatures": { "group_1": ["/path/file1.sig", "/path/file2.sig"], "group_2": ["/path/file2.sig", "/path/file3.sig"] }, "binaries" : ["group_1"] }
48. Yara  This is still beta, Heroku isn't using this in prod  Only in exploration right now  No idea what performance impact this has
49. osquery.conf  JSON configuration file  holds the scheduled queries  file paths to monitor  read from local filesystem or https osqueryd Configuration
50. osquery.conf osquery.flags  configures osquery administration  where do logs go  where is the pidfile  where to find JSON config file osqueryd Configuration
51. Managing Intelligence with Packs  osquery query packs are groups of queries to be added to the osquery schedule  Lets you group queries for easier management and distribution  Basically a specialized osquery.conf
52. Why Packs?  Make use of  included intelligence & management  Share and collaborate on intelligence  Overall easier management without an unwieldy osquery.conf
53. Building Your Own Packs Basic Template (stolen from osquery.io/docs/packs) Extend and add queries to your 's content! { "platform": "darwin", "version": "1.1.17", "queries": { "example_query": { "query": "select * from kernel_extensions;", "interval": "86400", "description": "Identifies a systems kext files", "value": "Kext's are a common OSX malware persistence mechanism" } } }
54. Using Non-Facebook Packs  Place on file system  Point to in  ???  Profit! osquery.config
55. Performance Impact & Testing heavy queries and how to avoid them  query profiler  cgroups
56. Query Profiler  Python script that reads your config & rates queries  1 to 5 rating in CPU, Memory, File I/O.  RequirHow Heroku is Using osquery
57. cgroups Control Groups: a feature of the Linux kernel  LXC uses cgroups to do it's work  Allows you to segment a group of processes into their own process space or file system
58. cgroups Root cgroup - everything runs here  by default everything gets 1000 cpu shares  smallest you can provide is 2 shares  set up during init script
59. cgroups Create the cgroups for memory and cpu cgcreate -g cpu:osquery cgcreate -g cpuacct:osquery cgcreate -g memory:osquery
60. cgroups Set the cpu and memory limits cgset -r cpu.shares=2 osquery cgset -r memory.limit_in_bytes=1073741824 cgset -r memory.kmem.limit_in_bytes=104857600 cgset -r memory.kmem.tcp.limit_in_bytes=104857600 cgexec -g cpuset:osquery osqueryd
61.  use osquery at scale
62. How Heroku is Using osquery  Only on the servers  Our environment is very uniform  Look for things that are not uniform  Lookup IOCs just to get a heads up
63. How GitHub is Using osquery  Very very very work in progress  osquery on OSX laptops hunting known IOCs  osquery on Linux servers hunting known IOCs, anomalies, and doing auditing
64. Deploying osquery on Linux (w/Puppet or Chef) Ask your friendly neighborhood devops engineer...
65. Deploying osquery on OSX  Deploy the osquery package (or a custom version)  Set and/or  Start osqueryd  Figure out how to collect logs (or don't!) osquery.conf osquery.flags
66. Deploying osquery on OSX #!/bin/sh set -e echo "==> Installing osquery (This requires the root password...)" curl https://osquery-packages.s3.amazonaws.com/darwin/osquery.pkg > osquery.pkg sudo installer -pkg osquery.pkg -target / echo "==> Setting osquery.conf" sudo mkdir -p /var/osquery/ sudo cp ./osquery.conf.json /var/osquery/osquery.conf echo "==> Setting the osquery.flags" sudo mkdir -p /etc/osquery/ sudo cp ./osquery.flags.txt /etc/osquery/osquery.flags echo "==> Cleaning up" rm osquery.pkg echo "==> Start osquery" sudo osqueryctl start echo "\nNow check out '/var/log/osquery/osqueryd.results.log' for results."
67. Configuring Many Endpoints  Earlier mentioned that tells where to find config file  That config file can come from an https server osquery.flags osqueryd
68. Introducing Windmill Developed with  by Heroku & GitHub
69. Windmill  Open source Ruby TLS Configuration Endpoint  Get it now on  at heroku/windmill  Organizes endpoints into Configuration Groups  Enables intelligent endpoint management
70. Windmill: Configuration Groups
71. Windmill: Configuration Groups
72. Windmill  Focused on safety  versioned config files  canary deployments
73. Windmill: Canary Deploy
74. So... What now?  Transport: syslog, logstash forwarder, fluentd, etc  Analysis: Splunk, ELK, or a SIEM  You have logs now... GO FIND BAD STUFF!!!
75. Resources  osquery.io, facebook/osquery, & osquery-python  heroku/windmill  blackfist/osq_simulator
76. Summary
77.  osquery  osquery use  osquery use at scale
78. Questions?
79. Thanks