OSSEC at Scale

Presentation given on applying DevOps mindset to Security with OSSEC as the Focus.

Transcript

1. at Scale Making Computers Work for You!
2. Brad Lhotsky • Systems Security Team Lead
 at Booking.com" " • Recovering" • Perl Programmer" • Linux Systems Admin" • Network Security Specialist" • PostgreSQL Administrator" • ElasticSearch Janitor" • DNS Voyeur" • Starving Author" • OSSEC Team Member!
3. From 30,000 Feet ‣The Joy in Obsoleting Yourself" ‣OSSEC Instrumentation" ‣Instrumentation with OSSEC" ‣Pitfalls and Caveats" ‣Q & A
5. DevOps Proliferation
6. DevOps Goals ‣ Infrastructure as Code" ‣ Continuous Release" ‣ Testing ‣ Easy Deployment and Rollback ‣ Monitoring" ‣ Reduce “Bus Factor”" ‣ Replayability" ‣ Beer
7. OSSEC Instrumentation ‣ Identify common pain points ‣ Write a script to solve the problem ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!
8. Pain: client.keys
9. Easy Solution ‣ Server" ‣ ossec-authd ‣ Clients" ‣ agent-auth Or is it?" Agents still need manual run of agent-auth!
10. So fix it! exec { 'agent-auth': path => [ '/var/ossec/bin' ], timeout => 10, command => "agent-auth -m ${::ossec_server_ip} -A ${::fqdn}", creates => '/var/ossec/etc/client.keys', notify => Service['ossec-hids'], require => Package['ossec-hids-client']; } Example with Puppet
11. Pain: Monitoring OSSEC
12. Use Existing Tools Graphite (https://github.com/graphite-project)
13. Simple Graphite Script #!/bin/bash # Crontab: * * * * * /path/to/this-script.sh LIST_AGENTS=‘/var/ossec/bin/list_agents’ CARBON_HOST=‘graphite.example.com’ CARBON_PORT=2003 " prefix=“security.ossec.$(hostname -s)” ts=$(date +%s) all=$($LIST_AGENTS -a |wc -l) connected=$($LIST_AGENTS -c | wc -l) " echo <<EOM $prefix.available $all $ts $prefix.connected $connected $ts EOM | nc $CARBON_HOST $CARBON_PORT
14. Nagios can monitor values in Graphite and alert on thresholds.
15. Kibana (http://www.elasticsearch.org/overview/kibana/)
16. If you missed Vic’s Presentation ..
17. Pain: Relocating Clients to a New Server
18. Puppetry $ossec_server_ip = extlookup(‘ossec_server_ip’); " file { '/etc/facter/facts.d/ossec.txt': content => inline_template("prev_ossec_server=<%= @ossec_server_ip %>\n"), require => Service[‘ossec-hids']; } " if ( $ossec_server_ip != $::prev_ossec_server ) { ossec::reset { $ossec_server_ip: } }
19. Reset the Client " define ossec::reset() { notify { "OSSEC SERVER RESET: $name (prev:$::prev_ossec_server)": } # Remove the Client Keys exec { "ossec-stop": path => [ '/sbin', '/bin', '/usr/bin', '/var/ossec/bin' ], timeout => 10, command => "ossec-control stop"; " "ossec-remove-client-keys": before => Exec['agent-auth'], onlyif => 'test -f /var/ossec/etc/client.keys', command => '/bin/rm -f /var/ossec/etc/client.keys’, require => Exec['ossec-stop']; " "ossec-rids-reset": path => [ '/bin', ‘/usr/bin' ], timeout => 10, command => "rm -f /var/ossec/queue/rids/*", require => Exec['ossec-stop'], notify => Service['ossec-hids']; } } }
20. Now Auto-Distribute! " $ossec_servers = extlookup(‘ossec_servers’) # Now an array " file { ‘/etc/facter/facts.d/ossec.txt': content => template(‘ossec/ossec_server_fact.erb’); require => Service[‘ossec-hids']; } " # ossec_server_fact.erb <% uuid = scope.lookupvar("::uniqueid"); # Convert HEX to Integer seed = [uuid].pack(‘H*’).unpack('l')[0]; # Use UUID as Random Seed srand(seed); # Get seeded random number in range idx = rand(0 .. @ossec_servers.length-1); -%> prev_ossec_server=<% @ossec_servers[idx] %>
21. Instrumentation with OSSEC
22. DevOps ‣ Configuration Management has States ‣ Configuration Files ‣ Application Versions ‣ Resource Status and Definitions ‣ OSSEC has States too! ‣ Log data ‣ System status ‣ Process status ‣ Network status Instrumentation with a Security Tool? SEC
23. ‣ Configuration Management has Actions ‣ Resource CRUD ‣ Also “Run this script, kthxbye.” ‣ OSSEC has Actions too! ‣ ActiveResponse!!! ‣ Mostly, “Run this script, kthxbye.” Instrumentation with a Security Tool? DevOpsSEC
24. File Integrity Monitoring ‣ Noisy, at best ‣ Things messing with your files are mostly legit: ‣ System Updates ‣ Configuration Management ‣ Software Deployments ‣ Sysadmins Saving the Day
25. OSSEC v2.8+ ‣ ActiveResponse passes alert->filename ‣ Write a script which does your job ‣ Commit that script ‣ Deploy that script ‣ Re-use that script ‣ Automatically!
26. Game Plan ‣ Demote FIM Alerts to Level 1, disable email ‣ Fire an ActiveResponse that takes filename ‣ Emit a new log message in our script ‣ Decode the new log message ‣ Alert / Log based on that ‣ … ‣ PROFIT!
27. FIM Alerting <global> <alerts> <log_alert_level>1</log_alert_level> <email_alert_level>9</email_alert_level> </alerts> </global> " <syscheck> <!-- Alert Enhancements --> <auto_ignore>no</auto_ignore> <alert_new_files>yes</alert_new_files> <!-- Normal Stuff —> </syscheck>
28. Splaying Scan Time <% uuid = scope.lookupvar("::uniqueid"); seed = [uuid].pack('H*').unpack('l')[0]; srand(seed); " # Set our runtime for syscheckd hour = rand(4 .. 7); minute = sprintf("%02d", rand(0 .. 59)); -%> <syscheck> <!-- Better Scheduling --> <scan_on_start>no</scan_on_start> <scan_time><%= hour %>:<%= minute %>am</scan_time> <frequency>82800</frequency> <!-- Normal Stuff —> </syscheck>
29. FIM Alerting cont’d <rule id="106002" level="1"> <category>ossec</catgeory> <if_group>syscheck</if_group> <options>no_email_alert</options> <description>Verify file changes. </description> </rule>
30. Problem Solved! No more emails!
31. Fire ActiveResponse <command> <name>ossec-ar-verify-file</name> <executable>ossec-ar-verify-file.py</executable> <expect>filename</expect> <timeout_allowed>no</timeout_allowed> </command> " <active-response> <command>ossec-ar-verify-file</command> <location>local</location> <rules_id>106002</rules_id> </active-response>
32. Verify Logs Sep 15 00:26:10 ether ossec-ar-verify: file ok (/etc/ cron.d/puppet-job) " " Sep 15 00:26:10 ether ossec-ar-verify: file managed by RPM (/etc/mcollective/facts.yaml) changed outside of RPM " " Sep 15 00:26:10 ether ossec-ar-verify: file unmanaged (/ etc/postfix/aliases.db) changed
33. FIM Decoder <decoder name="ossec-ar-verify"> <program_name>ossec-ar-verify</program_name> </decoder> " <decoder name="ossec-ar-verify-file"> <parent>ossec-ar-verify</parent> <prematch>^file </prematch> <regex offset="after_prematch">^(\S+)</regex> <order>action</order> </decoder>
34. Parent Rule <rule id="107000" level="1"> <decoded_as>ossec-ar-verify</decoded_as> <description>Verification</description> <group>verify,</group> </rule>
35. Everything is OK <rule id="107001" level="3"> <if_sid>107000</if_sid> <match>^file ok </match> <description>File was changed intentionally. </description> </rule>
36. Managed File Change <rule id="107002" level="11"> <if_sid>107000</if_sid> <match>^file managed </match> <description>Verified: Unauthorized File Change </description> </rule>
37. Unmanaged File Change <rule id="107003" level="11"> <if_sid>107000</if_sid> <match>^file unmanaged </match> <description>Verified: Unknown File Change </description> <options>no_email_alert</options> </rule>
38. OSSEC FIM Results 96% Reduction in Alerting
39. Pitfalls and Caveats ‣ Who controls inputs? ‣ How resource intensive are your checks? ‣ What if 1,000,000 fire simultaneously? ‣ On the same server? ‣ Think, test, then get some to try to break it.
40. CVE-2014-5284 ‣ host-deny.sh created files in /tmp ‣ cp /tmp/hosts.$$.deny /etc/hosts.deny ‣ Didn’t properly manage permissions ‣ Would copy, as root, the contents of that file to /etc ‣ Moved from /tmp to /var/ossec and added randomness to file name
41. Find Me GitHub: https://github.com/reyjrar/ Twitter: @reyjrar Blogging: http://edgeofsanity.net Email: brad.lhotsky@gmail.com