Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Headers for Hackers

Headers for Hackers

The web is growing up and getting faster and more secure. Making that the default is hard to achieve when you have to be backwards compatible, and some of the stuff we built 10 years ago is now a serious security liability. The answer: headers. Lots of headers.

Andrew Betts

March 09, 2018
Tweet

More Decks by Andrew Betts

Other Decks in Technology

Transcript

  1. 2 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com

    HTTP/1.1 200 OK Date: Tue, 27 Feb 2018 13:28:47 GMT Content-Type: text/html Content-Length: 34882 <html> ...
  2. 3 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com

    HTTP/1.1 200 OK Content-Type: text/html Content-Length: 34882 Cache-Control: private, max-age=3600 Access-Control-Allow-Origin: * Accept-Ranges: bytes Last-Modified: Fri, 02 Feb 2018 07:21:05 GMT X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=31536000; includeSubdomains; Content-Security-Policy: default-src 'self'; report-uri https://csp.example.com/ Expect-CT: max-age=86400, enforce, report-uri="https://ect.example.com/" Alt-Svc: h2="new.example.com:443"; ma=600 Link: /script/bundle.rev-983c15.js>;rel=preload;as=script;charset=UTF-8; Accept-CH: DPR, Width, Viewport-Width Feature-Policy: vibrate 'none'; geolocation 'none'; unsized-media ‘none’ <html> ...
  3. 4

  4. 8 • Alexa top 1,000,000 websites • Around 500,000 pages

    analysed • Over 50 million requests per run • Captures full request and response data, timing metrics etc. • Runs using WebPageTest • Makes raw result data available in BigQuery
  5. 10

  6. 14

  7. 15 (I'm going to mess up saying this on stage

    aren't I) Platform for Privacy Preferences Project? • Intended as a declaration of privacy policy • Too hard for users to understand/use • Only ever implemented by Internet Explorer, to gate access to third party cookies in IFRAMEs. – ... but not validated • Commonly set to “this is not a P3P policy” which satisfies the check
  8. 16

  9. 19

  10. 20 “Note: if a response includes a Cache-Control field with

    the max-age directive, a recipient MUST ignore the Expires field.”
  11. 21

  12. 22 Domains sending Expires 78% Domains sending Expires and Cache-Control

    with max-age: 64% https://bigquery.cloud.google.com/savedquery/598614557294:98e14323d29740678fe1b3012c9186db
  13. 23 Expires: Thu, 01 Dec 1994 16:00:00 GMT Cache-Control: private,

    no-store, no-cache, no-transform, must-revalidate, max-age=0, post-check=0, pre-check=0 Pragma: no-cache
  14. 26 Headers being emitted by > 5000 domains in HTTP

    Archive: Meaningless to the browser https://bigquery.cloud.google.com/savedquery/598614557294:2463981d0f444b6ba6c1a8c376079b90 x-cache x-aspnet-version x-varnish x-request-id x-cache-hits x-cacheable x-aspnetmvc-version x-runtime x-generator x-drupal-cache host referer x-served-by x-proxy-cache server x-type x-cache-group x-cache-status x-accel-version
  15. 27 Removing the cruft if (!req.http.Reveal-Debug && !req.http.Cookie:RevealDebug) { unset

    resp.http.Server; unset resp.http.X-Powered-By; unset resp.http.X-Cache; // ... etc } app.disable('x-powered-by'); ExpressJS (Node): VCL (Varnish / Fastly):
  16. 29 X-Frame-Options: SAMEORIGIN Stop anyone from framing your site: Content-Security-Policy:

    frame-ancestors 'self' But... equivalent to... https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/frame-ancestors
  17. 34 Via in summary The bad parts • It prevents

    forwarding loops – but might cause upstream servers to disable some HTTP features – and you can strip it in CDN config if you want to disable loop detection • It's both a request and a response header – but the response part is kind of useless • It provides for protocol negotiation – but only for H0 vs H1. HTTP2 negotiation happens a different way • It uses the name of the proxy engine – but that can be a bit misleading, eg Fastly sets 'varnish', and Heroku 'vegur', both of which are open-source products
  18. 37

  19. 38

  20. 44 HTTP/1.1 504 Gateway Timeout Proxy-Status: connection_timeout; proxy=SomeCDN; origin=abc; tries=3

    Reasons for obscure proxy errors: DNS Timeout DNS Error Destination Not Found Destination Unavailable Destination IP Prohibited Destination IP Unroutable Connection Refused Connection Terminated Connection Timeout Connection Read Timeout Connection Write Timeout Connection Limit Reached HTTP Response Status HTTP Incomplete Response HTTP Protocol Error HTTP Response Header Block Too Large HTTP Response Header Too Large HTTP Response Body Too Large HTTP Response Transfer-Coding Error HTTP Response Content-Coding Error HTTP Response Timeout TLS Handshake Error TLS Untrusted Peer Certificate TLS Expired Peer Certificate TLS Unexpected Peer Certificate TLS Missing Proxy Certificate TLS Rejected Proxy Certificate TLS Error HTTP Request Error HTTP Request Denied HTTP Upgrade Failed Proxy Internal Response Proxy Internal Error
  21. 47 Content-Security-Policy: default-src 'self'; img-src *; media-src media1.com media2.com; script-src

    userscripts.example.com Simple CSP example: By default, only allow access to the same origin as the page. But allow images to come from anywhere. Media and scripts from a whitelist of specific origins.
  22. 48 Content-Security-Policy: default-src 'self'; font-src 'self' data: http://*.gstatic.com https://*.gstatic.com https://*.symantec.com

    https://*.criteo.com http://*.resultspage.com https://*.resultspage.com; child-src 'self' https://www.google.com/ads/ https://*.listrak.com https://www.google.pl/ads/ https://www.google.ua/ads/ http://*.google.com.ua/ads/ https://www.youtube.com/ https://www.googleadservices.com/ https://googleads.g.doubleclick.net/pagead/ https://*.symantec.com https://*.criteo.com http://*.rfksrv.com https://*.rfksrv.com http://*.resultspage.com https://*.resultspage.com; object-src 'self' http://*.verisign.com http://*.iesnare.com https://*.verisign.com https://*.iesnare.com https://*.symantec.com https://*.zmags.com; img-src 'self' data: https://ssl.emailcli.com *.sli-spark.com https://*.veinteractive.com http://*.bazaarvoice.com/ http://blog.natchezss.com/ https://blog.natchezss.com/ http://*.avmws.com http://*.websecurity.norton.com http://*.google.com.ua http://*.natchezss.com https://www.pepperjamnetwork.com/ http://*.gstatic.com http://*.ywxi.net https://www.google.com/ads/ http://s3.amazonaws.com/product.reflektion.com/ http://*.reflektion.com/ https://stats.g.doubleclick.net/ https://d26opx5dl8t69i.cloudfront.net/ http://*.google-analytics.com/ https://seal.networksolutions.com/ http://*.listrakbi.com/ http://*.chartbeat.net/ http://*.scanalert.com/ http://*.avantlink.com https://*.bazaarvoice.com/ https://*.listrakbi.com/ https://*.chartbeat.net/ https://*.google-analytics.com/ https://*.reflektion.com/ https://*.bazaarvoice.com/ https://*.google.com.ua https://*.listrakbi.com/ https://*.chartbeat.net/ https://*.scanalert.com/ https://*.avantlink.com https://*.amazonaws.com/ https://*.scanalert.com/ https://*.norton.com https://*.ywxi.net/ https://*.cloudfront.net/ https://*.zmags.com/ https://*.adnxs.com http://*.adnxs.com https://*.symantec.com https://*.r1cdn.com https://ad.doubleclick.net https://go.flx1.com https://*.g.doubleclick.net https://*.optimizely.com https://*.yahoo.com https://*.sitescout.com https://*.1rx.io https://*.tubemogul.com https://*.simpli.fi https://*.ipredictive.com https://*.wtp101.com https://*.pubmatic.com https://*.media.net https://*.demdex.net https://*.smartclip.net https://*.bit.ly https://*.criteo.com http://*.rfksrv.com https://*.rfksrv.com http://*.resultspage.com https://*.resultspage.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' *.sli-r.com *.sli-spark.com *.resultspage.com *.resultsstage.com tagmanager.google.com *.googletagmanager.com https://*.veinteractive.com http://*.avmws.com http://*.optimizely.com http://*.iesnare.com https://*.listrak.com https://www.googleadservices.com/ http://*.verisign.com http://*.googleapis.com http://*.nr-data.net http://*.newrelic.com http://*.ywxi.net http://*.bazaarvoice.com/ http://*.cloudfront.net/ http://*.listrakbi.com/ https://seal.networksolutions.com/ http://*.google-analytics.com/ https://ping.chartbeat.net/ http://*.reflektion.com/ http://*.chartbeat.com/ https://*.listrak.com/ https://product.reflektion.com/ https://display.ugc.bazaarvoice.com/ https://www.google-analytics.com/ https://cdn.listrakbi.com/ https://*.verisign.com https://*.googleapis.com https://*.nr-data.net https://*.newrelic.com https://*.ywxi.net https://*.bazaarvoice.com/ https://*.cloudfront.net/ https://*.listrakbi.com/ https://*.google-analytics.com/ https://*.reflektion.com/ https://*.chartbeat.com/ https://*.optimizely.com https://*.avmws.com/ http://*.zmags.com https://*.zmags.com https://*.dpmsrv.com http://*.dpmsrv.com https://*.adnxs.com/ http://*.adnxs.com/ https://*.doubleclick.net http://*.doubleclick.net https://*.google.com https://*.gstatic.com/ https://*.symantec.com https://*.statsstory.com https://*.hotjar.com https://*.cloudfront.net https://*.jsdelivr.net https://c.vepxl1.net https://c.flx1.com https://go.flx1.com https://*.youtube.com https://s3.amazonaws.com https://*.ytimg.com http://*.criteo.com https://*.criteo.com http://*.criteo.net https://*.criteo.net http://*.resultspage.com https://*.resultspage.com https://*.sli-r.com; style-src 'self' 'unsafe-inline' tagmanager.google.com http://*.amazonaws.com/ http://*.reflektion.com/ http://*.googleapis.com http://*.bazaarvoice.com/ https://*.bazaarvoice.com/ https://*.amazonaws.com/ https://*.listrakbi.com http://*.listrakbi.com https://*.symantec.com https://*.cloudfront.net https://*.criteo.com http://*.rfksrv.com https://*.rfksrv.com http://*.resultspage.com https://*.resultspage.com; frame-src 'self' http://*.bazaarvoice.com http://*.listrak.com http://*.youtube.com/ http://*.zmags.com/ woobox.com https://*.bazaarvoice.com https://*.listrak.com https://*.youtube.com/ https://*.zmags.com/ https://*.listrakbi.com https://*.googleadservices.com http://*.googleadservices.com https://*.doubleclick.net http://*.doubleclick.net https://*.google.com https://*.google.ad https://*.google.ae https://*.google.com.af https://*.google.com.ag https://*.google.com.ai https://*.google.al https://*.google.am https://*.google.co.ao https://*.google.com.ar https://*.google.as https://*.google.at https://*.google.com.au https://*.google.az https://*.google.ba https://*.google.com.bd https://*.google.be https://*.google.bf https://*.google.bg https://*.google.com.bh https://*.google.bi https://*.google.bj https://*.google.com.bn https://*.google.com.bo https://*.google.com.br https://*.google.bs https://*.google.bt https://*.google.co.bw https://*.google.by https://*.google.com.bz https://*.google.ca https://*.google.cd https://*.google.cf https://*.google.cg https://*.google.ch https://*.google.ci https://*.google.co.ck https://*.google.cl https://*.google.cm https://*.google.cn https://*.google.com.co https://*.google.co.cr https://*.google.com.cu https://*.google.cv https://*.google.com.cy https://*.google.cz https://*.google.de https://*.google.dj https://*.google.dk https://*.google.dm https://*.google.com.do https://*.google.dz https://*.google.com.ec https://*.google.ee https://*.google.com.eg https://*.google.es https://*.google.com.et https://*.google.fi https://*.google.com.fj https://*.google.fm https://*.google.fr https://*.google.ga https://*.google.ge https://*.google.gg https://*.google.com.gh https://*.google.com.gi https://*.google.gl https://*.google.gm https://*.google.gp https://*.google.gr https://*.google.com.gt https://*.google.gy https://*.google.com.hk https://*.google.hn https://*.google.hr https://*.google.ht https://*.google.hu https://*.google.co.id https://*.google.ie https://*.google.co.il https://*.google.im https://*.google.co.in https://*.google.iq https://*.google.is https://*.google.it https://*.google.je https://*.google.com.jm https://*.google.jo https://*.google.co.jp https://*.google.co.ke https://*.google.com.kh https://*.google.ki https://*.google.kg https://*.google.co.kr https://*.google.com.kw https://*.google.kz https://*.google.la https://*.google.com.lb https://*.google.li https://*.google.lk https://*.google.co.ls https://*.google.lt https://*.google.lu https://*.google.lv https://*.google.com.ly https://*.google.co.ma https://*.google.md https://*.google.me https://*.google.mg https://*.google.mk https://*.google.ml https://*.google.com.mm https://*.google.mn https://*.google.ms https://*.google.com.mt https://*.google.mu https://*.google.mv https://*.google.mw https://*.google.com.mx https://*.google.com.my https://*.google.co.mz https://*.google.com.na https://*.google.com.nf https://*.google.com.ng https://*.google.com.ni https://*.google.ne https://*.google.nl https://*.google.no https://*.google.com.np https://*.google.nr https://*.google.nu https://*.google.co.nz https://*.google.com.om https://*.google.com.pa https://*.google.com.pe https://*.google.com.pg https://*.google.com.ph https://*.google.com.pk https://*.google.pl https://*.google.pn https://*.google.com.pr https://*.google.ps https://*.google.pt https://*.google.com.py https://*.google.com.qa https://*.google.ro https://*.google.ru https://*.google.rw https://*.google.com.sa https://*.google.com.sb https://*.google.sc https://*.google.se https://*.google.vu https://*.google.ws https://*.google.rs https://*.google.co.za https://*.google.co.zm https://*.google.co.zw https://*.google.cat https://*.symantec.com https://*.mcafeesecure.com https://*.veinteractive.com https://*.hotjar.com https://*.youtube.com https://*.hotjar.com https://*.criteo.com *.dotomi.com http://*.rfksrv.com https://*.rfksrv.com; connect-src 'self' https://*.statsstory.com https://*.hotjar.com wss://*.hotjar.com https://*.veinteractive.com https://*.servicebus.windows.net https://*.optimizely.com https://*.youtube.com https://*.criteo.com; 9643 bytes! unsafe-inline unsafe-eval *.cloudfront.net tagmanager.google.com *.amazonaws.com
  23. 49

  24. 52 Referrer-Policy What to send in referrals from this page

    https://w3c.github.io/webappsec-referrer-policy/
  25. 53

  26. 54 Referrer policy options Policy Referrer (same origin) Referrer (to

    foreign origin) no-referrer omitted omitted no-referrer-when-downgrade https://example.com/page.html?query https://example.com/page.html?query origin https://example.com https://example.com origin-when-cross-origin https://example.com/page.html?query https://example.com same-origin https://example.com/page.html?query omitted Or just take my word for it and use 'origin-when-cross-origin'
  27. 56 Access-Control-Allow-Origin: requesting-origin.example.com Access-Control-Allow-Credentials: true Access-Control-Max-Age: 86400 Access-Control-Expose-Headers: X-Debug Access-Control-Allow-Headers:

    DNT, User-Agent Allow authenticated cross-origin requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/frame-ancestors
  28. 58

  29. 59 Accept-CH Requests client hints data be sent in future

    http://httpwg.org/http-extensions/client-hints.html
  30. 60 Accept-CH: DPR, Width, Viewport-Width, Save-Data Accept-CH-Lifetime: 86400 Request that

    client send specified CHs: Sec-CH-DPR: 2.0 Sec-CH-Width: 320 Sec-CH-Viewport-Width: 320 Sec-CH-Save-Data: 1 Subsequent requests:
  31. 62 Proposed new client hints! Sec-CH-UA: "Google Chrome 74" Sec-CH-UA-Platform:

    "macOS 12" Sec-CH-UA-Arch: "ARM64" Sec-CH-UA-Engine: "Blink" Sec-CH-Lang: "en-US", "en", "de"
  32. 63 Chrome has already started sending these sec-ch-ua is sent

    by default: not subject to the Accept-CH opt-in
  33. 64 Link: rel=preload Declare a resource that’s important early on

    https://w3c.github.io/preload/#x2.link-type-preload
  34. 68 The status code problem DNS Lookup TLS TTFB Status

    code + LINK headers received Database Auth Templating API queries
  35. 70 HTTP/1.1 103 Early Hints Link: <some-font-face.woff2>; rel="preload"; as="font"; crossorigin

    Link: <main-styles.css>; rel="preload"; as="style" HTTP/1.1 200 OK Date: Fri, 26 May 2017 10:02:11 GMT Content-Length: 1234 Content-Type: text/html; charset=utf-8 Link: <some-font-face.woff2>; rel="preload"; as="font"; crossorigin Link: <main-styles.css>; rel="preload"; as="style" <!doctype html> Get your fonts and styles down even sharper:
  36. 71 Preloaded resources start preloading earlier DNS Lookup TLS Early

    hints + Link headers received Real status received
  37. 74 Server-timing metrics appear in developer tools But.. no sequencing,

    and the only unit for numeric values is milliseconds
  38. 77 Server-Timing: customerID;desc=1234567, edition;desc="EMEA" Server-Timing: paywallArticlesRemaining;desc=8 Tell the browser…. anything?:

    performance.getEntriesByType('navigation')[0] .serverTiming .reduce((soFar, entry) => ({ ...soFar, [entry.name]: entry.description }), {}) ; {customerID: "1234567", edition: "EMEA", paywallArticlesRemaining: "8"} So we could, like, embed post-cache metadata into headers?
  39. 78 But server-timing might be unknown in headers DNS Lookup

    TLS TTFB Headers sent Only now we know: Total bytes sent Last byte timestamp Client data rate This is a bit like the the Early-Hints situation!
  40. 80

  41. 82 HTTP/1.1 200 OK Date: Tue, 27 Feb 2018 13:28:47

    GMT Content-Type: text/html Content-Length: 12 Server-Timing: miss, db;dur=53, app;dur=47.2 Trailers: Server-Timing Hello world! Server-Timing: sentduration;dur=234 Headers, but like, at the end?
  42. 85 Feature-Policy: autoplay 'none'; camera 'self'; unsized-media some3rdparty.com Turn off

    bad practices No-one can autoplay video on this page Only I can use the camera. Images only take size from their contents when loaded from some3rdparty.com
  43. 86 What policies are available? document.featurePolicy.allowedFeatures().sort().join('\n') accelerometer, ambient-light-sensor, autoplay, camera,

    ch-device-memory, ch-downlink, ch-dpr, ch-ect, ch-lang, ch-rtt, ch-ua, ch-ua-arch, ch-ua-model, ch-ua-platform, ch-viewport-width, ch-width, document-access, document-domain, document-write, downloads-without-user-activation, encrypted-media, execution-while-not-rendered, execution-while-out-of-viewport, focus-without-user-activation, font-display-late-swap, forms, fullscreen, geolocation, gyroscope, hid, idle-detection, layout-animations, lazyload, loading-frame-default-eager, magnetometer, microphone, midi, modals, orientation-lock, oversized-images, payment, picture-in-picture, pointer-lock, popups, presentation, scripts, serial, sync-script, sync-xhr, top-navigation, unoptimized-lossless-images, unoptimized-lossless-images-strict, unoptimized-lossy-images, unsized-media, usb, vertical-scroll, vr, wake-lock, xr
  44. 87 Feature-Policy: accelerometer 'none'; ambient-light-sensor 'none'; animations 'none'; autoplay 'none';

    camera 'none'; document-write 'none'; encrypted-media 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; image-compression 'none'; legacy-image-formats 'none'; magnetometer 'none'; max-downscaling-image 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture 'none'; speaker 'none'; sync-script 'none'; sync-xhr 'none'; unsized-media 'none'; usb 'none'; vertical-scroll 'none'; vr 'none'; Today's tightest feature-policy (in Chrome):
  45. 88

  46. 90 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com

    HTTP/1.1 200 OK Date: Tue, 27 Feb 2018 13:28:47 GMT Content-Type: text/html Content-Length: 34882 <html> ...
  47. 91 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com

    HTTP/1.1 200 OK Content-Type: text/html Content-Length: 34882 Cache-Control: private, max-age=3600 Last-Modified: Fri, 02 Feb 2018 07:21:05 GMT X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=31536000; includeSubdomains; Content-Security-Policy: default-src 'self'; report-uri https://csp.example.com/ Expect-CT: max-age=86400, enforce, report-uri="https://ect.example.com/" Alt-Svc: h2="new.example.com:443"; ma=600 Link: /script/bundle.rev-983c15.js>;rel=preload;as=script;charset=UTF-8; Accept-CH: DPR, Width, Viewport-Width Feature-Policy: vibrate 'none'; geolocation 'none'; unsized-media ‘none’ <html> ...
  48. 92 { "headers": [ { "name": "Content-Security-Policy", "value": "script-src 'self'

    https://cdn.example.com", "type": "fallback" }, { "name": "Referrer-Policy", "value": "origin-when-cross-origin", "type": "fallback" }, { "name": "Content-Security-Policy", "value": "object-src 'none'; frame-ancestors 'none'", "type": "baseline" }, { "name": "Strict-Transport-Security", "value": "max-age=10886400; includeSubDomains; preload", "type": "baseline" }, { "name": "X-Content-Type-Options", "value": "nosniff", "type": "baseline" } ], "cors-preflight": { "origins": "*" } } /.well-known/origin-policy/policy-1
  49. 93 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com

    HTTP/1.1 200 OK Content-Type: text/html Content-Length: 34882 Cache-Control: private, max-age=3600 Last-Modified: Fri, 02 Feb 2018 07:21:05 GMT X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=31536000; includeSubdomains; Content-Security-Policy: default-src 'self'; report-uri https://csp.example.com/ Expect-CT: max-age=86400, enforce, report-uri="https://ect.example.com/" Alt-Svc: h2="new.example.com:443"; ma=600 Link: /script/bundle.rev-983c15.js>;rel=preload;as=script;charset=UTF-8; Accept-CH: DPR, Width, Viewport-Width Feature-Policy: vibrate 'none'; geolocation 'none'; unsized-media ‘none’ <html> ...
  50. 94 $ telnet bank.example.com 80 GET /statement HTTP/1.1 Host: bank.example.com

    HTTP/1.1 200 OK Content-Type: text/html Content-Length: 34882 Cache-Control: private, max-age=3600 Last-Modified: Fri, 02 Feb 2018 07:21:05 GMT Sec-Origin-Policy: "policy-1" Vary: sec-origin-policy <html> ...
  51. 95 The whole talk in one slide Use these: •

    Clear-Site-Data • Content-Security-Policy • Strict-Transport-Security • Referrer-Policy • Access-Control-* (CORS) • Accept-CH • Link (rel=preload) • Server-Timing • P3P • Expires • X-Cache • X-Frame-Options • Via Don't use these: Follow these: • Feature-Policy • Origin-Policy • CDN-Loop (if you're a CDN) • Cache • Proxy-Status