Share
Embed
Introduction to Software Supply Chain Security & Tooling
You’re familiar with containers and Kubernetes, but suddenly people are talking about supply chains, and you’re pretty sure they don’t mean the Suez Canal. Join Jeffrey Sica and Bob Killen for an in-depth look at the nuts, bolts, and metaphorical shipping containers that make up modern container supply chain security. They’ll demo sigstore and show how container signing automation works with Kubernetes. You’ll leave prepared to make decisions that will keep your org unstuck.
Bob Killen
August 05, 2022
Transcript
104 slides
-
$ whoami - Bob Bob Killen [email protected] OSS Program Manager
@ Google Github: @mrbobbytables Twitter: @mrbobbytables Site: mrbobbytabl.es -
$ whoami - Jeff Jeffrey Sica [email protected] DevEx @ CNCF
Github: @jeefy Twitter: @jeefy Site: jeefy.dev -
Before We Begin Pt. 2 Docker will be required to
follow along with the demos. Docker is not required to use the tooling shown. https://docs.docker.com/engine/install/#desktop -
Before We Begin Pt. 3 KinD (Kubernetes in Docker) and
Helm will be used for later demos. https://kind.sigs.k8s.io/docs/user/quick-start/#installation https://helm.sh/docs/intro/install/ -
Containers: Galaxy-scale overview • Package an application with all of
its dependencies (container image) • Run instances of the packaged application (container) • Think “Virtual Machine” but sharing kernel space and OS libraries (cgroups) -
Kubernetes: Galaxy-scale overview • “Container Orchestration” – It takes an
entire orchestra to play a symphony • Manages containers and their dependencies (Storage, Networking, third-party resources) in a declarative way • 100% API Driven -
Solarwinds Attack • SolarWinds Orion network management tool’s build servers
compromised • Hack injected a malicious DLL at build time that was signed and distributed to clients automatically when they updated • 18,000 SolarWinds customers including many Fortune 500 Companies, Government Agencies and Tier 1 Network Providers were impacted Project Trebuchet: How SolarWinds is Using Open Source to Secure Their Supply Chain in the Wake of the Sunburst Hack -
Solarwinds Attack • SolarWinds Orion network management tool’s build servers
compromised • Hack injected a malicious DLL at build time that was signed and distributed to clients automatically when they updated • 18,000 SolarWinds customers including many Fortune 500 Companies, Government Agencies and Tier 1 Network Providers were impacted Project Trebuchet: How SolarWinds is Using Open Source to Secure Their Supply Chain in the Wake of the Sunburst Hack NO source code compromised -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between the Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between the Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between the Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities Work on OSS? Want a security key? Find me after and get a free Titan key. -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between the Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities -
US Gov Policy Changes Executive Order 14028 - (Sec. 2)
Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities -
Supply Chain Attacks are on the rise and the Software
Development Lifecycle has become a popular vector for attacks. -
Source Integrity Build Integrity Source Build Package Dev Dependencies Users
Software Supply Chain Artifact Process -
Source Integrity Build Integrity Source Build Package Dev Inject bad
code (A) Compromise source control (B) Build from modified sources (C) Dependencies Compromise build system (D) Bypass CI/CD, inject bad artifact (F) Compromise package repository (G) Use bad package (H) Users Software Supply Chain: Vulnerability Points Artifact Process Inject bad or vulnerable dependency (E) -
Source Integrity Build Integrity Source Build Package Dev Inject bad
code (A) Compromise source control (B) Build from modified sources (C) Dependencies Compromise build system (D) Bypass CI/CD, inject bad artifact (F) Compromise package repository (G) Use bad package (H) Users Software Supply Chain: Trust Boundaries Artifact Process Inject bad or vulnerable dependency (E) -
SLSA Levels Automation & Provenance Build must be fully scripted/automated
and generate provenance Version Control & Signed Provenance Requires using version control and hosted build service that generates authenticated provenance Non-falsifiable, Ephemeral Builds are fully trustworthy, with identity attestations of underlying build infrastructure/hardware. Ephemeral builds leave nothing behind. Hermetic Builds, Review All build inputs/dependencies are specified upfront with no internet egress during the build. Two-party reviews. https://slsa.dev/spec/v0.1/requirements -
Build Provenance Set of metadata that describes a software artifact
and how it was built. SLSA spec fields* - subject - artifact(s) information (name+digest) - builder - the entity that produced the artifact(s) - invocation - execution command / information - buildConfig - record of steps executed during build - materials - other artifacts/dependencies used during build -
{ "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":
{ "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} -
{ "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":
{ "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} -
{ "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":
{ "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} builder - Entity that produced the artifact Examples: "builder": { "id": "mailto:[email protected]" } "builder": { "id": "https://github.com/Attestations/GitHubHostedActions@v1" } "builder": { "id": "https://gitlab.com/foo/bar/-/runners/12345678" } -
{ "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":
{ "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} invocation - Execution command / information Example: "invocation": { "configSource": { "uri": "git+https://github.com/foo/bar.git, "digest": { "sha1": "1234..."}, //git commit hash "entryPoint": "build.yaml:build" }, "parameters": {"inputs": {}}, "environment": { “arch”: “amd64”, "env": { "GITHUB_RUN_ID": "1234", "GITHUB_RUN_NUMBER": "5678", "GITHUB_EVENT_NAME": "push" } } } -
{ "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":
{ "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} materials - other artifacts/dependencies used during build Example: "materials": [{ "uri": "https://example.com/example-1.2.3.tar.gz", "digest": {"sha256": "1234..."} }] "materials": [{ "uri": "git+https://github.com/foo/bar.git", "digest": {"sha1": "abc..."} }] -
{ "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":
{ "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} buildConfig - record of steps executed during build Example: "buildConfig": { "steps": [ { "image": "pkg:docker/make@sha256:244fd47e07d1004f0aed9c", "arguments": ["build"] } ] } "buildConfig": { "commands": [ "./configure --enable-some-feature", "make foo.zip" ], "shell": "bash" } -
{ "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":
{ "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} subject - Artifact(s) information (name+digest) Examples: "subject": [{"name": "_", "digest": {"sha256": "5678..."}}] "subject": [ { "name": "herpderp.exe", "digest": {"sha256": "1234..."} }] -
Software Attestation A software attestation is a signed set metadata
(provenance) about one or more software artifacts. { "payload": "Gew9gICJzdWJqZWN0IjogWwogICAg...", "payloadType": "application/vnd.in-toto+json", "signatures": [{ "keyid": "my-awesome-builder-key", //optional "sig": "Re4ya66MyFyc9Y..." }] } -
{ "payload": "Gew9gICJzdWJqZWN0IjogWwogICAg...", "payloadType": "application/vnd.in-toto+json", "signatures": [{ "keyid": "my-awesome-builder", "sig":
"Re4ya66MyFyc9Y..." }] } { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ "name": "_", "digest": {"sha256": "5678..."} }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate": { "builder": { "id": "https://github.com/Attestations/GitHubHostedActions@v1" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" } … -
{ "payload": "Gew9gICJzdWJqZWN0IjogWwogICAg...", "payloadType": "application/vnd.in-toto+json", "signatures": [{ "keyid": "my-awesome-builder", "sig":
"Re4ya66MyFyc9Y..." }] } { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ "name": "_", "digest": {"sha256": "5678..."} }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate": { "builder": { "id": "https://github.com/Attestations/GitHubHostedActions@v1" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" } … Thanks SLSA! -
An Overview Of sigstore • fulcio ◦ Identity • rekor
◦ Immutable log with artifact metadata • cosign ◦ Utility to automate combining fulcio, rekor, and OCI-compliant artifacts ◦ Utilizes underlying sigstore libraries -
fulcio Free Root-CA for code signing certs - issuing certificates
based on an OIDC email address fulcio only signs short-lived certificates that are valid for under 20 minutes https://docs.sigstore.dev/fulcio/overview https://github.com/sigstore/fulcio -
fulcio Free Root-CA for code signing certs - issuing certificates
based on an OIDC email address. fulcio only signs short-lived certificates that are valid for under 20 minutes. https://twitter.com/jacques_chester/status/1506332697387491335 -
rekor Greek for “Record” rekor is a transparency log where
anyone can find and verify signatures, and check whether someone’s changed the source code, the build platform or the artifact repository. https://docs.sigstore.dev/rekor/overview https://github.com/sigstore/rekor -
cosign cosign: Container Signing, verification and storage in an OCI
registry “Make signatures invisible infrastructure” Pluggable components https://docs.sigstore.dev/cosign/overview https://github.com/sigstore/cosign -
Demo Recap • Built a container • Auth’d using cosign
• Signed a container using cosign • Inspected the container’s signature using cosign -
Kubernetes Admission Webhooks Lightweight extensible way to “do something” with
API Server requests Two kinds: - Validating Admission Webhook - Mutating Admission Webhook -
Kubernetes Admission Webhooks Lightweight extensible way to “do something” with
API Server requests Two kinds: - Validating Admission Webhook - Mutating Admission Webhook -
Kubernetes Admission Webhooks … webhooks: - name: "pod-policy.example.com" rules: -
apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE"] resources: ["pods"] scope: "Namespaced" -
Kubernetes Admission Webhooks Lightweight extensible way to “do something” with
API Server requests Two kinds: - Validating Admission Webhook - Mutating Admission Webhook -
sigstore Policy Controller … apiVersion: policy.sigstore.dev/v1beta1 kind: ClusterImagePolicy metadata: name:
image-policy spec: images: - glob: "**" -
sigstore Policy Controller … apiVersion: policy.sigstore.dev/v1beta1 kind: ClusterImagePolicy metadata: name:
image-policy spec: images: - glob: "**" Beta -
Demo Recap • Installed Kyverno • Attempted to create a
Pod with an unsigned container image • Successfully created a Pod with a signed container image Task Failed Successfully! -
Demo Recap • Installed the Kyverno • Attempted to create
a Pod with an unsigned container image • Successfully created a Pod with a signed container image Task Failed Successfully! -
GitHub Action jobs: test_cosign_action: runs-on: ubuntu-latest permissions: {} name: Install
Cosign and test presence in path steps: - name: Install Cosign uses: sigstore/cosign-installer@main - name: Check install! run: cosign version https://github.com/sigstore/cosign-installer -
GitHub Action … - name: Sign image with a key
run: | cosign sign --key env://COSIGN_PRIVATE_KEY ${TAGS} env: TAGS: ${{ steps.docker_meta.outputs.tags }} COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}} COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}} https://github.com/sigstore/cosign-installer -
Batteries included support for cosign Create a secret called signing-secrets
with the following structure: • cosign.key • cosign.password cosign generate-key-pair k8s://tekton-chains/signing-secrets Tekton (Chains) -
Demo Recap • Set up GitHub Action • Use cosign
to sign the image • Use SLSA tooling to generate attestation • User cosign to attach attestation in rekor • Inspected and verified the image and SLSA payload -
gitsign Keyless Git signing with Sigstore! Homebrew: brew install sigstore/tap/gitsign
Go: go install github.com/sigstore/gitsign@latest -
gitsign # Sign all commits git config --global commit.gpgsign true
# Sign all tags git config --global tag.gpgsign true # Use gitsign for signing git config --global gpg.x509.program gitsign # gitsign expects x509 args git config --global gpg.format x509 https://github.com/sigstore/gitsign -
Demo Recap • Signed a git commit with gitsign •
Inspected the rekor log of the commit -
Tutorial Recap • Supply chain security is an increasing concern
when building, shipping, and running code. • The SLSA framework can be used to enable or improve trust in the software that we build and run. -
Tutorial Recap • sigstore looks to solve supply chain security
with an end to end solution based on linking Identity and TLS certificates • Automating SLSA provenance generation and artifact signing lowers the mental load on developers shipping software -
Tutorial Recap • Starts with Containers / OCI-compliant artifacts •
Already moving further down the stack to packages -
SLSA Framework Build Integrity • Modification of code after source
control • Compromised build platforms • Bypassing CI/CD Source Integrity • Available change history • Code review • Compromised source control systems Dependencies • Applying SLSA checks recursively to dependencies • Dependency confusion