Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Securing and managing dependencies at scale

Securing and managing dependencies at scale

PayPay Corporation.

July 07, 2021
Tweet

More Decks by PayPay Corporation.

Other Decks in Technology

Transcript

  1. Ho YuehHsun - Joined PayPay at 2019/08. - DevSecOps team.

    - Focus on Dev side security. - Interests: Travel in Japan.
  2. - Reduce 70% percentage of vulnerabilities. - Finish migration on

    ~180 services in a quarter. - Finished all front-facing services. - Finished all backend services. Reduce vulnerabilities at scale Backend migration Front-end migration
  3. • A vulnerability is a bug which can be exploited

    by an attacker. ◦ Known vulnerabilities is one of the most critical cyber risks. ◦ CVEs (Common Vulnerabilities and Exposure) identify and catalog vulnerabilities. ▪ Severity level: Critical, High, Medium, Low ◦ An attack can facilitate data loss or server takeover. ▪ Equifax breach in 2017 ◦ Known vulnerabilities have exploit kits. Vulnerabilities and Exploits
  4. • How to prevent: ◦ Reducing threat surface. ▪ Remove

    unneeded dependencies, components, files, and documentation. ◦ Continuously monitor CVEs for vulnerabilities in components. ◦ Fix or upgrade underlying platforms, frameworks, and dependencies regularly. Vulnerabilities and Exploits
  5. - Micro-Service architecture - Services built from many 3rd-party and

    open-source dependencies. PayPay’s infrastructure Payment Finnet Wallet App POS Online ©PayPay Corporation.All rights reserved. CLM ...
  6. • Upgrade the version in gradle build file ◦ Teams

    doesn’t always know the best way to fix a vulnerability. ▪ One by one, each by each ▪ Resolution Strategy (Gradle) impl 'com.fasterxml.jackson.core:jackson-core:2.12.2' // previously 2.12.0 How to upgrade to a non-vulnerable version configurations.all { resolutionStrategy.eachDependency { DependencyResolveDetails details -> if (details.requested.group == 'org.apache.tomcat.embed') { details.useVersion '8.5.60' details.because 'Security Fixes (Multiple Issues)' } } }
  7. Dependency Management in build system • Maven ◦ BOM (Bills

    of Materials): ▪ BOM is a kind of POM file that is used to suggest the versions of a projects dependencies and provide a central place to define and update those versions. ▪ The version can also be overridden.
  8. spring-boot-dependencies BOM: <dependencyManagement> ... <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-test</artifactId> <version>2.3.11.RELEASE</version> </dependency> <dependency>

    <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-test-autoconfigure</artifactId> <version>2.3.11.RELEASE</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-actuator</artifactId> <version>2.3.11.RELEASE</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-actuator-autoconfigure</artifactId> <version>2.3.11.RELEASE</version> </dependency> ... </dependencyManagement>
  9. • Import a BOM in Maven • Don’t need to

    declare the version if the BOM is imported <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> </dependencies> <dependencyManagement> <dependencies> <dependency> <!-- Import dependency management from Spring Boot --> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-dependencies</artifactId> <version>2.3.11.RELEASE</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement>
  10. Dependency Management in build system - Gradle - Dependency Management

    Plugin - A plugin created by the Gradle team to implement Maven’s dependency management. plugins { id 'org.springframework.boot' version '2.3.11.RELEASE' } ext { springBootVersion = ‘2.3.11.RELEASE’ } dependencyManagement { imports { mavenBom('org.springframework.boot:spring-boot-dependencies:${springBootVersion}') } }
  11. • Gradle Platform ◦ A platform is a software component

    in gradle system which can be used to control, suggest, enforce dependency versions. ◦ Maven BOM is also a kind of platform that gradle supports. ext { springBootVersion = ‘2.3.11.RELEASE’ } implementation platform(“org.springframework.boot:spring-boot-dependencies:${springBootVersion}”)
  12. Solution: BOM • A dependency management artifact that allows us

    to control the versions for a set of core dependencies. • Advantages: ◦ Reduce the burden for team to maintain dependencies individually. ◦ Define non-vulnerable versions of dependencies. ◦ Introduce consistency of dependencies across services. ◦ Version can be overridden.
  13. Solution: BOM impl ('jp.ne.paypay:paypay-lib1:2.3.5') - CVE1, CVE2 impl ('jp.ne.paypay:paypay-lib2:4.1.7') -

    CVE3 impl ('com.foo:foo-bar:1.0.1') impl platform('paypay-bom:1.0.0') impl ('jp.ne.paypay:paypay-lib1') (0 CVE) impl ('jp.ne.paypay:paypay-lib2') (0 CVE) impl ('com.foo:foo-bar') PayPay BOM - 'jp.ne.paypay:paypay-lib1:2.4.0' CVE1, CVE2 - 'jp.ne.paypay:paypay-lib2:4.2.0' CVE3 - 'com.foo:foo-bar:1.0.1'
  14. Solution: BOM PayPay BOM lib1:1.0.0 - Fix CVE1 CRITICAL lib2:1.1.0

    - Fix CVE2 CRITICAL lib3:2.1.1 - Fix CVE3 CRITICAL … libX:2.0.0 - Fix CVEX LOW Payment Finnet Wallet POS Online ©PayPay Corporation.All rights reserved. CLM ...
  15. Create the BOM • Preliminary work: ◦ Build System: gradle

    or maven? ◦ Dependency selection. ◦ Dependency constraints selection. ◦ Naming and versioning.
  16. ◦ Spring Boot and Spring Cloud dependencies based. ▪ Spring

    Boot: 2.3.X, 2.4.X. ▪ Spring Cloud: corresponding versions. ◦ Internal PayPay Libraries. ◦ Internal PayPay BOMs. ◦ Fixed version of vulnerable dependencies that is critical or high severity level. Create the BOM: dependency selection
  17. • Dependency constraints selection ◦ Vulnerable dependencies: ▪ The minimum

    version constraints that could fix the vulnerability. ▪ Newest version is preferred. com.lib:lib.bar:1.2.0 - CVE 1 - CVE 2 com.lib:lib.bar:1.2.1 - CVE 1 - CVE 2 com.lib:lib.bar:1.3.0 - CVE 1 - CVE 2 ✅ Create the BOM: dependency constraint selection
  18. ◦ Name: paypay-spring-bom ◦ Version: ▪ Format: ▪ Use Spring-Boot

    version as part of the BOM version. • Developers can easily determine the Spring-Boot version. • example: ▪ Add a patch number: PPX to the version indicating minor changes. • X is a number and increased by 1 when there’s a minor change. ${SpringBootVersion}-PPX Create the BOM: naming and versioning paypay-spring-bom:2.3.6
  19. BOM: 2.3.6-PP1 impl 'paypay-lib1' - CVE1 CRITICAL impl 'paypay-lib2' impl

    'com.foo:foo-bar' BOM: 2.3.6-PP2 impl 'paypay-lib1' - CVE1 impl 'paypay-lib2' impl 'com.foo:foo-bar' Upgrade dependencies included in BOM fix CVE1 Minor Changes (1)
  20. BOM: 2.3.6-PP1 impl 'paypay-lib1' impl 'paypay-lib2' impl 'com.foo:foo-bar' BOM: 2.3.6-PP2

    impl 'paypay-lib1 impl 'paypay-lib2' impl 'com.foo:foo-bar' impl 'paypay-lib3' Additional new dependencies to the BOM that do not require an upgrade to the Spring-Boot version. add paypay-lib3:1.0.0 Minor Changes (2)
  21. How to create a BOM using gradle • Use Java

    Platform Plugin ◦ constraints, api to limit the version of a dependency ◦ platform to source other BOMs. ◦ Declare myPlatform in the publishing task to publish the artifact. dependencies { constraints { api 'commons-httpclient:commons-httpclient:3.1' api platform("org.springframework.boot:spring-boot-dependencies:2.3.11.RELEASE") } } publishing { publications { myPlatform(MavenPublication) { from components.javaPlatform } } }
  22. BoM project structure • Multi-module structure ◦ A parent build.gradle

    file at root level controls all the version information for each BOMs. ◦ Under each BOM folder there’s a build.gradle file for respective BOMs. project |--/boot-2.3 | |--/build.gradle // for 2.3 BOM |--/boot-2.4 | |--/build.gradle // for 2.4 BOM | |--... |--build.gradle
  23. • The root level build.gradle contains the following for each

    BOM: ◦ Single dependency constraints. ◦ Platform source versions and published version. // keyword: api dependentMap = [ 'boot-2.3': [ 'org.mybatis.spring.boot:mybatis-spring-boot-starter:2.1.4', // fix CVE 'org.hibernate.validator:hibernate-validator:6.1.7.Final', // fix CVE 'com.alibaba:fastjson:1.2.76' ], 'boot-2.4': [ ... ] ] // keyword: platform versions = [ 'boot-2.3': [ 'paypaySpringBomVersion': '2.3.11-PP2', // published version 'springbootVersion': '2.3.11.RELEASE', // platform source version ], 'boot-2.4': [ ... ] ] project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle
  24. // do this to all subprojects subprojects { javaPlatform {

    allowDependencies() } dependencies { constraints { // check for subproject.name if (dependentMap.containsKey(project.name)) { dependentMap[project.name].each { dep -> api(dep) } } } } } project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle
  25. • Apply the version constraints to each BoM version =

    "${paypaySpringBomVersion}" // published version dependencies { ... api platform("org.springframework.boot:spring-boot-dependencies:${springbootVersion}") api platform("org.springframework.cloud:spring-cloud-dependencies:${springbootVersion}") } project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle
  26. BOM verification • Integration test ◦ To catch regression. ◦

    To catch transitive dependency problem (this will happen). ◦ Testing are shared among teams. ▪ @EnableScheduling R |-- B:1.0.0 // newer Spring Boot doesn’t include this |-- C:1.0.0
  27. BoM migration • Goal: Services migrate to BOM and remove

    vulnerabilities and make it easier to remove new ones. • Two phases: ◦ Front-facing Service: ▪ Services that connect to outside world directly. ▪ API gateways, third-party connections. ◦ Backend services: ▪ All Java services.
  28. - Reduce 70% percentage of vulnerabilities. - Finish migration on

    ~180 services in a quarter. - Finished all front-facing services. - Finished all backend services. BoM migration: Result Backend migration Front-end migration