Securing and managing dependencies at scale

Transcript

1. Securing and managing dependencies at scale Ho YuehHsun
2. Ho YuehHsun - Joined PayPay at 2019/08. - DevSecOps team. - Focus on Dev side security. - Interests: Travel in Japan.
3. - Reduce 70% percentage of vulnerabilities. - Finish migration on ~180 services in a quarter. - Finished all front-facing services. - Finished all backend services. Reduce vulnerabilities at scale Backend migration Front-end migration
4. • A vulnerability is a bug which can be exploited by an attacker. ◦ Known vulnerabilities is one of the most critical cyber risks. ◦ CVEs (Common Vulnerabilities and Exposure) identify and catalog vulnerabilities. ▪ Severity level: Critical, High, Medium, Low ◦ An attack can facilitate data loss or server takeover. ▪ Equifax breach in 2017 ◦ Known vulnerabilities have exploit kits. Vulnerabilities and Exploits
5. • How to prevent: ◦ Reducing threat surface. ▪ Remove unneeded dependencies, components, files, and documentation. ◦ Continuously monitor CVEs for vulnerabilities in components. ◦ Fix or upgrade underlying platforms, frameworks, and dependencies regularly. Vulnerabilities and Exploits
6. - Micro-Service architecture - Services built from many 3rd-party and open-source dependencies. PayPay’s infrastructure Payment Finnet Wallet App POS Online ©PayPay Corporation.All rights reserved. CLM ...
7. • Upgrade the version in gradle build file ◦ Teams doesn’t always know the best way to fix a vulnerability. ▪ One by one, each by each ▪ Resolution Strategy (Gradle) impl 'com.fasterxml.jackson.core:jackson-core:2.12.2' // previously 2.12.0 How to upgrade to a non-vulnerable version configurations.all { resolutionStrategy.eachDependency { DependencyResolveDetails details -> if (details.requested.group == 'org.apache.tomcat.embed') { details.useVersion '8.5.60' details.because 'Security Fixes (Multiple Issues)' } } }
8. Dependency Management in build system • Maven ◦ BOM (Bills of Materials): ▪ BOM is a kind of POM file that is used to suggest the versions of a projects dependencies and provide a central place to define and update those versions. ▪ The version can also be overridden.
9. spring-boot-dependencies BOM: <dependencyManagement> ... <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-test</artifactId> <version>2.3.11.RELEASE</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-test-autoconfigure</artifactId> <version>2.3.11.RELEASE</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-actuator</artifactId> <version>2.3.11.RELEASE</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-actuator-autoconfigure</artifactId> <version>2.3.11.RELEASE</version> </dependency> ... </dependencyManagement>
10. • Import a BOM in Maven • Don’t need to declare the version if the BOM is imported <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> </dependencies> <dependencyManagement> <dependencies> <dependency> <!-- Import dependency management from Spring Boot --> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-dependencies</artifactId> <version>2.3.11.RELEASE</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement>
11. Dependency Management in build system - Gradle - Dependency Management Plugin - A plugin created by the Gradle team to implement Maven’s dependency management. plugins { id 'org.springframework.boot' version '2.3.11.RELEASE' } ext { springBootVersion = ‘2.3.11.RELEASE’ } dependencyManagement { imports { mavenBom('org.springframework.boot:spring-boot-dependencies:${springBootVersion}') } }
12. • Gradle Platform ◦ A platform is a software component in gradle system which can be used to control, suggest, enforce dependency versions. ◦ Maven BOM is also a kind of platform that gradle supports. ext { springBootVersion = ‘2.3.11.RELEASE’ } implementation platform(“org.springframework.boot:spring-boot-dependencies:${springBootVersion}”)
13. Solution: BOM • A dependency management artifact that allows us to control the versions for a set of core dependencies. • Advantages: ◦ Reduce the burden for team to maintain dependencies individually. ◦ Define non-vulnerable versions of dependencies. ◦ Introduce consistency of dependencies across services. ◦ Version can be overridden.
14. Solution: BOM impl ('jp.ne.paypay:paypay-lib1:2.3.5') - CVE1, CVE2 impl ('jp.ne.paypay:paypay-lib2:4.1.7') - CVE3 impl ('com.foo:foo-bar:1.0.1') impl platform('paypay-bom:1.0.0') impl ('jp.ne.paypay:paypay-lib1') (0 CVE) impl ('jp.ne.paypay:paypay-lib2') (0 CVE) impl ('com.foo:foo-bar') PayPay BOM - 'jp.ne.paypay:paypay-lib1:2.4.0' CVE1, CVE2 - 'jp.ne.paypay:paypay-lib2:4.2.0' CVE3 - 'com.foo:foo-bar:1.0.1'
15. Solution: BOM PayPay BOM lib1:1.0.0 - Fix CVE1 CRITICAL lib2:1.1.0 - Fix CVE2 CRITICAL lib3:2.1.1 - Fix CVE3 CRITICAL … libX:2.0.0 - Fix CVEX LOW Payment Finnet Wallet POS Online ©PayPay Corporation.All rights reserved. CLM ...
16. Create the BOM • Preliminary work: ◦ Build System: gradle or maven? ◦ Dependency selection. ◦ Dependency constraints selection. ◦ Naming and versioning.
17. ◦ Spring Boot and Spring Cloud dependencies based. ▪ Spring Boot: 2.3.X, 2.4.X. ▪ Spring Cloud: corresponding versions. ◦ Internal PayPay Libraries. ◦ Internal PayPay BOMs. ◦ Fixed version of vulnerable dependencies that is critical or high severity level. Create the BOM: dependency selection
18. • Dependency constraints selection ◦ Vulnerable dependencies: ▪ The minimum version constraints that could fix the vulnerability. ▪ Newest version is preferred. com.lib:lib.bar:1.2.0 - CVE 1 - CVE 2 com.lib:lib.bar:1.2.1 - CVE 1 - CVE 2 com.lib:lib.bar:1.3.0 - CVE 1 - CVE 2 ✅ Create the BOM: dependency constraint selection
19. ◦ Name: paypay-spring-bom ◦ Version: ▪ Format: ▪ Use Spring-Boot version as part of the BOM version. • Developers can easily determine the Spring-Boot version. • example: ▪ Add a patch number: PPX to the version indicating minor changes. • X is a number and increased by 1 when there’s a minor change. ${SpringBootVersion}-PPX Create the BOM: naming and versioning paypay-spring-bom:2.3.6
20. BOM: 2.3.6-PP1 impl 'paypay-lib1' - CVE1 CRITICAL impl 'paypay-lib2' impl 'com.foo:foo-bar' BOM: 2.3.6-PP2 impl 'paypay-lib1' - CVE1 impl 'paypay-lib2' impl 'com.foo:foo-bar' Upgrade dependencies included in BOM fix CVE1 Minor Changes (1)
21. BOM: 2.3.6-PP1 impl 'paypay-lib1' impl 'paypay-lib2' impl 'com.foo:foo-bar' BOM: 2.3.6-PP2 impl 'paypay-lib1 impl 'paypay-lib2' impl 'com.foo:foo-bar' impl 'paypay-lib3' Additional new dependencies to the BOM that do not require an upgrade to the Spring-Boot version. add paypay-lib3:1.0.0 Minor Changes (2)
22. Final format: paypay-spring-bom:${SpringBootVersion}-PPX
23. How to create a BOM using gradle • Use Java Platform Plugin ◦ constraints, api to limit the version of a dependency ◦ platform to source other BOMs. ◦ Declare myPlatform in the publishing task to publish the artifact. dependencies { constraints { api 'commons-httpclient:commons-httpclient:3.1' api platform("org.springframework.boot:spring-boot-dependencies:2.3.11.RELEASE") } } publishing { publications { myPlatform(MavenPublication) { from components.javaPlatform } } }
24. BoM project structure • Multi-module structure ◦ A parent build.gradle file at root level controls all the version information for each BOMs. ◦ Under each BOM folder there’s a build.gradle file for respective BOMs. project |--/boot-2.3 | |--/build.gradle // for 2.3 BOM |--/boot-2.4 | |--/build.gradle // for 2.4 BOM | |--... |--build.gradle
25. • The root level build.gradle contains the following for each BOM: ◦ Single dependency constraints. ◦ Platform source versions and published version. // keyword: api dependentMap = [ 'boot-2.3': [ 'org.mybatis.spring.boot:mybatis-spring-boot-starter:2.1.4', // fix CVE 'org.hibernate.validator:hibernate-validator:6.1.7.Final', // fix CVE 'com.alibaba:fastjson:1.2.76' ], 'boot-2.4': [ ... ] ] // keyword: platform versions = [ 'boot-2.3': [ 'paypaySpringBomVersion': '2.3.11-PP2', // published version 'springbootVersion': '2.3.11.RELEASE', // platform source version ], 'boot-2.4': [ ... ] ] project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle
26. // do this to all subprojects subprojects { javaPlatform { allowDependencies() } dependencies { constraints { // check for subproject.name if (dependentMap.containsKey(project.name)) { dependentMap[project.name].each { dep -> api(dep) } } } } } project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle
27. • Apply the version constraints to each BoM version = "${paypaySpringBomVersion}" // published version dependencies { ... api platform("org.springframework.boot:spring-boot-dependencies:${springbootVersion}") api platform("org.springframework.cloud:spring-cloud-dependencies:${springbootVersion}") } project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle
28. BOM verification • Integration test ◦ To catch regression. ◦ To catch transitive dependency problem (this will happen). ◦ Testing are shared among teams. ▪ @EnableScheduling R |-- B:1.0.0 // newer Spring Boot doesn’t include this |-- C:1.0.0
30. BoM migration • Goal: Services migrate to BOM and remove vulnerabilities and make it easier to remove new ones. • Two phases: ◦ Front-facing Service: ▪ Services that connect to outside world directly. ▪ API gateways, third-party connections. ◦ Backend services: ▪ All Java services.
31. - Reduce 70% percentage of vulnerabilities. - Finish migration on ~180 services in a quarter. - Finished all front-facing services. - Finished all backend services. BoM migration: Result Backend migration Front-end migration
32. Q & A Thank you !