Share
DAMOS - The Smart Cruise Control for RAM
Last year, we saw how the Linux kernel’s DAMON subsystem lets us safely watch the real memory usage of applications (data access pattern) without slowing them down. But looking at charts is only half the battle.
What do we actually do with that data? This talk introduces DAMOS (DAMON-based Operation Schemes) — the engine that turns passive observation into automatic action. We will explain how DAMOS works using simple mental models, showing how it automatically parks “cold” memory to save money and optimizes “hot” memory for raw speed.
Designed for kernel veterans and curious beginners alike, this session will show you how to write basic rules to make your Linux systems automatically manage their own memory footprint safely, predictably, and with zero code changes.
SJ Park
Kernel Recipes PRO
Transcript
74 slides
-
TL; DR • DAMOS is a Linux kernel feature for
automated access-aware system operations Image is retrieved from https://pollackgroup.com/wp-content/uploads/2023/07/62bcd9b81b5cfd7bf26cb65d_self-drivingcars-1-jpg.webp -
Table of Contents Why Smart Control for RAM (6 minutes)
• Begins: Origin to Productizing (10 minutes) • Darknight: DAMOS Auto-tuning (10 minutes) • Rises: DAMOS Usages (2 minutes) • Revolution: Ongoing Changes (2 mins) • QnA (10 mins) • -
RAM Efficiency Matters • Data: Source of money – Getting
bigger :D • RAM (Memory): Sink of money – Getting more expensive :’( • Goal: More data with less memory Images are retrived from https://i.imgflip.com/244jt6.jpg?a497136 -
Real World Story • In a cloud provider far, far
away • CPU: Pay-as-you-go • Memory: Fixed price • Plan: pay-as-you-go for memory – Charge for only working set Image is retrieved from https://imgflip.com/i/b1mr1j -
Real World Story • In a cloud provider far, far
away • CPU: Pay-as-you-go • Memory: Fixed price • Plan: pay-as-you-go for memory – Charge for only working set “Everyone Has a Plan Until They Get Punched in the Mouth”, Mike Tyson Image is retrieved from https://imgflip.com/i/b1mr1j -
Problem: Working Set Detection Working set != Allocated memory •
Much more difficult than it looks like • – Definition of working set – Monitoring overhead – Memory-greedy system • Common and essential for memory efficiency Image is retrieved from https://magicmeme.com/_next/image? url=https%3A%2F%2Fcdn.magicmeme.com%2Ftemplates%2Ficeberg.png&w=1200&q=75 -
DAMON: A New Hope • A PhD student’s side project
– For the issue in main research – Developed a kernel subsystem as a solution • Named DAMON: Data Access MONitor Image is retrieved from https://media.makeameme.org/created/jedi-master-why.jpg -
DAMON in One Sentence “DAMON is a Linux kernel subsystem
for efficient access monitoring and access-aware system operations.” Image is generated by DAMON -
High Level Implementation • Spawn a kernel thread that does
below • See if each page is accessed, every 5ms • Inform users the findings, every 100ms • Time intervals can be [auto-]tuned $ ls mm/damon/ acma.c Makefile ops-common.h core.c modules-common.c paddr.c Kconfig modules-common.h reclaim.c lru_sort.c ops-common.c stat.c $ wc -l mm/damon/*.c | tail -n 1 16000 total sysfs.c sysfs-common.c sysfs-common.h sysfs-sample.c sysfs-schemes.c tests vaddr.c -
Providing Access Information • Location • Frequency • Stability •
Recency Access pattern heatmap of Splash2x/Raytrace -
Design • • • Goals: Accurate, Lightweight, Tunable and Automated
Mechanisms: Sampling, Adaptive regions, Sysfs interface, Feedback loops Results – <0.1% CPU overhead on products -
Availability and Usages • Available on >=5.15 upstream kernels •
Enabled on most distro kernels • Being used for products and researches -
Community • Public channels – Mailing list: [email protected] – Project
website: https://damonitor.github.io/ • Private channels – Maintainer email: [email protected] – DAMON Beer/Coffee/Tea Meetup -
But This is Not About DAMON • DAMON was presented
in detail at Kernel Recipes’25 -
Table of Contents Why Smart Control for RAM (6 minutes)
• Begins: Origin to Productizing (10 minutes) • Darknight: DAMOS Auto-tuning (10 minutes) • Rises: DAMOS Usages (2 minutes) • Revolution: Ongoing Changes (2 mins) • QnA (10 mins) • -
Developer’s Expectation • • DAMON only provides Access Pattern Users
do hard lifts – Decide optimum system memory size – Optimize programs for access pattern Image is retrieved from https://encrypted-tbn0.gstatic.com/images? q=tbn:ANd9GcSOuahkkiJvsAe56zRaoB0KSAqMdowUS5XB36ULDvioD0HGmRe2d4GJYILc&s=10 -
DAMON User Reaction in Real • • • Users are
busy, even in hyper-scalers No time to run a tool for no direct wins FAQ: Can DAMON provide wins itself? Image is retrieved from https://encrypted-tbn0.gstatic.com/images? q=tbn:ANd9GcSOuahkkiJvsAe56zRaoB0KSAqMdowUS5XB36ULDvioD0HGmRe2d4GJYILc&s=10 -
madvise() System call • Gives memory advice to kernel •
Supported actions • – MADV_PAGEOUT, MADV_HUGEPAGE, MADV_WILLNEED, ... Image is retrieved from https://i.imgflip.com/1itm1d.jpg?a495600 -
DAMON+madvise() • Let DAMON do madvise() calls • For memory
of a given pattern – Pattern: Access frequency & stability • Target pattern+madvise() action: DAMon-based Operation Scheme (DAMOS) Image is retrieved from https://static0.hotcarsimages.com/wordpress/wp-content/uploads/2018/05/6138666188_f86c3cdc 67_b.jpg?q=50&fit=crop&w=825&dpr=1.5 -
DAMON+madvise() Example • Proactive cold memory reclaim – ‘damo’ is
DAMON user-space tool $ sudo ./damo start $(pidof my_workload) \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max -
DAMON+madvise() Example • Proactive cold memory reclaim $ sudo ./damo
start $(pidof my_workload) \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max -
DAMON+madvise() Example • Proactive cold memory reclaim $ sudo ./damo
start $(pidof my_workload) \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max -
More than madvise() • • No reason to support only
madvise() actions DAMOS is later extended for general system operations – Moving folio inactive → active LRU list – Migrate folio to a given node -
What If DAMOS Out Of Control • Access pattern is
dynamic – Target memory can suddenly increase – DAMOS can consume huge system resource – Critical in real-world products Image is retrieved from https://inews.co.uk/wp-content/uploads/2022/01/JGD_1214107.jpg?w=1200&h=675&crop=1 -
DAMOS Quota • Max speed of DAMOS actions – E.g.,
reclaim cold memory up to 128 MiB/s • Regions for action under quota – Selected via access pattern – Hottest or coldest memory, depend on action Image is retrieved from https://i.pinimg.com/736x/ba/31/88/ba3188f5a6f7a1b6227262a14d9841ae.jpg -
Special Requests • Special products have unique requests – Access
pattern is useful but not everything – Example requests • Reclaim cold page cache but anon pages • Reclaim cold memory of cgroup X but Y Image is retrieved from https://encrypted-tbn0.gstatic.com/images? q=tbn:ANd9GcRuwoXpDo3YjK9u8JQ8r0VVP5YWcNENt9NaxBS2uicKEH4nG-ef_YAIUo4&s=10 -
DAMOS Filter • • • Non-access pattern target spec Allow/reject
memory of given types Support multiple filters combination – “Reclaim anon pages of cgroup A” Image is retrieved from https://coffeeculture.asia/wp-content/uploads/2020/12/Coffee-CultureCoffee-Filter-Papers-Buy-Coffee-equipment-online-in-thailand.jpg -
DAMOS Quota,Filters Example • Speed-bounded, Targeted proactive cold memory reclaim
$ sudo ./damo start \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive -
DAMOS Quota,Filters Example • Speed-bounded, Targeted proactive cold memory reclaim
$ sudo ./damo start \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive -
DAMOS Quota,Filters Example • Speed-bounded, Targeted proactive cold memory reclaim
$ sudo ./damo start \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive -
DAMOS Quota,Filters Example • Speed-bounded, Targeted proactive cold memory reclaim
$ sudo ./damo start \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive -
DAMOS Quota,Filters Example • Speed-bounded, Targeted proactive cold memory reclaim
$ sudo ./damo start \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive -
DAMOS Quota,Filters Example • Speed-bounded, Targeted proactive cold memory reclaim
$ sudo ./damo start \ --damos_action pageout \ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive -
Table of Contents Why Smart Control for RAM (6 minutes)
• Begins: Origin to Productizing (10 minutes) • Darknight: DAMOS Auto-tuning (10 minutes) • Rises: DAMOS Usages (2 minutes) • Revolution: Ongoing Changes (2 mins) • QnA (10 mins) • -
Problem: Tuning Difficulties • DAMOS users should... – Understand DAMON/DAMOS
internal – Understand their workload access pattern – Setup optimum scheme for the workload • Complicated, repetitive for each workload Image is retrieved from https://media.makeameme.org/created/whats-that-sounds.jpg -
Why So Difficult? It’s Question! • DAMOS asks users “how”
(mechanism) • Effectively programming request • Flexible but difficult Image is retrieved from https://i.pinimg.com/736x/4c/25/6f/4c256f103aca4785544b78e430c079aa.jpg -
Change The Question • Idea: Ask users “what” (policy, or
vibe) – What they want to achieve • E.g., 5% free memory, 0.05% memory pressure – Intuitive for service owners • DAMOS works in darkness – Auto-tune parameters to achieve the goal -
Knobs to tune: Access Pattern? • Min/max access frequency and
stability • Four parameters; too complicated Image is retrieved from https://subsonic.com/img/p/1/0/7/4/1074.jpg -
Knobs to Tune: Quota! • • Single parameter; much simple
Effectively auto-tune access pattern parameters Image is retrieved from https://thumbs.static-thomann.de/thumb/padthumb600x600/pics/bdb/_28/284224/10785205_800.jpg -
DAMOS quota auto-tuning • Users set goals of the quota
– Goal: a target value of a system metric • Auto-tune quota using a feedback loop – Increase/decrease quota when the goal is under/over-achieved – Multiple tuning algorithms available -
Auto-tuned DAMOS Example $ sudo ./damo start \ --damos_action pageout
\ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive --damos_quota_goal some_mem_psi_us 5ms -
Auto-tuned DAMOS Example $ sudo ./damo start \ --damos_action pageout
\ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive --damos_quota_goal some_mem_psi_us 5ms -
Auto-tuned DAMOS Example $ sudo ./damo start \ --damos_action pageout
\ --damos_access_rate 0% 0% --damos_age 2m max \ --damos_quota_space 128M --damos_quota_interval 1s \ --damos_filter reject anon \ --damos_filter reject memcg /workload_latency_sensitive --damos_quota_goal some_mem_psi_us 5ms -
DAMOS for Hot Memory • DAMOS is not only for
cold memory • Memory tiering: one of DAMOS use cases Image is retrieved from https://i.pinimg.com/736x/51/8f/09/518f09467228e2ac87c0f2d299d02fc5.jpg -
Memory Tiering 101 • Two tiers: DRAM and CXL memory
– DRAM: Fast, small – CXL: Slow, large • Goal: Hot data on DRAM, cold data on CXL Image is retrieved from https://www.researchgate.net/publication/361162182/figure/fig1/AS:1164536778358786@1654658612492/Different-tiers-has-differentcharacteristics-with-varied-memory-access-latency-in-a.png -
Transparent Page Placement • Utilize DRAM as much as possible
– It is more powerful, no reason to under utilize • • Keep headroom free memory in DRAM TPP-DAMON: DAMOS-based TPP implementation -
TPP-DAMON damo start \ --numa_node 0 --monitoring_intervals_goal 4% 3 5ms
10s \ --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_free_bp 0.5% 0 \ --damos_filter reject young \ --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \ --damos_action migrate_hot 0 --damos_access_rate 5% max \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
TPP-DAMON damo start \ Start DAMON in --numa_node 0 --monitoring_intervals_goal
4% 3 5ms 10s \ kernel. --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ Dear DAMON user--damos_apply_interval 1s \ space--damos_quota_interval tool, 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_free_bp 0.5% 0 \ --damos_filter reject young \ --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \ --damos_action migrate_hot 0 --damos_access_rate 5% max \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
TPP-DAMON damo start \ --numa_node 0 --monitoring_intervals_goal 4% 3 5ms
10s \ --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ --damos_apply_interval 1s \ Create one DAMON --damos_quota_interval 1s --damos_quota_space 200MB \ With auto-tuned worker thread, monitoring --damos_quota_goal node_mem_free_bp 0 \ monitoring0.5% intervals. for node 0 (upper tier) young \ --damos_filter reject --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \ --damos_action migrate_hot 0 --damos_access_rate 5% max \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
TPP-DAMON damo start \ --numa_node 0 --monitoring_intervals_goal 4% 3 5ms
10s \ --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_free_bp 0.5% 0 \ And ask it to migrate un-accessed --damos_filter reject young \(access_rate 0%) memory to node 1 (lower tier), coldest one first.4% 3 5ms 10s \ --numa_node 1 --monitoring_intervals_goal IOW,migrate_hot demote cold memory. --damos_action 0 --damos_access_rate 5% max \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
TPP-DAMON damo start \ --numa_node 0 --monitoring_intervals_goal 4% 3 5ms
10s \ --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ And do that once per second. --damos_quota_goal node_mem_free_bp 0.5% 0 \ --damos_filter reject young \ --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \ --damos_action migrate_hot 0 --damos_access_rate 5% max \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
TPP-DAMON damo start \ --numa_node 0 --monitoring_intervals_goal 4% 3 5ms
10s \ --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_free_bp 0.5% 0 \ --damos_filter reject young \ --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \ --damos_action migrate_hot 0 --damos_access_rate 5% max \ Within one second, do the migration only up to 200 MiB of memory. --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
TPP-DAMON damo start \ --numa_node 0 --monitoring_intervals_goal 4% 3 5ms
10s \ --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_free_bp 0.5% 0 \ --damos_filter reject young \ --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \ --damos_action migrate_hot 0 --damos_access_rate 5% max \ Under the 200MiB/1s hard limitation, auto-tune the real soft limitation --damos_apply_interval 1s \ to make node 0’s free1s memory ratio becomes 0.5% soon. --damos_quota_interval --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
TPP-DAMON damo start \ --numa_node 0 --monitoring_intervals_goal 4% 3 5ms
10s \ --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_free_bp 0.5% 0 \ --damos_filter reject young \ --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \ --damos_action migrate_hot 0 --damos_access_rate 5% max \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
TPP-DAMON damo start \ --numa_node 0 --monitoring_intervals_goal 4% 3 5ms
10s \ --damos_action migrate_cold 1 --damos_access_rate 0% 0% \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_free_bp 0.5% 0 \ --damos_filter reject young \ --numa_node 1 --monitoring_intervals_goal 4% 3 5ms 10s \ --damos_action migrate_hot 0 --damos_access_rate 5% max \ --damos_apply_interval 1s \ --damos_quota_interval 1s --damos_quota_space 200MB \ --damos_quota_goal node_mem_used_bp 99.7% 0 \ --damos_filter allow young \ --damos_nr_quota_goals 1 1 --damos_nr_filters 1 1 \ --nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1 -
Table of Contents Why Smart Control for RAM (6 minutes)
• Begins: Origin to Productizing (10 minutes) • Darknight: DAMOS Auto-tuning (10 minutes) • Rises: DAMOS Usages (2 minutes) • Revolution: Ongoing Changes (2 mins) • QnA (10 mins) • -
Proactive Memory Reclaim • AWS Aurora Serverless v2 • Proactively
reclaim cold memory in VMs • Pay for only remaining non-cold memory Image is retrieved from https://i.ytimg.com/vi/xKFA6PJgp0o/maxresdefault.jpg -
DRAM/CXL Memory Tiering • SK Hynix HMSDK – Migrate hot/cold
data across tiers • Micron dynamic interleaving – Interleave hot/cold memory for bandwidth Images retrieved from https://upload.wikimedia.org/wikipedia/fr/c/c1/Sk-hynix_-_Logo.svg and https://upload.wikimedia.org/wikipedia/commons/8/8e/Micron_Logo.png -
More Non-public Usages • Real DAMOS usages are unknown •
A few more usages are privately shared • Make your own creative usage -
The Classes • Primary class: Access information • Secondary class:
Data attributes – DAMOS filters for page level properties -
Is It Fair (and Useful)? • • Why access info
is always primary? Other attributes are primary interests for some users -
DAMON Revolution • Re-define DAMON – Data Attributes Monitoring/Operation eNgine
• Every attribute including access is equal • DAMOS get revolutionized together Image is retrieved from https://encrypted-tbn0.gstatic.com/images? q=tbn:ANd9GcShaoxj5LAQw3g2Ed4w83aLruDWdiIAN1TRojKmw49EKQ&s=10 -
Revolution In Progress • Monitoring framework is extended • More
support of attributes is ongoing – Now: access, Anon page, memcg, hugepage • Minimum support in DAMOS yet -
Wrapup • • • DAMOS is a Linux kernel feature
for automated access-aware system operations Being used for products A revolution for making it for every attribute is ongoing -
QnA • Don’t forget – [email protected] – [email protected] – https://damonitor.github.io
– Just poke me in hallway