Share
Embed
Hardware and its Concurrency Habits
This talk will present a high-level overview of the hardware structure of modern computer systems, and then will use this structure to show how the laws of physics put limitations on the design and implementation of concurrent software. Venue permitting, it will include a stark visual exposition of these limitations. Finally, this talk will compare how well various synchronization use cases are aligned with modern hardware structure and the laws of physics. Audience members will gain a firmer intuition on the costs and benefits of different synchronization mechanisms.
Paul E McKenney
Kernel Recipes PRO
September 30, 2023
Transcript
92 slides
-
Hardware and its Concurrency Habits © 2023 Meta Platforms Paul
E. McKenney, Meta Platforms Kernel Team Kernel Recipes, September 27, 2023 https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html Chapter 3 -
2 Recette Pour le Codage Simultané • Une pincée de
connaissance des lois de la physique • Compréhension modeste du matériel informatique • Compréhension approfondie des exigences • Conception soignée, y compris la synchronisation • Validation brutale https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html Chapitre 3 -
3 Recette Pour le Codage Simultané • Une pincée de
connaissance des lois de la physique • Compréhension modeste du matériel informatique • Compréhension approfondie des exigences • Conception soignée, y compris la synchronisation • Validation brutale https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html Chapitre 3 -
7 “Let Them Run Free!!!” CPU Benchmark Trackmeet CPU Benchmark
Trackmeet Sadly, it is now more of an obstacle course than a track... -
9 Don’t Make ‘em Like They Used To 4.0 GHz
clock, 20 MB L3 cache, 20 stage pipeline... The only pipeline I need is to cool off that hot- headed brat. -
10 4.0 GHz clock, 20 MB L3 cache, 20 stage
pipeline... The only pipeline I need is to cool off that hot- headed brat. Don’t Make ‘em Like They Used To! • No cache No cache • Shallow pipeline Shallow pipeline • In-order execution In-order execution • One instruction at a time One instruction at a time • Predictable (slow) Predictable (slow) execution execution • Large cache Large cache • Deep pipeline Deep pipeline • Out of order Out of order • Super scalar Super scalar • Unpredictable (fast) Unpredictable (fast) execution execution -
11 4.0 GHz clock, 20 MB L3 cache, 20 stage
pipeline... The only pipeline I need is to cool off that hot- headed brat. Don’t Make ‘em Like They Used To! • No cache No cache • Shallow pipeline Shallow pipeline • In-order execution In-order execution • One instruction at a time One instruction at a time • Predictable (slow) Predictable (slow) execution execution • Large cache Large cache • Deep pipeline Deep pipeline • Out of order Out of order • Super scalar Super scalar • Unpredictable (fast) Unpredictable (fast) execution execution “Tiny Bulldozer” “Semi Tractor-Trailer” -
12 4.0 GHz clock, 20 MB L3 cache, 20 stage
pipeline... The only pipeline I need is to cool off that hot- headed brat. Don’t Make ‘em Like They Used To! • No cache No cache • Shallow pipeline Shallow pipeline • In-order execution In-order execution • One instruction at a time One instruction at a time • Predictable (slow) Predictable (slow) execution execution • Large cache Large cache • Deep pipeline Deep pipeline • Out of order Out of order • Super scalar Super scalar • Unpredictable (fast) Unpredictable (fast) execution execution What would be the computing-systems equivalents of a freight train? -
13 “Good Olde Days” CPU Architecture 80386 Architecture (Wikipedia user
“Appaloosa” GDFL, simplified and reformatted) 32 bit Protection Test Unit ALU • Barrel Shifter • Multiply/Divide • Register File Segmentation Bus Control 32 Bit Paging Prefetcher / Limit Checker Code Queue (16 Bytes) Instruction Decoder 3-Decoded Instruction Queue Decode and Sequencing Control ROM Dedicated ALU Bus 34 bit 32 bit Effective Address Status Flags ALU Control -
16 Pipelined Execution For The Win!!! (Wikipedia user “Amit6” CC
BY-SA 3.0, reformatted) -
17 Superscalar Execution For The Win!!! Intel Core 2 Architecture
(Wikipedia user “I, Appaloosa” CC BY-SA 3.0, reformatted) 128 entry ITLB 32 KB Instruction Cache (8 way) 32 Byte Pre-Decode, Fetch Buffer 18 Entry Instruction Queue 128 Bit 6 Instructions Instruction Fetch Unit Micro- code Simple Decoder Simple Decoder Simple Decoder Complex Decoder 1μop 1μop 1μop 4μops 7 Entry μop Buffer 4μops Register Alias Table and Allocator 96 Entry Reorder Buffer (ROB) Retirement Register File 4μops 4μops Store Data Store Address SSE ALU ALU Branch SSE Shuffle MUL ALU SSE Shuffle ALU ALU Load Address 4μops 32 Entry Reservation Station 128 Bit FMUL FDIV 128 Bit FADD Memory Ordering Buffer (MOB) 128 Bit 32 KB Dual Ported Data Cache (8 way) 16 entry DTLB Store 128 Bit Load 256 entry L2 DTLB Shared L2 Cache (16 way) Shared Bus Interface Unit 256 Bit -
19 Laws of Physics: Atoms Are Too Big!!! Each spot
is an atom. Qingxiao Wang/UT Dallas ca. 2016. -
20 Laws of Physics: Atoms Are Too Big!!! Each spot
is an atom. Qingxiao Wang/UT Dallas ca. 2016. Speed controlled by base thickness: At least one atom thick!!! -
21 Laws of Physics: Light Is Too Slow!!! “One nanosecond
per foot” courtesy of Grace Hopper (https://www.youtube.com/watch?v=9eyFDBPk4Yw) https://en.wikipedia.org/wiki/List_of_refractive_indices A 50% sugar solution is “light syrup”. • Following the footsteps of Admiral Hopper: – Light goes 11.803 inches/ns in a vacuum • Or, if you prefer, 1.0097 lengths of A4 paper per nanosecond • Light goes 1 width of A4 paper per nanosecond in 50% sugar solution – But over and back: 5.9015 in/ns – But not 1GHz! Instead, ~2GHz: ~3in/ns – But Cu: ~1 in/ns, or Si transistors: ~0.1 in/ns – Plus other slowdowns: prototols, electronics, ... -
22 Laws of Physics: Data Is Slower!!! CPUs Caches Interconnects
Memories DRAM & NVM -
23 Laws of Physics: Data Is Slower!!! CPUs Caches Interconnects
Memories DRAM & NVM Light is way too slow in Cu and Si and atoms are way too big!!! -
24 Laws of Physics: Data Is Slower!!! CPUs Caches Interconnects
Memories DRAM & NVM Protocol overheads (Mathematics!) Multiplexing & Demultiplexing (Electronics) Clock-domain transitions (Electronics) Phase changes (Chemistry) Light is way too slow in Cu and Si and atoms are way too big!!! -
25 Laws of Physics: Summary • The speed of light
is finite (especially in Cu and Si) and atoms are of non-zero size • Mathematics, electronics, and chemistry also take their toll • Systems are fast, so this matters -
26 Laws of Physics: Summary • The speed of light
is finite (especially in Cu and Si) and atoms are of non-zero size • Mathematics, electronics, and chemistry also take their toll • Systems are fast, so this matters “Gentlemen, you have two fundamental problems: (1) the finite speed of light and (2) the atomic nature of matter.” * * Gordon Moore quoting Stephen Hawking -
27 Why All This Hardware Complexity? CPUs Caches Interconnects Memories
DRAM & NVM Protocol overheads (Mathematics!) Multiplexing & Demultiplexing (Electronics) Clock-domain transitions (Electronics) Phase changes (Chemistry) Slow light and big atoms create modern computing obstacle course!!! Light is way too slow in Cu and Si and atoms are way too big!!! -
28 Account For All CPU Complexity??? • Sometimes, yes! (Assembly
language!) • But we also need portability: CPUs change – From family to family – With each revision of silicon – To work around hardware bugs – As a given physical CPU ages -
29 One of the ALUs Might Be Disabled Intel Core
2 Architecture (Wikipedia user “I, Appaloosa” CC BY-SA 3.0, reformatted) 128 entry ITLB 32 KB Instruction Cache (8 way) 32 Byte Pre-Decode, Fetch Buffer 18 Entry Instruction Queue 128 Bit 6 Instructions Instruction Fetch Unit Micro- code Simple Decoder Simple Decoder Simple Decoder Complex Decoder 1μop 1μop 1μop 4μops 7 Entry μop Buffer 4μops Register Alias Table and Allocator 96 Entry Reorder Buffer (ROB) Retirement Register File 4μops 4μops Store Data Store Address SSE ALU ALU Branch SSE Shuffle MUL ALU SSE Shuffle ALU ALU Load Address 4μops 32 Entry Reservation Station 128 Bit FMUL FDIV 128 Bit FADD Memory Ordering Buffer (MOB) 128 Bit 32 KB Dual Ported Data Cache (8 way) 16 entry DTLB Store 128 Bit Load 256 entry L2 DTLB Shared L2 Cache (16 way) Shared Bus Interface Unit 256 Bit X X -
30 Thus, Simple Portable CPU Model 128 entry ITLB 32
KB Instruction Cache (8 way) 32 Byte Pre-Decode, Fetch Buffer 18 Entry Instruction Queue 128 Bit 6 Instructions Instruction Fetch Unit Micro- code Simple Decoder Simple Decoder Simple Decoder Complex Decoder 1μop 1μop 1μop 4μops 7 Entry μop Buffer 4μops Register Alias Table and Allocator 96 Entry Reorder Buffer (ROB) Retirement Register File 4μops 4μops Store Data Store Address SSE ALU ALU Branch SSE Shuffle MUL ALU SSE Shuffle ALU ALU Load Address 4μops 32 Entry Reservation Station 128 Bit FMUL FDIV 128 Bit FADD Memory Ordering Buffer (MOB) 128 Bit 32 KB Dual Ported Data Cache (8 way) 16 entry DTLB Store 128 Bit Load 256 entry L2 DTLB Shared L2 Cache (16 way) Shared Bus Interface Unit 256 Bit CPU CPU Store Store Buffer Buffer Cache Cache Intel Core 2 Architecture (Wikipedia user “I, Appaloosa” CC BY-SA 3.0, reformatted and remixed) -
31 And Lots Of CPUs Per System!!! Last-Level Cache Last-Level
Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Interconnect Interconnect CPUs 0-27 & CPUs 224-251 CPUs 28-55 & CPUs 252-279 CPUs 56-83 & CPUs 280-307 CPUs 84-111 & CPUs 308-335 CPUs 112-139 & CPUs 336-363 CPUs 140-167 & CPUs 364-391 CPUs 168-195 & CPUs 392-419 CPUs 196-223 & CPUs 420-447 -
33 Obstacle: Pipeline Flush PIPELINE ERROR PIPELINE ERROR BRANCH MISPREDICTION
Running at full speed requires perfect branch prediction -
34 Obstacle: Memory Reference A single fetch all the way
from memory can cost hundreds of clock cycles -
35 Obstacle: Atomic Operation Atomic operations require locking cachelines and/or
busses, incurring significant delays -
36 Obstacle: Memory Barrier Memory barriers result in stalls and/or
ordering constraints, again incurring delays Memory Barrier Memory Barrier -
37 Obstacle: Thermal Throttling Efficient use of CPU hardware generates
heat, throttling the CPU clock frequency -
38 Obstacle: Cache Miss Cache misses result in waiting for
data to arrive (from memory or other CPUs) CACHE- MISS TOLL BOOTH CACHE- MISS TOLL BOOTH -
39 Obstacle: Input/Output Operation And here you thought that cache
misses were slow... -
40 Which Obstables To Focus On? 1) I/O operations (but
often higher-level issue) 2) Communications cache misses 3) Memory barriers and atomic operations 4) Capacity/geometry cache misses (memory) 5) Branch prediction -
41 Which Obstables To Focus On? 1) I/O operations (but
often higher-level issue) 2) Communications cache misses 3) Memory barriers and atomic operations 4) Capacity/geometry cache misses (memory) 5) Branch prediction These obstacles can (usually) be overcome in a portable manner. -
43 Location Really Matters!!! Last-Level Cache Last-Level Cache Last-Level Cache
Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Last-Level Cache Interconnect Interconnect CPUs 0-27 & CPUs 224-251 CPUs 28-55 & CPUs 252-279 CPUs 56-83 & CPUs 280-307 CPUs 84-111 & CPUs 308-335 CPUs 112-139 & CPUs 336-363 CPUs 140-167 & CPUs 364-391 CPUs 168-195 & CPUs 392-419 CPUs 196-223 & CPUs 420-447 -
50 Can Hardware Help??? Source Drain Sub-Atomic Base Vacuum-gap transistor:
At these scales, the atmosphere is a vacuum!!! -
51 We Really Can Hand-Place Atoms... Actually a carbon monoxide
molecule that I moved across a few planes of copper -
52 We Really Can Hand-Place Atoms... But not trillions of
them in a cost-effective manner!!! Does Not Scale -
54 Hardware 3D Integration 3 cm 1.5 cm Half the
distance, twice the speed!!! Both stacked chiplets and lithographically stacked transistors. -
57 Hardware 3D Integration * Give or take issues with
power, cooling, alignment, interconnect drivers, and so on. 3 cm 1.5 cm Half the distance, twice the speed * -
58 Hardware Integration is Helping Q3 2017: 56 CPUs with
~100ns latencies -
59 Hardware Integration is Helping November 2008: 16 CPUs with
~100ns latencies: More than 3x in nine years!!! -
61 Hardware Accelerators, Theory Data Accelerator Unidirectional data flow, no
out and back, twice the speed!!! -
62 Hardware Accelerators, Practice Data Accelerator Sadly, back to request-response,
but better latency with local memory? Accelerator-local memory System main memory -
63 So Why Hardware Accelerators??? • Optimized data transfers (e.g.,
larger blocks) • Optimized hard-wired computation • Better performance per watt • Better performance per unit capital cost -
65 What Hardware Is Up Against CPUs Caches Interconnects Memories
DRAM & NVM Protocol overheads (Mathematics!) Multiplexing & Demultiplexing (Electronics) Clock-domain transitions (Electronics) Phase changes (Chemistry) Light is way too slow in Cu and Si and atoms are way too big!!! -
67 Simple Portable CPU Model Redux Intel Core 2 Architecture
(Wikipedia user “I, Appaloosa” CC BY-SA 3.0, reformatted and remixed) 128 entry ITLB 32 KB Instruction Cache (8 way) 32 Byte Pre-Decode, Fetch Buffer 18 Entry Instruction Queue 128 Bit 6 Instructions Instruction Fetch Unit Micro- code Simple Decoder Simple Decoder Simple Decoder Complex Decoder 1μop 1μop 1μop 4μops 7 Entry μop Buffer 4μops Register Alias Table and Allocator 96 Entry Reorder Buffer (ROB) Retirement Register File 4μops 4μops Store Data Store Address SSE ALU ALU Branch SSE Shuffle MUL ALU SSE Shuffle ALU ALU Load Address 4μops 32 Entry Reservation Station 128 Bit FMUL FDIV 128 Bit FADD Memory Ordering Buffer (MOB) 128 Bit 32 KB Dual Ported Data Cache (8 way) 16 entry DTLB Store 128 Bit Load 256 entry L2 DTLB Shared L2 Cache (16 way) Shared Bus Interface Unit 256 Bit CPU CPU Store Store Buffer Buffer Cache Cache (Not yet) -
68 Read-Side Hardware Help (1/7) CPU 0 Cache CPU 3
Cache x=42,y=63 CPU 1 CPU 2 r1 = READ_ONCE(x) r2 = READ_ONCE(y) r3 = READ_ONCE(x) Request cacheline x -
69 Read-Side Hardware Help (2/8) CPU 0 Cache CPU 3
Cache x=42,y=63 CPU 1 CPU 2 Request cacheline x r1 = READ_ONCE(x) r2 = READ_ONCE(y) r3 = READ_ONCE(x) -
70 Read-Side Hardware Help (3/8) CPU 0 Cache CPU 3
Cache x=42,y=63 CPU 1 CPU 2 Request cacheline x r1 = READ_ONCE(x) r2 = READ_ONCE(y) r3 = READ_ONCE(x) -
71 Read-Side Hardware Help (4/8) CPU 0 Cache CPU 3
Cache CPU 1 CPU 2 Cacheline x = 42, y = 63 r1 = READ_ONCE(x) r2 = READ_ONCE(y) r3 = READ_ONCE(x) -
72 Read-Side Hardware Help (5/8) CPU 0 Cache x=42,y=63 CPU
3 Cache CPU 1 CPU 2 Cacheline x = 42, y = 63 r1 = READ_ONCE(x) r2 = READ_ONCE(y) r3 = READ_ONCE(x) -
73 Read-Side Hardware Help (6/8) CPU 0 Cache x=42,y=63 CPU
3 Cache CPU 1 CPU 2 r1 = READ_ONCE(x) r2 = READ_ONCE(y) r3 = READ_ONCE(x) -
74 Read-Side Hardware Help (7/8) CPU 0 Cache x=42,y=63 CPU
3 Cache CPU 1 CPU 2 Caches help beat laws of physics given spatial locality!!! r1 = READ_ONCE(x) r2 = READ_ONCE(y) r3 = READ_ONCE(x) -
75 Read-Side Hardware Help (8/8) CPU 0 Cache x=42,y=63 CPU
3 Cache CPU 1 CPU 2 Caches help beat laws of physics given temporal locality!!! r1 = READ_ONCE(x) r2 = READ_ONCE(y) r3 = READ_ONCE(x) -
76 Levels of Cache on My Laptop Index Line Size
Associativity Size 0 64 8 32K 1 64 8 32K 2 64 4 256K 3 64 16 16,384K When taking on the laws of physics, don’t be afraid to use a few transistors -
77 Levels of Cache on Large Old Server Index Line
Size Associativity Size 0 64 8 32K 1 64 6 32K 2 64 16 1,024K 3 64 11 39,424K When taking on the laws of physics, don’t be afraid to use lots of transistors -
80 Write-Side Hardware Help (Summary) • Store buffers for the
win!!! Sort of… – Cache line for variable x is initially at CPU 3 – CPU 0 writes 1 to x, but doesn't have cacheline • So holds the write in CPU 0's store buffer • And requests exclusive access to the cacheline (which takes time) – CPU 3 reads x, obtaining “0” immediately from cacheline – CPU 0 receive's x's cacheline • And CPU 0's write finally gets to the cacheline • Overwriting the value that CPU 3 read, despite the write starting earlier • Writes only appear to be instantaneous!!! -
81 Simple Portable CPU Model Redux Intel Core 2 Architecture
(Wikipedia user “I, Appaloosa” CC BY-SA 3.0, reformatted and remixed) 128 entry ITLB 32 KB Instruction Cache (8 way) 32 Byte Pre-Decode, Fetch Buffer 18 Entry Instruction Queue 128 Bit 6 Instructions Instruction Fetch Unit Micro- code Simple Decoder Simple Decoder Simple Decoder Complex Decoder 1μop 1μop 1μop 4μops 7 Entry μop Buffer 4μops Register Alias Table and Allocator 96 Entry Reorder Buffer (ROB) Retirement Register File 4μops 4μops Store Data Store Address SSE ALU ALU Branch SSE Shuffle MUL ALU SSE Shuffle ALU ALU Load Address 4μops 32 Entry Reservation Station 128 Bit FMUL FDIV 128 Bit FADD Memory Ordering Buffer (MOB) 128 Bit 32 KB Dual Ported Data Cache (8 way) 16 entry DTLB Store 128 Bit Load 256 entry L2 DTLB Shared L2 Cache (16 way) Shared Bus Interface Unit 256 Bit CPU CPU Store Store Buffer Buffer Cache Cache Now!!! -
82 Write-Side Hardware Help (1/7) CPU 0 Store Buffer Cache
CPU 3 Store Buffer Cache x=0 CPU 1 CPU 2 WRITE_ONCE(x, 1) READ_ONCE(x) -
83 Write-Side Hardware Help (2/7) CPU 0 Store Buffer x=1
Cache CPU 3 Store Buffer Cache x=0 CPU 1 CPU 2 Request cacheline x WRITE_ONCE(x, 1) READ_ONCE(x) The store buffer allows writes to completes quickly!!! Take that, laws of physics!!! -
84 Write-Side Hardware Help (3/7) CPU 0 Store Buffer x=1
Cache CPU 3 Store Buffer Cache x=0 CPU 1 CPU 2 Request cacheline x WRITE_ONCE(x, 1) READ_ONCE(x) Except that later read gets older value... -
85 Write-Side Hardware Help (4/7) CPU 0 Store Buffer x=1
Cache CPU 3 Store Buffer Cache x=0 CPU 1 CPU 2 Request cacheline x WRITE_ONCE(x, 1) READ_ONCE(x) -
86 Write-Side Hardware Help (5/7) CPU 0 Store Buffer x=1
Cache CPU 3 Store Buffer Cache CPU 1 CPU 2 WRITE_ONCE(x, 1) READ_ONCE(x) Respond with cacheline x = 0 -
87 Write-Side Hardware Help (6/7) CPU 0 Store Buffer x=1
Cache x=0 CPU 3 Store Buffer Cache CPU 1 CPU 2 WRITE_ONCE(x, 1) READ_ONCE(x) Respond with cacheline x = 0 -
88 Write-Side Hardware Help (7/7) CPU 0 Store Buffer Cache
x=1 CPU 3 Store Buffer Cache CPU 1 CPU 2 WRITE_ONCE(x, 1) READ_ONCE(x) Quick write completion, sort of. Laws of physics: Slow or misordered!!! -
89 Misordering? Or Propagation Delay? CPU 0 CPU 1 CPU
2 CPU 3 WRITE_ONCE(x, 1); r1 = READ_ONCE(x) == 0; X == 0 X == 1 fr Time -
90 And Careful What You Wish For!!! CPU 0 CPU
1 CPU 2 CPU 3 WRITE_ONCE(x, 1); r1 = READ_ONCE(x) == 0; X == 0 X == 1 fr Time Hardware tricks help reduce the red triangle. But too bad about Meltdown and Spectre... -
92 Can Software Help? • Increasingly, yes!!! • Use concurrent
libraries, applications, subsystems, and so on – Let a few do the careful coding and tuning – Let a great many benefit from the work of a few • Use proper APIs to deal with memory ordering – Chapter 14: “Advanced Synchronization: Memory Ordering” https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html -
94 Summary • Modern hardware is highly optimized – Most
of the time! – Incremental improvements due to integration – But the speed of light is too slow and atoms too big • Use concurrent software where available • Structure your code to avoid the big obstacles – Micro-optimizations only sometimes needed -
95 For More Information • “Computer Architecture: A Quantitative Approach”,
Hennessey & Patterson (Recent HW) • “Parallel Computer Architecture: A Hardware/Software Approach”, Culler et al. – Includes SGI Origin and Sequent NUMA-Q • “Programing Massively Parallel Processors: A Hands-on Approach”, Kirk & Hwu – Primarily NVIDIA GPGPUs • https://developer.nvidia.com/educators/existing-courses – List of NVIDIA university courseware • https://gpuopen.com/professional-compute – List of AMD GPGPU-related content • “Is Parallel Programming Hard, And, If So, What Can You Do About It?” – https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html -
96 L'antidote de Codage Simultané de la Femme de Paul
• Cordial de mûres sauvages de l'Himalaya – Mettre deux litres de mûres dans un pot de quatre litres – Ajouter cinq huitièmes litres de sucre – Remplissez le pot de vodka – Secouez tous les jours pendant cinq jours – Secouez chaque semaine pendant cinq semaines – Passer au tamis: Ajoutez des baies à la glace, consommez le liquide filtré comme vous voulez -
97 Paul’s Wife’s Concurrency Antidote • Wild Himalayan Blackberry Cordial
– Put 8 cups wild himalayan blackberries in 1 gallon jar – Add 2½ cups sugar – Fill jar with vodka – Shake every day for five days – Shake every week for five weeks – Pour through sieve: Add berries to ice cream, consume filtered liquid as you wish -
98 Paul’s Wife’s Concurrency Antidote • Wild Himalayan Blackberry Cordial
– 8 cups wild himalayan blackberries in 1 gallon jar – Add 2½ cups sugar – Fill jar with vodka – Shake every day for five days – Shake every week for five weeks – Pour through sieve: Add berries to ice cream, consume filtered liquid as you wish If there is no right tool, invent it!!! Questions?