Gerber Genome Institute Singapore Ólafur Haukur Flygenring WuXi NextCode, Island Jonathan Sheffi Google Genomics, USA Rad Suchecki CSIRO, Australia Luke Goodsell Achilles Therapeutics, UK
input: file x from this_ch """ your_command $x """ } proces bar { input: file x from that_ch """ your_command $x """ } Channel.fromPath('/something/*') .fork(2) .set{ this_ch } proces foo { input: file x from this_ch """ your_command $x """ } proces bar { input: file x from this_ch """ your_command $x """ } →
from something_ch onSuccess: // do something onError: // do something else metadata_ch << [ name: task.name, failed: true, sample: pair_id, cmd_ver: task.version ] onTerminate: // an action when the last task has complete script: """ your_command $x """ }
temporary data • Deleting it affects the resume ability • Work in progress to cleaned it up at runtime w/o breaking the resume (issues # 452) p1 q1 t3 t2 t1 ω1 fromPath('/data/*') p1 t2 t1 q1
usage and inputs metadata for first n tasks • Predict following task execution of previous trained model process foo { time auto memory auto input: file 'some-in.txt' from in_ch output: file 'some-out.bam' into out_ch script: """ your_command --mem $task.memory """ }
<?> from ch_1 output: file <?> into ch_2 input: file <?> from ch_2 output: file <?> into ch_3 prj/foo + prj/bar + ... proces foo { script: workflow('prj/foo') } process bar { script: workflow('prj/bar') }
• Data will be fragmented and siloed in many different public clouds and private orgs • Heterogeneous computing environment eg. clouds, HPC clusters, interactive notebooks, etc. • Hardly there could be one-size fit-all solution • Increasingly need of portable workflows and hybrid computing enabling transparent multi-cloud and platform deployments