Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The state of Nextflow

The state of Nextflow

A quick update on the project status, recent changes and open challenges presented at the Nextflow Workshop 2018.

Paolo Di Tommaso

November 22, 2018
Tweet

More Decks by Paolo Di Tommaso

Other Decks in Technology

Transcript

  1. Apr 2018 Alexander 
 Peltzer Phil Ewels Andreas Wilm Jun

    2018 Google 
 Summer of Code Edgar 
 Garriga
  2. HEALTH METRICS • 76 citations • 90k LoC (+35%) •

    650 stars and
 136 forks on GitHub (+100%)
  3. NEW CONTRIBUTORS Sven Fillinger Quantitative Biology Center, 
 Germany Lorenz

    Gerber
 Genome Institute 
 Singapore Ólafur Haukur 
 Flygenring
 WuXi NextCode, Island Jonathan Sheffi
 Google Genomics, USA Rad Suchecki
 CSIRO, Australia Luke Goodsell
 Achilles Therapeutics, UK
  4. SYNTAX POLISHING Channel.from( .. ) Channel.fromPath( .. ) Channel.fromFilePairs( ..

    ) channel.of( .. ) channel.ofFiles( .. ) channel.ofFilePairs( .. ) → → → proces foo { input: file x from file('/some/path') """ your_command $x
 """ } proces foo { input: path x from '/some/path' """ your_command $x
 """ } →
  5. CHANNELS DEDUP Channel.fromPath('/something/*') .into{ this_ch; that_ch } proces foo {

    input: file x from this_ch """ your_command $x
 """ } proces bar { input: file x from that_ch """ your_command $x
 """ } Channel.fromPath('/something/*') .fork(2) .set{ this_ch } proces foo { input: file x from this_ch """ your_command $x
 """ } proces bar { input: file x from this_ch """ your_command $x
 """ } →
  6. EVENTS & METADATA proces foo { input: set pair_id, file(reads)

    from something_ch onSuccess: // do something onError: // do something else metadata_ch << [ name: task.name, failed: true, sample: pair_id, cmd_ver: task.version ] onTerminate: // an action when the last task has complete script: """ your_command $x
 """ }
  7. EVENTS & METADATA (2) workflow.onComplete { // action } process.onComplete

    { // action } process.onSuccess { // action } process.onError { task -> metadata_ch << [ name: task.name, failed: true, sample: task.context.pair_id, // ??? cmd_ver: task.version ] }
  8. RUNTIME CLEANUP • Workflow execution may produce a lot of

    temporary data • Deleting it affects the resume ability • Work in progress to cleaned it up at runtime w/o breaking the resume (issues # 452) p1 q1 t3 t2 t1 ω1 fromPath('/data/*') p1 t2 t1 q1
  9. RESOURCES HANDLING process foo { time memory errorStategy 'retry' input:

    file 'some-in.txt' from in_ch output: file 'some-out.bam' into out_ch script: """ your_command --mem $task.memory """ } { 1.h * task.attempt } { 10.GB * task.attempt }
  10. RESOURCES PREDICTION • Train a prediction model with real resources

    usage and inputs metadata for first n tasks • Predict following task execution of previous trained model process foo { time auto memory auto input: file 'some-in.txt' from in_ch output: file 'some-out.bam' into out_ch script: """ your_command --mem $task.memory """ }
  11. WORKFLOW COMPOSITION? nextflow run prj/foo nextflow run prj/bar input: file

    <?> from ch_1 output: file <?> into ch_2 input: file <?> from ch_2 output: file <?> into ch_3 prj/foo + prj/bar + ... proces foo { script: workflow('prj/foo') } process bar { script: workflow('prj/bar') }
  12. DATA INTEGRATION • Sequence Read Archive (NCBI SRA) • GA4GH

    HTSGET • SQL/NoSQL big datasources (e.g. AWS Athena or Google BigTable, etc.) • Domain specific databases e.g. iRODS
  13. MY TAKE • Genomics data and health records are exploding

    • Data will be fragmented and siloed in many different public clouds and private orgs • Heterogeneous computing environment eg. clouds, HPC clusters, interactive notebooks, etc. • Hardly there could be one-size fit-all solution • Increasingly need of portable workflows and hybrid computing enabling transparent multi-cloud and platform deployments