Upgrade to Pro — share decks privately, control downloads, hide ads and more …

52 weeks after Digdag operation

amesho
February 21, 2018

52 weeks after Digdag operation

About one year passed since Digdag was introduced. How did batch operation change during that period? How did the management consciousness change? What was going to be seen? What is troubled at present? Plugin We announce you made.

amesho

February 21, 2018
Tweet

More Decks by amesho

Other Decks in Programming

Transcript

  1. Quant 4 constitution 4 rails 4 redshift 4 aurora 4

    backend 4 Dependency complexity
  2. Agenda 4 How has batch operation changed? 4 How has

    the situation changed? 4 Problems with operation.
  3. Problems of CRON 4 It is hard to understand which

    server is running 4 It is hard to understand startup time 4 Difficult to understand dependency 4 Elapsed time is difficult to understand
  4. How has the situation changed 4 It became easy to

    match recognition with your boss and colleagues 4 People who can not write programs can join 4 Safe to understand what you are doing
  5. Note 4 If you look closely at the document you

    will find detailed notes etc 4 Some things are not written in the document
  6. API 4 It exists under digdag-server Ϋϥε ϝιου API ֓ཁ

    SessionResource GET /api/sessions List sessions from recent to old GET /api/sessions/{id} Get a session by id GET /api/sessions/{id}/attempts List attempts of a session AdminResource GET /api/admin/attempts/{id}/userinfo AttemptResource GET /api/attempts list attempts from recent to old GET /api/attempts?include_retried=1 list attempts from recent to old GET /api/attempts?project=<name> list attempts that belong to a particular project GET /api/attempts?project=<name>&workflow=<name> list attempts that belong to a particular workflow GET /api/attempts/{id} show a session GET /api/attempts/{id}/tasks list tasks of a session GET /api/attempts/{id}/retries list retried attempts of this session PUT /api/attempts starts a new session POST /api/attempts/{id}/kill kill a session
  7. Sample API access #!/bin/env ruby require 'net/http' require 'uri' require

    'json' require 'time' url = URI.parse('http://localhost:65432/') res = Net::HTTP.start(url.host, url.port) {|http| http.get('/api/schedules') } schedules = JSON.parse(res.body) schedules['schedules'].sort_by { |s| Time.strptime(s['nextRunTime'],'%Y-%m-%dT%H:%M:%S%z').to_i }.each do |row| next if row['disabledAt'] printf("%s\t%s/%s\n",Time.strptime(row['nextRunTime'],'%Y-%m-%dT%H:%M:%S%z').localtime,row['project']['name'],row['workflow']['name']) end
  8. storelastresults: BOOLEAN 4 Whether to store the query results to

    redshift.last_results parameter. Default: false. 4 Setting first stores the first row to the parameter as an object (e.g. ${redshift.last_results.count}). 4 Setting all stores all rows to the parameter as an array of objects (e.g. $ {redshift.last_results[0].name}). If number of rows exceeds limit, task fails.
  9. redshift operator is not implemented It is described in comments

    // TODO store_last_results should be io.digdag.standards.operator.jdbc.StoreLastResultsOption // instead of boolean to be consistent with pg> and redshift> operators but not implemented yet. this.storeLastResults = params.get("store_last_results", boolean.class, false);
  10. end