Upgrade to Pro — share decks privately, control downloads, hide ads and more …

データ分析基盤Crois

 データ分析基盤Crois

Machine Learning Casual Talks #10 での発表資料です。

リクルートの横断組織であるリクルートコミュニケーションズで開発しているデータ分析基盤のCroisについて紹介させていただきました。

Recruit Communications

May 29, 2019
Tweet

More Decks by Recruit Communications

Other Decks in Technology

Transcript

  1.    o D n T mf , ,

    ,, , T D T m e c a T T D r k k 18
  2. @7 8-,. •5C3B <'$%* +4 /  !& " 9A

    •>1!) :62?; # $()/ •&'$' =0  * & +49A 19
  3.   •     YAML tasks: -

    name: task-1 module: sleep params: hoge: fuga - name: task-2 module: sleep 33
  4. SFN =  AWS Batch  !   !

     !   describe-job Lambda Lambda Lambda 38
  5. SFN  =   AWS Batch " $ 

    #!  SFN $ Lambda Lambda Lambda 39
  6.   •  •SFNParallel  tasks: - name: task-1

    module: sleep - parallel: - tasks: - name: task-3 module: sleep - tasks: - name: task-4 module: sleep - tasks: - name: task-5 module: sleep = 40
  7. AWS Batch  module  agent- downloader Instance mount agent

    Module run.sh execute data- bucket download upload secrets Agent     download Agent project-role 43
  8. SFN Batch9, %(" •&$1+5E ? •'#( - • 500DF &$1++

    67   !%(" •&$ACAPI> 8  •DynamoDB.4@2;  •AWS Batch))B3/0  •Fargate= :<* 44
  9.  •Crois " $ • KMSIAM  •  

    Decrypt! project-role api project-key  #%! other-role #%! 47
  10.  #% •"S3(. !" ! $ •S3 21&), -' •

    ! IAM! +* 01  agent data- bucket download upload project-key project-role Container 01/, 48
  11.  • ./%"+& •(- #' • IAM!  *, 

    api + project-key agent project-role Container $ *,)& 49
  12. API09>=2  •AWS Batch •RegisterJobDefinition@B) ( "5- D% • &!%*

    •Step Functions •4/8:API'!&$!?  • #!A3  •IAM RoleKMS Key Policy •Role61,2030;< KMS Key PolicyC.Role*  • &!%+7 51
  13. SFN  &*!+' •SFN Parallel $%   &*! +')

    •Parallel"  &*!#  (+  52 Lambda   
  14. D<5103? >6 •)*-510/%   " /!#IT' +/$C • @>6

    ;E: •GitHub, Docker, =79A  OK •CI(Drone)84 C GF2  •1,(#,1)*-51&. B 56
  15.  (& •1   ! • ")'*  •

    Drone(CI))'* • /%   build, push# • git diff+ • ,  .- module- registry Drone ghe hook (&$ push push git diff & docker build 57
  16. BS@Q2A=)-!$'* •CroisP1;4 9MBS@Q2A= )-!$'* •Web(-<1LG • 7O8#6K> • BS@Q+#/ ,%0

    • @QFILN= • @QD+#/H ER •CroisAPIC? BS@Q<1: 3CroisJ  • -&."50'- C? 59
  17. 2:#,' •< 2:*2:# '  •Crois( 4  • &<2:

    %+ • $;"8)!70 0  • 2: ' • -.< 53 •69$/12769 60
  18. 2-1.0 •$"+5')#%&!( Crois2-  •Crois 4    

    /* •Crois 4   3, 62
  19. +-4' •& •AWS Batch, Step Functions6.0 E/@1"!# 8A •https://www.rco.recruit.co.jp/career/e ngineer/blog/detail_of_job_scheduler_

    on_aws/ • < *:4' •)CD 3?7= B> 2,(;95 $% %1)C# Crois •http://hatenanews.com/articles/2019/ 03/04/103000 65