Travisify - Project life cycle in the own repository

Miguel Ángel's (@magmax9) talk at PyConEs'14

Transcript

1. TRAVISIFY PROJECT LIFE CICLE IN THE OWN REPOSITORY
2. ABOUT ME Miguel Ángel García SRE (Systems Reliability Engineer) at @magmax9 http://magmax.org
3. ONCE UPON A TIME...
4. JUST ONE BIG REPOSITORY
5. TESTING EVOLUTION No testing at all Some manual testing performed by QA team Some Acceptation tests written by QA team Unit, Integration and Acceptation tests written by developers (no QA team)
6. The bad guy at the begining Time saver later Deployments center right now
7. TESTING Ad hoc testing environment One team to manage the testing environment: Testing Framework (TFW)
8. PROBLEMS It doesn't scale to other repositories/projects The team was not able to deal with problems developers are used to. Developers want to improve to that environment
9. THE PRESENT
10. A BUNCH OF REPOSITORIES
11. PROBLEMS Impossible to be managed by just one team Too different technologies: PHP: fpm, hphpi Java: tomcat, jboss, android Erlang, Python, Ruby, ... make, rake, ant, maven, gradle, ...
12. Another aproach is required:
13. TRAVIS IDEA The repository contains more than code: The definition of the testing environment The test execution commands
14. OUR REQUIREMENTS One job per branch Dynamic reconfiguration Special jobs for releases Just one testing environment per project Some projects require several environments (client- server)
15. OUR WISHLIST Reuse Jenkins if possible... ... and avoid coupling if possible
17. APROACH Jenkins has a Python API Jenkins job definition is made in XML
18. APROACH 1. Create a XML template 2. Parameterize the XML template 3. Use a YAML file in the repo to store values
19. Later on, we found that our library must communicate with Jenkins in order to: ERRORS Check if a plugin is installed Check a plugin version Still to be done...
20. USE CASE 1: CREATION We created the "tu-ci" (Tuenti Continuous Integration) command: Reads information about working copy Sends a message to Flow to create the repository Flow reads the .ci.yml and reconfigures the Jenkins Job
21. FLOW Is our centralized deployments tool Django You must be tired to hear us talking about it
22. USE CASE 2: UPDATE We added a hook in our git and mercurial repositories: It just sends a curl to Flow Flow regenerates the Jenkins Job
23. USE CASE 3: LAUNCH The Jenkins Job: 1. Generates a VirtualEnv environment 2. Installs cigen 3. Uses the cigen runner to parse(again) the .ci.yml
24. TELL ME WHAT TO DO And not where, when or how Write meta-commands in order to help developers: Create resources for servers (BBDDs, memcache, ...) Deploy the server side for clients Launch VMs
25. META FEATURES We discovered that: Some commands must be executed to setup a job Some commands should be executed on success/failure
26. EASY TO IMPLEMENT WITH CURRENT DESIGN Just added new mandatory stages!
27. SHOW ME THE CODE! Not yet :(
28. SCHEMA Platform where it should run Workflows are lists of Stages Stages are lists of Steps Steps are shell scripts Plugins are... a best effort :D
29. SCHEMA PYTHON RULES!! d e f p r o j e c t _ f r o m _ s t r e a m ( s t r e a m , p r o j e c t = N o n e , y a m l _ l i b = N o n e ) : y a m l _ l i b = y a m l _ l i b o r y a m l p r o j e c t = p r o j e c t o r P r o j e c t ( ) c o n t e n t = y a m l _ l i b . l o a d ( s t r e a m ) p r o j e c t . p l a t f o r m = c o n t e n t . g e t ( ' p l a t f o r m ' , ' d e f a u l t ' ) p r o j e c t . t i m e o u t = c o n t e n t . g e t ( ' t i m e o u t ' , p r o j e c t . t i m e o u t ) p r o j e c t . p a r a m e t e r s = c o n t e n t . g e t ( ' p a r a m e t e r s ' , { } ) p r o j e c t . p l u g i n s = c o n t e n t . g e t ( ' p l u g i n s ' , { } ) p r o j e c t . s t a g e s = c o n t e n t . g e t ( ' s t a g e s ' , { } ) p r o j e c t . w o r k f l o w s = c o n t e n t . g e t ( ' w o r k f l o w s ' , { } ) r e t u r n p r o j e c t
30. SCHEMA EXAMPLE p l a t f o r m : a n d r o i d w o r k f l o w s : f a s t : [ c o m p i l e , u n i t ] f u l l : [ c o m p i l e , u n i t , i n t e g r a t i o n ] r e l e a s e : [ c o m p i l e , u n i t , p u b l i s h ] s t a g e s : c o m p i l e : - f i n d - n a m e " * . p y c " - d e l e t e - m a k e u n i t : p y . t e s t t e s t s / u n i t i n t e g r a t i o n : p y . t e s t t e s t s / i n t e g r a t i o n p u b l i s h : p y t h o n s e t u p . p y b d i s t _ w h e e l u p l o a d p l u g i n s : c o v e r a g e : f i l e : c o v e r a g e . x m l s o n a r :
31. PLUGINS Junit/PHPUnit Coverage Sonar Automatic versioning
32. USE CASE 1: CREATION
33. c l a s s T e m p l a t e L o a d e r ( o b j e c t ) : d e f g e t _ t e m p l a t e ( s e l f , n a m e ) : a s s e r t i s i n s t a n c e ( n a m e , s t r ) t e m p l a t e _ f i l e = s e l f . _ t e m p l a t e _ f i l e ( n a m e ) l o g g e r . d e b u g ( " L o a d i n g t e m p l a t e % s " , t e m p l a t e _ f i l e ) w i t h o p e n ( t e m p l a t e _ f i l e ) a s f d : r e t u r n T e m p l a t e ( f d . r e a d ( ) ) d e f _ t e m p l a t e _ f i l e ( s e l f , n a m e ) : c u r r e n t = o s . p a t h . d i r n a m e ( _ _ f i l e _ _ ) f i l e n a m e = n a m e + ' . j i n j a ' r e t u r n o s . p a t h . j o i n ( c u r r e n t , ' t e m p l a t e s ' , f i l e n a m e ) c l a s s G e n e r a t o r ( o b j e c t ) : T E M P L A T E = N o n e d e f _ _ i n i t _ _ ( s e l f , p r o j e c t , l o a d e r = N o n e ) : s e l f . p r o j e c t = p r o j e c t s e l f . _ l o a d e r = l o a d e r o r T e m p l a t e L o a d e r ( ) d e f g e n e r a t e ( s e l f , r e l a t i v e _ p r o j e c t _ f i l e _ p a t h , w o r k f l o w ) : t e m p l a t e = s e l f . _ l o a d e r . g e t _ t e m p l a t e ( s e l f . T E M P L A T E ) r e t u r n t e m p l a t e . r e n d e r ( p r o j e c t = s e l f . p r o j e c t , c r e a t i o n _ t i m e = t i m e . t i m e ( ) , p r o j e c t _ f i l e _ p a t h = r e l a t i v e _ p r o j e c t _ f i l e _ p a t h , w o r k f l o w = w o r k f l o w , c i g e n = C I G E N , )
34. c l a s s J e n k i n s G e n e r a t o r ( G e n e r a t o r ) : T E M P L A T E N A M E = ' j e n k i n s '
35. p r o j e c t = c i g e n . l o a d e r . p r o j e c t _ f r o m _ f i l e ( a r g s . p r o j e c t _ f i l e ) p r o j e c t . r e p o s i t o r y . u r l = ' h t t p : / / e x a m p l e . o r g / f o o / b a r ' p r o j e c t . r e p o s i t o r y . k i n d = c i g e n . p r o j e c t . R e p o s i t o r y . G I T p r o j e c t . r e p o s i t o r y . b r a n c h = a r g s . r e p o _ b r a n c h g e n e r a t o r = c i g e n . g e n e r a t o r . J e n k i n s G e n e r a t o r ( p r o j e c t ) p r i n t g e n e r a t o r . g e n e r a t e ( a r g s . p r o j e c t _ f i l e , a r g s . w o r k f l o w )
36. USE CASE 3: CREATION
37. d e f r u n ( s e l f ) : s e l f . _ c o n f i g u r e _ p l u g i n s ( ) s e l f . _ u p d a t e _ e n v i r o n m e n t ( ) i f ' s e t u p ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f a u l t S e t u p ' , s e l f . _ c o n f i g [ ' s e t u p ' ] ) t r y : f o r s t a g e i n s e l f . _ p r o j e c t . w o r k f l o w s [ s e l f . _ w o r k f l o w ] : s t e p s = s e l f . _ p r o j e c t . s t a g e s . g e t ( s t a g e , N o n e ) i f s t e p s i s N o n e : l o g g e r . w a r n i n g ( ' E r r o r o n S t a g e " % s " ' , s t a g e ) c o n t i n u e s e l f . _ r u n _ s t a g e ( s t a g e , s t e p s ) s e l f . _ r u n _ s t a g e ( ' D e f a u l t S u c c e s s ' , s e l f . _ c o n f i g . g e t ( ' o n _ s u c c e s s ' , [ ] ) , p l u g i n = ' o n _ s ' ) l o g g e r . i n f o ( ' S U C C E S S ' ) e x c e p t E x c e p t i o n : i f ' o n _ f a i l u r e ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f . F a i l u r e ' , s e l f . _ c o n f i g [ ' o n _ f ' ] ) r a i s e f i n a l l y : i f ' t e a r d o w n ' i n s e l f . _ c o n f i g : s e l f . _ r u n _ s t a g e ( ' D e f . T e a r d o w n ' , s e l f . _ c o n f i g [ ' t . ' ] ) s e l f . _ p r i n t _ r e p o r t ( )
38. THE FUTURE
40. IMPROVEMENTS More configurable by developers: Plots Multi-platform (server+client) Artifacts to store Notifications
41. NEW REQUIREMENTS Cigen should communicate to Jenkins Check if a plugin is installed Check plugin version and compatibility Modularize the current (HUGE) template Move logic to code instead of template Improve integration: Go-CI, Travis, ... Jira, Taiga, ...
42. DEPLOYMENTS Just one idea: To manage deployments in the same way
43. GOAL
44. GOAL
45. GOAL
46. GOAL
47. GOAL
48. GOAL
49. GOAL
50. GOAL
51. REQUIREMENT Software knows WHAT to be deployed But not WHERE, WHEN or HOW
52. VAGRANT
53. DOCKER
54. PANAMAX
55. MESOS
56. GOAL
58. CONTACT Miguel Ángel García @magmax9 http://magmax.org magarcia@tuenti.com