Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automate Yo' Self (OpenWest 2016)

Automate Yo' Self (OpenWest 2016)

Fine-tuning your development environment means more than just getting your editor set up just so -- it means finding and setting up a variety of tools to take care of the mundane housekeeping chores that you have to do -- so you have more time to program, of course! I'll share the benefits of a number of yak shaving expeditions, including using App::GitGot to batch manage _all_ your git repos, App::MiseEnPlace to automate getting things _just_ so in your working environment, and a few others as time allows.

John SJ Anderson

July 13, 2016
Tweet

More Decks by John SJ Anderson

Other Decks in Programming

Transcript

  1. Automate Yo' Self! OpenWest 2016 Sandy UT John SJ Anderson

    @genehack if anybody has any questions or i'm going too fast, please throw up a hand and ask -- or i'm here all week, grab me on the hallway track
  2. @genehack i go by genehack most places on line. today

    i'm going to talk to you today about some tools and tricks i have for being productive while developing. but before i do that, i need to explain a bit about why i needed it. my life is busy.
  3. I've got a lot of balls in the air don't

    have time for trivial nonsense, so i use a lot of automation and other "lifehacks"
  4. Basic Principles whenever i'm trying to do this, there are

    some basic principles that i try to keep in mind
  5. Don't Make Me Think none of this automation stuff should

    require me to think -- if i have to think about it, it's not really saving me any time
  6. This is my "you made me think" face. the whole

    point is not having to think.
  7. Consistency is Good. so, in those terms, one thing that's

    essential: make everything the same. have a standard directory layout under $HOME. you shouldn't have to think about what system you're on, what shell, what project. things should just be the same -- or at least *correct* -- all the time
  8. Idempotence is better! Idempotence: the property of certain operations in

    mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application
  9. App::MiseEnPlace so, here's an example of idempotence, a utility i

    wrote to manage symlinks and directories in my home directory
  10. App::MiseEnPlace "everything in its place" "mise en place" is a

    French phrase meaning "everythng in its place" -- it comes from cooking, and the principle that you should have all your ingredients prepped and ready to go before you start cooking. i wanted something to make sure i always had my standard directory layout under $HOME as well as setting up various symlinks
  11. % cat ~/.mise --- manage: - doc - etc -

    private - proj/* - src/* create: directories: - bin - proj - proj/go/src/github.com/genehack - src - var links: - Desktop: var/tmp - Desktop: tmp here's what the top level config for mise looks like -- it goes in .mise in your home directory. we have a list of directories we want to manage (more on that in a minute) and a set of directories and symlinks to create. links are source:target
  12. % cat proj/emacs/.mise --- create: links: - DIR: ~/.emacs.d -

    bin/build-most-recent-emacs: BIN - bin/e: BIN - bin/ec: BIN - bin/git-blame-from-line-num: BIN - bin/map-test-lib: BIN this is a per-project config file. mise has a couple special keywords DIR and BIN, that refer to the directory containing the .mise file and ~/bin, respectively the advantage of this is you don't need to have a huge gnarly $PATH with a bunch of project directories in it, everything is just symlinked into ~/bin
  13. % mise [LINK] created ~/proj/emacs -> ~/.emacs.d [LINK] created ~/proj/emacs/bin/build-most-recent-emacs

    -> ~/bin/build-most-recent-emacs [LINK] created ~/proj/emacs/bin/e -> ~/bin/e [LINK] created ~/proj/emacs/bin/ec -> ~/bin/ec [LINK] created ~/proj/emacs/bin/git-blame-from-line-num -> ~/bin/git-blame-from-line-num [LINK] created ~/proj/emacs/bin/map-test-lib -> ~/bin/map-test-lib % mise % rm ~/bin/e % mise [LINK] created ~/proj/emacs/bin/e -> ~/bin/e when we run mise for the first time, you can see it creates all those links. if we run it again, it does NOTHING. Idempotency for the win! If you remove a link and run it again, it creates _just_ that link
  14. App::MiseEnPlace get it from your favorite CPAN mirror available on

    CPAN, minimal dependencies, works on any perl from this decade.
  15. smartcd ok, so that handles getting a consistent directory structure,

    and linking project binaries. what if you have other per-project stuff that you want to set up? environment variables or other stuff?
  16. smartcd Automatically run code when entering or leaving directories or

    sub-directories enter smartcd, which hooks the 'cd' command and then runs scripts when you enter or leave a directory (or even a subdirectory)
  17. % smartcd show enter /Users/genehack/.smartcd/scripts/Users/genehack/fake-node-proj/bash_enter exists ------------------------------------------------------------------------- ######################################################################## # smartcd

    enter - /Users/genehack/fake-node-proj # # This is a smartcd script. Commands you type will be run when you # enter this directory. The string __PATH__ will be replaced with # the current path. Some examples are editing your $PATH or creating # a temporary alias: # # autostash PATH=__PATH__/bin:$PATH # autostash alias restart="service stop; sleep 1; service start" # # See http://smartcd.org for more ideas about what can be put here ######################################################################## autostash NODE=4.2.3 autostash PATH=$PATH:__PATH__/node_modules/.bin/ nvm use $NODE ------------------------------------------------------------------------- smartcd has 'edit' and 'show' subcmds (and a bunch of others), and 'leave' and 'enter' scripts. here's an enter script for an arbitrary node project. the autostash keyword sets up an environment variable that will be *unset* when you leave this directory. you can also run arbitrary commands; the 'nvm' command here selects a version of node to use
  18. % cd ~/fake-node-proc Now using node v4.2.3 (npm v2.14.7) %

    echo xx$NODE xx4.2.3 % cd .. % echo xx$NODE xx here's what it looks like when you cd into that directory. and you can see the env var is set. if we change back out of the directory, the environment variable is unset.
  19. smartcd https://github.com/cxreg/smartcd pretty cool, available on github. works with bash

    and zsh. really easy to install. i can pay this the highest possible compliment: i haven't forked it or needed to patch it in any way, i just use a checkout from the upstream repo
  20. perlbrew plenv nvm App::GitGitr build-most- recent-emacs here are some of

    the ones i've used or use now. perlbrew and plenv let you have multiple perls. nvm does the same thing for node. similar tools exist for python, ruby, etc. then there's GitGitr, which I wrote while maintaining Git wrapper library. I needed to be able to quickly install arbitrary Git versions while responding to bug reports -- so I wrote a little tool that does that. Similarly, I'm an Emacs user. If a new version is released, I want to upgrade, across all my systems - so I scripted that.
  21. Consistency Corollary: Don't trust system binaries Back at the beginning,

    I said "consistency is good". Now, if you're developing on MacOS and deploying to Linux (or dev-ing on Ubuntu and deploying to Debian), you're probably not going to have the same version of tool from the OS (and if you do now, it's not going to last). Even if the versions are the same, there may have been vendor patches applied.
  22. Automate building your critical tools. No, if you really want

    to be consistent, the best approach is to build the tools that are most critical for your project. ("Build" in this case may just mean automating the install; it doesn't have to mean "build from source".) Note: only do this for the *important* stuff. (include examples)
  23. The Silver Searcher Speaking of tools, here's a tool that

    has literally improved my entire development life -- the silver searcher. Anybody here using this?
  24. grep? ok, so, grep -- everybody knows grep, right? let's

    you search for text inside files, which is something you do a lot while developing code.
  25. grep? powerful, speedy, indiscriminate so, grep is super powerful in

    terms of what you can search for, and it's pretty quick, but it's not very selective. you can list all the files you want to search, or search whole dir trees, but you quickly realize this sucks, because of things like .git directories. anybody ever do a recursive grep on a big git checkout? yeah.
  26. ack? powerful, selective, slow it's just as powerful as grep

    in terms of what you can search for, but it's recursive by default (which is what you want) and it's smart about ignoring .git and SVN stuff. the problem is, it's pretty slow, particularly to start up (because Perl)
  27. ag! powerful, selective, FAST ag works much like ack (not

    _exactly_, but close enough), but it's written in C and it's oh so fast.
  28. The Silver Searcher https://github.com/ggreer/the_silver_searcher available on Github, also packaged in

    several Linux distros. again, no higher compliment than to say I just build from a checkout of the upstream. have never needed to fork or patch
  29. My biggest productivity/automation tip so, here's my single biggest tip

    in this whole talk. are you ready? brace yourselves.
  30. Revision control $HOME this is it. anybody know what this

    is? this is your home directory under revision control
  31. Why bother? no, but seriously. it's a little bit of

    a pain to develop the discipline but once you get used to having _everything_ under revision control, it's nice. you don't have to worry about experimenting with anything, backing stuff up, or dealing with cross-machine variation in your environments
  32. say automation again originally i had series of kludgy shell

    scripts to manage repo updates and checkouts super ugly and not worth sharing and then, inspiration: Ingy döt Net talking about App::AYCABTU @ PPW2010
  33. Ingy döt Net this is ingy - he's a crazy

    awesome open source hacker guy who has done a whole bunch of stuff. probably best known for being one of the inventors of YAML
  34. Things I wanted to steal the thing ingy had developed

    had a bunch of stuff i wanted to steal: The basic idea The interface Info about repositories in config file Flexible ways of selecting repos for operations – by #, by name, by tag
  35. Things I wanted to add Support for more than just

    Git Locate repositories in arbitrary locations Easily add and remove repositories Ability to easily extend with more subcommands Most importantly: better name!
  36. GitGot Thus was born GitGot http://search.cpan.org/dist/App-GitGot/ Installs a ‘got’ command

    Uses Moo and App::Cmd under the covers Add new subcommands by writing a single class!
  37. got add you tell got about repos using the 'add'

    command, from inside the git repo
  38. % got add Name: foo URL: Path: /Users/genehack/foo Tags: bar

    it'll prompt you for required info, and supply sensible defaults. note that you can also apply tags (space-delimited)
  39. got add -D or you can just add the '-D'

    switch and it'll automatically use the defaults
  40. got clone <REPO URL> you can also clone a remote

    repo, which will check it out into the working directory and add it to got, prompting you for details
  41. % got clone [email protected]:genehack/app-gitgot.git Name: [app-gitgot]: Path: [/Users/genehack/app-gitgot]: Tags: :

    Cloning into '/Users/genehack/app-gitgot'... it'll prompt you for required info, and supply sensible defaults. note that you can also apply tags (space-delimited)
  42. got fork <GITHUB URL> finally, you can give got a

    github url, and it will fork that project under your github id, then check it out into the current directory and add it to got
  43. ~/.gitgot all the info about the repos managed by got

    lives in this .gitgot file in your home directory
  44. - name: App-Amylase path: /Users/genehack/proj/App-Amylase repo: [email protected]:genehack/App-Amylase.git type: git -

    name: Git-Wrapper path: /Users/genehack/proj/Git-Wrapper repo: [email protected]:genehack/Git-Wrapper.git tags: git type: git - name: HiD path: /Users/genehack/proj/HiD repo: [email protected]:genehack/HiD.git type: git - name: Perl-Build path: /opt/plenv/plugins/perl-build repo: git://github.com/tokuhirom/Perl-Build.git type: git it's just a simple YAML formatted line, totally hand-editable (although you shouldn't _need_ to do that, you can) note that repos can be located anywhere on the disk, don't have to under a common dir or in your home or whatever. anywhere you can write to is fair game
  45. But now what? ok, so you've added all your git

    repositories to got. what now?
  46. 1) App-Amylase git [email protected]:genehack/App-Amylase.git 2) Git-Wrapper git [email protected]:genehack/Git-Wrapper.git 3) HiD

    git [email protected]:genehack/HiD.git 4) Perl-Build git git://github.com/tokuhirom/Perl-Build.git 5) Perl-Critic git [email protected]:genehack/Perl-Critic.git 6) STAMPS git [email protected]:genehack/STAMPS.git 7) advanced-moose-class git ssh://[email protected]/train/advanced-moose-class.git 8) app-gitgitr git [email protected]:genehack/app-gitgitr.git 9) app-gitgot git [email protected]:genehack/app-gitgot.git that'll get you this sort of listing.
  47. got ls -q if you don't want to see the

    upstream repo info, you can use the '-q' or '--quiet' switch
  48. 1) App-Amylase 2) Git-Wrapper 3) HiD 4) Perl-Build 5) Perl-Critic

    6) STAMPS 7) advanced-moose-class 8) app-gitgitr 9) app-gitgot and that'll get you this output. note the numbers - those will give you a way to select repos for other commands
  49. got ls [repos] easiest way to demo that is with

    an example. you can restrict the listing
  50. 5) Perl-Critic also, note that the list is always sorted

    the same way, so the numbers will be stable (unless you add new repos)
  51. got ls -t git or by using tags. can specify

    multiple tags with multiple '-t' switches. they combine with 'or' semantics.
  52. 2) Git-Wrapper 8) app-gitgitr 9) app-gitgot here are all the

    repos tagged with 'git' (at least in our example)
  53. got ls 5-8 HiD 21 -t git finally, you can

    combine all of these selection methods together. here we're asking for repos 5 thru 8, the repo named HiD, repo 21, and all repos tagged with git
  54. 2) Git-Wrapper 3) HiD 5) Perl-Critic 6) STAMPS 7) advanced-moose-class

    8) app-gitgitr 9) app-gitgot 21) etc and this is what we get note that most commands operate on all repos, and any that do, you can use these techniques to restrict the command to a subset.
  55. 1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD

    : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK that'll get you output like this. again, note the use of color to give quick visual cues
  56. 1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD

    : Dirty 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK Dirty got status will let you know if a repo has uncommitted changes. super handy if, for example, working on one machine and are going to move to another one and want to see what hasn't been committed.
  57. 1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD

    : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK Ahead by 1 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK 1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : Dirty 5) Perl-Critic : OK 6) STAMPS : OK 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK Dirty 1) App-Amylase : OK 2) Git-Wrapper : OK 3) HiD : OK 4) Perl-Build : OK 5) Perl-Critic : OK 6) STAMPS : OK Ahead by 1 7) advanced-moose-class : OK 8) app-gitgitr : OK 9) app-gitgot : OK Ahead by 1 it'll also tell you if you have local commits that haven't been pushed to the remote yet
  58. got st -q finally, you can use the '-q' switch

    to hide the "uninteresting" stuff
  59. got st -q 3) HiD : Dirty 6) STAMPS :

    OK Ahead by 1 Dirty Ahead by 1 which in this case is all the repos that don't have changes and are up to date
  60. got up which abbreviates to 'up' and yeah, i should

    have called it pull but i'm a dummy and we're stuck with it now.
  61. 1) App-Amylase : Up to date 2) Git-Wrapper : Up

    to date 3) HiD : Up to date 4) Perl-Build : Updated Updating 7f25f89..72587c8 Fast-forward lib/Perl/Build.pm | 14 +++++++++++++- script/perl-build | 14 +++++++++++++- 2 files changed, 26 insertions(+), 2 deletions(-) 5) Perl-Critic : Up to date Updated it'll do pretty much what you expect (and it supports '-q' too)
  62. got update_status finally, there's a command that combines both those,

    because it's something i do pretty frequent -- update everything, then look at the status of everything
  63. got upst -q and it also supports the '--quiet' option

    to only show you the interesting stuff
  64. got fetch if you're not a fan of the way

    'git pull' works you can also run 'git fetch' via got.
  65. got push you can even do a push across all

    your repos at once. (personally, this strikes me as insane but somebody sent in a patch, so...)
  66. got this at some point, somebody sent in a patch

    to add 'got this' -- which tells you if the current directory is under got control
  67. got that <DIRECTORY> this provoked somebody else to send in

    a 'got that' command, which does the same thing, but takes a path to check
  68. got chdir finally, there are a number of commands that

    help you jump to the directory of a project. got chdir
  69. got cd also spelled 'got cd', will change your current

    working directory to the given repo (note that this is one of the few got commands that requires a single repo)
  70. got tmux we also have tmux integration -- 'got tmux'

    will open a new tmux window with the working directory in your repo. this _can_ be done with multiple repos. better, the tux window is persistent; as long as it's open 'got tmux' will just select the already open window, not open a new one
  71. got tmux -s you can also spawn whole new tmux

    sessions if you prefer those to windows -- and again, those will be re-used as long as they're around
  72. BLATANT PUG If you're not sure what 'cpanm' is, come

    to my Unfrozen Paleolithic Perl Programmer talk tomorrow at 4pm! blatant plug
  73. Find me at OpenWest and I'll help you install! or

    you can find me on the hallway track and i'll help you get it installed on your machine
  74. package App::GitGot::Command::chdir; # ABSTRACT: open a subshell in a selected

    project use 5.014; use App::GitGot -command; use Moo; extends 'App::GitGot::Command'; use namespace::autoclean; sub command_names { qw/ chdir cd / } sub _execute { my( $self, $opt, $args ) = @_; unless ( $self->active_repos and $self->active_repos == 1 ) { say STDERR 'ERROR: You need to select a single repo'; exit(1); } my( $repo ) = $self->active_repos; chdir $repo->path or say STDERR "ERROR: Failed to chdir to repo ($!)" and exit(1); exec $ENV{SHELL}; } 1;
  75. sub _execute { my( $self, $opt, $args ) = @_;

    unless ( $self->active_repos and $self->active_repos == 1 ) { say STDERR 'ERROR: You need to select a single repo'; exit(1); } my( $repo ) = $self->active_repos; chdir $repo->path or say STDERR "ERROR: Failed to chdir to repo ($!)" and exit(1); exec $ENV{SHELL}; }
  76. package App::GitGot::Command::chdir; # ABSTRACT: open a subshell in a selected

    project use 5.014; use App::GitGot -command; use Moo; extends 'App::GitGot::Command'; use namespace::autoclean; sub command_names { qw/ chdir cd / } sub _execute { my( $self, $opt, $args ) = @_; unless ( $self->active_repos and $self->active_repos == 1 ) { say STDERR 'ERROR: You need to select a single repo'; exit(1); } my( $repo ) = $self->active_repos; chdir $repo->path or say STDERR "ERROR: Failed to chdir to repo ($!)" and exit(1); exec $ENV{SHELL}; } 1;
  77. Suggestions welcome! areas for improvement: support for other VCSen better

    config management tools any other crazy workflow improvement you can think of!
  78. Thanks OpenWest organizers Ingy döt Net Yanick Champoux Michael Greb

    Rolando Pereira Chris Prather photo credits: all photos by speaker except Ingy döt Net photo - https://www.flickr.com/photos/bulknews/389986053/ and pug - https://upload.wikimedia.org/wikipedia/commons/d/d7/Sad-pug.jpg and automate yo'self - somewhere on the net
  79. Questions? https://joind.in/event/openwest-2016/automate-yoself as i said, i'm here all week and

    i *love* to talk to people about this productivity type stuff, so grab me on the hallway track. i'm friendly.