Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
NACIS 2016 - Practical Cartography Day
Search
Seth Fitzsimmons
October 19, 2016
Design
1
550
NACIS 2016 - Practical Cartography Day
Bash and Make tips for scripting reproducible workflows
Seth Fitzsimmons
October 19, 2016
Tweet
Share
More Decks by Seth Fitzsimmons
See All by Seth Fitzsimmons
Working with OpenStreetMap using Apache Spark and GeoTrellis - SotMUS 2018
mojodna
0
320
OpenMapKit + POSM
mojodna
0
2k
Portable OpenStreetMap at SotM US 2016
mojodna
0
200
Print Cartography in a Multi-Resolution World
mojodna
4
800
How can open data save the world? How can we help?
mojodna
1
120
Trail Assist
mojodna
0
170
Worldwide Terrain and Cloud Infrastructure: an update on Stamen's Knight Grant
mojodna
0
270
Running Your Own Rendering Infrastructure (FOSS4G 2014 Edition)
mojodna
0
260
What We've Made Together
mojodna
0
160
Other Decks in Design
See All in Design
数理的アプローチで挑むスマホUIのデザイン改善:タップ成功率推定ツール「Tappy」の社内活用事例 / Improving Smartphone UI Design with a Mathematical Approach: In-house Use Case of the Tap Success Rate Estimation Tool "Tappy"
lycorptech_jp
PRO
0
880
root COMPANY DECK / We are hiring!
root_recruit
PRO
1
24k
デザインシステムの「種」を使って、受託開発を加速させる
akane___ui
0
11k
OSO2025-マサカリと太陽:伝え方の情報デザイン
majimasachi
0
440
What makes a great Director?
_limex_
0
330
【PoCで終わらない】運用フェーズまで見据えたAI駆動UIデザイン/フロントエンド開発実践
kitami
1
400
組織の右腕として共創する ー デザインと経営の二つの視点から見えた、新しい支援のかたち/ Designship2025_Nishimura
root_recruit
PRO
0
170
Liquid GlassとApp Intents
touyou
0
450
Goodpatch Tour💙 / We are hiring!
goodpatch
31
900k
Yahoo!フリマ:生成AI利用機能ならではのインターフェース設計について / Yahoo! JAPAN Flea Market: Interface Design Specific to Generative AI Utilization Features
lycorptech_jp
PRO
0
520
Meet, Learn, Grow × AI ― コミュニティで加速するスキル循環 「コミュニティと関わり方」
tame
0
280
【最新】マズロー安達の弟子実績(1期-6期の37人分)
maslow_akkun
0
3.1k
Featured
See All Featured
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.7k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Imperfection Machines: The Place of Print at Facebook
scottboms
269
13k
Raft: Consensus for Rubyists
vanstee
140
7.2k
GraphQLの誤解/rethinking-graphql
sonatard
73
11k
KATA
mclloyd
PRO
32
15k
Site-Speed That Sticks
csswizardry
13
930
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
Rebuilding a faster, lazier Slack
samanthasiow
84
9.2k
Navigating Team Friction
lara
190
15k
Designing for humans not robots
tammielis
254
26k
Principles of Awesome APIs and How to Build Them.
keavy
127
17k
Transcript
$ echo “Make, etc.” | \ wall -g PCD Seth
Fitzsimmons Stamen Design, etc.
[email protected]
# slides $ open \ http://bit.ly/2eFFvJt
# why? # # * repeatable # * self-documenting #
* transformations # <> data changes
# on Windows 10? # no worries # # “Bash
on Ubuntu # on Windows”
# shell fundamentals $ /bin/sh -c theory
# is a comment $ is a prompt
# <cmd> ––help usually # works # man <cmd> is
detailed
# do this, then that $ this; that
# if this, then that $ this && that
# that, unless this $ this || that
# standard file # descriptors # (stdio)
# stdin (fd 0) $ cat > greeting Hi! ⌃d
$ cat greeting Hi!
# stdout (fd 1) $ echo “Hi PCD!”
# stderr $ >&2 echo Error
# a data black hole /dev/null
# redirection $ echo hi > greeting
# append $ echo hey >> greeting
# stderr → stdout $ thing 2>&1 logs
# pipe # this means everything # can be combined!!!
$ cat /etc/passwd | \ grep -i calendar
# exit codes $ thing; echo $?
# success (0) $ thing && echo $? 0
# failure (not 0) $ thing || echo $? 1
$ make basics
# same input (repeatedly), # same output $ make idempotency
Yes $ make idempotency Yes
# target this: touch $@
# prerequisite this: that # that exists cp that this
$ make -j2 vars fns
# target this: # outputs “this” echo $@
# all prereqs this: that the_other # “that the_other” echo
$^
# first prereq this: that the_other # “that” echo $<
# first prereq this: that the_other # “that” echo $<
# nth prereq this: that the_other # “the_other” echo $(word
2, $^)
# catch-all %: echo 42 > $@
$ ls /bin
# explore
# who am I? $ whoami seth
# where am I? $ pwd /home/seth
# what time is it? $ date Tue Oct 18
23:58:42 MDT 2016
# what’s here? $ ls -lh total 8 -rw-r--r-- 1
seth wheel …
# what’s in that file? $ cat /etc/passwd …
# …a page at a time $ less /etc/passwd …
# …just the beginning $ head /etc/passwd …
# …just the end $ tail /etc/passwd …
# what variables are set? $ env HOME=/home/seth …
# where’s that file? $ find . -type f -name
hi <list of files>
# where’d that file go? # (full-text search, macOS) $
mdfind Seattle <list of files>
# get help $ man man …
# manipulation
# create a file $ touch file
# copy a file $ cp file file2
# move a file $ mv file2 file3
# delete a file $ rm file3
# create a directory $ mkdir -p my/stuff
# remove a directory $ rmdir my/stuff
# remove a directory # and everything in it $
rm -r my
# find lines in a file $ grep -i name
file.txt
# find non-matching lines $ grep -v name file.txt
# count words, lines, # characters $ wc file.txt
# pretty-print JSON $ jq . file.json
# extract fields $ jq .name file.json
# replace things $ sed 's/this/that/' file
# extract columns $ cut -d , -f 1,3 file.csv
# extract columns $ cut -d , -f 1,3 file.csv
# compression
# open a zip file $ unzip file.zip
# list zip contents $ unzip -v file.zip
# create a zip $ zip file file.zip
# open a tarball $ tar zxf file.tar.gz
# list a tarball $ tar ztf file.tar.gz
# create a tarball $ tar zcf file.tar.gz stuff/
# compress with gzip $ gzip file.tar
# uncompress with gzip $ gzip -d file.tar.gz
# misc
# always exit 0 $ true; echo $? 0
# always exit non-0 $ false; echo $? 1
# fetch and fail # if appropriate $ curl -f
nacis.org
# download $ wget nacis.org
# download $ wget nacis.org
# “open” (macOS) $ open nacis.org
# display progress $ cat /etc/passwd | \ pv |
wc -l
# also write to a file $ echo hi |
tee file
$ man bash
#!/usr/bin/env bash
set -eo pipefail
set -x
NACIS=2016
# assignment NACIS=“2016”
# capture a command NACIS=$(curl nacis.org)
echo $NACIS
echo ${NACIS}
# set a default value echo ${NACIS:-PCD}
# replace echo ${NACIS/2016/2017}
# replace all echo ${NACIS//2016/2017}
# substring echo ${NACIS:2:2}
# remove suffix $ filename=“world.tif” $ echo ${filename%.tif} world
# do math $ echo $[2 ** 3] 8
if [[ “this” != “that” ]]; then echo Control Flow
elif [[ ! -f file ]]; then touch file elif [[ $six -le $five ]]; then false else rm -f file fi
while true; do echo Control Flow done
for f in $(ls); do echo $f done
# man test
# misc
# extract filename $ basename /etc/passwd passwd
# extract directory $ dirname /etc/passwd /etc
$ make recipes
$ make convert source.json: source.shp ogr2ogr \ -t_srs EPSG:4326 \
-f GeoJSON \ $@ \ $<
$ make reproject output.tif: source.tif gdalwarp \ -q \ -t_srs
EPSG:3857 \ $< \ $@
$ make wilderness data/S_USA.Wilderness.zip: @mkdir -p $$(dirname $@) @curl -sfL
http://data.fs.usda.gov/ geodata/edw/edw_resources/shp/ S_USA.Wilderness.zip \ -o $@
$ make table db/wilderness: sql/wilderness.sql psql \ -c "\d $(subst
db/,,$@)" \ > /dev/null \ 2>&1 || \ psql \ -v ON_ERROR_STOP=1 \ -qX1f \ $<
# resources • https://bost.ocks.org/mike/make/ • http://www.gregreda.com/2013/07/15/unix- commands-for-data-science/ • https://google.github.io/styleguide/shell.xml •
https://github.com/stamen/toner-carto/blob/ master/Makefile (WARNING!) • http://mojodna.net/2015/01/07/make-for-data- using-make.html