Share
Embed
Get Power From Command Line
It's the deck for a modular course at NYCU. The goal of this course is to help the students and engineers who have focused on programming or data science to catch up the command line skills.
This deck covers an introduction of command line, TSV, CSV, JSON processing (using sed, awk, jq, etc.), file, process, container management (using Docker), system monitoring, SSH, SSH tunneling, and developing on command line.
This deck is not going to help you become a professional system administrator, but it will give you a wonderful head start on becoming a superb command line user!
Mosky Liu
October 28, 2022
Transcript
130 slides
-
Mosky Architect, Pinkoi • Data Science With Python • Statistical
Regression With Python • Hypothesis Testing With Python • Practicing Python 3 • The mediums. • The PyCons, etc. -
1. Introduction to command line 2. Text processing: e.g., CSV,
JSON, parallelizing. 3. System management: e.g., fi les, processes, monitoring. 4. Work on a remote machine: e.g., SSH, SSH tunneling. 5. Developing on CLI Outline -
What commands? • POSIX • Portable Operating System Interface •
IEEE Std 1003.1-2008 • List of Unix commands – Wikipedia • Ubuntu 22.04 LTS Server • curl, gawk, git, htop, less, lsof, man - db, rsync, strace, tmux, vim, ssh, etc. • The de facto standard • tree, ripgrep, jq, etc. • The modern alternatives • bat (cat), fd - f i nd ( fi nd), etc. -
How to use this deck? • Practice makes perfect: •
Try every single command in this deck, and downloading the deck as a PDF may make it easier. • Be careful of the spaces and newlines after paste. • Visit the links, but don't read; scan them to index in your brain. -
“Command Notes” • Memorize all the commands? No. • You
may try to note them, and make them searchable. • Inputs & Outputs. • Tags. • Like → -
Get your terminal emulator • iTerm for macOS • See
also: Mosky’s Preferences in macOS § iTerm and search “Bash 5” in the doc! • Windows Terminal • GNOME Terminal / Konsole -
Connect to ... • macOS with GNU/Linux commands • Install
the GNU/Linux commands on macOS • WSL • Windows Subsystem for Linux • Install Linux on Windows with WSL • Ubuntu • Or, -
The container • Get Docker • $ docker pull ubuntu:22.04
$ docker network create ubuntu - n $ docker run - dit - p 2222 : 22 - p 8000 : 8000 - - network ubuntu - n - - name ubuntu - c ubuntu:22.04 $ docker exec - it ubuntu - c bash • ubuntu - c# yes | unminimize && apt install curl gawk git htop less lsof man - db rsync strace tmux vim ssh - y -
The container • Get Docker • $ docker pull ubuntu:22.04
$ docker network create ubuntu - n $ docker run - dit - p 2222 : 22 - p 8000 : 8000 - - network ubuntu - n - - name ubuntu - c ubuntu:22.04 $ docker exec - it ubuntu - c bash • ubuntu - c# yes | unminimize && apt install curl gawk git htop less lsof man - db rsync strace tmux vim ssh - y $ hints as a normal user. ubuntu - c# hints as a superuser and in the Ubuntu container. They are default in the Ubuntu Server, but not in the container image. -
Checkpoint: Log into the container • You shall see →
• Hints: • $ docker stop ubuntu - c; docker rm ubuntu - c # to start over. • $ docker start ubuntu - c $ docker exec - it ubuntu - c bash # after reboot if any. -
Read the manual • $ man man # Try
`\ - k` <n> <u> # and <h> <q> `\ - k` • $ man - k ls • $ ls - - help | less # <q> to exit. • $ ls - - help | grep ' - l' • $ help for • $ type man • $ type ls • $ type for • Google: “man yes” • explainshell.com • Bash Reference Manual -
Read the manual • $ man man # Try
`\ - k` <n> <u> # and <h> <q> `\ - k` • $ man - k ls • $ ls - - help | less # <q> to exit. • $ ls - - help | grep ' - l' • $ help for • $ type man • $ type ls • $ type for • Google: “man yes” • explainshell.com • Bash Reference Manual Two spaces. -
Edit commands like a ninja • Bash Keystrokes – Mosky’s
Notes • $ ec<Tab>Hello, World! • $ <Up><C-w>Ctrl-W! • $ <Up><M-C-h>H! • $ <Up><C-h>? • Also works in Python shell and many other places. • See also: • “CTRL-p CTRL-q” to detach – Docker Docs • macOS Shortcuts – Mosky’s Notes • Windows Shortcuts – Mosky’s Notes -
Edit commands like a ninja • Bash Keystrokes – Mosky’s
Notes • $ ec<Tab>Hello, World! • $ <Up><C-w>Ctrl-W! • $ <Up><M-C-h>H! • $ <Up><C-h>? • Also works in Python shell and many other places. • See also: • “CTRL-p CTRL-q” to detach – Docker Docs • macOS Shortcuts – Mosky’s Notes • Windows Shortcuts – Mosky’s Notes Or <M-Rubout>. The <Rubout> is <Delete> or <Backspace>. Or <C-p>, if not from the docker. “Space” vs. “Alphanumeric” The <M> is <Opt> or <Alt>. -
Checkpoint: “Hey, Ctrl-K!” • $ echo Hello, World! • $
<Up><your keystrokes> Hey, Ctrl-K! • Hints: • <C-a>, <M-f>, <C-k> -
Quoting • $ echo Hello, World! # 2 arguments. •
$ echo 'Hello, World!' # 1 argument. • $ echo Hello, $HOSTNAME! • $ echo "Hello, $HOSTNAME!" • $ echo "Hello, \$HOSTNAME!" • $ echo 'Hello, $'HOSTNAME! • $ echo $'Hello,\nWorld' # ANSI-C Quoting -
Pipe, AND, OR • $ echo 'Hello, World!' | grep
Hello • $ echo 'Hello, World!' | grep Hello && echo Found! • $ echo 'Hello, World!' | grep Hey && echo Found! • $ echo 'Hello, World!' | grep Hey || echo Not found. • $ echo 'Hello, World!' | grep Hello && echo Found! || echo Not found. • $ echo 'Hello, World!' | grep Hey && echo Found! || echo Not found. -
Checkpoint: Explain the echo by yourself • explainshell.com • $
echo 'Hello, World!' | grep Hello && echo Found! || echo Not found. -
Loops • $ cd ~ # cd: change directory •
~$ touch a.txt b.txt c.txt • $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done # It's a dry run. • $ for name in *.txt; do echo mv $name ${name/.txt/.csv}; done • $ while true; do date; docker container top ubuntu - c; echo; sleep 3; done • $ until docker start ubuntu - c; do sleep 3; done -
Redirections fi le descriptor:name = 0:stdin, 1:stdout, 2:stderr • $
cd ~ • ~$ echo Hello, File! >hello.txt • $ cat hello.txt • $ cat hello.txt | grep Hello • $ grep <hello.txt Hello • $ grep <hello.txt Hello >out.txt • $ grep <hello.txt Hello >>out.txt # appends to out.txt. -
• $ grep <hello.txt - - bad Hello >out.txt #
still prints errors. • $ grep <hello.txt - - bad Hello >out.txt 2>err.txt • $ grep <hello.txt - - bad Hello >out.txt 2>&1 • $ grep <hello.txt - - bad Hello &>both.txt • $ grep <hello.txt - - bad Hello 2>/dev/null -
• $ cat <<HEREDOC | grep paste > Hello,
Here Doc! > It's useful to paste something. > HEREDOC • $ cat <<<'Hello, Here String!' • $ python3 <<<'print(1234 * * 1234)' # C's unsigned long is 0 to 18446744073709551615, btw. # Useful! -
Expansions & Substitutions • $ echo x.{txt,csv} • $ echo
~ • $ echo ~root # or any username. • $ cd ~ • ~$ touch a.{txt,csv} • $ echo a.* • $ echo x.* # - > x.* • $ echo ?.* • $ echo [abcx].txt • $ echo [abcx].csv -
• $ echo $name • $ echo ${name} • $
echo ${name/.txt/.csv} • $ echo `echo 'Hello, Echo!'` • $ echo $(echo 'Hello, Echo!') • $ cat <(echo 'Hello, Cat!') • See also: • Bash scripting cheatsheet § Parameter expansions • Shell Style Guide § Variable expansion -
Environments & Parameters • Bash scans its own environment
variables and creates shell parameters. • $ echo $PATH • $ A=1 env | grep A= # - > A=1 • $ echo $A # A is empty. • $ A=2 • $ echo $A # - > 2 • $ env | grep A= # - > <empty> • $ export A • $ # export A=20 • $ env | grep A= # - > A=2 • $ A=3 echo $A # - > 2 • $ # = = = $ A=3 echo 2 • $ export - n A && env | grep A= -
Job Control • $ vi a.txt # and <C-z>. •
$ vi b.txt # and <C-z>. • $ jobs • $ fg # <CTRL-G> shows f i lename. • $ fg - # like Cmd-Tab or Alt-Tab. • $ fg 1 # `:q` to exit vim. • $ kill %2 • $ sleep 3 && echo Hi! & # = = = • $ bash - c 'sleep 3 && echo Hi!' # and <C-z>. • $ bg -
Aliases • $ alias ll • $ type ll •
$ ls - - help | grep ' -[alF]' • $ alias hi=history • $ hi 10 • $ alias . . = 'cd . . ' • $ cd /usr/bin • usr: User System Resources • $ . . -
History • <Up>, <Down>; <C-p>, <C-n> • <C-r>, <C-s>: search
history. • $ <a long long command> #TAG • $ <C-r>#TAG • $ stty - ixon # <C-s> needs it. • <C-c>: discard the editing. • $ history 10 • $ history - a • $ history - n • $ history - - help | grep ' -[an]' -
Checkpoint: Hack the which • $ which echo /usr/bin/echo
• $ ??? which echo bash: which: No such f i le or directory • Hints: • which searches the PATH for the executable fi le. -
TSV Create TSV fi le using cat • $ cd
• ~$ cat <<EOS >cmds.tsv > <Open https: / / en.wikipedia.org/wiki/List_of_Unix_commands, copy the table, and paste here.> > EOS -
TSV Create TSV fi le using vim • $ cd
• ~$ vi cmds.tsv • (vim) i # goes the insert mode. • (vim) <Open https: / / en.wikipedia.org/wiki/List_of_Unix_commands, copy the table, and paste here.> • (vim) <Esc> # backs to the normal mode. • (vim) :x<CR> # goes the command - line mode, write and quit. -
• If any garbled text: • ubuntu - c$ LANG=C.UTF-8
bash • or, • (host)$ docker exec - it ubuntu - c env LANG=C.UTF-8 bash -
• $ wc - l cmds.tsv # wc: word count
• $ cat cmds.tsv • $ head cmds.tsv • $ tail cmds.tsv • $ grep <cmds.tsv cat # = = = grep cat cmds.tsv • grep: g/re/p in the ed command, globally search for a regular expression and print matching lines. • $ grep <cmds.tsv '^.cat' • $ grep <cmds.tsv '^.cat' - o -
Checkpoint: Copy the head and paste to a sheet •
A sheet in Google Sheets, Excel, LibreO ff i ce Calc, like → • Hints: • Just like how did you paste the table from the browser to the terminal. • Copy the output on the terminal and paste it to the sheet. -
TSV (cont.) • $ seq 1 3 • $ seq
1 3 | column • $ seq 1 3 | column | hexdump - c • $ seq 1 3 | column - t -
• $ head <cmds.tsv | column - s '<C-v><TAB>' -
t • $ column - - help | grep ' -[st]' • $ head <cmds.tsv -1 # = = = $ head - n 1 cmds.tsv • $ head <cmds.tsv -1 | sed 's/(Option code)//' • sed: stream editor • $ sed <cmds.tsv - n - e 1p - e /^.cat/p • $ sed - - help | grep ' -[ne]' -
• $ cut <cmds.tsv - f 2 | head •
$ cut <cmds.tsv - f 2 | tail +2 | head • $ cut <cmds.tsv - f 2 | tail +2 | sort • $ cut <cmds.tsv - f 2 | tail +2 | sort | uniq • $ cut <cmds.tsv - f 2 | tail +2 | sort | uniq - c • $ cut <cmds.tsv - f 2 | tail +2 | sort | uniq - c | sort - n -
• $ awk <cmds.tsv -F $'\t' '{ print $2 }'
| head • $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/' | column - s $'\t' - t • $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/ { counts[$2]++ } END { for (cat in counts) print counts[cat] "\t" cat }' • $ awk <cmds.tsv -F $'\t' '$5 ~ /BSD/ { counts[$2]++ } END { for (cat in counts) print counts[cat] "\t" cat }' | sort - n -
• See also: • The GNU Awk User’s Guide •
6.3.2.2 Comparison Operators • 6.1.1.3 Regular Expression Constants • 8.1.5 Scanning All Elements of an Array -
How many mandatory text processing does POSIX have? Checkpoint: •
$ ??? <cmds.tsv ??? | wc - l ??? • Hints: • Either grep or awk is a good choice. • Try = = and & & if you choose awk. • “Text processing” • “Mandatory” -
TSV (cont.) • $ cut <cmds.tsv - f 2 |
head • $ awk <cmds.tsv '{ print $2 }' | head • $ vimdiff <(cut <cmds.tsv - f 2 | head) <(awk <cmds.tsv '{ print $2 }' | head) # `:qa` to exit. • $ awk <cmds.tsv -F $'\t' '{ print $2 }' | head -
• $ cut <cmds.tsv - f 2 | head >a
• $ awk <cmds.tsv '{ print $2 }' | head >b • $ vimdiff a b • $ diff a b • $ diff a b - y # using the side by side format. • $ diff a b - u # using the unif i ed format. • $ diff - - help | grep ' -[yu]' -
• $ sudo apt install bat - y # or,
• # apt install bat - y • $ alias bat=batcat • $ diff a b | bat # <q> to exit. -
Interact with the system clipboards • (macOS)$ pbpaste | tail
-1 | pbcopy • The way I make slides with highlighted codes: • $ pbpaste | highlight - u utf-8 -S python - s solarized - light - k Monaco -K 28 -O rtf | pbcopy # and then paste into a slide. • (X11)$ xclip - o | tail -1 | xclip • (WSL)$ powershell.exe Get-Clipboard | tail -1 | clip.exe • See also: vor0nwe/clip – Gist for a neat clip script in WSL. -
cURL • $ curl https://google.com • $ curl - v
https://google.com • $ curl -L https://google.com -
JSON with jq • $ sudo apt install jq -
y # or, • # apt install jq - y • jq is like sed for JSON. • $ cd • ~$ curl https://raw.githubusercontent.com/5etools - mirror-1/5etools - mirror-1.github.io/ master/data/bestiary/bestiary - mm.json >monsters.json # or, • ~$ curl https://raw.githubusercontent.com/hazmole/TheGiddyLimit.github.io/master/ data/bestiary/bestiary - mm.json >monsters_zh_tw.json • $ wc - l monsters.json # - > 70811 -
• $ jq <monsters.json • $ jq <monsters.json | less
• $ jq <monsters.json -C | less -R • $ grep <monsters.json dragon - - color=always | less -R • $ jq <monsters.json | bat - l json • $ bat monsters.json • $ vi monsters.json # `:q` to exit. -
• $ jq <monsters.json '.monster[] | .name' | head •
$ jq <monsters.json '.monster[] | select(.type == "dragon") | .name' | head • $ jq <monsters.json '.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]' | head -
• $ jq <monsters.json '.monster[] | select(.type == "dragon") |
[.hp.average, .name, .type, .page] | @tsv' | head • $ jq <monsters.json '.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page] | @tsv' - r | head • $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' - r | head • $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' - r >dragons.tsv -
• $ sort <dragons.tsv # Sort by HP. • $
sort <dragons.tsv - k 2 - - debug | head # Try to sort by Name. • $ sort <dragons.tsv - k 2,2 - - debug | head • $ sort <dragons.tsv - t $'\t' - k 2,2 - - debug | head -
What is the first undead monster in the book? Checkpoint:
• $ jq ??? | tail -1 ??? • Hints: • “undead” • “page” -
JSON with gron • $ sudo apt install gron -
y # or, • # apt install gron - y • $ jq <monsters.json '.monster[] | select(.type == "dragon") | .name' | head • $ jq <monsters.json '[ .monster[] | select(.type == "dragon") ]' >dragons.json • $ bat dragons.json • $ gron dragons.json • $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name =' • $ gron dragons.json | grep -E 'json\[[0-9]+\]\.name =' | gron - u -
. * [a - z] ^$ ? + | Fixed
string . * [a - z] ^$ ? + | $ grep -F BRE . * [a - z] ^$ ? + | $ grep $ grep -G ERE . * [a - z] ^$ ? + | $ grep -E Glob . * [a - z] ^$ ? + | Fixed string, BRE, ERE, vs. glob -
Test the regex types • $ echo 'i++' | grep
-F '.+' - o # f i xed string (not regex) <empty> • $ echo 'i++' | grep -G '.+' - o # basic regular expression (BRE) i + • $ echo 'i++' | grep -E '.+' - o # extended regular expression (ERE) i + + -
• See also: • Regex cheatsheet for BRE, ERE, Vim,
Python's re, PCRE (Perl). • regex101: a regex debugger for Python, JavaScript, Java, Golang, PHP. • jq Manual • tomnomnom/gron – GitHub • yq for YAML. -
Checkpoint: The regex type of less may be? • BRE-like
... ERE-like ... or not regex? • Hints: • How to put i + + into less? • How to search something in less? -
Bulk editing • $ cp dragons.tsv cats.tsv • $ cp
dragons.tsv cats_2.tsv • $ head cats * .tsv • $ sed s/Dragon/Cat/ cats * .tsv | less • $ sed - e s/Dragon/Cat/ - e s/dragon/cat/ cats * .tsv | less • $ sed - e s/Dragon/Cat/g - e s/dragon/cat/g cats * .tsv | less • $ sed - e s/Dragon/Cat/g - e s/dragon/cat/g cats * .tsv - i -
CSV with CSVKit • $ sudo apt install csvkit -
y # or, • # apt install csvkit - y • $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @tsv' - r >dragons.tsv • $ jq <monsters.json '["HP", "Name", "Type", "Page"], (.monster[] | select(.type == "dragon") | [.hp.average, .name, .type, .page]) | @csv' - r >dragons.csv • $ bat dragons.csv -
• $ csvcut <dragons.csv - c Name,HP • $ csvsort
<dragons.csv - c HP,Page • $ csvstat <dragons.csv - c HP,Name,Type • $ csvlook <dragons.csv # - > a Markdown table • $ csvgrep <dragons.csv - c Name - m 'Adult Black Dragon' • $ csvsql <dragons.csv - - query "select avg(HP) from stdin where Name like 'Adult%'" -
• See also: • Tutorial – csvkit • 1.4. in2csv:
the Excel killer • 3.3. csvsql and sql2csv: ultimate power -
Parallelizing • $ time sleep 1 • $ seq 3
| xargs - i echo sleep {} # xargs: extended arguments • $ seq 3 | xargs - i echo sleep 1 • $ time seq 3 | xargs - i sleep 1 # takes 3 seconds. • $ time seq 3 | xargs - i -P 3 sleep 1 # takes 1 second. • $ man xargs | less +'/ -[iI]' -
File management • $ cd • ~$ mkdir mydir &&
cd mydir • ~/mydir$ touch a b .h • $ mkdir - p x/y/z • $ cp b c • $ rm b • $ ls • $ ll • $ type ll • $ ll - t # List and sort by time. • $ ll -S - h # List and sort by size. -
• $ sudo apt install tree - y # or,
• # apt install tree - y • $ cd ~/mydir • ~/mydir$ ls • $ tree • $ tree - a # List all f i les. • $ tree -L 1 # List one level deep. • $ tree - f # Print full paths. -
• $ cd • ~$ ln - s mydir mydir_link
# Make a symbolic link. • $ ls mydir_link • $ ll mydir_link • $ cp - r mydir mydir_backup # Copy recursively. • $ ll mydir_backup -
• $ tar czf mydir.tar.gz mydir • $ rmdir mydir
# - > failed . . . not empty. • $ rmdir mydir/x/y/z # OK! • $ rm - r mydir/ • $ tar xzf mydir.tar.gz • $ tree • $ du - h - d 1 # du: disk usage • $ df - h # df: disk free -
Describe one difference between cp - r and cp -
a Checkpoint: • Read and search the manual! • Hints: • $ man cp # `/` to search; `n` to next; `h` to get help. • $ cp - - help | grep . . . -
Terminology EN program process thread concurrency parallelism TW 程式 程序
執⾏緒 並⾏性 平⾏性 CN 程序 进程 线程 并发性 并⾏性 -
Process management • $ ps # ps: process status •
$ ps a # a: includes other users. • $ ps ax # x: includes other ttys or no tty's. # tty: teletypewriter • $ ps axu # u: user - oriented format • $ ps f # f: "forest" • $ man ps | grep -E ' (VSZ|RSS|TTY|STAT|START|TIME) ' -
• $ vi test.txt # and <C-z>. • $ ps
u | grep vi • $ ps u | grep [v]i • $ ps u | sed - n - e 1p - e /[v]i/p • $ ps axu | grep [v]i | awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }' • (macOS)$ ps axu | grep [C]hrome | awk '{ rss += $6 } END { printf "%.4g MB\n", rss/1024 }' -
• $ vi test.txt # and <C-z>. • $ pgrep
v # - > <pid> • $ pgrep vi # - > <pid> • $ pgrep test # - > <empty> • $ pgrep - f test # - > <pid> • $ pgrep - f test - l • $ pgrep - - help | grep ' -[fl]' • $ # kill <pid> • $ pkill - f test # -SIGTERM • $ fg • $ vi test.txt # and <C-z>. • $ pkill - f test -SIGKILL • See also: • Signal (IPC) – Wikipedia -
Monitoring • $ sudo apt install iotop iftop - y
# or, # if: interface • # apt install iotop iftop - y • $ free - h • $ man free | grep -E ' (total|used|free|shared|buff/cache|available)' -A 1 • $ vi test.txt # and <C-z>. • $ top • $ htop • Select a process, and <F9> (kill) -
• $ cd • ~$ python3 - m http.server &>
my.log # <C-c> to interrupt. • Open http://localhost:8000/. • Open another terminal: • # iftop # <q> to exit. • $ tail - f ~/my.log • $ less +F ~/my.log # Try <C-c> ` - i` `/http` <F>. • $ watch tail -10 ~/my.log -
• The commands which are useful but not relevant to
containers: • $ sudo iotop • Important system logs: • $ sudo vi /var/log/syslog • $ sudo vi /var/log/auth.log -
Checkpoint: Monitor your HTTP server by less • Try to
get three new logs. • Hints: • Click on your browser! -
Monitoring (cont.) • $ python3 - c 'print(1234)' • $
strace python3 - c 'print(1234)' • $ sudo strace - p <pid> • $ ps x • $ pgrep bash • $ tty • $ pgrep bash - t pts/1 # pts: pseudo terminal slave • $ lsof - p `pgrep bash - t pts/1` # lsof: list open f i les • $ lsof /usr/bin/bash • $ man lsof | less - i +/^examples • $ htop • Select a process, and: • <s> (strace) • <l> (lsof) -
• $ sudo apt install sysstat - y # or,
• # apt install sysstat - y • $ sar 1 # Report CPU stats per second. • $ sar # Report today's CPU stats. • $ sar - f /var/log/sysstat/sa10 • $ sar - f /var/log/sysstat/sa10 - s 12 : 00 - e 15 : 00 • $ man sar | grep 'Report ' -B 1 • $ sar - f /var/log/sysstat/sa10 - s 12 : 00 - e 15 : 00 - r - - human # Report memory stats. • See also: • sysstat/sysstat – GitHub • Amazon CloudWatch • Grafana -
Container management • OCI: Open Container Initiative • Docker fi
le: part of OCI spec. • image: container image • Why? • Once you pack your app as a container image, it can be run anywhere seamlessly. -
• (host)$ docker container commit ubuntu - c ubuntu -
cli - power # = = = $ docker container commit . . . • $ docker image ls # = = = $ docker images -
• $ docker image save ubuntu - cli - power
>ubuntu - cli - power.tar • $ docker image rm ubuntu - cli - power # = = = $ docker rmi ubuntu - cli - power • $ docker image ls • $ docker image load <ubuntu - cli - power.tar • $ # docker image (save|load) = = = docker image (save|load) • $ docker image ls -
• $ docker container create - - name ls -
expert ubuntu - cli - power man ls • $ docker container ls - a # = = = $ docker ps - a • $ docker container start ls - expert • $ docker container ls - a # - > Exited (0) 4 seconds ago • $ docker container start - a ls - expert # Start and attach. • $ docker container rm ls - expert • $ # docker container (create|start|stop|rm) = = = docker container (create|start|stop|rm) -
• $ docker container cp ubuntu - c:/root/cmds.tsv ~/cmds.tsv
# = = = $ docker cp . . . • $ docker container cp ~/cmds.tsv ubuntu - c:/root/cmds_2.tsv • $ docker container run - dit - v ~/container:/root - - name volume - c ubuntu - cli - power # = = = $ docker run . . . -
Run programs after disconnect • $ tmux # tmux: terminal
multiplexer • (tmux)$ while true; do date; sleep 1; done # and then <C-b><d>. • $ tmux ls # ls: the alias of list - sessions. • $ tmux a # a: attach • (tmux) <C-b><?> # List key bindings; <q> to exit. • (tmux) <C-c><C-d> • $ tmux ls -
• The way I start the server of JupyterLab: •
$ tmux new - s jupyter - lab - d 'cd && jupyter lab - - no - browser' -
The local key pair • (local)$ ll ~/.ssh/id_rsa * •
If nothing there: • (local)$ ssh - keygen • (local)$ cat ~/.ssh/id_rsa.pub # pub: public key • id_rsa is the private key which proves who you are. • Do not share id_rsa – the private key – to anyone. • Simply generate a new pair when you have a little doubt about the old pair. -
Add a user in the remote host • (remote)# new_user=<new
username> • (remote)# useradd - m - s /bin/bash $new_user • (remote)# su $new_user • <new user>$ mkdir ~/.ssh <new user>$ cat <<EOS >~/.ssh/authorized_keys > <paste the content of id_rsa.pub> > EOS <new user>$ exit • (remote)# service ssh start -
Connect to a remote host • (local)$ ssh ssh://<user>@localhost:2222
# = = = $ ssh <user>@localhost - p 2222 # $ ssh <host> # if port is 22 and usernames are the same. • If there is a warning and you're sure you changed something in the remote: • $ sed <line no>d ~/.ssh/known_hosts - i -
ssh -A: Allow the remote to use the local key
pair • (local) $ ssh ssh://<user>@localhost:2222 • (remote)$ ssh ubuntu - c # . . . password: • (local) $ ssh ssh://<user>@localhost:2222 -A • (remote)$ ssh ubuntu - c # OK • The ubuntu-c works since the docker run --network enables the DNS resolution between containers. • See also: Consul, for the similar functionality without Docker. -
ssh -J: Jump to a host in a private network
• (local)$ ssh ubuntu - c # - > Could not resolve hostname ubuntu - c • (local)$ ssh -J ssh://<user>@localhost:2222 ubuntu - c • -J is securer than -A; use -J when applicable. -
Transfer files between local and remote • (remote)$ cp cmds.tsv
~<user> • (local) $ rsync - e 'ssh ssh://<user>@localhost:2222' :~/cmds.tsv ~ # $ rsync <user>@<host > : ~/cmds.tsv ~ # if port is 22. • (local) $ rsync - e 'ssh ssh://<user>@localhost:2222' ~/cmds.tsv :~/cmds_2.tsv • Deprecating scp – LWN.net, but scp is still a common way nowadays: • (local)$ scp scp://<user>@localhost:2222/~/cmds.tsv ~ # $ scp <user>@<host > : ~/cmds.tsv ~ # if port is 22. • (local)$ scp ~/cmds.tsv scp://<user>@localhost:2222/~/cmds_2.tsv -
Mount a remote directory • (local)$ sudo apt install sshfs
# or macFUSE for macOS. • (local)$ mkdir ~/sshfs_container • (local)$ sshfs <user>@localhost:/home/<user> - p 2222 ~/sshfs_container • (local)$ ls ~/sshfs_container • (local)$ umount ~/sshfs_container -
ssh -L: Forward a local port to a remote port
• As known as “SSH tunneling”. • (remote)$ python3 - m http.server 1234 # <C-c> to interrupt. • (local)$ ssh ssh://<user>@localhost:2222 -L 4321:ubuntu - c:1234 -Nf • Open http://localhost:4321/. • localhost:4321 → localhost:2222 → ubuntu-c:22 → ubuntu-c:1234 → http.server 1234 • (local)$ pgrep - f 4321 • (local)$ # pkill - f 4321 -
Access the HTTP server by http://localhost:8765/ Checkpoint: • Open http://localhost:8765/
and there should be a page. • Hints: • Use ssh -L. -
ssh -D: Start a proxy server • (local)$ ssh ssh://<user>@localhost:2222
-D 1080 -Nf • (remote)# iftop • Con fi gure the system proxy or Proxy SwitchyOmega for Chrome. • (remote)# python3 - m http.server 80 • Open http://ubuntu-c/. • For accessing the services in a private network by private DNS and standard ports! • If the server has another public IP, try Google “my ip”. • $ man ssh | grep ' -[LDNf]' -A 1 -
• See also: • A simple ~/.ssh/con fi g for
macOS • A simple Bash con fi g for both macOS and Linux -
Edit a config • $ cp ~/.bashrc ~/backup_bashrc • $
echo 'alias bat=batcat' >>~/.bashrc • Re-enter the shell to take e ff ect. • $ grep batcat ~/.bashrc - n • $ sed <line no>d ~/.bashrc - i • $ grep batcat ~/.bashrc - n -
• $ vi ~/.bashrc • (vim) <C-End> # <Fn-Ctrl-Right> •
(vim) i<Right><CR> # or `a<CR>` • (vim) alias bat=batcat # literally • (vim) <Esc> # backs to Normal • (vim) :x<CR> • $ grep batcat ~/.bashrc - n • $ vi ~/.bashrc • (vim) /cat<CR> • (vim) n . . . • (vim) dd • (vim) :x<CR> • $ grep batcat ~/.bashrc - n -
The mini-cheatsheet of Vim 1. / n • Search; Next.
2. i <Esc> :x • Go Insert; Back to Normal; Write and quit. 3. dd • Delete a line. • It's all you need to know now. Of course, the more you know, the more e ff i ciency you'll get. 💪 -
Spell checking • $ echo 'Oops. Does it spll like
this?'<C-x><C-e> • (vim) :set spell<CR> • (vim) /sp<CR> • (vim) z= # List spelling suggestions. • (vim) 1<CR> • (vim) :x<CR> -
• See also: • $ vimtutor # Learn Vim in
30 mins! • (vim) :help dd, for example. • Vim Keystrokes – Mosky’s Notes • Vim’s Search Tips – Mosky’s Notes • Vim’s Replace Tips – Mosky’s Notes • Mosky's Vim Con fi gs -
Checkpoint: Add your bat • (host)$ docker exec - it
ubuntu - c bash • ubuntu - c# bat ~/.bashrc # It should work immediately. • Hints: • You may use echo to add. • Use grep to con fi rm. • Re-enter the shell to take e ff ect. -
Developing Slak: Collect data from Slack like a pro. ⚡
1. System Tools: the common tools which are supposed to be installed in the system. • git: the version control system. • pip: the Python package manager. • pipenv: one of Python package environment managers. • The Docker (containers) provides OS-level isolation; Pipenv provides Python interpreter and package–level isolation. 2. Project Tools: the tools of this project, good to be managed within the project by, for example, pipenv. • ipdb: a simple yet beautiful debugger. • pytest: the testing framework. 3. Personal Tools: the tools of a developer, may be installed in one's system. • flake8: the static analyzer (linter). • mypy: the static type checker. • black: one of the formatters. -
• $ sudo apt install pip - y # or,
# System Tool • # apt install pip - y • $ pip install pipenv # System Tool • $ cd • ~$ git clone - - depth 1 https://github.com/moskytw/slak.git -
• ~$ cd slak • ~/slak$ bat Pipf i le
• $ pipenv sync - - dev • $ slak • $ python slak.py • $ for cmd in python python3 slak; do type $cmd; done • $ pipenv shell • (slak)$ slak • $ python slak.py • $ for cmd in python python3 slak; do type $cmd; done -
• (slak)$ python - m ipdb slak.py # Project Tool,
`q` to exit. • $ pytest # Project Tool • $ pytest - - cov • $ pytest - - cov - - cov - report html • $ cd htmlcov • ~/slak/htmlcov$ python - m http.server # <C-c> to interrupt. • Open http://localhost:8000/. • $ exit # or <C-d> to exit the shell of pipenv. -
• (my mac)$ for cmd in flake8 mypy black xxx;
do type $cmd; done flake8 is hashed (/usr/local/bin/flake8) mypy is aliased to `\mypy - - ignore - missing - imports' black is aliased to `black - - line - length 79 - - skip - string - normalization' - bash: type: xxx: not found • $ for cmd in flake8 mypy black xxx; do type $cmd >/dev/null; done - bash: type: xxx: not found -
• $ cd ~/slak • ~/slak$ flake8 slak.py • $
mypy slak.py • $ black slak.py -
• The di ff erent languages have di ff erent
toolsets; however, the concepts and the categories are similar, e.g.: • $ vi /etc/nginx/sites - enabled/your - site.conf • $ nginx - tc /etc/nginx/sites - enabled/your - site.conf # Test the conf i g f i le. -
• See also: • git - the simple guide: “no
deep shit ;)” • Git Documentation • Cheat Sheets • Pro Git, the book • Reference Manual • Mosky’s Pipenv Notes -
In a large project • It's all about locate the
path and line number: • f i nd/fd: locate by path keywords. • grep/rg: locate by content keywords. • ctags: locate by pre-built “tags”. • Language Server: a server keeps analyzing your code in background. -
• $ cd /usr/lib/python3.10 • /usr/lib/python3.10$ f i nd -
name ' * server * ' • $ sudo apt install fd - f i nd - y # or, • # apt install fd - f i nd - y • $ alias fd=fdf i nd • $ fd server # fd: f i nd -
• $ cd /usr/lib/python3.10 • /usr/lib/python3.10$ grep - r 'Serving
HTTP' - n • $ sudo apt install ripgrep - y # or, • # apt install ripgrep - y • $ rg 'Serving HTTP' # rg: ripgrep -
• $ f i nd - name 'server.py' - exec
grep 'Serving HTTP' - n {} + • $ fd '^server.py$' -X rg 'Serving HTTP' {} • $ man f i nd | less +'/ - exec ' • $ man fd | less +'/ -[Xx]' -
• Ctags & Language Server: • $ sudo apt install
universal - ctags $ ctags -R $ vim +'help tags | only' • A 10 MB–level solution. • neoclide/coc.nvim – GitHub for the Language Server solution. • A GB-level solution. -
The end • But hope it's the start of your
journey of getting the CLI power. • Having fun is still the key, as always. • May the man guide you! -
Image Credits • “The Power” https://www.facebook.com/nixcraft/photos/ meme-based-upon-iamnotanartist_-comic- for-linuxunix-users/3443131449033542/ • “The
Terminal” https://en.wikipedia.org/wiki/ Computer_terminal#/media/ File:DEC_VT100_terminal_transparent.png • “What Is a Container?” https://www.docker.com/resources/what- container/ • “The Cat” https://www.facebook.com/nixcraft/posts/ linux-explained-for-new-users-including-cat- command-explained-httpswwwcybercitib/ 3034767979869893/ • “The Dragon” https://unsplash.com/photos/F6920BvzrZE • “The Miniatures” https://unsplash.com/photos/Q4Honp3Pyqs • “The Banshee Appears” https://en.wikipedia.org/wiki/Banshee#/ media/File:The_Banshee_Appears_(1862).jpg • “The Earth” https://unsplash.com/photos/Q1p7bh3SHj8