✂️ Unix

February 28, 2019 · 11 min · 2180 words | source

Small information nuggets and recipies about Unix

Table of Contents

(most recent on top)

Quick curl recipes

Post json data

# `-d` implies `-X POST`
curl -H 'Content-Type: application/json' -d '{"key": "value"}' http://example.com

curl useful options

# Silence the progress bar
curl -s http://example.com

# Show response headers
curl -i http://example.com

# Follow redirects
curl -L http://example.com

Check and make sure a script is sourced

# Exit if this script was not sourced as an import
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    >&2 echo "Error: Must run as an import source"    # stderr
    >&2 echo "Try: source \"${BASH_SOURCE[0]}\""      # stderr
    exit
fi

Run two commands in parallel and wait for their result

command1 &
command2 &
wait %1 && wait %2
echo $?

Read secret into a variable

read -s TOKEN                     # silent typing
read -s -p "Token?" TOKEN         # … with prompt

Increment a counter / calculate expressions

… count from 0

let count=$count+1                # count = 1, 2, 3, ...
((count=count+1))                 # count = 1, 2, 3, ...
value=$((count-1))                # value = 0, 1, 2, ...

… count from N = 10

let count=${count:-10}+1          # count = 11, 12, 13, ...

Group and count number of multiple occurrences of the same line

sort file | uniq -c | sort -r
#   3 foo
#   2 bar
#   1 baz

Remove extension from a filename

filename=some.example.name.txt
echo ${filename%.*}               # => some.example.name

Default missing variable to a value

set -u                            # treat unset variables as errors
echo ${1:-}                       # => empty
echo ${1:-0}                      # => 0
echo ${1:-X}                      # => X
echo ${1:-$var}                   # => example

Use multiline heredoc, herestring, and var as input

… same output

content
string

… using heredoc

cat <<WORD
content
string
WORD

… using herestring

cat <<< "content
string"

… using var in herestring

text="\
content
string"
cat <<< "$text"

View zip or jar files

view some.jar

unzip -c some.jar path/to/file.txt

Transforming and parsing JSON

jq useful options

-c    --compact-output
-r    --raw-output
-S    --sort-keys

Feed json into jq

json='{"example":"value"}'

… from a variable

echo $json | jq .                 # using pipe
jq . <<< "$json"                  # using herestring

… from a file

echo "$json" > /tmp/example.json
jq . < /tmp/example.json

Convert to new array with a subset of keys

json='[
  {"name":"a", "id":1, "extra":"foo"}, 
  {"name":"b", "id":2, "extra":"bar"}
]'

jq -c '[.[] | { id, name }]' <<< "$json"
# [{"id":1,"name":"a"},{"id":2,"name":"b"}]

Select array elements

json='[{"key":"a"}, {"key":"b"}, {"key":"c"}]'

echo $json | jq -c '.[0]'         # => { "key": "a" }
echo $json | jq -c 'first'        # => { "key": "a" }
echo $json | jq -c 'nth(1)'       # => { "key": "b" }
echo $json | jq -c 'last'         # => { "key": "c" }

Array elements from key-value pairs

json='[{"key":"a", "value":1}, {"key":"b", "value":2}]'

jq -c 'from_entries' <<< "$json"  # => { "a": 1, "b": 2 }

Array elements from custom key-value strings

json='[
  {"name":"a", "id":1, "extra":"foo"},
  {"name":"b", "id":2, "extra":"bar"}
]'

jq -c '.[] | [{ "key": .name, "value": .id }] | from_entries' <<< "$json"
# { "a": 1, "b": 2 }

Default null to empty string

json='{"example":"value"}'

jq -r '.path.to.attr' <<< "$json"           # => null
jq -r '.path.to.attr // empty' <<< "$json"  # =>

Convert json to CSV

json='[
  {"name":"a", "id":1, "extra":"foo"}, 
  {"name":"b", "id":2, "extra":"bar"}
]'

jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' <<< "$json"
# => "extra","id","name"
# => "foo",1,"a"
# => "bar",2,"b"

Special tips about sed

sed separators are configurable

… useful to avoid escaping /

sed 's_http://_www._' file.txt
sed 's#http://#www.#' file.txt
sed 's|http://|www.|' file.txt
# http://example.com => www.example.com

Refer to the matched search in sed

sed 's/unix/& or linux/' file.txt
# unix os => unix or linux os

Bouncing systemd services

cd /etc/systemd/system            # => servname.service
sudo systemctl stop servname      # stop the service
sudo systemctl daemon-reload      # reload changes from disk
sudo systemctl start servname     # start the service
systemctl status ser*             # what’s going on?

Using color escape sequences with echo

echo $'\e[1;34m'"text"$'\e[0m'

… using a variable instead of the escape sequence does no work!

FOO='\e[1;34m'
echo $FOO"text"$'\e[0m'           # => \e[1;34mtext

Bash noop command (colon)

if [[ $a == $b ]]; then
    :                             # noop
fi

List wildcard directories without showing its contents

ls -d pattern*
# foo       bar       baz

ls -1d pattern*
# foo
# bar
# baz

Continue a command in the next line

commannd1 -arg1 -arg2 \
          -arg3 -arg4 \
          -arg5 -arg6

Read lines from a file, one at a time

… excludes the last line if it doesn’t end with a newline ⚠️

while read -r line; do
    echo $line
done < $filename

… does not stop to enter data manually if the loop is interactive (i.e. reads from stdin) ⚠️

while IFS='' read -r line || [[ -n "$line" ]]; do
    echo $line
    read -r input                 # grabs line from file
    echo $input
done < $filename

Where a script is located and from where was it called

readonly BASEDIR=$(cd "$(dirname "$0")" && pwd) # where the script is located
readonly CALLDIR=$(pwd)                         # where it was called from

Check for environment variables

… make sure var is defined

[[ $ISDEFINED ]]; echo $?         # => 0 (success)
[[ $UNDEFINED ]]; echo $?         # => 1 (error)

[[ $ISDEFINED ]] && echo "yes" || echo "not"    # => yes
[[ $UNDEFINED ]] && echo "yes" || echo "not"    # => not

if [[ $ISDEFINED ]]; then echo "yes"; else echo "not"; fi    # => yes
if [[ $UNDEFINED ]]; then echo "yes"; else echo "not"; fi    # => not

… make sure var is not defined

[[ -z $ISDEFINED ]]; echo $?      # => 1 (error)
[[ -z $UNDEFINED ]]; echo $?      # => 0 (success)

[[ -z $ISDEFINED ]] && echo "not" || echo "yes"    # => yes
[[ -z $UNDEFINED ]] && echo "not" || echo "yes"    # => not

if [[ -z $ISDEFINED ]]; then echo "not"; else echo "yes"; fi  # => yes
if [[ -z $UNDEFINED ]]; then echo "not"; else echo "yes"; fi  # => not

Recursively grep files with specific patterns

… filter by file extensions

grep -ri "pattern" . --include "*.txt" --include "*.text"
grep -ri "pattern" . --include "*.md" --exclude "README.*"

… filter by directories

grep "pattern" filemask* --exclude-dir "test" --exclude-dir "target"

Remove an environment variable

unset ENV_VAR

Pass boolean values as parameters to bash functions

bool=true
if [ "$bool" = true ]; then echo "yes"; fi            # => yes
if [[ "$bool" == true ]]; then echo "yes"; fi         # => yes

Return values from bash functions

readonly SUCCESS=0                # exit status of bash commands

… capture the return value of a function

get_value() {
  echo "$variable"
}
value=$(get_value)

… check if the last command executed with success

is_equal() {
  [[ "$1" == "$2" ]]
}

is_equal && echo ok || echo not ok          # => ok
is_equal a b && echo ok || echo not ok      # => not ok

is_valid() {
  grep -q "$looking_for" "$file"
}

is_valid && echo found || echo not found
# => not found [shows stderr msg]

if [[ is_valid == 0 ]]; then
  echo found
else
  echo not found
fi
# => not found [no stderr msg]

Dereference bash variable param substitution/expansion

TEST=foobar
var="TEST"

echo "${!var}"                    # => foobar

Check if a directory is empty

[[ $(find /some/dir/ -maxdepth 0 -type d -empty 2> /dev/null) ]] \
  && echo "Empty" \
  || echo "Not empty or missing"

[[ "$(ls -A /some/dir/ 2> /dev/null)" ]] \
  && echo "Not empty" \
  || echo "Empty or missing"

Look for empty directories

find . -type d -empty

Exclude hidden dot files and dirs when searching

find . -type file ! -name ".*"
find . -type file ! \( -name ".*" -or -path "*/.sync/*" \)

find . -type file -not -name ".*"
find . -type file -not -name ".*" -not -path "*/.sync/*"
find . -type file -not \( -name ".*" -or -path "*/.sync/*" \)

Show function definition

declare -f functionname

Check open files

lsof | grep file

Check open ports

sudo lsof -i -P | grep :8080

Monitor network traffic

sudo tcpdump -ien1 -q
sudo lsof -i:8080

vi survival commands

vi exit

… exit and save

⇧-Z-Z
:wq
:x

… exit without saving

⇧-Z-Q
:q!

vi settings

… in ~/.vimrc

:set backupdir=~/tmp              # backup files
:set background=dark              # use lighter colors
:set ff=unix|dos                  # unix or dos file format

Remove trailing newlines

echo -n "$(cat file)" > file

Convert date to/form epoch

… from epoch

date -r 1234567890
# => Fri Feb 13 23:31:30 WET 2009

date -r 1294897379 +"%Y-%m-%d %H:%M:%S"
# => 2011-01-13 05:42:59

… to epoch

date -jf "%Y-%m-%d %H:%M:%S" "2011-01-13 05:42:59" +%s
# => 1294897379

Difference between 2 dates

ruby -r 'date' -e 'puts "#{(Date.today - Date.new(2007,11,30)).to_i} days"'

python -c "import datetime as d; print((d.date.today() - d.date(2007,11,30)).days)"

Subtract days to a date

expr -1 \* 24 \* 60 \* 60 + `date +%s` | xargs date -r
expr -1 \* 24 \* 60 \* 60 + `date +%s` | xargs -I{} date -r {} +"%Y-%m-%d"

ruby -r 'date' -le 'puts "#{Date.today - 1}"'
ruby -r 'date' -le 'puts "#{Date.today << 1}"'

python -c "import datetime as d; print(d.date.today() - d.timedelta(days=1))"

Current week number

man strftime                      # list of formats

date +%V

ruby -e 'puts (Time.now).strftime("%V")'

python -c "import time; print(time.strftime('%W'))"
python -c "import datetime as d; print(d.date(2009,9,1).strftime('%W'))"

Files paches

… generate

diff -uraN olddir newdir > patch.diff

… apply

patch -p1 < patch.diff

Dos ➔ Unix ➔ Dos

# ^M = ctrl-V ctrl-M

… unix ➔ dos

 sed -e 's/$/^M/' unix.file > dos.file

… dos ➔ unix

sed -e 's/^M//' dos.file > unix.file

tr -d \\r < dos.file > unix.newfile

Test a POP3 server

$ telnet myserver 110
user myusername
pass mypassword
stat | list | retr <msgnbr> | dele <msgnbr>
quit

Manage postfix mail queues

sudo tail -f /var/log/postfix.log # check what postfix is doing
mailq                             # see what is in the queue (shows id)
postsuper -d id                   # delete a mail from the queue
postsuper -d ALL                  # delete all mail from the queue

Unix distribution name/version

… depending on the distro

dmesg | head -1
cat /proc/version
cat /etc/issue                    # human-readable
uname -a                          # full summary

Shell redirections

Redirect stdout

cmd > /dev/null                   # stdout to file
cmd >&2                           # stdout to stderr

Redirect stderr

cmd 2> /dev/null                  # stderr to file
cmd 2>&1                          # stderr to stdout

Redirect stdout and stderr

cmd &> /dev/null                  # both to file
cmd > /dev/null 2>&1              # both to file
cmd 2>&1 | cmd                    # both to pipe
cmd 2>&1 | tee file.txt           # both to console and file

Redirect stdout and stderr (bash 4)

cmd &>> /dev/null                 # both to file
cmd |& cmd                        # both to pipe

Common lines between two files

comm -12 $file1 $file2            # assumes files are sorted

grep -Fxf $file1 $file2

perl -ne 'print if ($seen{$_} .= @ARGV) =~ /10$/' $file1 $file2

Compile .po message files into .mo data files

msgfmt -o filename.mo filename.po

Create a fake blank file of any size

dd if=/dev/zero of=myfile bs=1024 count=12
# 12+0 records in
# 12+0 records out
# 12288 bytes transferred in 0.000225 secs (54613333 bytes/sec)

ls -lh myfile 
# -rw-r--r--  1 hfr7756  staff    12K Sep 17 13:56 myfile

du -h myfile  
# 12K myfile

du --si myfile
# 12k myfile

Split very long xml lines in tags

… brute force format by inserting newlines

# it's one single command line!
sed -e 's/\>\</\>\
\</g' input.xml > output.xml

Port forwarding through remote host

ssh server -L 4321:servername:80

Encode string using openssl

… from string

echo -n "qwerty" | openssl base64 # => cXdlcnR5
echo -n "qwerty" | openssl md5    # => d8578edf8458ce06fbc5bb76a58c5ca4
echo -n "qwerty" | openssl sha1   # => b1b3773a05c0ed0176787a4f1574ff0075f7521e

… from input

openssl base64
^D

openssl md5
^D

openssl sha1
^D