bsd | Hugo Ferreira https://hugo.ferreira.cc/tags/bsd/ Recent content in Bsd by Hugo Ferreira bsd | Hugo Ferreira https://hugo.ferreira.cc/hf-bw.jpg https://hugo.ferreira.cc/tags/bsd/ Hugo -- 0.124.1 en 2000–2024 by Hugo Ferreira · CC BY 4.0 Thu, 28 Feb 2019 01:04:32 +0000 ✂️ Unix https://hugo.ferreira.cc/unix/ Thu, 28 Feb 2019 01:04:32 +0000 https://hugo.ferreira.cc/unix/ Small information nuggets and recipies about Unix
  • bash cheatsheet
  • zsh cheatsheet

    • (most recent on top)

    Quick curl recipes

    Post json data

    # `-d` implies `-X POST`
curl -H 'Content-Type: application/json' -d '{"key": "value"}' http://example.com

    curl useful options

    # Silence the progress bar
curl -s http://example.com

# Show response headers
curl -i http://example.com

# Follow redirects
curl -L http://example.com

    Check and make sure a script is sourced

    # Exit if this script was not sourced as an import
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    >&2 echo "Error: Must run as an import source"    # stderr
    >&2 echo "Try: source \"${BASH_SOURCE[0]}\""      # stderr
    exit
fi

    Run two commands in parallel and wait for their result

    command1 &
command2 &
wait %1 && wait %2
echo $?

    Read secret into a variable

    read -s TOKEN                     # silent typing
read -s -p "Token?" TOKEN         # … with prompt

    Increment a counter / calculate expressions

    … count from 0

    let count=$count+1                # count = 1, 2, 3, ...
((count=count+1))                 # count = 1, 2, 3, ...
value=$((count-1))                # value = 0, 1, 2, ...

    … count from N = 10

    let count=${count:-10}+1          # count = 11, 12, 13, ...

    Group and count number of multiple occurrences of the same line

    sort file | uniq -c | sort -r
#   3 foo
#   2 bar
#   1 baz

    Remove path and extension from a filename

    full_path=/some/example/directories/some.example.name.txt
echo ${full_path##*/}              # => some.example.name.txt
echo ${full_path%.*}               # => /some/example/directories/some.example.name

    Default missing variable to a value

    set -u                            # treat unset variables as errors
echo ${1:-}                       # => empty
echo ${1:-0}                      # => 0
echo ${1:-X}                      # => X
echo ${1:-$var}                   # => example

    Use multiline heredoc, herestring, and var as input

    … same output

    content
string

    … using heredoc

    cat <<WORD
content
string
WORD

    … using herestring

    cat <<< "content
string"

    … using var in herestring

    text="\
content
string"
cat <<< "$text"

    View zip or jar files

    view some.jar

    unzip -c some.jar path/to/file.txt

    Transforming and parsing JSON

    jq useful options

    -c    --compact-output
-r    --raw-output
-S    --sort-keys

    Feed json into jq

    json='{"example":"value"}'

    … from a variable

    echo $json | jq .                 # using pipe
jq . <<< "$json"                  # using herestring

    … from a file

    echo "$json" > /tmp/example.json
jq . < /tmp/example.json

    Convert to new array with a subset of keys

    json='[
  {"name":"a", "id":1, "extra":"foo"}, 
  {"name":"b", "id":2, "extra":"bar"}
]'

    jq -c '[.[] | { id, name }]' <<< "$json"
# [{"id":1,"name":"a"},{"id":2,"name":"b"}]

    Select array elements

    json='[{"key":"a"}, {"key":"b"}, {"key":"c"}]'

    echo $json | jq -c '.[0]'         # => { "key": "a" }
echo $json | jq -c 'first'        # => { "key": "a" }
echo $json | jq -c 'nth(1)'       # => { "key": "b" }
echo $json | jq -c 'last'         # => { "key": "c" }

    Array elements from key-value pairs

    json='[{"key":"a", "value":1}, {"key":"b", "value":2}]'

    jq -c 'from_entries' <<< "$json"  # => { "a": 1, "b": 2 }

    Array elements from custom key-value strings

    json='[
  {"name":"a", "id":1, "extra":"foo"},
  {"name":"b", "id":2, "extra":"bar"}
]'

    jq -c '.[] | [{ "key": .name, "value": .id }] | from_entries' <<< "$json"
# { "a": 1, "b": 2 }

    Default null to empty string

    json='{"example":"value"}'

    jq -r '.path.to.attr' <<< "$json"           # => null
jq -r '.path.to.attr // empty' <<< "$json"  # =>

    Convert json to CSV

    json='[
  {"name":"a", "id":1, "extra":"foo"}, 
  {"name":"b", "id":2, "extra":"bar"}
]'

    jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' <<< "$json"
# => "extra","id","name"
# => "foo",1,"a"
# => "bar",2,"b"

    Special tips about sed

    sed separators are configurable

    … useful to avoid escaping /

    sed 's_http://_www._' file.txt
sed 's#http://#www.#' file.txt
sed 's|http://|www.|' file.txt
# http://example.com => www.example.com

    Refer to the matched search in sed

    sed 's/unix/& or linux/' file.txt
# unix os => unix or linux os

    Bouncing systemd services

    cd /etc/systemd/system            # => servname.service
sudo systemctl stop servname      # stop the service
sudo systemctl daemon-reload      # reload changes from disk
sudo systemctl start servname     # start the service
systemctl status ser*             # what’s going on?

    Using color escape sequences with echo

    echo $'\e[1;34m'"text"$'\e[0m'

    … using a variable instead of the escape sequence does no work!

    FOO='\e[1;34m'
echo $FOO"text"$'\e[0m'           # => \e[1;34mtext

    Bash noop command (colon)

    if [[ $a == $b ]]; then
    :                             # noop
fi

    List wildcard directories without showing its contents

    ls -d pattern*
# foo       bar       baz

    ls -1d pattern*
# foo
# bar
# baz

    Continue a command in the next line

    commannd1 -arg1 -arg2 \
          -arg3 -arg4 \
          -arg5 -arg6

    Read lines from a file, one at a time

    … excludes the last line if it doesn’t end with a newline ⚠️

    while read -r line; do
    echo $line
done < $filename

    … does not stop to enter data manually if the loop is interactive (i.e. reads from stdin) ⚠️

    while IFS='' read -r line || [[ -n "$line" ]]; do
    echo $line
    read -r input                 # grabs line from file
    echo $input
done < $filename

    Where a script is located and from where was it called

    readonly BASEDIR=$(cd "$(dirname "$0")" && pwd) # where the script is located
readonly CALLDIR=$(pwd)                         # where it was called from

    Check for environment variables

    … make sure var is defined

    [[ $ISDEFINED ]]; echo $?         # => 0 (success)
[[ $UNDEFINED ]]; echo $?         # => 1 (error)

    [[ $ISDEFINED ]] && echo "yes" || echo "not"    # => yes
[[ $UNDEFINED ]] && echo "yes" || echo "not"    # => not

    if [[ $ISDEFINED ]]; then echo "yes"; else echo "not"; fi    # => yes
if [[ $UNDEFINED ]]; then echo "yes"; else echo "not"; fi    # => not

    … make sure var is not defined

    [[ -z $ISDEFINED ]]; echo $?      # => 1 (error)
[[ -z $UNDEFINED ]]; echo $?      # => 0 (success)

    [[ -z $ISDEFINED ]] && echo "not" || echo "yes"    # => yes
[[ -z $UNDEFINED ]] && echo "not" || echo "yes"    # => not

    if [[ -z $ISDEFINED ]]; then echo "not"; else echo "yes"; fi  # => yes
if [[ -z $UNDEFINED ]]; then echo "not"; else echo "yes"; fi  # => not

    Recursively grep files with specific patterns

    … filter by file extensions

    grep -ri "pattern" . --include "*.txt" --include "*.text"
grep -ri "pattern" . --include "*.md" --exclude "README.*"

    … filter by directories

    grep "pattern" filemask* --exclude-dir "test" --exclude-dir "target"

    Remove an environment variable

    unset ENV_VAR

    Pass boolean values as parameters to bash functions

    bool=true
if [ "$bool" = true ]; then echo "yes"; fi            # => yes
if [[ "$bool" == true ]]; then echo "yes"; fi         # => yes

    Return values from bash functions

    readonly SUCCESS=0                # exit status of bash commands

    … capture the return value of a function

    get_value() {
  echo "$variable"
}
value=$(get_value)

    … check if the last command executed with success

    is_equal() {
  [[ "$1" == "$2" ]]
}

is_equal && echo ok || echo not ok          # => ok
is_equal a b && echo ok || echo not ok      # => not ok

    is_valid() {
  grep -q "$looking_for" "$file"
}

is_valid && echo found || echo not found
# => not found [shows stderr msg]

if [[ is_valid == 0 ]]; then
  echo found
else
  echo not found
fi
# => not found [no stderr msg]

    Dereference bash variable param substitution/expansion

    TEST=foobar
var="TEST"

echo "${!var}"                    # => foobar

    Check if a directory is empty

    [[ $(find /some/dir/ -maxdepth 0 -type d -empty 2> /dev/null) ]] \
  && echo "Empty" \
  || echo "Not empty or missing"

    [[ "$(ls -A /some/dir/ 2> /dev/null)" ]] \
  && echo "Not empty" \
  || echo "Empty or missing"

    Look for empty directories

    find . -type d -empty

    Exclude hidden dot files and dirs when searching

    find . -type file ! -name ".*"
find . -type file ! \( -name ".*" -or -path "*/.sync/*" \)

    find . -type file -not -name ".*"
find . -type file -not -name ".*" -not -path "*/.sync/*"
find . -type file -not \( -name ".*" -or -path "*/.sync/*" \)

    Show function definition

    declare -f functionname

    Check open files

    lsof | grep file

    Check open ports

    sudo lsof -i -P | grep :8080

    Monitor network traffic

    sudo tcpdump -ien1 -q
sudo lsof -i:8080

    vi survival commands

    vi exit

    … exit and save

    ⇧-Z-Z
:wq
:x

    … exit without saving

    ⇧-Z-Q
:q!

    vi settings

    … in ~/.vimrc

    :set backupdir=~/tmp              # backup files
:set background=dark              # use lighter colors
:set ff=unix|dos                  # unix or dos file format

    Remove trailing newlines

    echo -n "$(cat file)" > file

    Convert date to/form epoch

    … from epoch

    date -r 1234567890
# => Fri Feb 13 23:31:30 WET 2009

date -r 1294897379 +"%Y-%m-%d %H:%M:%S"
# => 2011-01-13 05:42:59

    … to epoch

    date -jf "%Y-%m-%d %H:%M:%S" "2011-01-13 05:42:59" +%s
# => 1294897379

    Difference between 2 dates

    ruby -r 'date' -e 'puts "#{(Date.today - Date.new(2007,11,30)).to_i} days"'

    python -c "import datetime as d; print((d.date.today() - d.date(2007,11,30)).days)"

    Subtract days to a date

    expr -1 \* 24 \* 60 \* 60 + `date +%s` | xargs date -r
expr -1 \* 24 \* 60 \* 60 + `date +%s` | xargs -I{} date -r {} +"%Y-%m-%d"

    ruby -r 'date' -le 'puts "#{Date.today - 1}"'
ruby -r 'date' -le 'puts "#{Date.today << 1}"'

    python -c "import datetime as d; print(d.date.today() - d.timedelta(days=1))"

    Current week number

    man strftime                      # list of formats

    date +%V

    ruby -e 'puts (Time.now).strftime("%V")'

    python -c "import time; print(time.strftime('%W'))"
python -c "import datetime as d; print(d.date(2009,9,1).strftime('%W'))"

    Files paches

    … generate

    diff -uraN olddir newdir > patch.diff

    … apply

    patch -p1 < patch.diff

    Dos ➔ Unix ➔ Dos

    # ^M = ctrl-V ctrl-M

    … unix ➔ dos

     sed -e 's/$/^M/' unix.file > dos.file

    … dos ➔ unix

    sed -e 's/^M//' dos.file > unix.file

    tr -d \\r < dos.file > unix.newfile

    Test a POP3 server

    $ telnet myserver 110
user myusername
pass mypassword
stat | list | retr <msgnbr> | dele <msgnbr>
quit

    Manage postfix mail queues

    sudo tail -f /var/log/postfix.log # check what postfix is doing
mailq                             # see what is in the queue (shows id)
postsuper -d id                   # delete a mail from the queue
postsuper -d ALL                  # delete all mail from the queue

    Unix distribution name/version

    … depending on the distro

    dmesg | head -1
cat /proc/version
cat /etc/issue                    # human-readable
uname -a                          # full summary

    Shell redirections

    Redirect stdout

    cmd > /dev/null                   # stdout to file
cmd >&2                           # stdout to stderr

    Redirect stderr

    cmd 2> /dev/null                  # stderr to file
cmd 2>&1                          # stderr to stdout

    Redirect stdout and stderr

    cmd &> /dev/null                  # both to file
cmd > /dev/null 2>&1              # both to file
cmd 2>&1 | cmd                    # both to pipe
cmd 2>&1 | tee file.txt           # both to console and file

    Redirect stdout and stderr (bash 4)

    cmd &>> /dev/null                 # both to file
cmd |& cmd                        # both to pipe

    Common lines between two files

    comm -12 $file1 $file2            # assumes files are sorted

    grep -Fxf $file1 $file2

    perl -ne 'print if ($seen{$_} .= @ARGV) =~ /10$/' $file1 $file2

    Compile .po message files into .mo data files

    msgfmt -o filename.mo filename.po

    Create a fake blank file of any size

    dd if=/dev/zero of=myfile bs=1024 count=12
# 12+0 records in
# 12+0 records out
# 12288 bytes transferred in 0.000225 secs (54613333 bytes/sec)

    ls -lh myfile 
# -rw-r--r--  1 hfr7756  staff    12K Sep 17 13:56 myfile

    du -h myfile  
# 12K myfile

    du --si myfile
# 12k myfile

    Split very long xml lines in tags

    … brute force format by inserting newlines

    # it's one single command line!
sed -e 's/\>\</\>\
\</g' input.xml > output.xml

    Port forwarding through remote host

    ssh server -L 4321:servername:80

    Encode string using openssl

    … from string

    echo -n "qwerty" | openssl base64 # => cXdlcnR5
echo -n "qwerty" | openssl md5    # => d8578edf8458ce06fbc5bb76a58c5ca4
echo -n "qwerty" | openssl sha1   # => b1b3773a05c0ed0176787a4f1574ff0075f7521e

    … from input

    openssl base64
^D

    openssl md5
^D

    openssl sha1
^D
    ]]>