Linux CLI for development

A list of useful CLI tools and some useful usage examples

Read

Read variable from CLI

read -p "Username: " -s username
echo $username

Read password into variable

read -p "Password: " -s password
echo $password

Grep

Search for a file in the current directory

find . | grep 'some-file'

Search in contents of all files recursively

grep -r 'some-text'

General grep stuff

Grep can be used in various ways

echo 'pipe some text into it' | grep 'text'
grep 'look-for-this-line' /inside/a/file
grep -r 'look-for-this-in-all-files-and-subfolders'

Make the search case insensitive

echo 'TeSt' | grep -i 'test'

Remember grep uses regular expressions, so we can for example do stuff like this to look at every line excluding the ones starting with #

grep '^[^#]' /etc/ssh/sshd_config

To learn more about regular expressions try to copy the pattern ^[^#] into the expression editor at regexr and look the explanation at the bottom of the page.

Awk

Count duplicates with awk

git log | grep '^Author' | sed -E 's/Author: (.*)/\1/' | awk '{ a[$0]++ } END { for(i in a) { print i, a[i] }}'

Sed

Sed print contents of regular expression capture group

git log | grep '^Author' | sed -E 's/Author: (.*)/\1/'

Replace in files

sed -i 's/replacethis/withthis/' in-file.txt

Replace in files with capture groups

This example is for replacing function name prefix on Powershell. For multiple files, the script name can be replaced with *

The line Function Get-FOOFile { will become Function Get-BARFile {

Captured groups are substituted by \1, \2 in the replacement expression.

To test the results of this before executing it on actual files, remove -i and it will return the entire file after substitution on stdout.

sed -E 's/(Function .*?-)FOO(.*?\s)/\1BAR\2/' -i test-script.ps1

Turn off indexing on all paths on all sites on an an Apache server

sudo sed -i 's/Options Indexes FollowSymLinks/Options FollowSymLinks\n        Options -Indexes/'  /etc/apache2/sites-available

Recursively replace become: yes with become: true in files

sed -i 's/become: yes/become: true/' $(grep -r "become: yes" | awk '{ print $1 }' | sed 's/.$//' |  sort | uniq)

Cut

Cut can be used to split strings in various ways.

This will split the string on , and select string2

"string1,string2,string3" | cut -d ',' -f 2

Xargs

Replace input to xargs into strings.

This example was used to copy the file ~/cert.pem into each directory under /var/app

ls /var/app | xargs -I '{}' cp ~/cert.pem '/var/app/{}/.'

Combined examples

Count number of commits by authors (uniq -c (count) requires the input to be sorted)

git log | grep '^Author' | sed -E 's/Author: (.*)/\1/' | sort | uniq -c

Same, but also sort by number of commits

sort -n (numerically) -k1 (on the first column)

git log | grep '^Author' | sed -E 's/Author: (.*)/\1/' | sort | uniq -c | sort -n -k1

Or by author name (here you can easily spot the noobs who didn’t set their git config email)

git log | grep '^Author' | sed -E 's/Author: (.*)/\1/' | sort | uniq -c | sort -k2

Search all apache logs for a particular site

ls -rt /var/log/apache2/site-access* | xargs zgrep 'GET /icon/favicon.ico' | cut -d: -f2-

Git

Show local config

git config --local -l

Show global config

git config --global -l

In a git repository you can override your global email. (Useful if you are using a single computer for work and private projects)

# Check the contents
cat .git/config

# Append your email
cat << EOF >> .git/config
[user]
    email=mail@example.com
EOF

openssl

Show the certificate chain for a server

echo "" | openssl s_client -connect cbsch.no:443 -showcerts 2>&1 | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p'

Connect to endpoind and view certificate

openssl s_client -showcerts -connect cbsch.no:443 </dev/null 2>/dev/null | openssl x509 -text

Some non-standard tools

fzf

Fzf is a fuzzy searcher

Fuzzy search all files:

find . | fzf

Search for a file and then open in vim

vim $(find -type f . | fzf)

Look for a folder and change directory

cd $(find -type d . | fzf)

ag-silversearcher

Much the same as grep -r, but understands .gitignore and ignores it by default

ag 'look for this in all subfiles and folders'

Powershell

Powershell is generally a lot easier to do more stuff with, but will be slower if the inputs are huge

Git author count again, now with powershell

git log | % { if ($_ -match "^Author: (.*)") { $matches[1] }} | group | select Name, Count

git log | Select-String "^Author: (.*)" | % { $_.Matches.Groups[1].Value } | group | select Name, Count