Karan Sharma - Today I Learned

Using slog for logging in Go

2024-03-18T00:00:00+00:00

Initialise a base logger#

// InitLogger initialies a logger.
func InitLogger(lvl, env, version string) *slog.Logger {
	var logLvl = new(slog.LevelVar)
	if lvl == "debug" {
		logLvl.Set(slog.LevelDebug)
	}
	// Replace internal attributes.
	replace := func(groups []string, a slog.Attr) slog.Attr {
		if a.Key == slog.SourceKey {
			if source, ok := a.Value.Any().(*slog.Source); ok {
				// Remove the function name from the source as it's too noisy.
				source.Function = ""
			}
		}
		return a
	}

	h := slog.NewJSONHandler(
		os.Stderr,
		&slog.HandlerOptions{
			AddSource:   true,
			Level:       logLvl,
			ReplaceAttr: replace,
		},
	).WithAttrs([]slog.Attr{
		slog.String("version", version),
		slog.String("environment", env),
	})
	return slog.New(h)
}

Logging SMTP connection timings

2024-02-22T00:00:00+00:00

import smtplib
import time

def print_time_taken(start, end, step):
    print(f"{step} took {end - start:.2f} seconds.")

def main():
    smtp_server = "faulty-smtp-host.com"
    smtp_port = 25

    total_start = time.time()

    # Connect to the SMTP server
    connect_start = time.time()
    server = smtplib.SMTP(smtp_server, smtp_port)
    connect_end = time.time()
    print_time_taken(connect_start, connect_end, "Connection")

    # Send EHLO
    ehlo_start = time.time()
    server.ehlo()
    ehlo_end = time.time()
    print_time_taken(ehlo_start, ehlo_end, "EHLO")

    # Send STARTTLS
    starttls_start = time.time()
    server.starttls()
    starttls_end = time.time()
    print_time_taken(starttls_start, starttls_end, "STARTTLS")

    # Re-send EHLO after STARTTLS
    ehlo2_start = time.time()
    server.ehlo()
    ehlo2_end = time.time()
    print_time_taken(ehlo2_start, ehlo2_end, "EHLO after STARTTLS")

    # Quit the session
    quit_start = time.time()
    server.quit()
    quit_end = time.time()
    print_time_taken(quit_start, quit_end, "QUIT")

    total_end = time.time()
    print_time_taken(total_start, total_end, "Total execution")

if __name__ == "__main__":
    main()

Produces an output like:

Connection took 20.09 seconds.
EHLO took 0.01 seconds.
STARTTLS took 0.04 seconds.
EHLO after STARTTLS took 0.01 seconds.
QUIT took 0.01 seconds.
Total execution took 20.15 seconds.

Golang: The Loop Variable Trap

2023-09-20T00:00:00+00:00

In Go, one of the most popular features is the ability to easily create lightweight concurrent threads using goroutines. While goroutines make concurrent programming simpler, they come with their own set of quirks that can be a source of subtle bugs if not understood well.

The Issue#

When you launch a goroutine inside a loop, and the goroutine function references the loop variable, you might encounter unexpected behavior. Instead of each goroutine using its respective loop iteration value, they might all end up using the last value of the loop variable.

Here’s a simple demonstration:

values := []string{"a", "b", "c"}
for _, v := range values {
    go func() {
        fmt.Println(v)
    }()
}

You might expect this to print (the order of the goroutines’ execution is not guaranteed, but you’ll still see 3 different values):

a
b
c

But instead you will see this:

c
c
c

The Reason#

The loop variable v is shared among all goroutines. Since the goroutines don’t execute immediately and are scheduled to run, by the time they start executing, the loop might have already completed its iterations, leaving v with its last value, “c”.

Here’s a step-by-step of what happens:

Loop iteration starts with v = "a".
A goroutine is launched which references v.
Before the goroutine has a chance to execute, the loop moves to the next iteration.
v now takes the value “b”.
Another goroutine is launched, also referencing v.
Again, before this new goroutine has a chance to execute, the loop moves to the next iteration.
v now takes the value “c”.
A third goroutine is launched, also referencing v.
At this point, depending on the scheduler and system, one or more of the goroutines might start executing. But all of them see v as “c” since they all reference the same memory location (the address of v), which by now has been set to “c” by the loop in the main goroutine.

So, when each of these goroutines tries to read the value of v, they all see “c”, hence printing “c” three times.

The Solution#

To ensure each goroutine receives the correct value from the loop iteration, pass the loop variable as an argument to the goroutine’s function:

values := []string{"a", "b", "c"}
for _, v := range values {
    go func(val string) {
        fmt.Println(val)
    }(v)  // passing the loop variable as an argument
}

By passing the value of v as an argument to the anonymous function, you effectively make a copy of the value for each iteration of the loop, so each goroutine will see the value of v that was present at the time it was launched.

Conclusion#

When working with goroutines inside loops, always be wary of capturing the loop variable directly. Passing it as an argument can help avoid this subtle trap and ensure your concurrent code behaves as intended.

Resolving UUID Conflicts when Mounting AWS EBS Volumes

2023-08-31T00:00:00+00:00

Today I Learned (TIL) about an interesting issue that can occur when mounting Amazon EBS volumes created from snapshots. The problem arises from UUID conflicts which can prevent the mounting process.

When you create an EBS volume from a snapshot, the new volume inherits the UUID of the original filesystem. If the original volume is still mounted on the same instance, this can cause a UUID conflict when you try to mount the new volume.

Let’s say you’re trying to mount a volume with XFS filesystem:

sudo mount -t xfs /dev/nvme2n1 /backup-check

And you encounter an error like this:

mount: /backup-check: wrong fs type, bad option, bad superblock on /dev/nvme2n1, missing codepage or helper program, or other error.

The issue might be due to a UUID conflict. To resolve this, you can use the -o nouuid option when mounting the volume:

sudo mount -t xfs -o nouuid /dev/nvme2n1 /backup-check

The -o nouuid option tells the mount command to ignore the UUID of the filesystem. This allows the volume to be mounted even if another volume with the same UUID is already mounted.

Check UUID#

You can check the UUID of a filesystem using the blkid command:

sudo blkid /dev/nvme2n1

This command will output something like this:

/dev/nvme2n1: UUID="1234abcd-12ab-34cd-56ef-123456abcdef" TYPE="xfs"

You can compare the UUIDs of your volumes to see if there’s a conflict.

Clean Up Outdated Local Git Branches

2023-08-16T00:00:00+00:00

To delete local branches that don’t exist in the upstream repository anymore, you can follow these steps:

Ensure you’re up-to-date: First, fetch the latest changes from the upstream:
```
git fetch --prune
```
The --prune option will remove any remote-tracking references that no longer exist on the remote. This will help you ensure that you know which branches are truly gone from the upstream.
List merged branches: You can see which branches are already merged into your current branch with:
```
git branch --merged
```
This will show you a list of branches that are safe to delete because their changes are already included in the current branch.
Delete merged branches: If you’re sure that the branches listed in step 2 are no longer needed (and you’ve checked that they don’t contain any unique work), you can delete them.
```
git branch --merged | egrep -v "(^\*|master|main)" | xargs git branch -d
```
- The egrep -v "(^\*|master|main)" part of this command ensures that you don’t try to delete the currently checked out branch (*) or typically important branches like master or main.
- The xargs git branch -d part of the command will attempt to delete each branch that’s passed to it. The -d flag ensures that Git will only delete the branch if it’s been merged. If you’re absolutely sure you want to force deletion, you can use the -D flag instead, but be careful with it.
Optional: To verify that you’ve successfully deleted the old local branches, you can list all your local branches with:
```
git branch
```

By following these steps, you’ll clean up your local branches, removing the ones that don’t exist in the upstream repository anymore. Always be cautious when deleting branches to ensure you don’t lose any important work.

Docker CLI Snippets

2023-02-02T00:00:00+00:00

Remove images locally with a wildcard:#

docker rmi --force $(docker images | grep <image-name> | tr -s ' ' | cut -d ' ' -f 3) `

Remove images using xargs#

I wanted to remove all containers that had the word outline in their name. I used the following command:

docker ps | grep outline | awk '{print $1}' | xargs docker rm -f

Dockerhub Rate Limits#

TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq --raw-output .token) && curl --head --header "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest" 2>&1 | grep ratelimit

Docker Disk Usage#

docker system df --verbose

Stop all containers#

docker stop $(docker ps -a -q)

Useful Terraform Snippets for AWS

2022-10-11T00:00:00+00:00

Get the list of AWS API actions required to run `terraform`#

Terraform uses AWS APIs to modify infrastructure. To find out a set of minimal IAM policies required for tf apply to run, we can follow this approach:

Give full permissions to your IAM user.
Run TF_LOG=trace terraform apply --auto-approve &> log.log
Run cat log.log | grep "DEBUG: Request"

Sometimes, the error message isn’t clear by Terraform Provider. In that case, the following command helps:

grep 'HTTP/1.1 403' -C 5 log.log

Delete all resources in a state#

terraform state list | cut -f 1 -d '[' | xargs -L 1 terraform state rm

Golang Gotchas

2022-09-24T00:00:00+00:00

HTTP Responses#

If you’ve a http.ResponseWriter object and using it to directly write the response to the connection, the order in which you write different parts of response is very important:

// First write headers.
wr.Header().Set("Content-Type", "text/plain; version=0.0.4")
// Then set the status code. And yes, this is the function name for setting status code!
wr.WriteHeader(http.StatusOK)
// Then finally write the response body.
wr.Write([]byte("hello world"))

NOTE

Headers should be written first. If any extra header is set after calling WriteHeader it’s a no-op.

Body should be after WriteHeader. Else it’s a no-op.

Working with cloud-init in AWS

2022-09-21T00:00:00+00:00

Run the script manually#

cd /var/lib/cloud/instance/scripts
sudo ./part-001

To view logs#

sudo journalctl -u cloud-final -b

To output logs to a different file#

#!/usr/bin/env bash
set -Eeuo pipefail
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

Logs will be available on /var/log/user-data.log.

Bash Scripting Templates: From Basic to Advanced

2022-09-21T00:00:00+00:00

Preferred Version#

I use the following template to quickstart any new shell script.

#!/usr/bin/env bash

set -Eeuo pipefail
readonly SCRIPT_DIR="$(cd "$(dirname "$${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "$0")"

function log {
  local readonly level="$1"
  local readonly message="$2"
  local readonly timestamp=$(date +"%Y-%m-%d %H:%M:%S")
  >&2 echo -e "$${timestamp} [$${level}] [$${SCRIPT_NAME}] $${message}"
}
function log_info {
  local readonly message="$1"
  log "INFO" "$${message}"
}
function log_warn {
  local readonly message="$1"
  log "WARN" "$${message}"
}
function log_error {
  local readonly message="$1"
  log "ERROR" "$${message}"
}

Elaborate Version#

Ref: https://betterdev.blog/minimal-safe-bash-script-template/

#!/usr/bin/env bash

set -Eeuo pipefail
trap cleanup SIGINT SIGTERM ERR EXIT

script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd -P)

usage() {
  cat << EOF # remove the space between << and EOF, this is due to web plugin issue
Usage: $(basename "${BASH_SOURCE[0]}") [-h] [-v] [-f] -p param_value arg1 [arg2...]

Script description here.

Available options:

-h, --help      Print this help and exit
-v, --verbose   Print script debug info
-f, --flag      Some flag description
-p, --param     Some param description
EOF
  exit
}

cleanup() {
  trap - SIGINT SIGTERM ERR EXIT
  # script cleanup here
}

setup_colors() {
  if [[ -t 2 ]] && [[ -z "${NO_COLOR-}" ]] && [[ "${TERM-}" != "dumb" ]]; then
    NOFORMAT='\033[0m' RED='\033[0;31m' GREEN='\033[0;32m' ORANGE='\033[0;33m' BLUE='\033[0;34m' PURPLE='\033[0;35m' CYAN='\033[0;36m' YELLOW='\033[1;33m'
  else
    NOFORMAT='' RED='' GREEN='' ORANGE='' BLUE='' PURPLE='' CYAN='' YELLOW=''
  fi
}

msg() {
  echo >&2 -e "${1-}"
}

die() {
  local msg=$1
  local code=${2-1} # default exit status 1
  msg "$msg"
  exit "$code"
}

parse_params() {
  # default values of variables set from params
  flag=0
  param=''

  while :; do
    case "${1-}" in
    -h | --help) usage ;;
    -v | --verbose) set -x ;;
    --no-color) NO_COLOR=1 ;;
    -f | --flag) flag=1 ;; # example flag
    -p | --param) # example named parameter
      param="${2-}"
      shift
      ;;
    -?*) die "Unknown option: $1" ;;
    *) break ;;
    esac
    shift
  done

  args=("$@")

  # check required params and arguments
  [[ -z "${param-}" ]] && die "Missing required parameter: param"
  [[ ${#args[@]} -eq 0 ]] && die "Missing script arguments"

  return 0
}

parse_params "$@"
setup_colors

# script logic here

msg "${RED}Read parameters:${NOFORMAT}"
msg "- flag: ${flag}"
msg "- param: ${param}"
msg "- arguments: ${args[*]-}"

Useful ClickHouse Query Snippets

2022-09-13T00:00:00+00:00

Check Table Size#

SELECT
    concat(database, '.', table) AS table,
    formatReadableSize(sum(bytes)) AS size,
    sum(rows) AS rows,
    max(modification_time) AS latest_modification,
    sum(bytes) AS bytes_size,
    any(engine) AS engine,
    formatReadableSize(sum(primary_key_bytes_in_memory)) AS primary_keys_size
FROM system.parts
WHERE active
GROUP BY
    database,
    table
ORDER BY bytes_size DESC

Access Management#

SHOW CREATE USER
SHOW ACCESS

Listing#

SHOW TABLES
SHOW CREATE TABLE
SHOW DATABASES

Show Mutations#

Pending#

SELECT *
FROM system.mutations
WHERE is_done = 0

Cardinality#

SELECT
    formatReadableQuantity(uniq(URL)) AS cardinality_URL,
    formatReadableQuantity(uniq(UserID)) AS cardinality_UserID
FROM
(
    SELECT
        user_id AS UserID,
        http_uri AS URL
    FROM logs.http
)

Useful Git commands

2022-09-02T00:00:00+00:00

Set the editor#

git config --global core.editor "vim"

Delete all local branches#

git branch --merged | grep -v \* | xargs git branch -D

Delete all branches except a few#

git branch | grep -v "develop" | grep -v "master" | xargs git branch -D

Delete a remote branch#

git push origin :develop

Force merge a branch#

When you have too many conflicts while merging branch feature to main, use this:

git checkout feature
git merge -s ours main
git push origin feature

git checkout main
git merge feature
git push origin main

Extracting Artifacts from Docker with Buildx

2022-08-08T00:00:00+00:00

I needed to build some binaries inside Dockerfile and copy the artifacts built inside the image to the host machine.

After some digging in, found buildx supports Custom Build Outputs

Here’s an example workflow on how to achieve the same:

FROM ubuntu:22.04 AS base

RUN mkdir -p /data
RUN touch /data/hello.txt

# Export the data to host using buildx
FROM scratch AS export
COPY --from=base /data .

docker buildx build --file Dockerfile --output data .

After the above command builds the image, you can see ./data got created in our host machine.

.
├── Dockerfile
└── data
    └── hello.txt

Ref

AWS CLI Snippets

2022-08-05T00:00:00+00:00

Fetch a list of `running` instances#

aws ec2 describe-instances --output json | jq -r '.Reservations[].Instances[] | select(.State.Name == "running") | { instance_id: .InstanceId, instance_type: .InstanceType, private_ip: .PrivateIpAddress, name: .Tags[]|select(.Key=="Name")|.Value, env: .Tags[]|select(.Key=="env")|.Value, role: .Tags[]|select(.Key=="role")|.Value } | [.instance_id,.name,.instance_type,.private_ip,.env,.role] | @csv '

Selects only running instances
Add tags to the metadata.
Outputs to CSV.

Size of S3 Bucket#

aws s3 ls s3://{{BUCKET_NAME}} --recursive  | grep -v -E "(Bucket: |Prefix: |LastWriteTime|^$|--)" | awk 'BEGIN {total=0}{total+=$3}END{print total/1024/1024" MB"}'

Format List Output#

aws s3 ls s3://mybucket --recursive --human-readable --summarize

Network usage of EC2 Instance#

Find out the total Network Out (in GBs) of an EC2 instance for a given time period

 aws cloudwatch get-metric-statistics --metric-name NetworkOut --start-time 2022-07-01T00:00:00.000Z --end-time 2022-08-01T00:00:00.000Z --period 86400 --namespace AWS/EC2 --statistics Sum --dimensions Name=InstanceId,Value=i-instance-id-xxx --region ap-south-1 --output text

Using the Nomad API to Manage Allocations

2022-07-26T00:00:00+00:00

List allocations#

curl -vvvv --get http://localhost:4646/v1/allocations\?namespace\=\*

Filter#

By using --filter param, we can restrict the list of allocations with specific filter queries.

curl -vvvv --get http://localhost:4646/v1/allocations\?namespace\=\* \
    --data-urlencode 'filter=NodeID == "a7542cd4-491d-26c8-64d4-db4db979f61b"'

Working with the Linux Filesystem

2022-07-11T00:00:00+00:00

Check disk size of a directory#

du -sh /var/log/*

List files with modification timestamp#

ls -laht

To see the files with the oldest file modification timestamp, use -r

ls -lahtr

Nomad Template Snippets

2022-07-08T00:00:00+00:00

Service List#

I needed to fetch a list of upstream services from Nomad’s service registry and render in a .toml file like this. This is essentially a comma separated list of services

servers = ["1.1.1.1:53","8.8.8.8:53"]

Figuring this out with Go template took some time.

servers = [{{ range $index, $element := nomadService "my-app-svc" }}{{if $index}},{{end}}"{{.Address}}:{{.Port}}"{{- end }}]

Testing Open File Descriptors Limit in Golang

2022-07-01T00:00:00+00:00

I wanted a quick way to test if ulimit -n (max number of open files per process) works or not.

This Go snippet creates n files and opens them. It prints the underlyng File Descriptor ID but doesn’t close the file handle. This way we can test the max number of open files this Go program can ever create:

Running#

package main

import (
	"fmt"
	"os"
)

var (
	FILES_TO_OPEN = 99999
)

func main() {
	fmt.Println("writing")
	for i := 1; i < FILES_TO_OPEN; i++ {
		fdF, err := os.Create(fmt.Sprintf("/tmp/fd_go%d.txt", i))
		if err != nil {
			panic(err)
		}
		_, err = fdF.Write([]byte("a"))
		if err != nil {
			panic(err)
		}
		fdF.Close()
	}
	fmt.Println("opening")
	for i := 1; i < FILES_TO_OPEN; i++ {
		fdF, err := os.Open(fmt.Sprintf("/tmp/fd_go%d.txt", i))
		if err != nil {
			panic(err)
		}
		fmt.Println(fdF.Fd())
	}
}

Exceeding the Limit#

If you exceed the limit, it’ll give an error like:

1018
1019
1020
1021
1022
1023
panic: open /tmp/fd_go1018.txt: too many open files

goroutine 1 [running]:
main.main()
	/home/karan/Code/Infra/fd-test/main.go:29 +0x286
exit status 2

In my host ulimit -n was set to 1024, so it failed to create more files than that.

Python Version#

See [[Python/Open File Descriptors]]

Handling Open File Descriptors Limit in Python

2022-07-01T00:00:00+00:00

See Golang/Open File Descriptors version for context.

>>> [open("FD_TEST_%d.txt" % i, "w") for i in range(1, 1025)]

OSError: [Errno 24] Too many open files: 'FD_TEST_1021.txt'
>>>

How to Identify the EBS Volume ID on an AWS EC2 Instance

2022-06-27T00:00:00+00:00

When managing EC2 instances with multiple EBS volumes, especially if they have similar sizes, it can be challenging to identify each one by its volume ID.

There’s a handy command for this:

lsblk -o +SERIAL

This command will display the EBS Volume ID for all attached volumes. It’s especially useful when you need to modify or resize a specific volume without any mix-ups.

Understanding AWS EC2 Pricing: A Quick Reference Guide

2022-06-23T00:00:00+00:00

For a swift lookup of EC2 instances, their pricing, and alternative instance types, I highly recommend ec2instances.github.io. It’s a comprehensive, user-friendly list and an indispensable tool for cloud professionals.

Controlling Screen Brightness in Linux

2022-06-21T00:00:00+00:00

Get a list of devices connected#

xrandr | grep " connected" | cut -f1 -d " "

Tweak the brightness value#

xrandr --output eDP-1 --brightness 0.7

Creating a MacOS-Like Screenshot Effect in Linux

2022-06-21T00:00:00+00:00

One thing which I miss from my OSX (now called macOS) days is it’s ability to create “pretty” screenshots out of the box. OSX screenshots have a nice drop-shadow effect and it centers the image with a faux border.

Not exactly same, but quite close is this imagemagick command to achieve the same on Linux systems:

convert screenshot.png \( +clone -background black -shadow 50x50+30+30 \) +swap -background white -layers merge +repage result.png

Source: StackOverflow

Pretty happy with the result!

Converting UTC to Local Time in Python

2022-06-21T00:00:00+00:00

Often I’ve felt the need to convert UTC time to IST but a quick google search reveals some shitty websites which makes such a seemingly simple task irritating and difficult. I tried to search for any small scripts or tools but thought I can whip one up real quick in Python and use it locally for myself.

Here’s the script I wrote, you can customise according to your needs if you wish to:

import argparse
import time
from datetime import datetime
from sys import exit

from dateutil.parser import parse


def datetime_from_utc_to_local(utc_datetime):
    now = time.time()
    offset = datetime.fromtimestamp(now) - datetime.utcfromtimestamp(now)
    return utc_datetime + offset


parser = argparse.ArgumentParser(description="Convert UTC time to Local time.")
parser.add_argument(
    "--timestamp", required=True, help="timestamp of UTC string"
)

args = parser.parse_args()

try:
    utc = parse(args.timestamp)
except Exception as e:
    print(e)
    exit(1)

res = datetime_from_utc_to_local(utc)
print(res.strftime("%H:%M:%S"))

Usage#

python utc-ist.py --timestamp 03:00
python utc-ist.py --timestamp 03:00:10
python utc-ist.py --timestamp 03:00:15.100

It takes a --timestamp parameter which should be a string depicting the UTC time format.

Alias#

ist: aliased to python /home/karan/Code/Infra/utc-ist/utc-ist.py --timestamp

$ ist 3:45
09:15

Generating Random Strings in Shell Scripts

2022-06-21T00:00:00+00:00

Generating random strings in Shell can be useful in various scenarios such as creating random passwords, unique identifiers, and more. Here are a couple of methods you can use:

Using tr and /dev/urandom:
```
tr -dc A-Za-z0-9 </dev/urandom | head -c 13 ; echo ''
```
This command will generate a 13-character random string using alphanumeric characters.
Using openssl:
```
openssl rand -base64 12
```
This command will generate a random string in base64 format.

Remember to replace the numbers in these commands with the length of the string you want to generate.

Extracting Filename in Bash

2022-06-21T00:00:00+00:00

A little while ago, I’d to extract the filename (without the extension) from a particular JSON file. There’a a neat basename utility that I didn’t know about.

However, it cannot accept STDIN so if you’re using it with pipes, you need to use xargs.

Get the filename with extension#

cat ~/app.json | jq '.file_path' | xargs basename

Remove the extension#

cat ~/app.json | jq '.file_path' | xargs basename  -s .txt

Counting Lines in Files with Bash

2022-06-21T00:00:00+00:00

To get wc -l for each file in the directory:

wc -l *.csv

To output the number of lines, sorted in decreasing order:

wc -l *.csv | sort -rn

Analyzing Most Frequently Used Shell Commands

2022-06-21T00:00:00+00:00

Using history and a bunch of other Linux-fu commands, we can find out the top 10 most used commands in the shell history:

history | awk '{CMD[$2]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' | grep -v "./" | column -c3 -s " " -t | sort -nr | nl |  head -n10

Here’s mine (last updated on Jun 2021)

     1	1376  13.6251%     git
     2	670   6.63432%     cd
     3	635   6.28775%     make
     4	630   6.23824%     clear
     5	502   4.97079%     kcp
     6	500   4.95099%     ls
     7	420   4.15883%     ssh
     8	369   3.65383%     sudo
     9	323   3.19834%     kc
    10	286   2.83196%     kcl

There’s a fancier output with bars too using:

history | tr -s ' ' | cut -d ' ' -f3 | sort | uniq -c | sort -n | tail | perl -lane 'print $F[1], "\t", $F[0], " ", "▄" x ($F[0] / 12)'

kcl	286 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
kc	323 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
sudo	365 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
ssh	419 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
ls	479 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
kcp	502 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
clear	626 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
make	635 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
cd	653 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
git	1348 ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄

NOTE
kc, kcl, kcp etc are aliases for K8s CLI commands.

Source

How to View CSV Files in Terminal

2022-06-21T00:00:00+00:00

column -s, -t < my.csv | less -#2 -N -S

Useful cURL Snippets

2022-06-14T00:00:00+00:00

Show only headers#

curl has an option to display only headers using -I. This flag is better than using a combination of -s and redirecting the output to /dev/null or 2>&1 as it’s easier to remember.

An important thing to note is that -I sends a HEAD request. As shown here:

$ curl -I  -vvvv https://mrkaran.dev 2>&1 | grep -C 3 'HEAD'
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55dff1e30920)
} [5 bytes data]
> HEAD / HTTP/2
> Host: mrkaran.dev
> user-agent: curl/7.76.1
> accept: */*

However it’s easy to override the HTTP method using -X. For eg, to send a GET request but only display the headers:

curl -I -X GET -vvvv https://mrkaran.dev

Send a JSON payload from file#

curl -i -XPOST -H "Content-Type: application/json" -d @mock_payload.json http://localhost:6000/endpoint

Fail if response is 4xx/5xx#

Using the --fail flag, curl can detect if the response code is an error (includes 4xx,5xx with an exception for 401 and 407 as they are authentication related). This is handy to use in scripts. Read this for more.

❯ curl -i --fail https://httpbin.org/status/200
HTTP/2 200 
date: Tue, 02 Nov 2021 10:43:38 GMT
content-type: text/html; charset=utf-8
content-length: 0
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

❯ curl -i --fail https://httpbin.org/status/502
curl: (22) The requested URL returned error: 502 

❯ curl -i --fail https://httpbin.org/status/400
curl: (22) The requested URL returned error: 400

Show stats of the request#

curl -kso /dev/null https://mrkaran.dev -w "==============\n\n 
| dnslookup: %{time_namelookup}\n 
| connect: %{time_connect}\n 
| appconnect: %{time_appconnect}\n 
| pretransfer: %{time_pretransfer}\n 
| starttransfer: %{time_starttransfer}\n 
| total: %{time_total}\n 
| size: %{size_download}\n 
| HTTPCode=%{http_code}\n\n"

Configuring Custom Subnets in Docker

2022-06-13T00:00:00+00:00

Docker uses 172.17.0.0/16 as the CIDR for it’s own network and all the other bridge network it creates. It maybe sometimes useful to change the default subnet to a custom one, in case it conflitcts with other resources (like AWS VPC) in your infra. Not just this, it can also happen if you’ve multiple docker-compose projects in your server and you face an error similar to:

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

$ ip a show docker0

8: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:18:b7:60:80 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:18ff:feb7:6080/64 scope link
       valid_lft forever preferred_lft forever

Docker-compose#

If you’re using docker-compose, then you can simply update the subnet for the bridge network created in that file by giving custom IPAM options in the network section of the file.


services:
  app:
    image: app/app:latest
    networks:
      - monitor-net

networks:
  monitor-net:
    ipam:
      driver: default
      config:
        - subnet: 192.168.96.0/27

Docker settings#

If you wish to update the base address of docker0 interface and define these subnets globally, you can update daemon.json settings.

Add the following to /etc/docker/daemon.json. Create the file if it doesn’t exist.

{
  "bip": "10.200.0.1/24",
  "default-address-pools":[
    {"base":"10.201.0.0/16","size":24},
    {"base":"10.202.0.0/16","size":24}
  ]
}

sudo systemctl restart docker

Verify the settings#

ip a show docker0

You should see 10.200.0.1

Explanation#

"bip": "10.200.0.1/24"
- bip stands for “Bridge IP”.
- This specifies the IP address and subnet for the Docker daemon’s default bridge network. The default bridge network is used for communications between the Docker host and containers that do not specify a network.
- 10.200.0.1 is the IP address assigned to the bridge interface on the Docker host.
- /24 indicates that the subnet mask is 255.255.255.0, which means that the IP addresses from 10.200.0.1 to 10.200.0.254 are available for use by containers connected to this bridge.
"default-address-pools": [...]
- This is an array defining pools of network addresses that Docker can use for creating user-defined networks (i.e., networks created using docker network create).
Inside the default-address-pools array, we have two objects, each specifying a base subnet and a size for the network pools:
- {"base":"10.201.0.0/16","size":24}
  - This pool defines a range of IP addresses starting with the base 10.201.0.0/16.
  - /16 means that any address from 10.201.0.0 to 10.201.255.255 can be used to create smaller subnets.
  - "size":24 specifies that when Docker creates a user-defined network from this pool, it should use a subnet size of /24. Therefore, each user-defined network created from this pool will have a range of IP addresses like 10.201.x.0/24, where x is a variable that increments for each new network, providing 254 usable addresses for each subnet.
- {"base":"10.202.0.0/16","size":24}
  - Similar to the first pool, but this uses the 10.202.0.0/16 range.
  - Again, the "size":24 means that Docker will create user-defined networks with a /24 subnet from this range, for example 10.202.x.0/24, where x is an incrementing value.

Ref#

Interacting with Nomad using the CLI

2022-06-13T00:00:00+00:00

Some common commands which are used to interact with the jobs/allocations.

Please refer to CLI Docs to see the common Environment Variables to use in case you’re accessing a production Nomad cluster.

Create a deployment

nomad job run file.nomad

Stop a job

nomad job stop -purge <job-name>

View job status

nomad job status <job-name>

View alloc status

nomad alloc status -stats <alloc-id>

View logs

nomad alloc logs -f -stderr <alloc-id>

Exec inside alloc

nomad alloc exec -i -t -task <task-name> <alloc-id> /bin/bash

Setting Up Your Editor for Nomad Job Specifications

2022-06-13T00:00:00+00:00

For formatting HCLv2 files (that Nomad job spec uses), you can use the following setup to neatly format them automatically.

VSCode#

Install this extension.
Add the following lines to your settings.json:

    "files.associations": {
        "*.nomad": "hcl",
        "*.nomad.tpl": "hcl",
        "*.tf": "terraform",
    }

Reload VSCode.

Manual#

Install hclfmt using go (they don’t provide pre-compiled binaries, so building from source is only option):

go install github.com/hashicorp/hcl/v2/cmd/hclfmt@latest

Format with hclfmt. Since it only works on a single file by default, we need to use a wrapper script to cover all files and nested directories.

#!/usr/bin/env bash
set -e
for file in *; do
  hclfmt -w $file
done

Save the above as run-hclfmt in /usr/local/bin and chmod +x /usr/local/bin/run-hclfmt
Now, you’ll be able to run this script from anywhere in your directory of Nomad job specs using run-hclfmt. It includes sub directories as well.

Setting Up a Single Node Nomad Environment

2022-06-13T00:00:00+00:00

I use the following server/client config to quickly prototype Nomad jobs:

datacenter = "dc1"
data_dir = "/opt/nomad/data"

log_level = "DEBUG"

bind_addr = "0.0.0.0"

server {
  enabled          = true
  bootstrap_expect = 1
}

client {
  enabled = true
  
  reserved {
    cores          = 2
    memory         = 1024
    disk           = 1024
    reserved_ports = "22"
  }

  meta {
    env   = "dev"
    stack = "personal"
  }
}

plugin "docker" {
  config {
    allow_privileged = true
    volumes {
      enabled = true
    }
    extra_labels = ["job_name", "job_id", "task_group_name", "task_name", "namespace", "node_name", "node_id"]
  }
}

plugin "raw_exec" {
  config {
    enabled = true
  }
}

Save the file as nomad.hcl (which will be referenced later).

Instructions#

Get Nomad from https://learn.hashicorp.com/tutorials/nomad/get-started-install.
Run the local agent:

sudo nomad agent -config=nomad.hcl

Check node status

nomad node status

Run example deployment

nomad init
nomad run example.nomad

Ansible FAQ

2022-06-12T00:00:00+00:00

Tags#

Tags are opt-in. If you’ve an Ansible Playbook, the task will always run. To ensure it only runs if a tag is given and never otherwise, use the never tag.

- never
- my_custom_tag

If a task is defined with above tags, then only if my_custom_tag is there, the task will run.

To make this simple, the playbook run command should have a list of tags that are to be executed. --tags='abc,def' is the way to supply these tags on the command-line.

!!! tip Use Makefiles whenever possible.

If you add a tag to a playbook, then the tag is appended to the role.

  roles:
    - name: clickhouse
	  tags: [check_me_out]

You can see this check_me_out tag is appended to each task in the role.

ansible-playbook -i inventory playbook.yml --list-tasks

playbook: playbook.yml

  play #1 (clickhouse): clickhouse      TAGS: []
    tasks:
      clickhouse : Install multiple packages    TAGS: [check_me_out, clickhouse_install]
      clickhouse : Add the APT Key for ClickHouse.      TAGS: [check_me_out, clickhouse_install]
      clickhouse : Add ClickHouse APT sources   TAGS: [check_me_out, clickhouse_install]
      clickhouse : Install ClickHouse   TAGS: [check_me_out, clickhouse_install]
      clickhouse : Config | Set ClickHouse configuration file   TAGS: [check_me_out, clickhouse_configure]
      clickhouse : Config | Set users configuration file        TAGS: [check_me_out, clickhouse_configure]

Hence, the best way is to have a Makefile with different tags supplied on the CLI:

install-clickhouse:
	ansible-playbook -i inventory playbook.yml --tags "clickhouse_install"

configure-clickhouse:
	ansible-playbook -i inventory playbook.yml --tags "clickhouse_configure"

Naming Strategy#

Since Ansible Tags use OR (and not AND) as the joining condition, it becomes tricky to have a role with tasks like:

- clickhouse
- install

- clickhouse
- configure

In a case where you want to only run configure, you just have to pass configure. If you pass clickhouse,configure, even the install.yml will be run. This is important to note as it can cause issues when running playbooks.

The workaround then is to have a tag called clickhouse_configure and clickhouse_install and only pass that.

It’s possible to preview the list of tasks which will be executed with:

ansible-playbook example.yml --tags "configuration,packages" --list-tasks

Roles#

!!! tip Always ensure you fetch the remote roles again if you made any changes to them.

Inventory#

Aliases#

Aliases are helpful in cases where the ansible_host is same (bastion host), but the target host is defined with ansible_ssh_extra_args. In such cases, Ansible is unable to differentiate between two hostnames and it thinks both the hosts are same.

bastion-host ansible_ssh_port=22 ansible_ssh_user=ubuntu ansible_ssh_extra_args="-o 'Hostname 1.1.1.1'"
bastion-host ansible_ssh_port=22 ansible_ssh_user=ubuntu ansible_ssh_extra_args="-o 'Hostname 2.2.2.2'"

The fix to this is to use an Inventory Alias.

app_1 ansible_host=bastion-host ansible_ssh_port=22 ansible_ssh_user=ubuntu ansible_ssh_extra_args="-o 'Hostname 1.1.1.1'"
app_2 ansible_host=bastion-host ansible_ssh_port=22 ansible_ssh_user=ubuntu ansible_ssh_extra_args="-o 'Hostname 2.2.2.2'"

So with the alias app_1 and app_2, Ansible is able to differentiate b/w two hostnames.

Ansible Snippets

2022-06-12T00:00:00+00:00

Assert a list of variables#

---
- name: assert if all template variables are present
  assert:
    that:
      - "{{item}} is defined"
      - "{{item}} | length > 0"
    quiet: false
  with_items:
    - my_var
    - another_var
  no_log: true

Execute a task before executing a role#

- hosts: "my_server"
  become: yes
  # Assert if variables are present.
  pre_tasks:
    - import_tasks: ../tasks/assert.yml
  roles:
    - role: nginx

Wait for apt-get lock before installing packages#

# https://github.com/ansible/ansible/issues/51663#issuecomment-752286191
# A common issue, particularly during early boot or at specific clock times
# is that apt will be locked by another process, perhaps trying to autoupdate
# or just a race condition on a thread. This work-around (which can also be
# applied to any of the above statements) ensures that if there is a lock file
# engaged, which is trapped by the `msg` value, triggers a repeat until the
# lock file is released.
- name: Install OS dependencies
  apt:
    name: "{{ consul_os_packages }}"
    state: present
  register: apt_action
  retries: 100
  until: apt_action is success or ('Failed to lock apt for exclusive operation' not in apt_action.msg and '/var/lib/dpkg/lock' not in apt_action.msg)

Donwload and run a project from an external source#

The following playbook downloads a GitHub release, templates a config file and starts a systemd-service

  - name: Register working directory
    set_fact:
      http_script_bin_dir: "/home/{{ansible_ssh_user}}/services/http-script-executor"
      http_script_config_dir: /etc/http-script-executor

  - name: Create project directory
    ansible.builtin.file:
      path: "{{http_script_bin_dir}}"
      state: directory
      mode: '0755'
      owner: "{{ansible_ssh_user}}"
      group: "{{ansible_ssh_user}}"

  - name: Create config directory
    ansible.builtin.file:
      path: "{{http_script_config_dir}}"
      state: directory
      mode: '0755'
      owner: "{{ansible_ssh_user}}"
      group: "{{ansible_ssh_user}}"

  - name: Download latest release
    get_url:
      url: "https://github.com/iamd3vil/http-script-executor/releases/download/v0.1.0/http-script-executor_0.1.0_Linux_x86_64.tar.gz"
      dest: "{{http_script_bin_dir}}/http-script-executor.tar.xz"
      force: yes
      owner: "{{ansible_ssh_user}}"
      group: "{{ansible_ssh_user}}"

  - name: Unarchive the binary
    ansible.builtin.unarchive:
      src: "{{http_script_bin_dir}}/http-script-executor.tar.xz"
      dest: "{{http_script_bin_dir}}"
      remote_src: yes
      owner: "{{ansible_ssh_user}}"
      group: "{{ansible_ssh_user}}"

  - name: Copy config file.
    template:
      src: ./templates/http_script_executor/config.toml.j2
      dest: "{{http_script_config_dir}}/config.toml"
      owner: "{{ansible_ssh_user}}"
      group: "{{ansible_ssh_user}}"
    register: output

  - name: Copy binary to PATH
    ansible.builtin.copy:
      src: "{{http_script_bin_dir}}/http-script-executor"
      dest: "/usr/local/bin/http-script-executor"
      remote_src: yes
      owner: "{{ansible_ssh_user}}"
      group: "{{ansible_ssh_user}}"
      mode: +x
    register: output


  - name: Cleanup project folder
    file:
      state: absent
      path: "{{http_script_bin_dir}}"

  - name: Add systemd script for program
    template:
      src: ./templates/http_script_executor/http-script-executor.service.j2
      dest: "/etc/systemd/system/http-script-executor.service"
      owner: root
      group: root
      mode: 0644
    register: systemd_unit

  - name: Reload systemd
    systemd:
      daemon_reload: true
    when: systemd_unit is changed

  - name: Start service
    service:
      name: http-script-executor
      state: started
      enabled: yes

  - name: Restart service
    service:
      name: http-script-executor
      state: restarted
    when: output.changed

Systemd example#

[Unit]
Description= A simple HTTP server which executes scripts
After=network.target

[Service]
ExecStart=/usr/local/bin/http-script-executor --config=/etc/http-script-executor/config.toml
User={{ansible_ssh_user}}
Group={{ansible_ssh_user}}

[Install]
WantedBy=multi-user.target

Ansible Vault

2022-06-12T00:00:00+00:00

Vault let’s you encrypt sensitive values in your playbook. These can be DB Passwords, API tokens etc that need to referenced inside the roles/task etc.

There are 2 ways to encrypt:

File
String

IMHO encrypting with a file is better even for a few variables.

How To#

Create a file called secrets.enc and add the contents in plain-text:

password: secret
another_password: supersecret

Encrypt with ansible-vault encrypt secrets.enc
Store the password inside .vault_password (untracked from git) if you don’t want to provide the password everytime.
Run the playbook with ansible-playbook -i inventory playbook.yml -e @secrets.enc --vault-password-file=.vault_password

That’s all! {{password}} and {{another_password}} will be available inside tasks normally, no extra work needed.

Managing Secrets#

You can use these commands to view/edit secret file.

view-secrets:
	ansible-vault view secrets.enc --vault-password-file=.vault_password

edit-secrets:
	EDITOR=vim ansible-vault edit secrets.enc --vault-password-file=.vault_password

Accessing AWS EC2 Metadata within a Docker Container

2022-06-12T00:00:00+00:00

If you try to access EC2 Metadata endpoint from a Docker container which is bridged to the default network interface on your host, you would assume things to just work, right? Wrong.

Here’s what happens when you do this command on the host (it works fine):

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/

Here’s what happens when you do it inside a container:

docker run --rm -it alpine:sh
apk add bash curl

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/

It is stuck and will eventually time out.

The Fix#

This is because the response comes with a packet of TTL as 1. (TTL at packet level means number of hops). Since this response has to eventually pass to your container, you need to increase it higher. (For eg, 3 is fine).

Refer to docs

By default, the response to PUT requests has a response hop limit (time to live) of 1 at the IP protocol level. You can adjust the hop limit using the modify-instance-metadata-options command if you need to make it larger. For example, you might need a larger hop limit for backward compatibility with container services running on the instance. For more information, see modify-instance-metadata-options in the AWS CLI Command Reference.

There’s a nice read on how traceroute works, which kind of explains the above TTL concept as well: https://alexanderell.is/posts/toy-traceroute/

Modifying in Terraform#

Inside the aws_instance resource block, add this block:

  metadata_options {
    http_put_response_hop_limit = 3
    http_endpoint               = "enabled"
  }

Setting Custom DNS in Docker

2022-06-12T00:00:00+00:00

Docker makes it impossible to supply a --dns flag with docker build which makes it harder to use an internal service in your org. There are workarounds like --add-host=xyz.internal:<IP> but imagine doing this for dozen of services. Yep it’s not maintainable, especially if the DNS records of these services change (for eg if they are behind ALB).

A cleaner solution is to forward the queries to your host DNS server itself. You can use various techniques like Split Tunnel (if your VPN supports that) or custom routing to different upstream DNS servers (eg with CoreDNS).

To make this happen, you need to edit the /etc/docker/daemon.json file:

{
        "dns": ["x.x.x.x"]
}

Docker uses 8.8.8.8 as a default, but if you override the above setting, it will start using the DNS server you provided.

SSH Deployment with GitLab CI

2022-06-12T00:00:00+00:00

SSH to a server#

Create a public/private key pair on the remote instance
Copy the private key as an env variable in Gitlab CI
Use the following snippet to SSH:

stages:
  - prepare
  - deploy

default:
  image:
    name: public.ecr.aws/ubuntu/ubuntu:20.04

.ssh-commands: &ssh-commands
  - apt-get -y update && apt-get install -y rsync
  - 'which ssh-agent || ( apt-get install -y openssh-client )'
  - eval $(ssh-agent -s)
  - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh

sync-deploy-scripts:
  before_script:
    - *ssh-commands
  stage: deploy
  script:
    - rsync --cvs-exclude -ravz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" ./ ubuntu@<remote_ip>:/home/ubuntu/app
  only:
    - main

Useful Kubernetes CLI Commands

2022-06-12T00:00:00+00:00

Trigger a cronjob manually#

kubectl create job --from=cronjob/<cronjob-name> <unique-job-name>

Get events sorted by timestamp#

kubectl get events --sort-by=.metadata.creationTimestamp

Taint a node#

kubectl taint nodes {{name}} key1=value1:NoSchedule

Untaint a node#

kubectl taint nodes {{name}} key1=value1:NoSchedule-

Resource Utilisation#

Check Resource Requests/Limits on each node#

In case you need to check how much of Resource Requests/Limits are made across all the nodes in the cluster, following command is helpful:

alias util='kubectl get nodes --no-headers | awk '\''{print $1}'\'' | xargs -I {} sh -c '\''echo {} ; kubectl describe node {} | grep Allocated -A 5 | grep -ve Event -ve Allocated -ve percent -ve -- ; echo '\'''

Outputs:

ip-192-x-x-x
  Resource                    Requests      Limits
  cpu                         14472m (91%)  21970m (138%)
  memory                      21Gi (35%)    33070Mi (54%)

ip-192-x-x-x
  Resource                    Requests       Limits
  cpu                         14632m (92%)   21695m (136%)
  memory                      27448Mi (45%)  46854Mi (77%)

ip-192-x-x-x
  Resource                    Requests       Limits
  cpu                         15297m (96%)   22445m (141%)
  memory                      18819Mi (31%)  36278Mi (60%)

Troubleshooting Kubernetes CNI Issues

2022-06-12T00:00:00+00:00

Official guides to troubleshoot common issues:

https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/troubleshooting.md
https://aws.amazon.com/premiumsupport/knowledge-center/eks-failed-create-pod-sandbox/

One of the most common issue is getting out of available IPs in the subnet pool. You can check the logs here:

kubectl -n kube-system exec -it aws-node-XXX-- tail -f /host/var/log/aws-routed-eni/ipamd.log | tee ipamd.log

Next, go to the subnet and check the available IPs. Quite likely it’ll be 0. Add some kind of pod affinity to schedule the pod on another subnet using (.spec.affinity section of Pod config)

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - ap-south-1b

Here, the affinity is towards 1b node when the pod is scheduled. You can see all the different kind of Node labels/selectors applied using kubectl describe node | grep topology

You can even label certain nodes, and then use Node selectors instead of affinity to guarantee scheduling only on those selectors.

Troubleshooting Missing Logs in systemd-journald on Linux

2022-06-12T00:00:00+00:00

Debugging steps when journal logs are missing#

Follow these debugging steps, when systemd service is running, but logs are missing in journalctl:

sudo -i if not already.
Try running journalctl -b to see messages from the current boot.
If you still get -- No entries --, run journalctl --verify.
If you get No journal files were found, something is corrupted with the journal service itself. Run systemctl status systemd-journald*
If the services are all “green” (active/running), something is borked with the log files in /var/log/journal/<hash>. Try running the following to recreate them:

systemctl restart systemd-journald.service

https://unix.stackexchange.com/a/538881

In my case, I had a fresh droplet created on DO but journal entries were missing for all services. Doing a sudo systemctl restart systemd-journald.service fixed this weird as hell issue.

Essential Linux Networking Commands

2022-06-12T00:00:00+00:00

tcpdump#

Check for packets flowing out from an interface for a specific CIDR#

sudo tcpdump -v -i wlo1 dst net 192.168.0.0/16

Explanation:

-i is for interface. wlo1 is the interface name.
dst specifies destination IP
net specifies it’s a CIDR range.

ip#

List routing table#

ip route

Find which interface and route a particular IP is taking#

ip route get 192.168.1.1

ifconfig#

Find the primary IP of local machine#

Get the IP of all interfaces:#

ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'

Get the IP of a specific interface:#

For eg (wlo1):

ifconfig wlo1 | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'

Source

Managing Processes in Linux

2022-06-12T00:00:00+00:00

List of running process#

ps aux

Kill a process#

kill -9

Kill a process by name#

pkill -9 -f '././bin/app.bin'

Useful rsync Command Snippets for Linux

2022-06-12T00:00:00+00:00

Copying files locally#

rsync -avhW --no-compress --progress /src/ /dst/

Passing SSH Extra Options#

rsync -avzhP --stats -e 'ssh -o "Hostname <ip>"' <bastion_host>:/src/ /dst/

Useful sed Command Snippets for Linux

2022-06-12T00:00:00+00:00

Edit in place#

sed -i 's/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g' filename

Match only the word#

\b in regex is used to match word boundaries (i.e. the location between the first word character and non-word character).

sed -i 's/\bsuper_specific_phrase\b/STRING_TO_REPLACE_IT/g' filename

Minimal SystemD Starter Template

2022-06-12T00:00:00+00:00

Here’s a barebone/re-usable systemd service unit snippet that I often use to copy/paste for registering new services:

[Unit]
Description=Program Description
Documentation=https://program.url
After=network-online.target
Requires=network-online.target
StartLimitIntervalSec=0

[Service]
User=ubuntu
Group=ubuntu
ExecStart=/usr/local/bin/app.bin --config=/etc/app/config.toml
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Determining CPU Cores in Bash

2022-06-12T00:00:00+00:00

nproc is part of coreutils package that gives the number of cores available in an easy to consume way (no more grepping and parsing ).

I wanted this to set VECTOR_THREADS variable for vector CLI to 1/2 of what is available on system.

#!/bin/bash

set -e

cores=`nproc --all`
cap=2
VECTOR_THREADS=$((cores / cap))

vector --config /etc/vector

Executing Commands as Different Users in Bash

2022-06-12T00:00:00+00:00

Here’s an example of running echo but as a consul user:

sudo -H -u consul bash -c 'echo "I am $USER, with uid $UID"'

Useful Vector Configuration Snippets

2022-06-12T00:00:00+00:00

Output to File#

[sinks.file_out]
  # General
  type = "file" # required
  inputs = ["log_files"]
  compression = "none" # optional, default
  path = "./app-logs/access_token-%Y-%m-%d.log" # required

  # Encoding
  encoding.codec = "ndjson" # required

  # Healthcheck
  healthcheck.enabled = true # optional, default

Dummy Sink#

[sinks.blackhole]
  type = "blackhole"
  inputs = ["format_haproxy_logs"]

Console Sink#

[sinks.console_out]
  type = "console"
  inputs = ["log_files"]
  target = "stdout"
  encoding.codec = "json"

CLI Commands#

$ vector vrl
$ vector top

Securing Your ClickHouse Server Using RBAC

2022-05-02T00:00:00+00:00

Clickhouse offers a wide variety of managing access control using RBAC. As mentioned in the docs, Clickhouse offers two ways to manage users, roles, policies etc:

SQL driven approach
Using configuration files like users.xml

The docs mention SQL driven approach is more recommended.

Clickhouse comes with a default user which unfortunately is widely permissive. It is attached to a default profile, which allows access to entire DB:

CREATE USER default IDENTIFIED WITH plaintext_password SETTINGS PROFILE default
GRANT ALL ON *.* TO default WITH GRANT OPTION

Here’s a checklist to follow in case you want to secure your Clickhouse installation.

TL;DR:

Enable access-management for default user.
Create a new admin user with SHA-256 password.
- Restrict to local network interface.
Revoke permissions or remove the default user.

1. Restrict the `default` user to connect only via local network interface.#

The default configuration allows the default user to connect from any IP. This is dangerous since Clickhouse also offers an HTTP API (running on port 8123) by default). If there’s no token/basic_auth passed in the HTTP request to this API, the default user is picked up. So, this means if you don’t revoke permissions from default user, anyone who can access Clickhouse API has access to entire DB.

To fix this, edit /etc/clickhouse-server/users.xml:

<networks>
    <ip>::/0</ip>
</networks>

And make it:

<networks>
  <ip>::1</ip>
  <ip>127.0.0.1</ip>
</networks>

2. Create an Admin user#

CREATE USER IF NOT EXISTS admin IDENTIFIED WITH SHA256_PASSWORD BY '{{ADMIN_PASSWORD}}' HOST LOCAL;
GRANT ALL ON *.* TO admin WITH GRANT OPTION;

This creates an admin user with and stores the SHA-256 hash of the password in /var/lib/clickhouse/access. It also additonally only allows connection from a local interface using HOST LOCAL directive.

3. Remove default user#

You can safely remove the default user as it’s not really required.

Edit users.xml and remove everything inside <users><default> block.

<users>
</users>

Clickhouse automatically picks up changes from this file, so you don’t need to restart the server.

Creating Additional Users#

Follow the SQL driven approach to add more users. If the users have some common access policies, you can create a role with these policies and assign roles to the user.

The offficial docs are really good to help you further.

Decoding JSON in Golang: Handling String Encoded Data Types

2022-05-02T00:00:00+00:00

I had to unmarshall a POST body JSON payload into a struct but the payload had all fields as string and I wanted to have proper data types for the keys.

This is how the payload looked like:

{
    "bufsize": "512",
    "class": "IN",
    "do": "false",
    "duration": "0.000072844",
    "id": "33531",
    "level": "INFO",
    "name": "mrkaran.dev.",
    "proto": "udp",
    "rcode": "NOERROR",
    "rflags": "qr,aa,rd,ra",
    "rsize": "83",
    "server_addr": "127.0.0.1",
    "server_port": "53256",
    "size": "29",
    "type": "A"
}

Some of the fields like bufsize, size, id should be of type int32. To decode these string fields into the types defined in struct, we can use json:",string".

This is how the struct looks:

type Log struct {
        BufSize    int     `json:"bufsize,string"`
        Class      string  `json:"class"`
        DO         bool    `json:"do,string"`
        Duration   float64 `json:"duration,string"`
        ID         int     `json:"id,string"`
        Level      string  `json:"level"`
        Name       string  `json:"name"`
        Proto      string  `json:"proto"`
        RCode      string  `json:"rcode"`
        RFlags     string  `json:"rflags"`
        RSize      int     `json:"rsize,string"`
        ServerAddr string  `json:"server_addr"`
        SererPort  int     `json:"server_port,string"`
        Size       int     `json:"size,string"`
        Type       string  `json:"type"`
}

Quoting from the docs:

The “string” option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:

Inspecting HTTP Headers in Golang

2022-05-02T00:00:00+00:00

The following snippet is used to create a mini HTTP server to dump all HTTP headers of an incoming request which can be useful for debugging.

package main  
  
import (  
       "fmt"  
       "log"  
       "net/http"  
       "net/http/httputil"  
)  
  
func index(w http.ResponseWriter, req *http.Request) {  
       // Save a copy of this request for debugging.  
       requestDump, err := httputil.DumpRequest(req, true)  
       if err != nil {  
               fmt.Println(err)  
               fmt.Fprintf(w, "oops! check logs\n")  
               return  
       }  
       fmt.Fprintf(w, "%v", string(requestDump))  
}  
  
func ping(w http.ResponseWriter, req *http.Request) {  
       fmt.Fprintf(w, "pong\n")  
}  
  
func main() {  
  
       http.HandleFunc("/", index)  
       http.HandleFunc("/ping", ping)  
  
       port := ":6666"  
       log.Printf("starting server at %s", port)  
       http.ListenAndServe(port, nil)  
}

Profiling Golang Applications with pprof

2022-05-02T00:00:00+00:00

Creating a profile#

To profile Golang applications, pprof package is widely used. There’s a nice wrapper around it https://github.com/pkg/profile which allows you to just drop one line in your application and start generating profiles:


package main

func main() {

  defer profile.Start(profile.ProfilePath("."), profile.NoShutdownHook).Stop()

  // rest of your application
}

profile.NoShutdownHook: This is important to pass to profile.Start if you’re shutting down your application with a SIGINT/SIGTERM and not handling profile storage directly. This will ensure the .pprof file gets saved correctly.

Visualizing#

go tool pprof -http=":8000" ./cpu.pprof

This allows you to visualize the pprof file in Graphviz and the more helpful Flamegraph visualizations.

How to Update Your Golang Version

2022-05-02T00:00:00+00:00

The below shell script is helpful to grab a specific Go version and update it locally.

To use it you can tweak the following variables:

VERSION
PLATFORM

#! /bin/bash
set -euf -o pipefail

# Download latest Golang release for AMD64
# https://golang.org/dl/

VERSION=1.16
PLATFORM=linux-amd64

# Download Go
cd `mktemp -d`
echo "Downloading Go version ${VERSION}..."
curl -sL -o go${VERSION}.${PLATFORM}.tar.gz https://golang.org/dl/go${VERSION}.${PLATFORM}.tar.gz

# Remove old Go
echo "Removing old Go version"
sudo rm -rf /usr/local/go

# Install new Go
sudo tar -C /usr/local -xzf go"${VERSION}".${PLATFORM}.tar.gz
echo "Creating the skeleton for your local users go directory..."
mkdir -p ~/go/{bin,pkg,src}
# echo "Setting up GOPATH"
# echo "export GOPATH=~/go" >> ~/.profile && source ~/.profile
# echo "Setting PATH to include golang binaries"
# echo "export PATH='$PATH':/usr/local/go/bin:$GOPATH/bin" >> ~/.profile && source ~/.profile

# Cleanup
rm go"${VERSION}".${PLATFORM}.tar.gz

First Time#

If you’re installing for the first time, you can uncomment this blob from the above script:

# echo "Setting up GOPATH"
# echo "export GOPATH=~/go" >> ~/.profile && source ~/.profile
# echo "Setting PATH to include golang binaries"
# echo "export PATH='$PATH':/usr/local/go/bin:$GOPATH/bin" >> ~/.profile && source ~/.profile

Mofified it from a gist I found here.

Creating a Table of Images in Markdown

2022-05-02T00:00:00+00:00

<table>
  <tr>
    <td><img src="image.png" width="200" style="display: inline" /></td>
    <td><img src="image.png" width="200" style="display: inline" /></td>
  </tr>
  <tr>
    <td><img src="image.png" width="200" style="display: inline" /></td>
  </tr>
</table>

Resolving Pritunl Connection Issues with a Custom DNS Server

2022-05-02T00:00:00+00:00

Problem#

I faced this issue when I disabled systemd-resolved on my system (Ubuntu 20.04) but Pritunl tried to update the DNS entry via systemd-resolvd and it failed when the service wasn’t running.

sudo systemctl stop systemd-resolved         
sudo systemctl disable systemd-resolved

Logs where it tried to look for the service:

tail -f ~/.config/pritunl/profiles/*
<14>Mar 10 23:54:25 a65e97e8075b8b500c5e9cfe29083e4d-up.sh: Link 'tun0' coming up
<14>Mar 10 23:54:25 a65e97e8075b8b500c5e9cfe29083e4d-up.sh: Adding IPv4 DNS Server 1.1.1.1
<14>Mar 10 23:54:25 a65e97e8075b8b500c5e9cfe29083e4d-up.sh: SetLinkDNS(30 1 2 4 1 1 1 1)
Call failed: Unit dbus-org.freedesktop.resolve1.service not found.
<8>Mar 10 23:54:25 a65e97e8075b8b500c5e9cfe29083e4d-up.sh: 'busctl' exited with status 1
Wed Mar 10 23:54:25 2021 WARNING: Failed running command (--up/--down): external program exited with error status: 1
Wed Mar 10 23:54:25 2021 Exiting due to fatal error

You can see it fails at the step Call failed: Unit dbus-org.freedesktop.resolve1.service not found.

Solution#

The solution is to disable systemd-resolved at the NetworkManager level. You need to add the following config block which overrides these settings at /etc/NetworkManager/NetworkManager.conf:

[main]
...
dns=default
systemd-resolved=false
...

Restart network-manager:

sudo systemctl restart NetworkManager

Now, when you try to connect to Pritunl, it will work fine :)

Generating a Million Rows of Random Data in PostgreSQL

2022-05-02T00:00:00+00:00

I wanted a quick way to generate some random data in a PostgreSQL database and insert 1mn rows.

The following query shows how to generate a UUIDv4 and insert 1mn rows:

INSERT INTO table (uuid, name)
SELECT uuid_in(overlay(overlay(md5(random()::text || ':' || clock_timestamp()::text) placing '4' from 13) placing to_hex(floor(random()*(11-8+1) + 8)::int)::text from 17)::cstring), 'load tests'
FROM generate_series(1, 1000000);

Resetting the WAL Log in PostgreSQL After a Crash

2022-05-02T00:00:00+00:00

In an event of a DB crash or an unclean shutdown (host node failure), sometimes Postgres cannot find a valid checkpoint to start. The logs look like:

2021-02-16 04:09:44.977 UTC [26] LOG:  database system was shut down at 2021-02-16 03:50:02 UTC
2021-02-16 04:09:44.978 UTC [26] LOG:  record with incorrect prev-link 1DF0000/7BB0000 at 0/D786C2B0
2021-02-16 04:09:44.978 UTC [26] LOG:  invalid primary checkpoint record
2021-02-16 04:09:44.978 UTC [26] PANIC:  could not locate a valid checkpoint record
2021-02-16 04:09:45.252 UTC [25] LOG:  startup process (PID 26) was terminated by signal 6: Aborted
2021-02-16 04:09:45.252 UTC [25] LOG:  aborting startup due to startup process failure
2021-02-16 04:09:45.254 UTC [25] LOG:  database system is shut down

The line to notice here is:

2021-02-16 04:09:44.978 UTC [26] PANIC:  could not locate a valid checkpoint record

In such a scenario, perform a WAL Log Reset with pg_resetwal:

Quoting from the docs:

pg_resetwal clears the write-ahead log (WAL) and optionally resets some other control information stored in the pg_control file. This function is sometimes needed if these files have become corrupted. It should be used only as a last resort, when the server will not start due to such corruption.

$ su postgres
$ pg_resetwal /var/lib/postgresql/data

Postgres should initialise now. But as the docs clarify further:

After running this command, it should be possible to start the server, but bear in mind that the database might contain inconsistent data due to partially-committed transactions. You should immediately dump your data, run initdb, and reload. After reload, check for inconsistencies and repair as needed.

Converting Python Requests to cURL Commands

2022-05-02T00:00:00+00:00

$ pip install curlify
...
import curlify
print(curlify.to_curl(r.request))  # r is the response object from the requests library.

Karan Sharma - Today I Learned

Using slog for logging in Go

Initialise a base logger#

Logging SMTP connection timings

Golang: The Loop Variable Trap

The Issue#

The Reason#

The Solution#

Conclusion#

Resolving UUID Conflicts when Mounting AWS EBS Volumes

Check UUID#

Clean Up Outdated Local Git Branches

Docker CLI Snippets

Remove images locally with a wildcard:#

Remove images using xargs#

Dockerhub Rate Limits#

Docker Disk Usage#

Stop all containers#

Useful Terraform Snippets for AWS

Get the list of AWS API actions required to run terraform#

Delete all resources in a state#

Golang Gotchas

HTTP Responses#

Working with cloud-init in AWS

Run the script manually#

To view logs#

To output logs to a different file#

Bash Scripting Templates: From Basic to Advanced

Preferred Version#

Elaborate Version#

Useful ClickHouse Query Snippets

Check Table Size#

Access Management#

Listing#

Show Mutations#

Pending#

Cardinality#

Useful Git commands

Set the editor#

Delete all local branches#

Delete all branches except a few#

Delete a remote branch#

Force merge a branch#

Extracting Artifacts from Docker with Buildx

AWS CLI Snippets

Fetch a list of running instances#

Size of S3 Bucket#

Format List Output#

Network usage of EC2 Instance#

Using the Nomad API to Manage Allocations

List allocations#

Filter#

Working with the Linux Filesystem

Check disk size of a directory#

List files with modification timestamp#

Nomad Template Snippets

Service List#

Testing Open File Descriptors Limit in Golang

Running#

Exceeding the Limit#

Python Version#

Handling Open File Descriptors Limit in Python

How to Identify the EBS Volume ID on an AWS EC2 Instance

Understanding AWS EC2 Pricing: A Quick Reference Guide

Controlling Screen Brightness in Linux

Get a list of devices connected#

Tweak the brightness value#

Creating a MacOS-Like Screenshot Effect in Linux

Converting UTC to Local Time in Python

Usage#

Alias#

Generating Random Strings in Shell Scripts

Extracting Filename in Bash

Get the filename with extension#

Remove the extension#

Counting Lines in Files with Bash

Analyzing Most Frequently Used Shell Commands

How to View CSV Files in Terminal

Useful cURL Snippets

Show only headers#

Get the list of AWS API actions required to run `terraform`#

Fetch a list of `running` instances#