Just another fancy blog.

Setting Up a Terragrunt with tfEnv, SOPS, TFLint, and pre-commit

Sun, 04 Aug 2024 17:49:09 GMT

Setting up a Terragrunt repository effectively maybe it’s hard, but it’s crucial for maintaining a clean, secure, and efficient infrastructure-as-code workflow. With this post, I wanted to share a quick tips, how in project maintained by me, through few years of experimentation I was able to structuring Terragrunt repository and configuring it with SOPS for secrets management, TFLint for Terraform linting, Pre-commit hooks for maintaining code quality, Tofuutils/Tenv for environment management and more.

Repository Structure

First things, firsts. Before we start going through entire process of setting up full scale Terraform project, I want to point, what additional programs and wrappers I use in my daily projects.

Recommended stack at start of the project:

tfenv / tenv for keeping same version of terraform in entire project
terragrunt is a state and variables maintaining terraform wrapper
tflint for Terraform modules linting
tfsec to keep the eye for any dangerous missed/misused resource configuration
sops because nobody is perfect and some variables can’t be stored in plain-text
pre-commit to make sure, that pushed changes are not malformed or need correcting after upload

Having that in mind, here’s an overview of the recommended repository structure:

. (root)
├── .pre-commit-config.yaml
├── .tflint.hcl
├── modules
│   ├── resource-group
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── output.tf
├── subscriptions
│   ├── .terraform-version
│   ├── global.hcl
│   ├── sops.global.enc.yml
│   ├── terragrunt.hcl
│   ├── nonprd
│   │   ├── sub.hcl
│   │   ├── sops.sub.enc.yml
│   │   ├── dev
│   │   │   ├── environment.hcl
│   │   │   ├── sops.environment.enc.yml
│   │   │   └── resource-group
│   │   │       └── terragrunt.hcl
│   │   └── tst
│   │       ├── environment.hcl
│   │       ├── sops.environment.enc.yml
│   │       └── resource-group
│   │           └── terragrunt.hcl

Setting up basics

For start we need to create few files and folders to ensure, that future state maintaining through Terragrunt and sorted by folders tree will reflect future project structure, used regions, environments and other splits of used infrastructure.

. (root)
├── modules
│   ├── resource-group
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── output.tf
├── subscriptions
│   ├── global.hcl
│   ├── terragrunt.hcl
│   ├── nonprd
│   │   ├── sub.hcl
│   │   └── dev
│   │       ├── environment.hcl
│   │       └── resource-group
│   │            └── terragrunt.hcl

First, and most important file

The most important file is surely subscriptions/terragrunt.hcl. This file store all necessary information about project, build variable, connect to right cloud provider and keep connect to a state files holding information about current IaC.

File itself is split into four parts, each providing different purpose.

locals take care of finding and using variables spread through project structure
provider take care of the connection to the right project subscription
remote_state is the set of the instructions, where and how to store project terraform state files
inputs provide all read variables as default inputs to simplifies future use of modules in the project

locals {
  global_vars      = read_terragrunt_config(find_in_parent_folders("global.hcl"))
  sub_vars         = read_terragrunt_config(find_in_parent_folders("sub.hcl"))
  environment_vars = read_terragrunt_config(find_in_parent_folders("environment.hcl"))

  sops_global_map      = try(yamldecode(sops_decrypt_file(find_in_parent_folders("sops.global.enc.yml"))), {})
  sops_sub_map         = try(yamldecode(sops_decrypt_file(find_in_parent_folders("sops.sub.enc.yml"))), {})
  sops_environment_map = try(yamldecode(sops_decrypt_file(find_in_parent_folders("sops.environment.enc.yml"))), {})

  resource_name_prefix = "${local.global_vars.locals.project_name}-${local.environment_vars.environment}"
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "azurerm" {
  features {}
  subscription_id = "${local.sub_vars.locals.subscription_id}"
}
EOF
}

remote_state {
  backend = "azurerm"
  generate = { path = "backend.tf", if_exists = "overwrite_terragrunt" }
  config = {
    subscription_id      = local.sub_vars.locals.subscription_id
    resource_group_name  = local.sub_vars.locals.terraform_resource_group
    storage_account_name = local.sub_vars.locals.storage_account
    container_name       = "terraform"
    key = "${path_relative_to_include()}/terraform.tfstate"
  }
}

inputs = merge(
  local.global_vars.locals, local.sub_vars.locals, local.environment_vars.locals,
  local.sops_global_map, local.sops_sub_map, local.sops_environment_map,
  {
    resource_name_prefix        = local.resource_name_prefix
    project_tags = merge({
        Environment   = local.environment_vars.environment
        IaC           = "Terraform"
      },
      lookup(local.subscription_vars.locals, "subscription_project_tags", {}),
      lookup(local.environment_vars.locals, "environment_project_tags", {}),
    )
  }
)

Specify the Terraform version

To ensure consistency across different environments, the best approach is to set one version of used program in the project. For this, it’s recommended to use tfenv or tenv (Fork of tfenv, supporting also OpenTOFU and Windows system). Setting of it is pretty simple. In the subscriptions folder, we need to create file .terraform-version. This file contains version, that is recommended to use in project. Everytime, that terragrunt use terraform version control, it look at this file and pick up right binary.

1.10.0

Variable files

To keep repeating variables in one place, we need to create those files:

subscriptions/global.hcl keep configurations that apply to all environments within the repository.
subscriptions/nonprd/sub.hcl Specific configurations for the non-production subscription.
subscriptions/nonprd/dev/environment.hcl Specific configurations for the only this environment.

After loading them into main subscriptions/terragrunt.hcl, they are generally viable in every default inputs field of used module. By this case, only additional inputs, that are required for working module are dependencies ones. Good example of variable simplifications is file subscriptions/nonprd/dev/resource-group/terragrunt.hcl showed below.

include {
  path = find_in_parent_folders()
}

terraform {
  source = "../../../../modules//resource-group"
}

Using SOPS for Secrets Management

SOPS is used to manage secrets securely. Encrypted files like sops.global.enc.yml, sops.sub.enc.yml, and sops.environment.enc.yml store sensitive data. To decrypt and use these files, user should usually have permissions to read keyvault keys on provided cloud or could verify its GPG fingerprint registered in SOPS files.

Maintain code quality with pre-commit

To ensure that code quality checks are performed before any commits, it’s good to setup right git hooks. Provided configuration below ensure, that code is formatted, keep good code structure and integrity, fixing minor git problems, and finally cleaning up cache so even other systems used in projects like ARM, x32, x86 could still run terragrunt commands without any additional added compiler steps.

repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: 1.92.1 # Get the latest from: https://github.com/antonbabenko/pre-commit-terraform/releases
    hooks:
      - id: terraform_fmt
      - id: terragrunt_fmt
#      - id: terraform_tfsec # Can be turned on later, in advanced stage of infrastructure
      - id: terraform_tflint
        args:
          - >
            --args=
            --color
            --config=__GIT_WORKING_DIR__/.tflint.hcl
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: fix-encoding-pragma
      - id: destroyed-symlinks
      - id: check-yaml
        args: [--allow-multiple-documents]
      - id: sort-simple-yaml
      - id: end-of-file-fixer
      - id: trailing-whitespace
  - repo: local
    hooks:
      - id: cache-clean-up
        name: cache-clean-up
        entry: bash -c 'find . -name ".terragrunt-cache" -type d -print0 | xargs -0 /bin/rm -fR && find . -name ".terraform.lock.hcl" -type f -print0 | xargs -0 /bin/rm -fR && exit 0'
        language: system
        types: [file]
        pass_filenames: false
        always_run: true

Later, to simply start using it, we can tun this command

pre-commit install

Lint your Terraform code according to best practices

TFLint is a great tool to keep code in good shape. There is a lot of guides how to set and use it, but for me this .tflint.hcl settings holding the most sense and not restricting writer too much.

config {
  force = false
}

plugin "azurerm" {
  enabled = true
  version = "0.20.0"
  source  = "github.com/terraform-linters/tflint-ruleset-azurerm"
}

# Disallow deprecated (0.11-style) interpolation
rule "terraform_deprecated_interpolation" {
  enabled = true
}

# Disallow legacy dot index syntax.
rule "terraform_deprecated_index" {
  enabled = true
}

# Disallow variables, data sources, and locals that are declared but never used.
rule "terraform_unused_declarations" {
  enabled = true
}

# Disallow // comments in favor of #.
rule "terraform_comment_syntax" {
  enabled = false
}

# Disallow output declarations without description.
rule "terraform_documented_outputs" {
  enabled = true
}

# Disallow variable declarations without description.
rule "terraform_documented_variables" {
  enabled = true
}

# Disallow variable declarations without type.
rule "terraform_typed_variables" {
  enabled = true
}

# Disallow specifying a git or mercurial repository as a module source without pinning to a version.
rule "terraform_module_pinned_source" {
  enabled = true
}

# Enforces naming conventions
rule "terraform_naming_convention" {
  enabled = true

  variable {
    format = "snake_case"
  }

  locals {
    format = "snake_case"
  }

  output {
    format = "snake_case"
  }

  resource {
    format = "snake_case"
  }

  module {
    format = "snake_case"
  }

  data {
    format = "snake_case"
  }
}

# Require that all providers have version constraints through required_providers.
rule "terraform_required_providers" {
  enabled = true
}

# Require that all providers are used.
rule "terraform_unused_required_providers" {
  enabled = true
}

# Ensure that a module complies with the Terraform Standard Module Structure
rule "terraform_standard_module_structure" {
  enabled = true
}

# terraform.workspace should not be used with a "remote" backend with remote execution.
rule "terraform_workspace_remote" {
  enabled = true
}

# Disallow terraform declarations without require_version.
rule "terraform_required_version" {
  enabled = false
}

Conclusion

By following this structure and using the specified tools, it’s easier to maintain a clean, secure, and efficient Terragrunt repository. Pre-commit hooks ensure code quality, tfEnv keep eye on use of correct terraform version, TFLint enforces best practices, and SOPS manages secrets securely.

Trip to Japan: Booking multiple connections tickets

Thu, 23 Feb 2023 17:50:34 GMT

In few weeks I plan to go take a flight to Japan. But to spare some money, I plan to use multiple connections over direct flights - what it also comes with some disadvantages.

Flying from my initial airport in Bangkok (Thailand), I plan to go to Japan via Ho Chi Minh (Vietnam) airport. However, there is a catch in booking system. If you have booked two separate tickets for your journey and need to transit at the Vietnam airport as me, you need to fulfill additional visa requirements.

Firstly, it is crucial to check the visa requirements for Vietnam before you book your tickets. Depending on your nationality and the length of your stay in Vietnam, you may need to apply for a visa in advance or obtain a visa on arrival. It is important to note that if you are transiting through Vietnam without leaving the airport, you may still require a Vietnam Transit Visa ($25 fee).

As official articles say, if you are transiting in an airport in Vietnam and omitting all Vietnam Transit Visa acquire problems, you need to fulfill this steps:

You have a ticket for a connecting flight within the next 24 hours
You will not leave the airport international transit area
You have the necessary documents for your final destination (passport, visa, etc) On the other hand, if your connecting flight is further than 24 hours away, or you would like to enter Vietnam while you wait, you will have to get a visa. This rule excludes all booked flights booked as separate tickets!

So, when booking separate tickets for your journey, it is important to leave sufficient time for transit between flights. If your first flight is delayed or canceled (happen a lot in SEASIA flights), you may miss your connecting flight by spending crucial time on going through security and immigration checks.

Also, important note about luggage: If you have it checked, it is important to confirm with the airline whether your bags will be automatically transferred to your connecting flight or whether you will need to collect them and re-check them - it’s possible that airlines have inner agreement to transfer your luggage between their flights like Vietnam Airlines <-> Etihad. If you need to collect and re-check it yourself, you have to go through customs and immigration again, as conveyer belt and international departure zone are on a different levels on airport - requiring you to leave international flight zone.

In summary, flying to Japan via a Vietnam airport can be a convenient and cheap option for travelers. However, if you have booked tickets for your journey, it is important to if you need to apply for Vietnam Transit Visa, leave sufficient time for transit, and be prepared for security and immigration checks. By following these tips, you can ensure a smooth travel experience and enjoy your trip.

How I discovered, that remote work not only allows you to work from home

Mon, 26 Dec 2022 17:50:34 GMT

It’s been a while since I last posted an update on the blog, but believe me, a lot has changed. For a few months, I’ve been trying to sort out things related to a trip to rather far country. The reason? To visit my friend who decided to move from the Netherlands to other place for a while. The journey… Well, what was originally planned as a quick test of remote work as a 3-week vacation + 3 weeks of remote work, to then return to my home, fell to pieces and turned into quite a long stay for me in the very center foreign country (when all my plans changed to stay here for only 2 months, then 3 months until the Christmas, to maybe 6 months until the end of winter - and as today, to not have plan to return that quickly to my country and thinking how to stay here longer).

During this adventure, I met many interesting people. I also have to say that people in this country are very friendly. A longer stay here was also made possible by a closer encounter with culture, much closer to my heart and hobby. Also, my dear partner who makes my life enjoyable. That also could be influenced by just changing rather coldish environments to warmer ones.

I was also able to work remotely, despite the initial shift in working hours. I reduce this time by shifting my work hours to different part of the day, which corresponds to classic 7:00 a.m. to 5:00 p.m. office hours in my country. This is a good alternative for me because in the later hours of the day I can concentrate better and perform assigned tasks more quickly. I hope that in the future I will be able to arrange fully asynchronous work - but time will show. At the moment, I am very happy with the current result.

Automate Instagram feed with GramAddict

Fri, 17 Jun 2022 17:27:13 GMT

Sometimes I don’t have time to browse Instagram, and as you know - spent time on your account will definitely help your SEO and getting your online activity noticed by others. Previously, for this purpose, I used to use projects based on the browser in the background, but unfortunately the repository after the Instagram changes died and does not have as many updates (there is also a problem with accepting PR, even if I would like to help in the public repo).

However, after searching the internet, I found an alternative, which is for me an even better solution and more resistant to failures. I’m talking about GramAddict, which uses the debug of the android phone and performs operations directly on device. Using the phone connected to the computer that was executing the script, I admit that it helped me to focus more on my work and not get distracted by the phone.

During setup, I sometimes had the problem of the cable disconnecting and losing my data transfer settings (probably broken connectors in phone USB port). This was quite annoying, especially when the device charging itself was left unchanged when the connection was interrupted. The best solution I came up with was to simply connect to the debugger via Wi-Fi, where the phone can now lie in any corner of the house and scroll the feed in peace.

Having tested this software for over the month, I would recommend it. GramAddict has lots of configuration options, had no problem with new Instagram media formats and has additional modules for reporting and saving previous operations.

Joining #100DaysOfHomeLab Challenge

Sun, 12 Jun 2022 22:47:45 GMT

At the beginning of the month, following the words of my friend, I started looking into putting up a HomeLab server. After many instances during this short journey, I finally managed to make a server to put up my home services, which I needed for a long time.

While building the server, in many cases I used tutorials, guides as well as best practice on YouTube. Turns out I also hit on the perfect opportunity to develop my current work with new event. #100DaysOfHomeLab, an event created by Techno Tim as a 100k celebration video, encourages you to spend 1 hour a day for 100 days developing your home environment. For the foreseeable future we may see a collaboration of the largest accounts dedicated to building a HomeLab. Each of the creators with a different approach to technology and the tools used - which can each be adapted to our needs. Along with this event, I have likewise planned new work that will expand my current setup.

Planned activities for near future:

adding openHAB
automated backups
converting HomeLab to k3s

We’ll see how many of these things I can get done. I will also try to give updates from week to week on the development of the project. I think this is a great option for everyone to join in creating their own HomeLab solution. To the next update!

Building ad-block home server from the scratch. Again...

Fri, 28 Jan 2022 22:58:33 GMT

After the recent chaos I found on my Raspberry Pi server and its unstable performance I made a fresh attempt to build a simple ad filter in my home using rPi.

First, remember to change default password of pi user!

sudo raspi-config

Install PiHole from official script, set/reset password for web admin and add auto-update PiHole line to crontab

curl -sSL https://install.pi-hole.net | bash
pihole -a -p

crontab -e
# 0 5 * * * pihole -up

I have to admit that PiHole’s basic blacklists do a good job of filtering ads, but there are places on the web that require more url restriction (like scammer sites). I previously used firebog list for this and manually tried to add lists of blocked sites to the tool. However, as it turns out, someone smart has added their python script to automatically add the appropriate list option from both firebog (blacklist) and/or anudeepND/whitelist (whitelist).

Also consider adding blocklists focused on your home smart devices:

Xiaomi DNS blocklist

sudo pip3 install pihole5-list-tool --upgrade
pihole5-list-tool

#     ┌──────────────────────────────────────────┐
#     │       π-hole 5 list tool  v0.6.0         │
#     └──────────────────────────────────────────┘
#     https://github.com/jessedp/pihole5-list-tool
# 
# ! docker not found running, continuing...
# 
# ? Gravity Db to Update:  /etc/pihole/gravity.db
# 
#   Blocks Enabled:  All=0   │   Ours=0   │   Allows Enabled:  All=0   │   Ours=0 
# 
# ? Options:  Manage Blocklists
# ? Blocklist action:  Add a list
# ? Where are the block lists coming from?  Firebog | Non-crossed lists : Use when someone is usually around to allow
# 
#     Do not hit ENTER or Y if a step seems to hang!
#     Use CTRL+C if you're sure it's hung and report it.
#     
# ? Add 45 block lists?  Yes
# 45 block lists added! 0 already existed.
#   Ad/Blocklist Stats   │  Top 3 by Comment                          
# ───────────────────────┼────────────────────────────────────────────
#    Total     :  45/45  │   Firebog | Non-crossed lists [ph5lt]  45  
#    Our Lists :  45/45  │                                            
#    Others    :  0/0    │                                            
#                        │                                            
# ? Are you finished?  Yes
# ? Update Gravity for immediate effect?  Yes

The final step is to auto-update the rPi itself. I wanted to use a standard crontab with the update+upgrade command to do this, but the problem here may be the occurrence of a dialog windows. A better solution to avoid this problem and update your rPi server with security payloads is to use the UnattendedUpgrades library

apt-get install unattended-upgrades
dpkg-reconfigure -plow unattended-upgrades

This is the most basic server setup, which should take no more than 30min. But it is also a good basis for future activities with the system that is currently set up and running smoothly. In the future, if I add something as a “base image” for the home server, I will add additional commands in the end of this post.

Map all AWS resources and make a vulnerability report using cloudmapper

Fri, 28 Jan 2022 21:11:38 GMT

Last time I remembered about a tool that allows you to create a big map of current resources present in your AWS account, and based on it make a report of potential vulnerabilities, threats, unused resources and much more. I can thank a colleague at work for finding it - such explorers are a real treasure!

The duolabs/cloudmapper repository is responsible for scanning and mapping resources on the current user account. It is a great tool to see our mistakes when releasing the environment or purging the client account of unused, often manually created resources. Using the program is best done with building and using a docker image.

git clone https://github.com/duo-labs/cloudmapper.git
docker build -t cloudmapper .

AWS_PROFILE=default
docker run -it \
        -e AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id --profile ${AWS_PROFILE}) \
        -e AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key --profile ${AWS_PROFILE}) \
        -p 8000:8000 \
        cloudmapper /bin/bash

Then run this command inside docker shell aws sts get-caller-identity to get data about your running account.

{
    "UserId": "JNSNATUSNPWEBAA8HNAPE",
    "Account": "830572105813",
    "Arn": "arn:aws:iam::830572105813:user/project-env"
}

After confirming right data, pass your UserId into YOUR_ACCOUNT_NUMBER field in first command below (check also additional report commands for unused resources or IAM data)

python cloudmapper.py configure add-account --config-file config.json --name YOUR_ACCOUNT --id YOUR_ACCOUNT_NUMBER
python cloudmapper.py collect --account YOUR_ACCOUNT
python cloudmapper.py report --account YOUR_ACCOUNT
python cloudmapper.py prepare --account YOUR_ACCOUNT
python cloudmapper.py webserver --public

After successful data fetch, you can find your results on those links:

You can find examples of fetched AWS accounts under those links:

Secure files using git-crypt

Mon, 20 Dec 2021 23:50:13 GMT

There is one major problem when working on a public repository. Anyone can see our access data in our project. It is impossible to hide, at some point, the project may lack another layer securing the data, or even a simple .env file with the data required by the project.

Git-crypt comes in handy, in which we can encrypt and decrypt files automatically and upload them to the repository with each commit. Each user requires registration and entering their GPG key into the repository. People who do not have the key will not be able to decode the content of the file.

Configuration which files we want to encrypt:

secretfile filter=git-crypt diff=git-crypt
*.key filter=git-crypt diff=git-crypt
secretdir/** filter=git-crypt diff=git-crypt

Initialization and unlocking of the repository:

git-crypt init
git-crypt add-gpg-user USER_ID
git-crypt unlock

Using Github 'actions'

Thu, 09 Dec 2021 17:47:29 GMT

When creating pipelines, we often use already-made code we’ve written ourselves in previous projects. Whether it is CircleCi, BitBucket Pipelines, AWS CodeBuild or GitHub Actions. The problem is when our cloud environment updates and no longer has the old libraries we used.

This is what happened to my blog repository. It turns out that every time the LTS version of the NodeJS library is released, it is added to the GitHub pipelined image. What caused this? Status failed on my site build pipeline. The solution was to update the packages or lock for the nodejs version. I originally wanted to fix this with another 3 steps in workflow that would allow me to enable the old version of NodeJS (along with apt-get update and other commands). The problem is that such steps increase the execution time of the action terribly. Fortunately, it turned out that it was also possible to use the main GitHub Actions feature, the actions themselves. As it turns out, with the use of simple 3 lines, we can be sure that our environment will have the correct version of NodeJS, and changing it is only one parameter.

steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
  with:
    node-version: '14'
- run: npm install
- run: npm test

The use of template actions is definitely a big advantage of GitHub. It will definitely come in handy for everyone more than once as performing some kind of automation, version lock, testing the environment and much more.

Launch multiple WordPress server in docker way

Sun, 21 Nov 2021 19:23:17 GMT

If you want to run multiple WordPress sites and other services but don’t have time to configure each one with their specific library versions, you can set all in one big docker config. To run this all, you’ll need an app image itself, reverse proxy preferably with SSL automation and in most cases’ database.

Example below should launch multiple instances of sites, configure them to connect to local database running on host machine, allow reverse-proxy and maintain SSL certificates.

version: '3'
services:

  #  -----------------------------------------------
  #  | Set reverse proxy to handle traffic
  #  -----------------------------------------------
  reverse-proxy:
    image: jwilder/nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./certs:/etc/nginx/certs
      - /etc/nginx/vhost.d
      - /usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
    labels:
      - com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy
    restart: unless-stopped

  #  -----------------------------------------------
  #  | Set letsencrypt to menage SSL
  #  -----------------------------------------------
  letsencrypt-nginx-proxy-companion:
    image: jrcs/letsencrypt-nginx-proxy-companion
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./certs:/etc/nginx/certs:rw
      - /var/run/docker.sock:/var/run/docker.sock:ro
    volumes_from:
      - reverse-proxy
    restart: unless-stopped

#  -----------------------------------------------
#  | Set services
#  -----------------------------------------------
  wordpress_app1_name:
    image: wordpress:php7.2
    environment:
      - WORDPRESS_DB_HOST=host.docker.internal:3306
      - WORDPRESS_DB_USER=db1user
      - WORDPRESS_DB_PASSWORD=password
      - VIRTUAL_HOST=www.app1_host.com
      - LETSENCRYPT_HOST=www.app1_host.com
      - LETSENCRYPT_EMAIL=confirmation_email@app1_host.com
    volumes:
      - ./app_1_volume:/var/www/html
      - ./app_1_uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
    restart: unless-stopped
    
    
  wordpress_app2_name:
    image: wordpress:php5.6
    environment:
      - WORDPRESS_DB_HOST=host.docker.internal:3306
      - WORDPRESS_DB_USER=db1user
      - WORDPRESS_DB_PASSWORD=password
      - VIRTUAL_HOST=www.app1_host.com
      - LETSENCRYPT_HOST=www.app1_host.com
      - LETSENCRYPT_EMAIL=confirmation_email@app1_host.com
    volumes:
      - ./app_2_volume:/var/www/html
      - ./app_2_uploads.ini:/usr/local/etc/php/conf.d/uploads.ini
    restart: unless-stopped

Training: Tiny Python Projects

Sun, 17 Oct 2021 16:48:09 GMT

Lastly I found a small repository related to teaching book about Python. I must say, that even if project is directed to beginners in this language, doing that tasks is refreshing ever for more advanced people. So if you don’t know what to do or want to clear your mind, you can always go to Tiny Python Projects site, and pick quick task to train your skills.

Installing Lutris on Debian from A to Z

Sat, 16 Oct 2021 08:32:54 GMT

After Nth attempt I finally managed to install fully working Lustris on Debian. There were some ups and downs during the installation, but finally I was able to get a fully working Wine with VulcanAPI support.

First, time to introduce my old but strong potato machine. It’s durable (hole in the wall confirm it) ThinkPad T450s. So… It’s components are not great and not terrible - i5-5300U with Intel HD 5500. And that added me additional problems to the list. As you can see, Vulcan have only partial support for older CPU Intel generations, so it could cause problems if you are not aware off different installation of dxvk library. For basic installation of dxvk on Debian with only Intel graphics you can follow mesa driver guide that is required also by the Lutris.

When installing Lutris, be aware, that you need ALL required components of Wine! Even when Lustris manage to download its own Wine libs, it’s still using some system libraries. That’s why you should follow fully Wine Dependencies guide.

As I mention before, I had problem with Debian installation of Wine on my laptop. Main problem was caused by wrongly committed version to debian repository - and it’s common problem for past few months in Wine community…

sudo apt-get install --install-recommends winehq-staging

The following packages have unmet dependencies:
 winehq-staging : Depends: wine-staging (= 6.17~focal-1)
E: Unable to correct problems, you have held broken packages.

To ‘fix’ it, just enter to download site of Wine, and download and install winehq-staging .deb package manually. Causing problem could occur by missing libfaudio0 library, which could not be installed with package manager… But thanks to this post on Ubuntu forum, I found open repository containing right library package.

After this, I finally could play my games running on DirectX 9 (But translated in air by Vulcan, which give additional 50% boost). So yea, game was pretty much playable with stable 20 fps (and with later expansion 17fps), on the lowest settings - but I didn’t expect much, and I was glad that it could ever work on this device. But I have one conclusion - If you want a Linux distro for gaming proposes, you should install Ubuntu or Pop!OS… It’s easier to set up.

Fixing my e-bike's speedometer and battery

Sun, 26 Sep 2021 21:47:27 GMT

In a word of introduction, I am the owner of an e-bike with a Bosch motor - as far as I can tell, it is the second version, powered by BatteryPack Classic 400 battery. I use the bike every day, and it allows me to get to many distant places in the city without much fatigue. Unfortunately, serving electronics for this type of equipment is a problem, and we are often left alone with repairing it. And in recent days one of the components failed.

But how did it happen? During the last transport of the bike I saw that the sensor of speedometer couldn’t send data to my control unit, and because of this, the motor on my bike stopped working. As it turns out, the magnet that is read by the sensor should be at the height of the line marked on the sensor itself. After a day of searching, I found this information on one of the YT videos

Unfortunately, correcting the magnet did not solve the problem. The second point was that my battery in the Bosch BatteryPack Classic 400 suddenly shut down, and I could not read the status of the current draw from the LED lights, and the charger would not allow me to recharge the battery. It took me a long time to find the cause, but I found a link on one of the bike forums of a person with a similar problem. One of the causes of a faulty battery could be that it is protected from shorting out or receiving too high/low a temperature. When battery gets stuck and can’t show charging status, just hold status button on it for 30s to give this component a soft reset. This way we can get the battery back in working order.

Given method fixed my bike, and now I can power up by control unit + read speed, so I can continue to enjoy riding. I wanted to write this note here because it was hard for me to find information how to fix battery problem. I don’t wish anyone to have this problem, but I will be glad if one day this post will help someone to solve this with his own equipment.

Check also this site to solve most common battery problems

Format the code automatically before commit

Fri, 24 Sep 2021 14:56:53 GMT

There is a situation where we forget to format our code to one standard before submitting changes. But what if git does it for us?

There are a couple of solutions, like plugging directly into git hooks commands, or using a framework like https://pre-commit.com/ which will visually help us check out the project, and add code checking plugins for us.

# Format code with terraform & terragrunt standard
terragrunt hclfmt
terraform fmt --recursive

# Add missing new-line in files
git ls-files -z | while IFS= read -rd '' f; do tail -c1 < "$f" | read -r _ || echo >> "$f"; done

It’s especially useful when we need to run a lot of commands responsible for testing, linter code and checking integration

Sync AWS CodeCommit with Bitbucket by running pipeline

Sun, 01 Aug 2021 23:01:48 GMT

If you don’t have access to the official integration of the two apps and their repositories, just do your own sync.

Set SSH

To prepare your repositories, enable pipelines in your main repository. Then in repository options let Bitbucket create new ssh key, that will be stored in /opt/atlassian/pipelines/agent/ssh/id_rsa path. Try to add generated public key to IAM user AWS CodeCommit keys in Security credentials tab in AWS Console

Create config file

Create new config that will be later used in our pipeline with: nano ~/.ssh/configBitbucket

Host git-codecommit.*.amazonaws.com
 Hostname git-codecommit.eu-central-1.amazonaws.com
 User YOUR_USER_SSH_KEY
 IdentityFile /opt/atlassian/pipelines/agent/ssh/id_rsa

Later try to encode it with: base64 ~/.ssh/configBitbucket

Set Environment Variables

Now with base64 result, use it in Bitbucket Environment Variables

CodeCommitConfig: OURBASE64CONFIG
CodeCommitHost: git-codecommit.aws-region.amazonaws.com
CodeCommitRepository: git-codecommit.aws-region.amazonaws.com/v1/repos/sample
CodeCommitUser: SSHUSERPUBLIC_KEY

Create Pipeline Step

Provided env’s will be used in pipeline step below

definitions: 
  steps:
    - step: &sync-codecommit
        name: Sync changes to CodeCommit
        script:
          - echo $CodeCommitConfig > ~/.ssh/config.tmp
          - base64 -d  ~/.ssh/config.tmp > ~/.ssh/config
          - cat ~/.ssh/config
          - set +e
          - ssh -o StrictHostKeyChecking=no $CodeCommitHost
          - set -e
          - git remote add codecommit ssh://$CodeCommitRepository
          - git push codecommit $BITBUCKET_BRANCH --force

pipelines:
  default:
    - step: *sync-codecommit

Now, whenever you push to Bitbucket, it will also push to CodeCommit. For more details and better step-by-step configuration you can check also this Medium post

Set access to k8s thru RBAC Token

Sun, 01 Aug 2021 20:28:38 GMT

Recently, I was trying to find a suitable way to share the k8s logs with the development team. After searching the internet, I finally found a suitable and current solution for the new kuberentes versions. With a little help and a mini-customization of the code, I present the result for everyone who will need help. Let the code be with you!

(Just watch out for TLS auth… Soo… To be fixed later… ) (ó﹏ò｡)

#!/bin/bash
if [ -z ${CLUSTERNAME} ] || [ -z ${NAMESPACE} ] || [ -z ${USERNAME} ] || [ -z ${K8S_CONTEXT} ]; then echo Usage: K8S_CONTEXT=admin.context CLUSTERNAME=project.name NAMESPACE=default USERNAME=dev ./script.sh && exit 1; fi;

# Command with context to admin user
KUBE_COMMAND="kubectl --context $K8S_CONTEXT"

$KUBE_COMMAND delete serviceaccount/$USERNAME-sa
$KUBE_COMMAND delete clusterrole.rbac.authorization.k8s.io/$USERNAME-cr
$KUBE_COMMAND delete rolebinding.rbac.authorization.k8s.io/$USERNAME-rb

cat <<EOF | $KUBE_COMMAND apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: $USERNAME-sa
  namespace: $NAMESPACE
EOF

# Set here what resources are allowed to enter
cat <<EOF | $KUBE_COMMAND apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: $USERNAME-cr
rules:
- verbs: ["get", "list", "watch"]
  resources: 
  - pods
  - deployments
  - jobs
  - cronjobs
  - pods/log
  apiGroups: ["", "apps", "batch"]
EOF


cat <<EOF | $KUBE_COMMAND apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: $USERNAME-rb
  namespace: $NAMESPACE
subjects:
- kind: ServiceAccount
  name: $USERNAME-sa
  namespace: $NAMESPACE
roleRef:
  kind: ClusterRole
  name: $USERNAME-cr
  apiGroup: rbac.authorization.k8s.io
EOF

TOKEN=$($KUBE_COMMAND describe -n default secrets "$($KUBE_COMMAND describe -n default serviceaccount $USERNAME-sa | grep -i Tokens | awk '{print $2}')" | grep token: | awk '{print $2}')

kubectl --kubeconfig=kubeconfig-$CLUSTERNAME config set-cluster $CLUSTERNAME --server=$($KUBE_COMMAND config view -o jsonpath='{.clusters[0].cluster.server}')
kubectl --kubeconfig=kubeconfig-$CLUSTERNAME config set-credentials $CLUSTERNAME-$USERNAME --token=$TOKEN --kubeconfig=kubeconfig-$CLUSTERNAME
kubectl --kubeconfig=kubeconfig-$CLUSTERNAME config set-context $CLUSTERNAME --cluster=$CLUSTERNAME --user=$CLUSTERNAME-$USERNAME
kubectl --kubeconfig=kubeconfig-$CLUSTERNAME config use-context $CLUSTERNAME

kubectl --kubeconfig=kubeconfig-$CLUSTERNAME --insecure-skip-tls-verify auth can-i get pods --namespace $NAMESPACE

GitFlow & Versioning

Thu, 01 Jul 2021 20:41:32 GMT

Trying to keep the branch in order, we can use the GitFlow formula and adjust our tactics for future product releases. Each version and its important aspect has its own initials and ever additional values as release candidates. In addition, the tagging system and its automation ensure the integrity of project tags even in the event of unwanted but often used hotfixes of the main branch.

The defined version is especially needed when releasing the application issuing the API because specifying this source can tell us about possible fixed things, new functionalities, or a new version incompatible with the previous one.

To be sure what version our code change should be, we can keep an additional changelog CHANGELOG.md which will help people find out what things are currently in the branch to be released and what things have been changed/fixed/added in recent commits

Tags Automation

The script below can be a nice help as it downloads the latest tags and then adds new ones as requested by the user. It is suggested to create and post tags on the target branch

#--------------------------------- Git ----------------------------------------------------#
GIT_LAST_TAG = $(lastword $(shell git tag --sort=taggerdate))
GIT_VERSION := $(subst -RC, , $(subst v,, $(subst ., ,$(GIT_LAST_TAG))))
GIT_MAJOR := $(word 1, $(GIT_VERSION))
GIT_MINOR := $(word 2, $(GIT_VERSION))
GIT_PATCH := $(word 3, $(GIT_VERSION))
GIT_RC := $(word 4, $(GIT_VERSION))
#--------------------------------- Git ----------------------------------------------------#

tag-major:
ifeq ($(GIT_RC),)
	$(eval tag = v$(shell echo $(GIT_MAJOR) + 1 | bc).0.0)
else
	$(eval tag = v$(GIT_MAJOR).0.0)
endif
	git tag $(tag) -m $(tag)

tag-major-rc:
ifeq ($(GIT_RC),)
	$(eval tag = v$(shell echo $(GIT_MAJOR) + 1 | bc).0.0-RC1)
else
	$(eval tag = v$(GIT_MAJOR).0.0-RC$(shell echo $(GIT_RC) + 1 | bc))
endif
	git tag $(tag) -m $(tag)

tag-minor:
ifeq ($(GIT_RC),)
	$(eval tag = v$(GIT_MAJOR).$(shell echo $(GIT_MINOR) + 1 | bc).0)
else
	$(eval tag = v$(GIT_MAJOR).$(GIT_MINOR).0)
endif
	git tag $(tag) -m $(tag)

tag-minor-rc:
ifeq ($(GIT_RC),)
	$(eval tag = v$(GIT_MAJOR).$(shell echo $(GIT_MINOR) + 1 | bc).0-RC1)
else
	$(eval tag = v$(GIT_MAJOR).$(GIT_MINOR).0-RC$(shell echo $(GIT_RC) + 1 | bc))
endif
	git tag $(tag) -m $(tag)

tag-patch:
ifeq ($(GIT_RC),)
	$(eval tag = v$(GIT_MAJOR).$(GIT_MINOR).$(shell echo $(GIT_PATCH) + 1 | bc))
else
	$(eval tag = v$(GIT_MAJOR).$(GIT_MINOR).$(GIT_PATCH))
endifgit tag $(tag) -m $(tag)

tag-patch-rc:
ifeq ($(GIT_RC),)
	$(eval tag = v$(GIT_MAJOR).$(GIT_MINOR).$(shell echo $(GIT_PATCH) + 1 | bc)-RC1)
else
	$(eval tag = v$(GIT_MAJOR).$(GIT_MINOR).$(GIT_PATCH)-RC$(shell echo $(GIT_RC) + 1 | bc))
endif
	git tag $(tag) -m $(tag)
	git tag $(tag) -m $(tag)

tag-patch-rc:
ifeq ($(GIT_RC),)
	$(eval tag = v$(GIT_MAJOR).$(GIT_MINOR).$(shell echo $(GIT_PATCH) + 1 | bc)-RC1)
else
	$(eval tag = v$(GIT_MAJOR).$(GIT_MINOR).$(GIT_PATCH)-RC$(shell echo $(GIT_RC) + 1 | bc))
endif
	git tag $(tag) -m $(tag)

Docker Images Versioning

When it comes to updating the version with Docker images, the best example is checking libraries that customers do not necessarily want to have in the latest version. One example is the python image.

When we look at the image side, we see a description of the shared, simplified, and individual tags. What’s the difference and what version is best to put in our Dockerfile?

Let’s assume that our project is currently using the newest python version 3.7.4. If we want the latest updates from the code side, we have several options. Downloading latest, 3, and 3.8 tags. Depending on how often we will update the code, we want to protect ourselves against possible “code deprecation” that may occur in future versions. narrowing down the tag versions to download specific trees can prevent us from getting the latest changes in the library, but also protect us against possible old functionalities removed (and even against changing to the full version of the library after a few years)

Therefore, it can be said that the safest version for the production environment is setting tags targeting patches (like python:3.8), and the development version for the minor or even major version (like python:3 or ever python:latest) - due to the possibility of testing even the expected changes in the used library, which can then be safely updated in production

Git Merging Strategies

Wed, 30 Jun 2021 18:23:03 GMT

Git is a great tool for managing your code version. It is also a tool where we can easily draw figures such as triangles, trapezoids, and other figures. How to find yourself on a railroad crossing? Try not to get lost at our train station?

Simpler branches with `git rebase`

When creating the code, we don’t pay attention to what our branch looks like. It is logical, after all, the code and changes are only ours, even if we have many commits - and you know, it’s better to have changes saved in the cloud than locally on the disk. On the other hand, at the time of merging, it causes some kind of discomfort for the committer and the code merger. In order not to bury ourselves in a thousand commits, before sending our final changes, we can use some useful commands that will improve the visibility of our branch The appearance of the tree after merging, depending on the method used

The most classic method of combining changes is classic merge. Executing a simple command is an ideal option for projects carried out by one-person teams. However, it has one problem, when detecting conflicts between two branches, it forces checking the data on both sides for each commit that the user committed and comparing the changes since disconnected from the parent branch.

git checkout master
git merge topic

To limit the number of visible changes after merging branches, you can use the squash method when merging branches. It causes that all the commits that we have done so far on the branch topic will be rewritten and merged as one new commit. this means that the displayed history of our changes will have only one summary commit in the tree.

git checkout master
git merge topic --squash

The last operation requires slightly more effort, but its result in the change tree looks the clearest. Especially in repositories with a lot of changes and more people working on it. This method is to overwrite the commit and the entire branch topic before submitting the changes for merge request/pull request/code review. First of all, in this method, we are sure that the uploaded changes are the latest from the branch we use as a parent. Secondly, it gives us control over the content of the commits by enabling them to be edited and arranged appropriately. The third argument is a very short tree branch that we will create after merging. At best, this branch will create a clear triangle of changes - that is, reading the changes from the parent’s latest commit, uploading our changes, and confirming the branch connection. Building such a branch is much clearer than scrolling through the entire page of a trapezoidal merged branch changes (changes that started, for example, 18 commits ago, but adding only one line of code in today’s commit). In addition to readability, the committing person is also sure that the changes he/she uploaded are up-to-date and will not have conflicts with the parent branch, because we already have the latest changes.

git checkout topic
git rebase --interactive HEAD~X # X is the number of commits that we want to rewrite (to be squashed or message edited)
# more about rebase and its use-cases https://git-scm.com/docs/git-rebase#_interactive_mode
git fetch origin/master # get lastest master commit without checking out to it
git rebase origin/master # rebase (rewrite) all commits on a branch to a new place above the lastest master changes
git push topic --force # rewritten branch must be pushed with force because it has whole new history and parent ID

Easy and fast file sharing from the command-line

Sat, 17 Apr 2021 23:17:32 GMT

Looking for a quick and easy-to-use option to share and send files, I came across transfer.sh. As it turns out, this is an ideal form of file sharing like WeTransfer or mega.nz, but for sending files via CLI. Obviously, the best option for sending files is direct data transfer using SCP or rsync. But for Polish people that we can go public or servers that don’t have direct IP or exit, this is an interesting option.

When it comes to sending files, curl is enough for us

# Upload using cURL 
$ curl --upload-file ./hello.txt https://transfer.sh/hello.txt 
https://transfer.sh/66nb8/hello.txt

On the website https://transfer.sh/ we can see many options for using the tool. We have automatic packing of files to tar.gz or zip archives, examples of data encryption before uploading, malware scanning, use of https://keybase.io/, and much more. The project even has a website at http://jxm5d6emw5rknovg.onion/

Not knowing Docker image configuration? Try runalike

Sat, 17 Apr 2021 19:34:54 GMT

There is a chance that after entering the server, we do not know with what command docker image was launched. The docker inspect will certainly come in handy, but the same one can little be unreadable - or it just requires the unreasonable rewriting of all values of the indicated result. Therefore, a simple image of assaflava / runlike was created which makes the work easier. The approach to the painting is simple. It is required to connect the socket docker and indicate the image. The output will be a printed result presented as a complete docker run command including all switches

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ 
    assaflavie/runlike YOUR-CONTAINER

If we use a given command often, we can additionally add it as an alias

alias runlike="docker run --rm -v/var/run/docker.sock:/var/run/docker.sock  
 assaflavie/runlike" 
runlike YOUR-CONTAINER

A simple option, and effective. It allowed me to save time and find a solution to overwrite the image environment variable when on the server I couldn’t find the file that launched the original image.

Want privacy? Use multiple nicknames

Sat, 20 Mar 2021 02:31:23 GMT

At one point in being on the Internet and creating dozens of accounts, we are faced with one disturbing fact. Our internet presence is not anonymous. What to do with this fact, how to deal with it, and plan our anonymity on the Internet?

As ordinary internet users, we are used to our nickname that we use every day. The problem begins if our nickname used in games is not separated from our so-called “private” accounts. Why? Let’s imagine that our in-game statement is quite controversial for one person we meet. Entering our nickname in the Google search engine itself may cause that the person who displayed results will indicate our profile on Facebook or LinkedIn. The same goes for forums like Reddit, the 9GAG community, or commenting on YouTube videos. The standard rule - “Nothing is lost on the Internet” and “If it’s on the internet, it’s no longer (only) yours” - is often overlooked in this case. However, it is worth remembering about additional security, such as having multiple nicknames, because thanks to this, we can feel comfortable on the Internet itself.

It’s good to remember, that additionally, we have to watch out for leaks. If an application for food delivery leaks address data, and our login is quite widespread on the Internet, we can expect that after one of the worse matches in the MMO game, we can expect a letter from your teammate with a “thank you” in the mailbox. Alternatively, using a similar method, we can check whether the seller of the item at the auction is not trying to cheat us (Like when I wanted to buy an item for $ 300). By checking his nickname in Google, I found his Facebook and LinkedIn public profiles - and thanks to such targeting, I knew that the purchased item will be in good and tidy condition (apart from the fact that it was clear where he often dined, what cuisine he likes and many other pieces of information… ).

An additional problem that can happen to us in the life of an online person is having all accounts attached to one email. As far as it is, it’s not about the risk of hacking our account and extracting data by using the password reminder method. We must remember that we are also dependent on one mail provider. Quite recently, at the beginning of 2021, it happened that Google services stopped working for a while. What if access to our data suddenly disappears? The second bad news is a shared account in Google services. What if we get banned from YouTube? Our account with videos will be lost - well, it happens if we accidentally break some rules or get a Copywrite strike 3 times. However, it turns out that the ban on the account does not stop only on the YouTube service - it affects our full private Google account - where we keep our photos, contacts, e-mail, integration with phones, browsing history, and have payment cards attached. All this can be lost like for a given user of the portal.

Recently, I started to use the principle of separation of accounts and try to migrate data. Some tips to start:

Separate nickname and email
- private accounts which I intentionally sign with my names, such as photography or blog
- discussion forums (Reddit)
- shopping accounts (Amazon / AliExpress)
- games and libraries
- video content created (Twitch / Youtube)
Just in case, I recommend sharing mailboxes between different providers, such as the main Google for our personal contacts, but also e.g. proton mail for everyone else
We may have our password bank in the cloud, but I recommend that you synchronize passwords with the KeePass file from time to time

Maybe the method works and I will not be afraid of unambiguously connecting my nickname in the mobile game with my private person in the first 3 minutes of any match. Time will tell, for now, I am adjusting the results and checking to what extent the given pool of accounts and addresses is sufficient and not burdensome for me.

Queuing jobs in apps is easier than you think

Thu, 10 Dec 2020 00:47:47 GMT

When trying to communicate with our application, we always expect that it is available and can handle our request at a given moment. However, if these things are not certain, or we need to operate in a distributed structure with no known source where the request is to go, we can use task queuing.

Queuing is most often used when starting actions that require more processing time and are not required to get immediate results. Therefore, when writing various types of websites, the handling of these orders is triggered in separate processes. The very implementation of the queuing system into your project is also not difficult.

Producers & Consumers

Queuing uses two modes. In the standard understanding of the process, it is sending a message to a queue created and selected by the user, and the other is reading the stored messages. Messages can be additionally tagged and sorted into appropriate collections working on one queue. Depending on the need to use the queuing server itself, you may want to consider a couple of data sending and reading patterns. This information can be found both in the updates of the tools themselves and in popular forums for developers.

One of the most popular queuing apps & services that are simple to set up and easy to use:

Knowledge books: Python

Tue, 01 Dec 2020 00:39:21 GMT

Knowledge books is a mini series of constantly updated articles in which I share my more interesting discoveries

Each language has its own nice extras. The same applies to the functions and usability of the basic functions of the programs. Of course, the important thing is that if some parts are not used on a daily basis, they may be forgotten. Therefore, here are some brief notes on this language functions and methods of implantation. Written knowledge cannot be forgotten.

Table of Contents:

Array operations
Code improvements
Command line arguments
Decorators
Docstring
Functions
Magic attributes
Loops & Enumerate
Object-oriented programming
Other topics
Opening file streams
Practice materials
References
Return statements & generators
String formatting
Testing
Threading

Array operations

Code improvements

Command line arguments

Python Command Line Arguments

Decorators

Primer on Python Decorators

Docstring

Example NumPy Style Python Docstrings

Functions

Mutable Default Parameter Values

In Python, default parameter values are defined only once when the function is defined (that is, when the def statement is executed). The default value isn’t re-defined each time the function is called. Thus, each time you call f() without a parameter, you’re performing .append() on the same list.

def f(my_list = []):
    my_list.append('###')
    return my_list

f()
# ['###']
f()
# ['###', '###']

As a workaround, consider using a default argument value that signals no argument has been specified. Most any value would work, but None is a common choice.

def f(my_list=None):
    if my_list is None:
        my_list = []
        my_list.append('###')
    return my_list

Mutable Objects

Argument passing in Python can be summarized as follows. Passing an immutable object, like an int, str, tuple, or frozenset, to a Python function acts like pass-by-value. The function can’t modify the object in the calling environment.

Passing a mutable object such as a list, dict, or set acts somewhat—but not exactly—like pass-by-reference. The function can’t reassign the object wholesale, but it can change items in place within the object, and these changes will be reflected in the calling environment.

Args

def f(*args):
    print(args)
    print(type(args), len(args))
    for x in args:
            print(x)


f(1, 2, 3)
# (1, 2, 3)        
# <class 'tuple'> 3
# 1
# 2
# 3

Argument Tuple Unpacking

Although this type of unpacking is called tuple unpacking, it doesn’t only work with tuples. The asterisk (*) operator can be applied to any iterable in a Python function call. For example, a list or set can be unpacked as well

def f(x, y, z):
    print(f'x = {x}')
    print(f'y = {y}')
    print(f'z = {z}')

f(1, 2, 3)
# x = 1
# y = 2
# z = 3

t = ('foo', 'bar', 'baz')
f(*t)
# x = foo
# y = bar
# z = baz

Argument Dictionary Packing

def f(**kwargs):
    print(kwargs)
    print(type(kwargs))
    for key, val in kwargs.items():
            print(key, '->', val)


f(foo=1, bar=2, baz=3)
# {'foo': 1, 'bar': 2, 'baz': 3}
# <class 'dict'>
# foo -> 1
# bar -> 2
# baz -> 3

Keyword-Only Any Arguments

def oper(x, y, *ignore, op='+'):
    if op == '+':
            return x + y
    elif op == '-':
            return x - y
    elif op == '/':
            return x / y
    else:
            return None


oper(3, 4, op='+')
# 7
oper(3, 4, op='/')
# 0.75
oper(3, 4, "I don't belong here")
# 7
oper(3, 4, "I don't belong here", op='/')
# 0.75

Keyword-Only No Additional Arguments

def oper(x, y, *, op='+'):
    if op == '+':
            return x + y
    elif op == '-':
            return x - y
    elif op == '/':
            return x / y
    else:
            return None


oper(3, 4, op='+')
# 7
oper(3, 4, op='/')
# 0.75

oper(3, 4, "I don't belong here")
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: oper() takes 2 positional arguments but 3 were given

oper(3, 4, '+')
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: oper() takes 2 positional arguments but 3 were given

Python Function Annotations

Annotations don’t impose any semantic restrictions on the code whatsoever. They’re simply bits of metadata attached to the Python function parameters and return value. Python dutifully stashes them in a dictionary, assigns the dictionary to the function’s __annotations__ dunder attribute, and that’s it. Annotations are completely optional and don’t have any impact on Python function execution at all.

To quote Amahl in Amahl and the Night Visitors, “What’s the use of having it then?”

For starters, annotations make good documentation. You can specify the same information in the docstring, of course, but placing it directly in the function definition adds clarity.

def f(a: int = 12, b: str = 'baz') -> float:
    print(a, b)
    return 3.5

f.__annotations__
# {'a': <class 'int'>, 'b': <class 'str'>, 'return': <class 'float'>}

Magic attributes

Magic Methods:

Construction and Initialization
- __init__(self, [...) The initializer for the class. It gets passed whatever the primary constructor was called with
- __new__(cls, [...) Is the first method to get called in an object’s instantiation. It takes the class, then any other arguments that it will pass along to __init__. __new__ is used fairly rarely, but it does have its purposes, particularly when subclassing an immutable type like a tuple or a string.
- __del__(self) It defines behavior for when an object is garbage collected. It can be quite useful for objects that might require extra cleanup upon deletion, like sockets or file objects.
Comparison magic methods
- __cmp__(self, other) Is the most basic of the comparison magic methods. It actually implements behavior for all of the comparison operators (<, ==, !=, etc.). Should return a negative integer if self < other, zero if self == other, and positive if self > other
- __eq__(self, other) Behavior: equality operator, ==
- __nq__(self, other) Behavior: inequality operator, !=
- __lt__(self, other) Behavior: less-than operator, <
- __gt__(self, other) Behavior: greater-than operator, >
- __le__(self, other) Behavior: less-than-or-equal-to operator, <=
- __ge__(self, other) Behavior: greater-than-or-equal-to operator, >=
Numeric magic methods
- Unary operators and functions
- Normal arithmetic operators
- Reflected arithmetic operators some_object + other That was “normal” addition. The reflected equivalent is the same thing, except with the operands switched around: other + some_object
- Augmented assignment
- Type conversion magic methods
Representing your Classes
- __str__(self) Defines behavior for when str() is called on an instance of your class.
- __repr__(self) The major difference between str() and repr() is intended audience. repr() is intended to produce output that is mostly machine-readable (in many cases, it could be valid Python code even), whereas str() is intended to be human-readable.
- __hash__(self)
Controlling Attribute Access
- __getattr__(self, name)
- __setattr__(self, name, value)
- __delattr__(self, name)
Making Custom Sequences
- The magic behind containers
Reflection
Callable Objects
- __call__(self, [args...]) Allows an instance of a class to be called as a function. Essentially, this means that x() is the same as x.__call__()
Context Managers
- __enter__(self) Defines what the context manager should do at the beginning of the block created by the with statement. Note that the return value of enter is bound to the target of the with statement, or the name after the as.
- __exit__(self, exception_type, exception_value, traceback)
Building Descriptor Objects
- __get__(self, instance, owner) Define behavior for when the descriptor’s value is retrieved. instance is the instance of the owner object. owner is the owner class itself.
- __set__(self, instance, value)
- __delete__(self, instance)
Copying
- __copy__(self)
- __deepcopy__(self, memodict={})
Other
- __annotations__(self)
- __doc__(self)

Loops & Enumerate

Python enumerate(): Simplify Looping With Counters

✓ Use Python’s enumerate() in your for loops
✓ Get values from enumerate() using argument unpacking
✓ Implement your own equivalent function to enumerate()\

Technical Detail: According to the Python documentation, an iterable is any object that can return its members one at a time. By definition, iterables support the iterator protocol, which specifies how object members are returned when an object is used in an iterator. Python has two commonly used types of iterables:

Content loop

values = ["a", "b", "c"]
for value in values:
    print(value)
# a
# b
# c

Content loop with manual index

index = 0
values = ["a", "b", "c"]
for value in values:
    print(index, value)
    index += 1
# 0 a
# 1 b
# 2 c

Range loop

values = ["a", "b", "c"]
for index in range(len(values)):
    value = values[index]
    print(index, value)

You should use enumerate() anytime you need to use the count and an item in a loop. Keep in mind that enumerate() increments the count by one on every iteration.

Enumerate loop

values = ["a", "b", "c"]
for count, value in enumerate(values, start=0):
    print(count, value)

Enumerate example usage

def even_items(iterable):
    """Return items from ``iterable`` when their index is even."""
    values = []
    for index, value in enumerate(iterable, start=1):
        if not index % 2:
            values.append(value)
    return values

Array strip examples

def even_items(iterable):
    return [v for i, v in enumerate(iterable, start=1) if not i % 2]

alphabet = "abcdefghijklmnopqrstuvwxyz"
even_items(alphabet)
# ['b', 'd', 'f', 'h', 'j', 'l', 'n', 'p', 'r', 't', 'v', 'x', 'z']
list(alphabet[1::2])
# ['b', 'd', 'f', 'h', 'j', 'l', 'n', 'p', 'r', 't', 'v', 'x', 'z']

Using iterator in enumerate

def alphabet():
    alpha = "abcdefghijklmnopqrstuvwxyz"
    for a in alpha:
        yield a

alphabet[1::2]
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: 'function' object is not subscriptable

even_items(alphabet())
# ['b', 'd', 'f', 'h', 'j', 'l', 'n', 'p', 'r', 't', 'v', 'x', 'z']

When you call enumerate() and pass a sequence of values, Python returns an iterator. When you ask the iterator for its next value, it yields a tuple with two elements. The first element of the tuple is the count, and the second element is the value from the sequence that you passed:

values = ["a", "b"]
enum_instance = enumerate(values)
enum_instance
# <enumerate at 0x7fe75d728180>
next(enum_instance)
# (0, 'a')
next(enum_instance)
# (1, 'b')
next(enum_instance)
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# StopIteration

In the for loop in this example, you nest zip() inside enumerate(). This means that each time the for loop iterates, enumerate() yields a tuple with the first value as the count and the second value as another tuple containing the elements from the arguments to zip(). To unpack the nested structure, you need to add parentheses to capture the elements from the nested tuple of elements from zip().

first = ["a", "b", "c"]
second = ["d", "e", "f"]
third = ["g", "h", "i"]
for one, two, three in zip(first, second, third):
    print(one, two, three)

# a d g
# b e h
# c f i

for count, (one, two, three) in enumerate(zip(first, second, third)):
    print(count, one, two, three)

# 0 a d g
# 1 b e h
# 2 c f i

Object-oriented programming (OOP)

Opening file streams

What is the python “with” statement designed for?

Practice materials

References

Pass by Reference in Python: Background and Best Practices

Return statements & generators

The Python return Statement: Usage and Best Practices

String formatting

Python String Formatting Best Practices

Old Style” String Formatting (% Operator)

'Hello, %s' % name
# "Hello, Bob"

'Hey %s, there is a 0x%x error!' % (name, errno)
# 'Hey Bob, there is a 0xbadc0ffee error!'

'Hey %(name)s, there is a 0x%(errno)x error!' % { "name": name, "errno": errno }
# 'Hey Bob, there is a 0xbadc0ffee error!'

This makes your format strings easier to maintain and easier to modify in the future. You don’t have to worry about making sure the order you’re passing in the values matches up with the order in which the values are referenced in the format string. Of course, the downside is that this technique requires a little more typing.

“New Style” String Formatting (str.format)

'Hello, {}'.format(name)
# 'Hello, Bob'

'Hey {name}, there is a 0x{errno:x} error!'.format(name=name, errno=errno)
# 'Hey Bob, there is a 0xbadc0ffee error!'

This also shows that the syntax to format an int variable as a hexadecimal string has changed. Now you need to pass a format spec by adding a :x suffix. The format string syntax has become more powerful without complicating the simpler use cases.

String Interpolation / f-Strings (Python 3.6+)

f'Hello, {name}!'
# 'Hello, Bob!'

f'Five plus ten is {a + b} and not {2 * (a + b)}.'
# 'Five plus ten is 15 and not 30.'

f"Hey {name}, there's a {errno:#x} error!"
# "Hey Bob, there's a 0xbadc0ffee error!"

Template Strings (Standard Library)

from string import Template
t = Template('Hey, $name!')
t.substitute(name=name)
# 'Hey, Bob!'

templ_string = 'Hey $name, there is a $error error!'
Template(templ_string).substitute(name=name, error=hex(errno))
# 'Hey Bob, there is a 0xbadc0ffee error!'

user_input = '${error.__init__.__globals__[SECRET]}'
Template(user_input).substitute(error=err)
# ValueError:
# "Invalid placeholder in string: line 1, col 1"

Testing

Threading

Why dockerizing production isn't bad idea

Tue, 24 Nov 2020 19:39:40 GMT

The use of docker is a common technique in dev teams. But really, creating and using the complete ready-made image shows its potential in the production environment. I think this post will convince some Ops people to look more positively at running closed images over the installation and putting projects directly on the servers.

There are many benefits to using images. For the development of the application, its tests, and automation, up to a running product. Along with the development of the industry, there were also many new technologies enabling the management of more servers as part of the application publishing itself. Some of them were designed directly by the current cloud operators. The emergence of a new branch of technology also forced the creation of a new profession - DevOps.

Each team looks for Docker and its derivative benefits in terms of work efficiency and simplification of the tasks assigned to them. Therefore, to better understand each of the teams and their point of view, each has its chapter. This will allow you to understand the possibility of using this technology in every step of the project.

Dev

Creating docker images while developing an application is currently one of the requirements in the hiring process. This is more or less because previously programmers had to build up their programming environment. Knowledge of Linux was a standard in the job description ever to know how to install the appropriate version of packages for the operation of their project. Unfortunately, the problem of installing main packages by programmers grew when they started working on several projects at the same time in different global package versions. Unfortunately, switching between versions was not convenient, it often required additional configuration files, and sometimes it forced to re-install them. Virtualization came to the rescue, which separated each project and did not require having specific versions of packages on the user’s main system. Due to the hardware consumption and number of layers in virtualization technology, Docker is the most used today. The development team needs to use application images because the same versions of the libraries are used on each workstation. To make editing the project code comfortable, it is common practice to link the folder with the code from the user’s machine to the running image. This operation makes the environment clean enough to allow the project to be built from scratch as if it were in a regular non-image environment. This allows for the utmost control over the project.

DevOps

This position is mostly related to the CI/CD function. The absolute automation of the application release process, updating images for security reasons, and control of application communication is what characterizes this work. With more projects to look after, the job requires a lot of management procedures. Recently, due to the growing interest among others in Kubernetes, DevOps is more and more often dealing with new tools offered by operators like Google Cloud or AWS. The arrangement of procedures in this profession is very important. Thanks to this, work with applications can be automated and images properly prepared. These people look after the internal environment of the application, test performance, check application statistics, and try to improve every point of it. The variation of tools for DevOps is huge, and they are often mixed - library changes can be compared to the same dynamically changing frontend technology. The Best description? One command should execute and handle each exception. Obtaining total automation in a project is the greatest reward.

Ops

The main task of this team is to create a safe system. Protection of any undesirable entrances, provision of external services, and monitoring the SLA of servers. Due to the profile of this profession, it is heavily responsible for data access and security. Increasingly, companies are moving away from burdening this team with app releases (due to the possibility of unexpected errors during updates, restoring the previous version, and consulting with the development team). Since these activities were a shared responsibility of the teams and usually required additional access for other departments, this function is postponed for safety. Instead, these actions are now being replaced with images of the application. The difference between the old and the new way can be broken down into points.

Classic prod environment update (application downtime):

blocking access to the application as part of updating the environment
[optional] software update to support new functionalities (~ 10 min)
updating application libraries (~ 5min)
[optional] migrations in the database (~ 5min)
unblocking access and admitting users
negative path
restoring the old code version (~ 1 min)
restoration of the old software version (if the functions were deprecated) (~ 10 min)
installation of application libraries (~ 5 min)

Updating the prod environment using Docker tools (application downtime):

download an image with the code included
replacing the version of the prepared image and restarting the configuration
blocking access to the application as part of updating the environment
docker facilitates this by using load balancing, which makes this point optimal
in the image, you can add code that will do some tasks before starting the motion in the application
[optional] database migrations (~ 2-5min)
when starting the image, check if you should perform the migration, if not - continue
negative path
run old version of the image (~ 1 min)

Seeing the steps given, you can see that preparing an image with built libraries, a good version of the software, and included code is a big profit. By adding release automation and even server conversion to a Kubernetes cluster, we simplify the issue of publishing drastically. Images created by teams are closed by default and it is up to us how we make them available to the world - or the environment configuration team will take care of it. By receiving such a simplification, people responsible for security and services can focus on their tickets and improving their work.

Knowledge books: Photography inspirations and cheat sheets

Mon, 16 Nov 2020 00:04:08 GMT

Knowledge books is a mini series of constantly updated articles in which I share my more interesting discoveries

It happens that in the photos with the model we have no idea for new frames, or the ones that were used are repeated in each session. When looking for alternatives of styles, poses, arrangement of light or the use of objects in web, we often come across links that are not needed at the moment. However, it is worth saving them for later and grouping them.

Every now and then, it’s a good idea to sit down for a day and gather ideas. We can then save the collected photos or links in specific styles and present them to other people. Thanks to a well-prepared modboard, which looks different depending on the person who uses it, we can predict in what style a particular session would be aimed at. Signing up is free and collecting ideas is really interesting. From one day and looking for similar photos, we can choose a lot of styles and inspirations.

Some of my boards:

Poses

Missing poses for photos in the background can be supplemented using materials for artists. They often use real photos or the person themselves. Using the resources of, among others, the Art Model Reference Images to Draw series created by New Masters Academy, with each episode we can be inspired by reference photos

Lighting setup

Behind the scenes

Tips

Knowledge books: Food recipes

Sun, 15 Nov 2020 19:08:58 GMT

Knowledge books is a mini series of constantly updated articles in which I share my more interesting discoveries

Free time is a really good thing. We can use it freely - including cooking something extraordinary. In this time we can approach cooking with great creativity and eagerness. Having the recipe, we can modify it at will, there are no bad results. When finding a good new recipe, I will certainly keep the list updated. In the end, I will also look here myself to recall even the list of ingredients or the next steps.

Chaliapin Steak Don

The recipe is obtained mainly from the Food Wars anime, which beautifully describes and imitates real recipes known all over the world. I had several approaches to this dish and I must admit, there are no bad results here - each variant tastes special. Don himself is really well described on the CHEFPK channel.

Ingredients:

beef meat
1 onion / 200g meat
rice
peper & salt

Extras:

maple syrup + miso for marinate
Japanese prunes or other bitter dried fruit

As a first step, we prepare the onion by cutting it into cubes. Put the first layer on the plate, then put the meat in it and cover it with the second layer of onion. The beef should then wait in this state for half to two hours. While waiting for the taste to soak and soften the meat, we can prepare the rice. After that time, remove the onion from meat and sauté it in a pan. Burn the meat in a frying pan and prepare it for the frying you want. The last step is to put rice, meat and a hill of onion into the bowl and pour the resulting sauce from the pan over it.

Modifications:

The meat can be prepared sous vide by enclosing it in a bag together with the marinade
An addition to rice can be rice sauce or dried Japanese cherry particles
While in a rice cooker, you can add additional ingredients to make rice soak in additional flavors

Massaman curry

It’s worth experimenting in the kitchen. Thanks to them, we have rediscovery tasteful combinations of flavors. Last time I was able to put together a good combination for the curry

Ingredients:

chicken/pork/beef meat
1 onion / 200g meat
massaman curry paste
rice
soy sauce & teriyaki sauce
salt, garlic, citric acid, ground paprika, ginger

Extras:

yakisoba sauce
dry wine
rice vinegar

Preparations should start with the onion glazing and rice cooking simultaneously. During frying, it is worth adding spices such as garlic, citric acid, and a pinch of salt to it. When the onion is almost ready, add the meat that will use the current sauce. To increase the flavor variation of the meat, you can add additional spices such as ground paprika or ginger. The next step is to pour coconut milk over the pan content and add the curry paste. Add soy and teriyaki sauce, rice vinegar, and reduce the sauce. Halfway through the process, add extra dry wine to enhance the umami of the curry. The finished product should be served in a bowl along with rice, cherry tomatoes, and chives. The variation in the preparation of the dish may simply change.

Modifications:

a flavor variation to the cooking rice
- yakisoba sauce and rice vinegar
- soy sauce and a mixture of ground paprika
curry works very well with the 5-taste variant (not 5-flavor variant), so add a different mix of spices
rice to increase its taste, you can fry it slowly with an egg in a pan (it’s worth adding spices to close them in the middle of the rice coated with egg protein)

Know what beef you want to add to the basket

Sun, 15 Nov 2020 18:00:50 GMT

Buying good meat is really important. It may turn out that the meat seen on the promotion or on the shelf may not completely match our dish. Therefore, it makes sense to have a list of pieces of meat that are suitable for specific dishes. It is also worth being aware of what the meat should look like or what price you can expect.

I make a list of all these types after I bought a weaker quality entrecote, which turned out to be stringy at the very end. The dish was saved, but the steaks turned into a stew. It is worth considering whether the quality and price of the meat will not affect the taste of the dish. Unfortunately, it did not work after a small thermal treatment today.

Entrecote (cote de boeuf)

It contains part of the longest back muscle and part of the intercostal muscles with very fine fibers. Additionally, it has beautiful hypertrophy of the so-called intramuscular fat. marble that makes the steaks particularly juicy.

26.50€ / kg
slices about 3cm thick

Rib Eye

The heart of the entrecote, after it is seasoned, the bone and the so-called outer sheath. rib eye is a slice of rib eye steak. It has a high degree of marbling and is therefore extremely delicate, full of flavor and aroma.

42.00€ / kg
slices about 3cm thick

TBone

King of steaks, adored by beef fans. It is the most popular steak in the US and England where it has become a legend. It owes its name to the characteristic T-shaped bone that divides the steak into 2 unequal parts. The smaller part is the tenderloin and the larger part is the roast beef. It requires careful grilling not to drag the part with the tenderloin. T-bone therefore has two flavors, roast beef and sirloin.

31.00€ / kg
slices about 3cm thick

Porterhouse

The steak is cut in a very similar way to a t-bon but much bigger than it. Porterhouse steak is cut in the place where the tenderloin is thickest. A portion of Porterhouse steak weighs about 650 g and should therefore be fried 6-8 minutes on each side. Most often served as a dish for 2 people, cut into smaller slices, cut from the bones. When frying, remember that sirloin comes much faster than roast beef and should not be too long.

35.50€ / kg
slices about 3cm thick

New York (roast beef)

A steak obtained from a part of the roast beef, with a great taste and meat structure. to preserve its full flavor, we cut it out with the bone. Considered by many experts as the tastiest Hereford steak.

24.00€ / kg
slices about 3cm thick

Sirloin (rump steak)

One of the most valuable types of beef for universal use, incl. in appetizers, soups, main courses as well as in snacks. Cut from cruciforms, they are slices slightly thinner than steaks that can be prepared on a grill or pan. A rump steak, stewed in a pan with caramelized onions and a delicate own sauce.

15.50€ / kg
slices about 1.5cm thick

Ligava

Part of a beef leg - a strong, lean muscle. It is sometimes referred to as “false sirloin”. It is well suited for cooking and stewing. Ligawa steaks should not be overcooked as they can get dry and unpalatable.

11.00€ / kg
slices about 1.5cm thick

Does the beginning of learning programming always have to be boring?

Mon, 09 Nov 2020 00:31:12 GMT

The beginnings are always difficult, whether when it comes to learning only programming or trying to learn a new language. Usually we are looking for a fresh idea to systematically develop it as part of the acquired knowledge. It’s hard to find something like this on the internet recently, so I’ll share my ideas.

The basis of each project is its future development. We can assume that our application, project or script has a chance for development, but also a change of direction. Therefore, the basic project itself must be quite flexible when it comes to introducing changes and allow for the implementation of many improvements and test solutions. This is very important, especially when developing your skills in a free way, where we should have our sandbox at our disposal.

The first thing that comes to my mind when it comes to use is building chatbots. By browsing bot repositories for Discord or Telegram, we can discover that there are really thousands of them. Why am I giving this idea first? I once created a bot repository for Telegram as part of a project with friends “Let’s write some cool code together”. Each person had a different idea, but this did not prevent us from implementing and developing the project. Each of us has evolved while writing code, team findings and understanding some procedures from inside out. Of course, if I come up with another project idea, I’d love to update it here.

Flexibility

Flexibility is the core of a design. By putting up a bot idea as the first, we can attach a development list to it. I will try to list the possibilities of project development divided into categories from the simplest to the more advanced. Of course, some categories presented may use similar solutions and tools. However, the point of looking at them changes here, and our knowledge, going through the previous points, can significantly improve our work on a more advanced topic. I will use python mentions in the list as now I am in the development stage of the language myself. Each of these solutions can also be used in a different language, and libraries will find alternatives. So no worries, although I think the references may bring some people closer to the subject than just a dry description.

Bot for Discord / Telegram / Other communicator

Read/write data

Possibility to create a list for a channel or for a user. Using commands we can add, subtract or edit items. So we need to save and later use this data somehow.

Writing data to a file?
How to get to them, edit, save?
Maybe a database? Setting up the entire database server instance or are there any libraries to use local database option?

Alternatively, maybe we can store some record of users who wrote in the chat, make a ranking who was active for how many days, an activity ranking for how many days in a row they were online.

Schedule tasks

The ability to run an operation at any time without the need to manually trigger an action. This is a really great option if we need to run some actions recurrently. The basic action could be notification of the collected data since the last time or sending a reminder. The activities for this module may increase with the next steps.

Are there any libraries to run this kind of functions?

Scrapping/fetching + parsing

Let’s read an RSS feed and send info about new posts. We just need to know which posts are new and which are already scrapped. The RSS feed can be long, we do not want it to give us a list of all posts every 15 minutes. It would be great if scrapping could work automatically in the background, so there will be no need to run the command manually in the command. RSS feeds are often saved in XML, so you probably need to convert this data from text to some kind of list or dict - somehow program have to read it.

Web / API:

After some data saved, we can get to them via API? It’s only here, but for Python projects i strongly suggest using Django or fastAPI

Test your code

Are you sure your code is 100% working? Are we getting good function results? Will the introduction of a new functionality break another part of the system, do all services communicate? We can answer all these questions by writing tests of our code. Thanks to this, we can rest easy and know that our changes have not damaged the integrity of the code.

Documentation

Well-described documentation is more valuable than gold. After some time, you can always come back to function and remember what it did, what arguments it accepted and what it returns. Each language has its own standards, it is worth seeing and applying the description in your project.

Advanced scrapping

It’s time to get data from real pages. For this, we can always use BeautifulSoup4 and Scrapy for tougher scripted sites

How we can get information from sites?
What is the best solution to save data as we know, that every site store them differently?

Queuing actions

What if we have a lot of services that outsource a task to be performed? We cannot allow other connections to slow down while waiting for the action to take place. Therefore, the option is to queue tasks and process them in the background in time, when the service itself receives information that a given action has been scheduled. RabbitMQ is a fairly well-designed and described queuing system that we can start with.

Tools

docker

We don’t know what we will encounter in the system, and we know that everyone on the computer has a different language version. It is worth solving with a virtual environment such as docker for both deployment and 24/7 service launch. You can immediately switch to the docker-compose library - it is more convenient to use, build and configure images.

VPS

It is always nice to have your own server if you want to put something to work permanently on it or for simple tests or sandbox. Google gives minimal server configuration for free — it works ok, I even have some services running there. It can handle docker too!

git

This tool is must-have! We have to save our projects somehow. I say right away that having app, app-v2, appv3, app-v3-final, app-v3-final2 folders does not work (I experienced it). That is why it is worth using repositories and having control over the versions of our code. For this, GitLab and Bitbucket give you free private repositories. Git is a necessary thing in the life of a programmer — even in the basics.

Summary

We must remember, that the most important thing is to try. The code we make is always good if it does what we asked for. It is important to have an initial idea of how to program. After some time, you can go back and improve / rewrite the code. If we see that the old code is ugly, it means that we already have more knowledge than at the beginning. If we feel good at programming, we can move on to new things and knowledge like tests, data structures, good coding practices, communication options, queuing, etc. — There are a lot of topics, but it is worth taking when you feel confident. At the beginning, write the code so that you can do something and develop it with each new idea. Check the data, validate if the user has entered something correctly and so on. Good luck everyone, ever me!

Detailed description of your API

Fri, 06 Nov 2020 22:38:31 GMT

A good described API is attitude. Especially when planning architecture, where other teams work on its successive elements. Therefore, it is so important to describe in detail each API path. For this, it is worth using the right tool. With a growing team, you need to be aware that communication between people must be at the highest level. To do this, it is often interfered with the introduction of many management tools. With such activities, you should also remember about the list of documentation and knowledge sharing. Swagger is one of such tool to help share information between developers about API endpoints in the project.

This tool is most often found in the form of various libraries for automatic building of API documentation from the current project code. Sure, this is one of the possibilities of this project, but often people overlook the standard operation of the main tool itself. For their integration, libraries ignore many additional functionalities provided by the basic swagger tool. Also, such libraries often may not allow for some kind of adding comments or define some kind of elements as additional or existing under some specific logic. Another thing is the modification of documentation that is scattered all over the code. Therefore, it is worth taking a look at the basic operation of the Swagger Editor itself.

Swagger Editor

The approach to Swagger can be difficult, but I encourage you to check what a well-structured endpoint description for a specific module or the entire project may look like. The example that can be found in the Swagger Editor tool describes all the cases encountered during the API description. As I can say editor allows to do a better description of the default fields, which this option from my experience is rarely found in integrated libraries. In addition, we can also download the desktop app beside web and cloud versions.

Summary

Generating the final documentation and forwarding it to the next department, which can be a frontend, for example, can be a great advantage for the implementation of the project. It also solves many questions whether and what the API data returns, reduces the number of tests, speeds up the project structure and may also allow for better planning of the API before writing it. The tool and the JSON file itself or the full output should become an integral part of any larger project.

Magic of whiptail and Squid proxy installation

Wed, 04 Nov 2020 21:50:47 GMT

When installing a bash open source project, I was always full of admiration for the developers who implemented the graphical interface. I was curious how it works and how I could apply it to my projects. The solution is simple. It is the whiptail.

whiptail

Whiptail is one of the popular programs that allows you to display dialog boxes in the shell. Proper preparation of steps and good ability to write shell scripts allows you to skillfully extract a lot of information from it and use it to build automated scripts.

The basic program boxes include

message box
yes / no box
input box
password box
text box
menu
check list
radio letter
progress gauge

So as you can see, we can ask the user for all his requirements for running the script.

Squid — install proxy server in few simple steps

Squid is a great program for creating a proxy server. It allows you to control the flow of information, maintain access at a specific time, manage access for users or bind specific network cards to appropriate ports and more. It is a very efficient tool and its configuration at a basic level does not require much time. Therefore, recently I installed this server on one of my VPS to get to specific sources by passing my data over other IP. Of course, while it takes a few steps to install, it’s always nice to have an automated task. Therefore, I created the base code using dialog boxes. In order not to take too much space on the article, it is also slimmed down with additional validation of variables and function results. After all, the code is designed to encourage work.

#!/bin/bash

whiptailInputBox() {
    # Open window with text input

    # Parameters:
    #   $1 Box title
    #   $2 Content
    #   $3 Default

    RESULT=`whiptail --inputbox "$2" 8 78 $3 --title "$1" 3>&1 1>&2 2>&3`
    return $?
    echo $RESULT
}

whiptailPasswordBox() {
    # Open window with password input

    # Parameters:
    #   $1 Box title
    #   $2 Content
    #   $3 Default

    echo `whiptail --passwordbox "$2" 8 78 $3 --title "$1" 3>&1 1>&2 2>&3`
    return $?
    echo $RESULT
}

whiptailMessageBox() {
    # Open window as message box 

    # Parameters:
    #   $1 Box title
    #   $2 Content
    
    whiptail --title "$1" --msgbox "$2" 8 78
}

preinstall() {
    if ! [ -x "$(command -v whiptail)" ]; then
        /usr/bin/apt update
        /usr/bin/apt -y install whiptail
    fi
}

install_squid() {
    /usr/bin/apt update
    /usr/bin/apt -y install apache2-utils squid3
}

squid_configuration() {
    # Revove old squid.conf config
    /bin/rm -f /etc/squid/squid.conf
    /usr/bin/touch /etc/squid/blacklist.acl

    # Update iptables and accept port
    /sbin/iptables -I INPUT -p tcp --dport 3128 -j ACCEPT
    /sbin/iptables-save

    # Reset service
    service squid restart
    update-rc.d squid defaults
}

preinstall
whiptailMessageBox "Info" "Auto installation script will guide you with configuration of squid package"
SQUID_USERNAME=$(whiptailInputBox "Squid Username" "Provide SQUID username" "")
if [ $? != 0 ]; then exit; fi
SQUID_PASSWORD=$(whiptailPasswordBox "Squid Password" "Provide SQUID password" "")
if [ $? != 0 ]; then exit; fi

install_squid
/usr/bin/htpasswd -b -c /etc/squid/passwd $SQUID_USERNAME $SQUID_PASSWORD
squid_configuration
whiptailMessageBox "Info" "Instalation complete"

With squid already installed, we can go to configuration. The current squid.conf allows us only to use the proxy locally on the server. To unlock its external use, we must remember that each service published by us should be secured. There are too many IP lists of proxy servers on the Internet that have been exposed to the public with different port values in hope that no one will guess why they are open.

To protect yourself against such situations, it is enough to unblock the proxy entry only for credentials that we created earlier using the script above. The record should be placed under the indicated comment in the /etc/squid/squid.conf file. After that, all you need it reset the service and you can connect to your proxy service.

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwd
auth_param basic children 5
auth_param basic realm Squid Basic Authentication
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
http_access allow auth_users

Tips

curl command can be used to test a connection through a proxy

curl -x http://user.password@domain_or_IP:3128 -I http://google.com

# When success (Message head)
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/

# When conection is secured with proxy (Message head)
HTTP/1.1 407 Proxy Authentication Required

Likewise, we can get our IP after going through any proxy we provide

wget -e use_proxy=yes -e http_proxy=http://user.password@domain_or_IP:3128 -qO- http://ipecho.net/plain | xargs echo

Your own VPN service in USA

Sun, 01 Nov 2020 21:37:05 GMT

Regional locks aren’t nice. They always cause a problem with access to certain sources. VPN service charges are often high, and the free versions only offer a certain usage limit or are bandwidth-constrained. Therefore, why not set up your own free VPN server overseas?

Virtal Private Server

There are a lot of VPS offers on the Internet. This time the search is directed a bit differently than the access and HTTP servers. The first thing we need to pay attention to is where the service is run, and then what is the suggested rental price of the resources. From large suppliers, we may be interested in well-known service providers such as OVH, AWS, Azure or Google. They can often have strong server rental deals or special free-tier offers. However, be careful whether the server will not be refunded after the end of the promotional period.

In paid offers from service providers you can find very cheap servers listed below

Vultr: Worldwide locations, IPv6 support, starting at $3.50/month
Hetzner: Germany, IPv6, 20 TB of traffic, starting at €3/month
Digital Ocean: Worldwide locations, IPv6 support, starting at $5/month
PulseHeberg: France, unlimited bandwidth, starting at €3/month
OVH: Worldwide locations, IPv6 support, starting at $3.50/month

Currently, the most attractive offer, in my opinion, can boast of google cloud, which offers a full free-tier for a machine located in Oregon, Iowa or South Carolina.

Free-tier in Google Cloud Platform

Basic server configuration starts with creating a Compute Engine instance. When creating it, we must remember about the appropriate region — although here the choice is more about continents. The instance that interests us is F1-micro, which will be enough for us to provide this and several other services. The next step will be to connect to the server via SSH and install OpenVPN. Unfortunately, for first-time users, installing OpenVPN can be a bit complicated. Large descriptions of functions in the documentation, no direct support for some distributions, configuration of files without full understanding. Unfortunately, this is not the simplest solution. Therefore, for users who do not want to play with the configuration of the server to support the entire company, but only for private purposes, very nice repositories have been created to auto-install this service.

After installation, all you need to remember is editing firewall rules to allow incoming traffic from another port. For GCP, we can find it at this link. It is suggested to add a new rule, assign the VPN ports, add a tag to the Compute Engine instance and reset the system to accept all changes.

Summary

If someone can install servers quickly or has their VPS abroad, this is a perfect opportunity to have a VPN for streaming services. Thanks to this, we will be able to use the advantages of service providers such as NordVPN or ExpressVPN without the need to buy a subscription or free providers with trial access and shady data usage. In addition, the data that passes through the server is ours, so there is less risk of it being viewed by an unknown person from an unknown proxy server.

Task runner that resolves missing pipeline events

Sun, 01 Nov 2020 17:01:53 GMT

When creating pipelines for releasing things to the environment, we may run into the problem of handling too few events. While searching the internet, I encountered an interesting task runner library. Upon closer acquaintance with the library, it will be possible to supplement the missing functions of pipelines offered in cloud solutions.

Makefile, bash or something different?

Bash or Makefile files are most often used when building commands to improve the output. While this solution runs all scripts and allows for a large management of functions, it lacks error handling or API communication. Pypyr is a task runner for automation pipelines script as sequential task workflow steps in yaml.

It has many functions, thanks to which we can properly automate the work of tasks. Most importantly, it has been quite successful at capturing and working on exceptions. Steps of activities can be combined, tasks described in detail, and also modified using environmental variables and data stored in the middle of the task. Additionally, as a basic functionality, we can also write logical conditions and even run python code. This solution gives a lot of possibilities for code actions. It is certainly easier to manage in already typical Python projects, where the work of publishing tasks does not have to be done by bash itself. While the first meeting on the website with the library may scare you off with its structure at the beginning, further delving into the topic shows that complex tasks can be much clearer for future activities in the project.

# This is an example showing the anatomy of a pypyr pipeline
# A pipeline should be saved as {working dir}/mypipelinename.yaml.
# Run the pipeline from {working dir} like this: pypyr mypipelinename

# optional. set this to pass cli arguments to the pipeline.
context_parser: my.custom.parser

# mandatory.
steps: # step-group. every pipeline starts with steps unless you tell pypyr differently.
  - my.package.my.module # simple step pointing at a python module in a package
  - mymodule # simple step pointing at a python file
  - name: pypyr.steps.default
    in:
      defaults:
        ifConinue: False # optional. Check if command have set on ifContinue variable, if not, set default.
  - name: my.package.another.module # complex step. It contains a description and in parameters.
    description: Optional description is for humans. It's any text that makes your life easier.
    in: # optional. Set these key-value pairs in context for this step.
      parameter1: value1
      parameter2: value2
    run: {ifContinue} # optional. Runs this step if True, skips step if False. Defaults True if not specified.
    skip: False # optional. Skips this step if True, runs step if False. Defaults False if not specified.
    swallow: False # optional. Swallows any errors raised by the step. Defaults False if not specified.

# optional.
on_success: # step-group
  - my.first.success.step
  - my.second.success.step

# optional.
on_failure: # step-group
  - my.failure.handler.step
  - my.failure.handler.notifier

Summary

The options provided by the pipeline configured in this way allow you to control the issue and with the appropriate status of a task or individual steps, we can even send an appropriate notification or withdraw changes from the server. Pypyr is a nice tool that can come in handy for more complex project steps and can become a good alternative to the Makefile’s all-in-one tasks.

Media library with 2TB cloud storage on VPS

Sat, 31 Oct 2020 13:50:26 GMT

I have to admit, having your own private Netflix-like service is really nice.

The project I have had contact with recently is Jellyfin. It is a very welcoming and multi-device media software system. The project, as in the case of kodi, allows you to manage the media library, download and complete information about the downloaded content, it has an automatic season organizer for TV Shows and allows you to watch Live TV & DVR and more. All this is available under the www panel placed on our home server or VPS. For all devices and access from everywhere.

Data storage problem

Ever if you install media center on the server and want to store a collection of movies to watch, you usually don’t have much disk space. It is understandable, because we do not attach large NAS drives or other heavy devices to each of our servers.

Unfortunately, for VPS servers, the matter of renting additional space becomes even more complicated Therefore, before approaching the topic, you should consider what problems you may encounter when accessing and reading/writing multimedia from the server.

Home server

The first solution is to have your own home server. However, this requires an investment in an additional large external drive, and the design itself does not look so nice. Of course, for most people, the very result of the work will be important. Such a server will have a problem with exposing the service to the world, but you can connect to it using DDNS or VPN service — which unfortunately makes the availability of the device difficult if we are away from home. Another disadvantage is the amount of session maintenance and usage of CPU for video encoding. It may turn out that it will not be possible to support more than two users at the same time. If these drawbacks are not that important to someone, you can always use this solution.

VPS

Purchasing a private server from the service provider will allow us to obtain more resources to use. It is also a good alternative, because we do not have to worry about access, and we know that the server itself is protected against failures. The only problem is usable space, which often ends with the offered 20 GB SSD.

Cloud Storage

There are a couple of options to get around this limit. The first one is to buy storage from the server provider itself. Unfortunately, this is probably the most expensive option. The second option is to purchase services for blop files. It works on the basis of file transfer in the form of file fragmentation of 20 MB. One such service is Wasabi, a good alternative to enlarge the server disk cheaper than Amazon S3, with no API or egress fees. If someone would like to test this service, there should still be a option for a first month free. The third option is to use a regular private cloud for data synchronization. This is quite an interesting option, but you have to read which clouds will be useful for use on the server. Private clouds have two modes, the first is to copy physical files 1:1, the second is to send files to the cloud and access them on demand, and we treat the file structure as links. For low capacity servers use the second option. After checking out the cloud offerings, I found that Dropbox was the easiest one (there are other service providers, of course). If you have an account with likely 2 TB of cloud data, it is good to connect it to one of the folders on the server. For known cloud services, it is strongly suggested using the rclone program for simple authorization on linux systems. In addition, it has the option of data encryption and decryption, so only people with the key will be able to see the real content of the cloud. After such an operation, our server with large capacity for data storage will be ready.

Summary

The last step is to install Jellyfin mentioned at the very beginning of post and select the folder of stored movie libraries. I also suggest installing an FTP server for easier file transfer to the server. Consequently, if someone is more advanced in console server operation, rsync is a good alternative. With such activities, what is left is to create an account, send files and enjoy the new private media service. I have been using the current solution for six months now and I must admit that I do not feel the need to switch to another platform. It is also especially good if several people work on one such website. Then it’s easier to find something new to watch and share your insights.

Reboot your old Raspberry Pi for home DNS + VPN

Tue, 27 Oct 2020 23:08:49 GMT

Every time I want to get on with the Raspberry Pi topic, it goes back to the closet after a few days. The problem with running on a home server is so bothersome that we have to do everything ourselves, from installation to configuration. So while the very idea of using the device is wonderful, its implementation and spent time can surpass us. Therefore, it is worth realizing that the most changes are introduced by the method of small steps. By reusing the Raspberry Pi lying in the corner, the first step might be to run a simple ad-filtering DNS server on it. The next step could be configuring a VPN service to get from anywhere to home network devices.

Current server situation

As you know, I deal with photography. The problem that this creates is storing photos for processing. I have had many attempts to store photos on physical, portable and cloud drives. Unfortunately, none of these solutions proved to be 100% successful. The reasons were different. From problems with synchronization, disk filling, downloading blop from the clouds, disk loss or keeping photos only on one device. Therefore, the current solution I have is to use the NAS server with 2 TB storage itself. Of course, I’m not saying that this is the perfect solution, but it meets all my requirements at the moment. Almost. This fact forced me to take action and install two really nice server projects for Debian devices. Thanks to the operation, from today I can use and view my photo collections from anywhere. And in fact, I’m happy with the current result.

PiHole

I omit the information that I am starting the topic in reverse order as I was installing services on the Raspberry Pi. I am simply aware that this topic will be more interesting and will be useful to a wider audience. There is one reason. This is the fastest way to get rid of bothersome ads while surfing the internet. Most of us probably use the AdBlock browser plugin. Unfortunately, the last year shown that there is a way to found if plugin is enabled, and some sites catch it’s and show us pop-up messages. The second problem is installing it on every device, bypassing the phone — unfortunately we have to use a clean browser instance here… Therefore, a great choice is to use the DNS service, which will block ads from showing up from the very network level. The advantages for sure will be:

no ads
no popups asking for whitelisting the page
faster page loading, because we do not load additional ad resources
statistics

The project that can be used is widely discussed PiHole. Its simple installation allows to quickly load settings with ready-to-go configuration. The last step is only to enter the router itself and, if possible, set static IP for our Raspberry and replace the primary DNS IP with the address of our server in DHCP options. After that, we could probably see our PiHole panel under pi.hole domain address. In a few hours, it could show that likely 30% of network traffic on websites are ads. And we are free of them.

PiVPN

PiVPN is an interesting project that I came across recently by accident. It is based on the philosophy of simple, automatic installation, just like PiHole. During configuration, we have the option of choosing two sites for installation. These will be OpenVPN services probably known by the majority and WireGuard. During the selection, I suggested information about easier WireGuard configuration on mobile devices. And in fact, as it turns out, configuration and performance as well as lower battery consumption are really an advantage of WireGuard. After a long analysis of these two solutions, it turns out that the program itself is very optimal (the entire code takes only 4000 lines, so you can read it at lunch and make sure it is safe). Additionally, there is an option to generate connection keys and a QR code for mobile devices. So it turns out to be a good competition to OpenVPN. After installation, it is worth remembering about redirecting ports to the server and, if we do not have a static public IP address, use DDNS services.

And so…

Without much describing how to install the projects, because the entire manual is on their pages, I can say that the whole process took me two hours. Of course, this includes finding a card reader and writing a clean image for the Raspberry Pi. So, if in the evening someone does not know what they could do, and has some time, I encourage trying to install these two services and improve own home network.

Any action to the future can improve access to information needed in time. Therefore, it is worth taking advantage of this opportunity and installing these services ever for the future tasks.

Just another fancy blog.

Setting Up a Terragrunt with tfEnv, SOPS, TFLint, and pre-commit

Repository Structure

Setting up basics

First, and most important file

Specify the Terraform version

Variable files

Using SOPS for Secrets Management

Maintain code quality with pre-commit

Lint your Terraform code according to best practices

Conclusion

Trip to Japan: Booking multiple connections tickets

How I discovered, that remote work not only allows you to work from home

Automate Instagram feed with GramAddict

Joining #100DaysOfHomeLab Challenge

Building ad-block home server from the scratch. Again...

Map all AWS resources and make a vulnerability report using cloudmapper

Secure files using git-crypt

Using Github 'actions'

Launch multiple WordPress server in docker way

Training: Tiny Python Projects

Installing Lutris on Debian from A to Z

Fixing my e-bike's speedometer and battery

Format the code automatically before commit

Sync AWS CodeCommit with Bitbucket by running pipeline

Set SSH

Create config file

Set Environment Variables

Create Pipeline Step

Set access to k8s thru RBAC Token

GitFlow & Versioning

Tags Automation

Docker Images Versioning

Git Merging Strategies

Simpler branches with git rebase

Easy and fast file sharing from the command-line

Not knowing Docker image configuration? Try runalike

Want privacy? Use multiple nicknames

Queuing jobs in apps is easier than you think

Producers & Consumers

Knowledge books: Python

Array operations

Code improvements

Command line arguments

Decorators

Docstring

Functions

Mutable Default Parameter Values

Mutable Objects

Args

Argument Tuple Unpacking

Argument Dictionary Packing

Keyword-Only Any Arguments

Keyword-Only No Additional Arguments

Python Function Annotations

Magic attributes

Loops & Enumerate

Content loop

Content loop with manual index

Range loop

Enumerate loop

Enumerate example usage

Array strip examples

Using iterator in enumerate

Object-oriented programming (OOP)

Other topics

Opening file streams

Practice materials

References

Return statements & generators

String formatting

Old Style” String Formatting (% Operator)

“New Style” String Formatting (str.format)

String Interpolation / f-Strings (Python 3.6+)

Template Strings (Standard Library)

Testing

Threading

Why dockerizing production isn't bad idea

Dev

DevOps

Simpler branches with `git rebase`