Jon couldn’t make it for this episode again, however he should be back next time!

Jerry mentions that he is using NetData, for monitoring his own infrastructure and also for his clients. He mentions how it can be used as a Prometheus Exporter, as a standalone package, and also has a Cloud/SaaS offering.

He mentions how it can pick up services automatically (if Netdata supports them – Integrations). RPM-based packages are available in EPEL and a third-party Debian repository (more information here).

Jerry mentions that it can run effectively as an agent to send metrics back to Netdata Cloud, which is different from how Prometheus has worked traditionally.

Stuart mentions that Prometheus are now adding a new feature called Agent mode. This is to solve the issue of needing to get access to Prometheus on a site, without necessarily wanting to open up every site in firewalls/security groups or running VPNs.

Jerry mentions issues he’s having with Let’s Encrypt currently, with Apache Virtual Hosts, specifically in how to automate it with Ansible.

Stuart mentions moving away from using Apache and starting to use Caddy, as he moving to using containers for deploying his publicly available services. Caddy comes out of the box with Let’s Encrypt support, removing one of the challenges in automation.

He also uses Traefik at home, as not everything is container-based and Traefik makes a mixed environment quite straightforward to use. Traefik is more complex than Caddy, but does have some extra features that Stuart makes use of.

Jerry mentions Dehydrated, a BASH implementation of an ACME server (what Let’s Encrypt is based upon).

Stuart mentions that he has been overhauling his home infrastructure. His aim was to move to using Git to define his infrastructure more, rather than the mixture of some configuration management, some adhoc, some scripts, with no consistency.

He mentions using Gitea for source control, and finding the awesome-gitea repository for what can be used alongside Gitea. He mentions using Drone for continuous integration, which has allowed him to move most tasks from manually-triggered to triggered on changes in his Git repositories.

He has put a series of posts on his blog about it here: –

• Introduction
• Moving to Ansible Roles
• SaltStack improvements
• Getting started with Drone CI

More posts on this are still to come!

Jerry asks about running Drone agents on something like Spot Instances or Spot Virtual Machines.

A discussion was had around our preferences on using an Open Source product with great documentation or a Commerial offering/SaaS with a support contract.

Stuart brought up the example of running something like Prometheus for monitoring (i.e. running a monitoring stack yourself) compared to something like Datadog that runs the monitoring stack for you.

Jerry mentions it is entirely dependent upon the service.

Stuart mentions that it can be nice to look through code to see where an issue might be that you are facing (and even contributing fixes).

No detailed show notes for this particular episode, as our intrepid stargazers are looking into the past to see how their predictions for 2021 fared… and then consulting their crystal balls to make their new predictions in technology for 2022.

Jon couldn’t make it for this podcast due to a recent job change, but will be back soon

Stuart and Jerry talk some about their new jobs.

Stuart is a Site Reliability Engineer for a VoIP/Communications company. He talks about using Puppet, Terraform, Nomad and Kubernetes. Jerry and Stuart both talk about the move to containers in both their jobs.

Jerry mentions learning Amazon AWS’s ECS (AWS managed Docker/Container solution) using Fargate. Stuart mentions using ECS previously, but using AWS EC2s rather than Fargate. Stuart also mentions that ECS is a lot simpler than Kubernetes, but the simplicity does have some trade offs.

Al mentions he has recently recertified his Azure Administrator Associate certiication. He mentions how the certifications are “point-in-time”, in that it doesn’t reflect some of the newer features.

Al also mentions the Late Night Linux Extra podcast episode featuring Martin Wimpress (of Ubuntu MATE and ex-Canonical fame) episode on Docker Slim

Al mentions Azure Web Apps, which are effectively Docker containers in the background.

Al asks an open question about monitoring and how it changes in the world of cloud, PaaS (Platform-As-A–Service) and microservices. He mentions how throwing machine resources at a problem doesn’t always fix an issue.

Stuart talks about the idea of contention in the cloud being desirable, compared to being avoided in on-premises environments. He mentions his issues with using purely thresholds for monitoring. He refers to distributed tracing to get insights into requests/services (especially when running across a number of microservices).

Stuart mentions the Golden Signals method of monitoring. He also refers to the Site Reliability Engineering handbook from Google.

Jerry mentions about using Prometheus for metrics, specifically the node_exporter as a lightweight agent for monitoring node metrics.

Stuart mentions OpenMetrics (which is the Prometheus metrics format but as an open standard) which can be exposed by any application, not just a specific exporter. He mentions adding this to his own applications, and writing exporters as well.

Stuart talks about eBPF, how it relates to monitoring, as well as tracing and forwarding packets. He mentions eBPF programs that are allowed to sit alongside the kernel itself, allowing direct kernel tracing or taking actions on network packets before they reach the kernel.

Stuart references Brendan Gregg and his website for information on eBPF usage and examples. He also later mentions Liz Rice for great information and tutorials on eBPF, having started learning eBPF because of her great tutorials.

Stuart mentions about start to learn C to be able to write eBPF programs. He also mentions that you can interact with eBPF programs using Go, Python, C and Rust, whereas the eBPF programs themselves are either in C or recently in Rust.

Al mentions that Azure Web Apps for PHP include Apache for PHP 7, and Nginx for PHP 8.

Jerry brings up Terragrunt, which is a thin wrapper for Terraform. Terragrunt extends Terraform with some useful features like being able to run Terraform across multiple directories, and to make Terraform DRY (Don’t Repeat Yourself). It can also show a graph of dependencies too. Stuart mentions why separating Terraform files into different directories is desirable, but comes with a trade off that Terragrunt can help resolve.

Jerry mentions how using Terragrunt to separate environments and parameterise Terraform helps significantly with keeping repitition of code lower.

Al talks about Terraform Workspaces as a way of separating environments.

Al brings up the subject of other podcasts we listen to, including: –

• Ship It – About deployment, infrastructure and the operation of software
• Rent, Buy, Build – About the cloud native world and whether to use a managed solution, an off-the-shelf solution, or building it yourself for different technologies
• Al’s Code Snippets Podcast – About Al’s journey into coding and his learnings along the way

We’re without Al again this episode, but we carry on regardless!

Stu talks about Puppet which is a configuration management system, comparable to Ansible, Salt Stack or Chef.

Like Chef and Salt, Puppet is predominently agent based, where the agent is installed on the endpoint, and it calls out to a central server, every X period of time (Jerry mentions 30 minutes at one point in the show, while Stu says 15 minutes) to get the state the device should be in, and it then tries to remediate all those items which are not compliant with the state.

Puppet is more like Chef than Ansible or Salt in that it uses a Ruby “Domain Specific Language” (or DSL) to define the configuration of the node, rather than YAML.

We then get into a more general conversation about configuration management software, including talking about how Salt Stack allows you to create entire tasks and variables using jinja2 templates, and Jon mentions he did something like this with Ansible variables. Jon mentions seeing a video from an early PuppetConf where a member of the board (he thinks the CTO) decided to learn Puppet by wiping and reinstalling his machine every day using Puppet. Sadly, he can’t find this video now, and would appreciate listeners pointing him to that video, if they can find it!

Jon talks about Architecture Decision Records (or “All” Decision Records) writing bash scripts, and using BATS to perform unit testing of bash files. He also mentions that it’s possible to “mock” specific commands in BATS.

Lastly, Stu proposes we talk at about using Cloud Native services in AWS, Azure, etc. versus using Infrastructure as a Service. A series of specific services on AWS and Azure are mentioned. We talk about how vendor-lock-in can occur and some of the things you can do to help prevent that. Jon mentions the books “The Phoenix Project” and “The Unicorn Project” by Gene Kim which discuss the idea of “Core” services (which make money for the company or project) and “Context” services (which don’t, and can be outsourced.) We also talk about the issues involved in not transforming your services when you “Lift and Shift” services into a cloud service.

We’re a member of the Other Side Podcast Network. The lovely Dave Lee does our Audio Production.

We want to remind our listeners that we have a Telegram channel and email address if you want to contact the hosts. We also have Patreon, if you’re interested in supporting the show. Details can all be found on our Contact Us page.

First episode of 2021! We’re in lockdown number 3 in England!

Jon admits to writing a private-only diary using WordPress (he doesn’t mention he also has a separate photo diary). Jerry mentions that another of his friends also has recently started a diary using WordPress, and suggests that maybe this is a new trend.

Jon is also Internet Famous due to a post he made on StatusNet in 2009 (mirrored to twitter) that got captured in the screenshot of a StatusNet client and posted to Wikipedia.

Jon wrote a post on his blog talking about how he got into his career. He would encourage anyone else to write something similar, particularly if they’ve taken an unusual route into their career!

Al asks the team what he should learn about. He talks about the tooling they’re using – Bamboo, Azure DevOps, Terraform, Ansible. We talk a bit about what Bamboo is, what a code pipeline entails, and how they’ve used it. Jon mentions that Lorna Jane Mitchell talks about moving from Travis to Github Actions on her Twitch Stream. We then drill into using Terraform modules.

Jon mentions about “Architecture Decision Records” and cites files in the gov.uk public repo as an example of this. It’s similar in principle to IETF RFCs. He found it via the Last Week in AWS newsletter issue 195 (which at the time of writing was only available to subscribers). He mentions the tooling (“adr-tools“) which you can use to write these records.

Al then asks where we find time to learn. We all talk about what we do, some at more length than others.

Al talks about being OK about being alone. He mentions about his life coach, the “Alonement” podcast, and the talk he gave at OggCamp about staying positive on a digital world.

Jon then reminds our listeners to check in with family, friends, colleagues and neighbours to make sure they’re OK.

In this bumper feedback episode we talk about line endings in files, OpenStack, secrets management, and protecting your network.

• Iain asks:

  Hello all,

  I hear a lot about Openstack but whenever I try to find out stuff about it, I get vague buzzword-laden and vague comments from “evangelists”. Could any of you guys explain to an IT-literate but not a sysadmin end user, what the hell Openstack is?

  Stu explains what OpenStack is built for, where it’s often deployed these days, and some of the issues he’s seen with it.

  Jon talks about some of the components inside OpenStack, and how OpenStack upgrades can have issues.

  Jon and Stu talk about companies who were selling OpenStack distributions, and comparisons to Kubernetes.

  Jerry mentions that many of the problems OpenStack was created to solve are now mostly solved by Kubernetes. He also mentions that we discussed Kubernetes in Episode 51.

  Jon mentions Eucalyptus, nominally as an alternative to using AWS S3 or OpenStack Swift (the object storage module), but also mentions it could be used to virtualize some of the other services provided by AWS.

• Al asked about “Dark Matter Engineering” which he’s heard about on Coder Radio. We presume it’s code that isn’t released into the public, or never gets any traction. We also discuss Linode and compare it to Digital Ocean as a result of the adverts run on Coder Radio.

• Jay provides some feedback:

  Hi, in your last podcast someone mentioned having an issue with VSCode in windows always saying that files were all edited.

  What’s probably going on is a wrong setting for the core.autocrlf setting.

  You can fix it by opening powershell in windows and running

  git config --global core.autocrlf input

  https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration#_formatting_and_whitespace

  There are 3 settings, but I always recommend the ‘input’ one, as it converts everything to LF endings on commit, and checks out without modification.

  Also, you may be interested in a recent networking video series I made: https://jaytuckey.name/2020/10/18/how-websites-load-a-deep-dive-into-the-ip-network-stack-and-how-it-is-used-to-connect-to-a-site/

  Jon talks about how he’s got Microsoft’s Windows Subsystem for Linux (WSL) setup and how he organizes his “My Documents” directory structure. He mentions “Symbolic Links” to easily find Windows directories in his WSL environment.

  We also talk about Line Endings (CR – Carriage Return, LF – Line Feed, CR+LF). Jon incorrectly recalls CR as ASCII character 10 (it should be 13) and LF as ASCII character 13 (it should be 10).

• WIE E asks for help with Secrets Management in a Continuous Integration (CI)/Continuous Delivery (CD)/Continuous Deployment (CD) environment.

  Stu talks about Gitlab, HashiCorp Vault, and AWS IAM roles, which Jerry extends to include Azure System Assigned Identities. Jerry mentions that you can use your provisioning system to create a per-system key during a build, which never commit to your version control system.

  Jon mentions about protecting CI/CD/CD systems and references the exploit of a CI/CD system on the Matrix.org project.

• Yannick asks:

  VPN: always on or not?

  How to protect the target network – i.e. does my machine becomes the weakest link in the network and what can/should I do to protect the network ?

  Jon talks about his views on always-on Client-to-Server Virtual Private Network (VPN) connections.

  Al mentions that he thought the question was talking about Site-to-Site VPNs, and Jon suggests that VPNs typically now auto-establish themselves when traffic is initiated from the “Encryption Domain” on one side of the network to the “Encryption Domain” on the other side of the network. Jon refers to IPsec Phase 1 and Phase 2 which are two stages of a VPN tunnel, dealing with the initial connection between the “left” and “right” sides of a VPN tunnel, and the connections between two encryption domains (subnets or hosts at either end of the tunnel). Jon also mentions about various encryption algorithms like DES, Triple DES, AES, and hashing algorithms like SHA1.

  Jerry quotes “Clarke’s Third Law“: Any sufficiently advanced technology is indistinguishable from magic.

  Jon mentions about the Diffie Hellman Key Exchange video, and then talks about browse-down management environments and references the National Cyber Security Centre (NCSC) End User Device security guidance for hardening machines. He also talks about segregating network segments for protecting trusted and untrusted networks, and then goes into “Zero Trust” networks, mentions “CASB“. Jon and Stu both talk about broadcast domains in a network, and how you can work around that.

  Jerry mentions about Bastion Hosts, and Jon explains why they’re not really a good idea.

  Jon butchers talking about User Behaviour Analytics (UBA) systems. He also mentions about a protocol break.

Wrap up

We’re a member of the Other Side Podcast Network. The lovely Dave Lee does our Audio Production.

We want to remind our listeners that we have a Telegram channel and email address if you want to contact the hosts. We also have Patreon, if you’re interested in supporting the show. Details can all be found on our Contact Us page.