Category Archives: Show Notes

Admin Admin Podcast #086 Show Notes – Committed to Cloud

The whole crew is back together for the first time in a while, talking about: Git commit hooks, Windows as a development environment, cloud network firewalls, and Azure DevOps.

What have we been up to?

  • Jerry started a new job, and he’s re-started using Windows 10 as part of his job after a few years of using OSX and Linux. He’s using Windows Subsystem for Linux (WSL). We talk a bit about the difference between WSL 1 and WSL 2, and comment about how WSL 2 uses Hyper-V and what that means compared to using VirtualBox or VMWare for virtual machines. We compare WSL 1 to a “reverse” WINE.
  • Stu mentions that you can’t run some network controls (like traceroute) with WSL 1 because some of the kernel calls are not available.
  • Jon notes that he used the terraform for Windows binary in WSL by mistake and couldn’t authenticate to AWS because he’d installed the AWS CLI for Linux. Installing Terraform for Linux and also putting the AWS configuration files into both the Linux expected path (~/.aws/config) and the Windows expected path (%UserProfile%\.aws) worked around this issue!
  • Jon then mentions using /etc/wsl.conf to configure mounting the Windows drives into WSL, and notes that you can configure it to permit POSIX style file permissions with this sample:
[automount]
options = "metadata"
  • Jon also creates a symbolic link between /mnt/c/Users/Jon/Documents and ~/Documents to “easily” get into the Windows paths that are backed up in Windows.
  • Jerry notes that he’s using the Windows version of Virtual Desktops. He’s also using the Microsoft Terminal application.
  • Al mentions that if you navigate to \\wsl$ in Windows Explorer, you can access the Linux file system from Windows Subsystem for Linux. Stu mentions he has this open in the left hand pane in Explorer all the time!
  • Al said he’s using Visual Studio Code (VSCode), and uses that to open Windows Subsystem for Linux, and he also mentioned that if you type in code in any path in WSL (or Command Prompt, for that matter [ed.]) it will open that folder in Visual Studio Code.
  • Jerry notes that he’s just moved to using VSCode, but has installed the Vim extension. Jon asks whether he’s installed the “Butterflies” extension, referring to an XKCD comic.
  • We talk briefly about using Git in VSCode, versus using it from the command line. Jon mentions a specific bug he has. Jon talks about the differences in line endings between Linux and Windows systems.
  • Al talks about using Azure DevOps with it’s pipelines. We talk about it’s history, and compare it to other products. Al mentions using Azure DevOps to trigger Terraform using PowerShell. Al also mentions using AWX (the open source upstream version of Ansible Tower), and having an agent for Azure Devops running on his AWX service.
  • Jon mentions the DevOps.fm podcast in the context of Azure DevOps. Stu asks about running PowerShell on Linux. Jerry mentions a Binary Times podcast episode where they interview the person, “dementor”/”the mentor” who runs the Powershell On Linux, Al mentions the Makers Corner podcast which also interviewed the same person.
  • Jon is writing Terraform to deliver 3rd party security appliances in AWS and Azure. He notes that most of the AWS appliances use a Transit Gateway to set this up. Jerry and Stu mention how they use Terraform Modules. Jerry mentions automating Jenkins with Terraform. Stu and Jon talk about using count and for each statements. Jon also mentions about defining which “providers” to use in the Terraform files. He also notes that you can get into a dependency loop if you have several modules with different provider files. Talking of Providers, Jon mentions using the “null” provider, but doesn’t explain what he uses it for.
  • Jon talks about Git hooks, and Jerry talks about a python project called pre-commit which can help to automate some of these pre-commit hooks, like calling a linter or a unit testing system (like ShellCheck) before the commit completes. Stu mentions using the GitLab Continious Integration (CI) system instead of using Pre-Commit hooks. Jon suggests when it might be preferable to use Pre-Commit hooks instead.
  • Stu mentions about SourceHut, which is an alternative to GitHub which uses email for patch sharing.
  • Al talks about using the Azure Firewall product, and Jon and Al drills down into how Azure Networking works. Jon then explains how High Availability events occur in AWS and Azure with 3rd party firewalls. Jon also mentions AWS Firewall Manager. Jon also mentions that Terraform and Ansible have code to write and amend AWS and Azure Firewall rules.

Feedback

  • We received feedback (although the source has now been lost) about how we pronounced “Oriented” as “Orientated”. These are both valid words in UK English and are interchangable in UK English.

Wrap up

We’re a member of the Other Side Podcast Network. The lovely Dave Lee does our Audio Production.

We want to remind our listeners that we have a Telegram channel and email address if you want to contact the hosts. We also have Patreon, if you’re interested in supporting the show. Details can all be found on our Contact Us page.

Admin Admin Podcast #085 Show Notes – Verbosely build your objects

No Jerry this time, but we do have Al back!

What have we been up to?

Feedback

  • Wayne (from the Binary Times podcast) contacted us to suggest that we’re not being very good at explaining what terms mean. We try to clear some of the terms up that we use!

Techniques for Rebuilding a machine using Post-Provisioning Tools (like Ansible)

  • Jon suggests a process of using Git on /etc/ and /home/<user>/ with Vagrant to test each stage of the build, and to see what files are changed by each action performed. Once you’ve got your build instructions based on that, you can use something like Ansible, SaltPuppet or Chef to apply pos-install statements.
  • Stu mentions using Chocolatey for installing packages in Windows. Jon mentions that using BoxStarter works well for automating Chocolately installs. He mentions using boxstarter paths which are currently not documented – https://boxstarter.org/package/<yourpackage> and https://boxstarter.org/package/url/?some_path_to_a_boxstarter_set_of_instructions.
  • Jon mentions the Ubuntu Server Vagrant Box file, and the Desktopify script written by Martin Wimpress. He also talks about provisioning Windows machines where Terraform renames machines and adds them to the Active Directory Domain.

Object Orientated Programming

  • Al wanted to know more about what Object Orientated Programming (OOP) is, as he’s started looking at ASP.NET and has previously only known Classic ASP. Jon talks about it from his previous PHP experience and perspective. Jon talks about when he used OOP in a previous role to perform network device backups. Jon mentions he’d used OOP in CCHits.net and the now defunct CampFireManager.

Wrap up

We’re a member of the Other Side Podcast Network. The lovely Dave Lee does our Audio Production.

We want to remind our listeners that we have a Telegram channel and email address if you want to contact the hosts. We also have Patreon, if you’re interested in supporting the show. Details can all be found on our Contact Us page.

Admin Admin Podcast #084 Show Notes – Git your stack here!

  • Al couldn’t make it for this recording cry
  • Jon broke his QNAP NAS with Debian. He doesn’t go into any details, and will leave it to another show (as long as he can remember what he did until then)!
  • Stu has been blogging, now on Consul, Saltstack and Prometheus on LOTS of different platforms.
  • Jerry started a new job. The CTO at his new firm is possibly a listener! His NAS has failed, and he’s building a ZFS mirror to move the data to. He used Syncthing to move the data off to another drive, and is using Backblaze to run a backup.
  • We discuss possible other uses of Syncthing, in particular, one YouTube creator talks about how he uses SyncThing for his video editing workflow (part 2).
  • Jon also mentions that he’s been watching some of Martin Wimpress’ YouTube channel, and in particular, the series where he created the Desktopify script for turning a Raspberry Pi Ubuntu Server image into a Desktop Flavour.
  • We talk about “Hashistack” (referring to the collection of tools released by Hashicorp, which are “Terraform“, “Packer“, “Consul“, “Vagrant“, “Vault” and “Nomad“). All the hosts provide summaries of how each of these tools work (except Nomad) and why you might use them.
  • We discuss using Git.
    • Jon talks about DangItGit (and a slightly more rudely named version of that site), and mentions a comic on XKCD about Git.
    • Then he mentions some of the things about git which may give it a bit of a bad name, like SubModules.
    • He also mentions that you can use “hooks” which are scripts that run before or after certain actions (like a commit or a push), and Stu talks about how he’s used that in the past.
    • Stu also talks about some of the tools in Github and Gitlab which are similar to hooks, that run when Github or Gitlab actions occur (like a pull/merge request being raised, or an issue being opened).
    • Stu also talks about CI/CD pipelines and Gitlab runners.
    • Stu and Jon talk about Pull Requests (Github terminology) and Merge Requests (Gitlab terminology). We all talk about Issues and Wikis within Github and Gitlab.
    • Jon talks about what a Fork is.
    • Stu reminds us that Git is not the only version control system, and that Subversion (SVN) is also out there. Jerry mentions CVS. Jon mentions Mercurial (HG) and we talk about where version control systems have come from.
    • We diverge into how Blockchain is similar to Git… and why it isn’t the same.

We’re a member of the Other Side Podcast Network. The lovely Dave Lee does our Audio Production.

We want to remind our listeners that we have a Telegram channel and email address if you want to contact the hosts. We also have Patreon, if you’re interested in supporting the show. Details can all be found on our Contact Us page.

Admin Admin Podcast #083 Show Notes – Parameterize This

  • Jon has reinstalled his QNAP NAS with Debian.
  • Jerry has been running Folding@Home with in his K8S environment.
  • Jon mentioned BOINC which he erroniously mentioned was what Folding@Home uses – it’s not, but there are other BOINC projects and that you can run, and that you can run BOINC on Android. You can’t run Folding@Home on Android.
  • Stu was blogging about managing Arista EOS with Ansible, and that he’s working on his next posts, firstly managing the MikroTik RouterOS in the same way, and also he’s looking at building a OpenBSD based equivelent too.
  • Al has been working with Terraform. He’s moving from using ARM templates to using Terraform Configuration Files.
  • Jerry suggests building a VM to a patched image, and then deploying the patched image instead of just building up a machine from a stock market image.
  • Al is doing something like that already.
  • Jon suggests some naming conventions with regards to Terraform configuration files. He also suggests using modules in Terraform. Stu and Jerry do the same thing.
  • Stu mentions that Terraform modules can be used with git tags.
  • Stu also suggests not provisioning EVERYTHING with the same directory of configuration files.
  • The team reviews the operations of the Terraform binary.
  • Stu mentions that Terraform has a lifecycle setting which may prevent accidental deletion of resources.
  • Al is also moving to using Ansible to post-provision the virtual machines.
  • Jon talks about how Ansible Tower and it’s upstream open source project, AWX, works, including scheduling, credential abstraction and the availability of web hooks.
  • Al talks about how he’s looking to use Ansible against his environment, and looks for some better practices in using Ansible.
  • Jerry suggests using inventory and roles. Stu suggests using tags to only run parts of the playbook. Jon uses conditionally included roles instead of using tags.
  • Jon explains about Ansible Galaxy, Roles and Collections, based on his attendance of a talk about the Ansible roadmap at Red Hat Summit. Stu mentions using ansible-galaxy to create role template directories.

Astute members of the community will notice that we’re now a member of the Other Side Podcast Network.

We want to remind our listeners that we have a Telegram channel and email address if you want to contact the hosts. We also have Patreon, if you’re interested in supporting the show. Details can all be found on our Contact Us page.

Admin Admin Podcast #082 Show Notes – The Four Amigos

TRIGGER WARNING: We mention the current Covid-19/Coronavirus situation a few times in the podcast, but without really going into any details about it.

We add Stu to our permanent line-up! Welcome Stu!

Al started a new job. He’s doing Agile working, with sprints and standups. They’re On-Prem and in Azure. He’s considering looking at Ansible with AWX to standardise their builds. He’s started using Slack, and noted that the company he works for uses Slack rather than Email for most conversations.

We talk about using GMail instead of Exchange. Jon mentions about a blog post talking about improving workflow in GMail following a comment in the Bad Voltage community slack.

Jerry mentions that Slack’s free plan has a limit on the number of messages you can recall. Stu mentions that his company were using Slack, but that they’ve started the migration to MS Teams. Jon mentioned that the backgrounds in Teams videocalls can be changed, or set to a blur. [New Path?]

Jon explains what CI/CD/CD stands for and explains what it can be used for. He also mentions that he wrote some AWX deployment scripts as part of a Gitlab and AWX demo which might be useful. He also mentions that he recorded a video about how AWX works.

Jon explains that he’s been writing documentation at work, and that outside work, he’s building a card playing game script that is based on the code he wrote for talk scheduling at OggCamp and inspired by the code he wrote for CCHits.net. Al also notes that Laravel is good for a PHP framework, and mentioned that Jon suggested it to him…

Al mentions playingcards.io as an alternative to writing his own game, and said he uses that to play Cards Against Humanity. Jon counters with houseparty.com .

Al then said that he’s using Git at work, which is the first time he’s using Git at work, rather than just in his personal life. Jon asks if Al’s signing his commits, and suggests using krypt.co to perform Two Factor Authentication (2FA) where you pair your phone to a browser and use the phone as the U2F authenticator, and it also has a mode where you can also pair the phone to enable signed git commits and use the phone as an separate SSH key provider too, if you turn the “developer” switch on in the phone app.

Stu talks about bypassing AWS network architecture moving to linux based routers, moving Prometheus/Consul into production, and why they’re doing that, and about some blogs he’s been writing about automating network products with Ansible. Jon talks about the Ansible modules moving out from Ansible core, and into Ansible Collections. Jon mentions looking at Nebula instead of changing the AWS network architecture, and explains how this works with NAT environments. He makes reference to a Pull Request he’s raised to add more documentation. We talked about Nebula in Episode 80.

Jerry has just got a new job, which is a permanent role, making a change from his previous freelance environment. Until that job starts he’s been writing some documentation on Disaster Recovery for sysadmin with VProtect, and also been looking at providing some support to a developer to provide configuration management tooling and new images with Packer [ ].

Al mentions that another podcast (the Mike Tech Show) had a question about using appliances that need IPv6, when you don’t have IPv6, like several of the hosts have with PlusNet. Jon used Hurricane Electric to create an IPv6 gateway. The downside to this was that the connection became much more flakey because you’re effectively using Hurricane Electric as a VPN provider. Stu mentions that this is likely to be because of “Happy Eyeballs“. We talked about Jon’s IPv6 gateway in Episodes 73 and 72.

Jerry mentions that he had an interesting situation because of his printer and was being detected on it’s IPv6 address, not on the IPv4 address. Jon makes some suggestions on alternatives using trunking or VLANs. We discuss how complicated our networks are, and what our partners/spouses will do if we’re not available in case of a disaster with that network.

We want to remind our listeners that we have a telegram channel and email address if you want to contact the hosts. We also have Patreon, if you’re interested in supporting the show. Details can all be found on our Contact Us page.

Admin Admin Podcast #081 Show Notes – Contain your enthusiasm

With the guys all back together, they talk about the Fully Automated Install (FAI) system, Kubernetes, and their recent projects.

Jerry mentions K3S – a simple Kubernetes (K8S) deployment, Jon mentions he’s reimaging Windows on his Laptop, and has been working on his AWX (he says Ansible Tower, but means AWX) install and configure Github Repo. Al has a new Job doing DevOps on Azure and mentions CI/CD (Continuous Integration and Continuous Delivery or Deployment) and Azure DevOps. The new job will be more Agile, and be working in Sprints.

Al talks briefly about SnapRaid and MergeFS. With the assistance of Stuart, who previously guest hosted, they have been building a dashboard for Prometheus with Node Exporter and Grafana that shows a lot of the automated tasks that Al previously received by email, and now he just has that as his opening tab on his browser.

Jerry talks about what he’s done with K3S. Jon mentions he also has done some stuff with K3S and that he has that published in a Git Repo. The Git Repo he’s created also includes a script to deploy to multiple machines and to include MetalLB to make K8S provide a load-balanced connection across multiple K3S nodes, without needing an external load balancer. MetalLB also lets you advertise addresses over BGP.

Jerry says that Plex can use multiple nodes to transcode. He also wants to mount persistent volumes with NFS, and so he’s experimenting with K8S to do this. Jon mentions Rook to do cross-cluster persistent volumes, and it can use Ceph to do that.

Al asks why use Kubernetes rather than Docker. Jerry and Jon give their viewpoints. Jon mentions a blog post called “‘Let’s use Kubernetes’, now you have 8 problems” and some courses on Pluralsight about the Container big picture, as well as deep dive courses on Docker and Kubernetes. Jerry mentions Podman.

Jon talks about the youtube video he recently recorded, and the inspiration for it, in a video by podcaster Chris Hartjes he found on Pluralsight. The video is about Vagrant, Ansible and Inspec. Alan Pope (@Popey from the Ubuntu Podcast and the User Error podcast) suggested publishing the video on Lbry too, which Jon did. Jon talks a little about Lbry. Jerry and Al talk about how they consume content, and Jon talks about his motivation (mostly because of a comment from Reggie from The Coolest Nerds in the Room Podcast).

We talk about a question from Yannick in the Telegram group, which is where he asks if we can advise on “Setting up a secure access to your home network : the bad way, the better way and the best way”. We talk about SSH, running VPNs (like OpenVPN) using PFSense, or using Raspberry Pis (using PiVPN). Streisand (which provides tools like IPsec with IKE, OpenVPN, OpenConnect, and Tor).

Jerry talks about FAI – the Fully Automated Install project that he has used at work as a tool to build Debian based systems and CentOS based systems.

We mention that we have a Patreon account, and encourage our listeners to join us in our Telegram group.

Admin Admin Podcast #080 Show Notes – Fired up about monitoring

In this episode, possibly the shortest since Jon joined the team, we have a conversation with Stuart (Mastodon | Twitter), who is a member of our Telegram community. We’re also missing Al.

Stuart talks about Prometheus, and compares it to Nagios. He talks about the differences between how Prometheus collects data, particularly how Prometheus talks to local exporters to collect metrics, rather than polling data every 5 minutes. He lists a collection of exporters from a whole range of products (too many to list here!) and then Jerry and Stuart discuss rewriting native data sources into a format that Prometheus works.

Stuart has linked to some additional sources of information about Prometheus:

Moving on with the show, we cover for the fact we’re missing Al by asking two questions on his behalf, the first covers how we believe Al is suffering from Alert Fatigue, and how he can collect results from scripts that run on his servers in a specific way. Stuart explains how he’d use Prometheus for this, Jerry mentions that he’d collect logs for later parsing and only forward logs in the case where the script has failed to run successfully. Jon mentions that he’d consider using Monit to run the tasks, as that will notify if the job fails to run. He also suggests using triggers for bash scripts to send an email on failure, and changing email titles based on the outcome of the task.

He also asks about monitoring disks on a homemade NAS. Jon mentions he’s used Monit with SmartMonTools (similar to this page) to monitor disk statuses in the past. Jerry and Stuart also mention that he could be using Prometheus for this. We also discuss that this may in fact be built into the NAS he’s trying to build. We discussed monitoring with Lucy in Episode 77.

Jon talks about the testing he’s been doing with Nebula, which is a meshed overlay VPN (Virtual Private Network) product, and compares it to a Hub-and-Spoke (or Star) VPN topology. He compares it, briefly, with ZeroTier and mentions that he needs to do more exploration into ZeroTier.

Jerry asks Stuart some questions about SaltStack, and compares it to Ansible.

As always, we’d encourage any listeners to join our Telegram Group, or contact us using the other links! We also have a Patreon which you can use to support the show if you’re so inclined.

Admin Admin Podcast #079 Show notes – A conversation with the coolest nerd in the room

In this episode, Al and Jon (no Jerry this time, sadly) have a conversation with Reggie from The Coolest Nerds in the Room Podcast.

Reggie is a Site Reliability Engineer (SRE). SRE was a term coined by Google in 2016. SREs will often perform operations roles, similar to those performed by “DevOps” or Operations teams, but are also responsible for reliability by monitoring the health of a service, an application or a node, and reacting to issues with a longer term view on solving those issues.

Reggie went into how he moved into an SRE role, and went into some details on the platforms he’s used in the past, including AWS, Azure and Google Cloud.

Reggie mentions the following terms:

  • Kubernetes (sometimes abbreviated to K8s) – A container orchestration tool, run by the Cloud Native Computing Foundation. Jon mentions MiniKube, which is a way to run Kubernetes on your local machine.
  • Stackdriver – a monitoring tool.
  • SLI – Service Level Indicator. An SLI is an indicator which is observed on a service component, like remaining storage capacity, CPU utilization by a specific application, number of errors returned by the application, response time to retrieve a specific page element, and so-on.
  • SLO – Service Level Objective. An SLO is the target for the SLI items on the host. For example, you might be looking for an SLO of < 5 non-OK HTTP responses in 1 hour, or perhaps that the login service returns a response in less than 3 seconds. This is typically a lower threshold than the SLA, and is the point where an SRE would be engaged to identify *why* the service was degraded before it becomes an issue.
  • SLA – Service Level Agreement. An SLA is a contractual agreement between the service provider and the service consumer, for example between a website and it’s user, or between a microservice and the overarching service it’s trying to deliver. The SLA might refer to SLO-like components, for example “logging in must take less than 5 seconds” or “no more than 10 minutes of outage time in a given month”.
  • Error Budget. This wasn’t explored particularly in the show, but seems to be an “acceptable” level of SLO failure that, if that threshold were crossed, should trigger the engagement of the SRE.

Next, we go into how Reggie started his podcast with Steph. We talk about how the podcast developed and how they keep their momentum in tech. This turns into a wider conversation about working in IT.

Reggie talks about how Kubernetes works, and how this has changed his workflow. We mention “Pets versus Cattle“, Microservices and Containers.

Reggie talks about how he learned about Kubernetes, and things he feels you need to understand about Kubernetes to be able to use it well. We mention that it’s worth learning about how Docker works (as a Container primitive), and then growing out to using Kubernetes. We mention that all the major cloud providers (AWS, Azure, Google) have Kubernetes platforms, that you can host Kubernetes in your hosting environment, and that you can also run MiniKube to learn Kubernetes on a small number of machines.

Reggie suggests that the Velocity Conference was very worthwhile getting to!

Reggie goes into more detail on what being an SRE is about, and talks about why Google and other large companies are moving towards using the SRE roles.

Reggie talks about bringing more diversity into tech, and that nerds are frequently very harsh about excluding people based on their choices and preferences. He also endorses bringing new people into your environments, and mentions that these can be good opportunities to examine why you do things and to ask if how they’re done is the right way to do them.

Reggie mentions that he puts videos on Instagram about tech basics, and encourages people to let him know when there’s something they don’t understand!

Wrapping up, we thank our Patreons, Dave for being our superproducer, and invite you to chat with our audience on Telegram, or directly to the team by email, especially asking any questions you want the podcast to answer! 

Admin Admin Podcast #078 Show notes – Unrolling OggCamp 2019

For this week’s episode we are sitting in a hotel lobby discussing OggCamp 19,  with special guest Gary Williams and Special thanks to Joe Ressington, standing in with his recording gear to record the podcast.

Al did a live demo for a talk and it did not work due to demo gods in: “How I use wireguard to connect to my VPS” but got it working after the event. More info can be found here.

We all agree this was the best talk at OggCamp “The power of change – learning to live as a “weirdo”” by Rachel Morgan-Trimmer.

The Oggcamp kids’ track continues to grow..

Al, Jerry and Gary mention about Talk “The MQTT, InfluxDB, NodeRED and Grafana stack, and natural intelligence” by Julian Todd and his @wheeliepad.

Al and Gary have a go at lock-picking.

Gary talk to us about how he migrated from being a SysAdmin to DevOps engineer.

Jon talks about “Noobs on Ubs (Ubuntu for Beginners) ” talk by Anna Dodson

We have many ways you can talk to us, including email and Telegram. Details for reaching us on these are on our contact page.

Admin Admin Podcast #077 Show notes – The one about monitoring

We introduce our guest – Lucy McGrother.

Lucy is a colleague of Jon’s, who worked in Windows Support, Enterprise Management and now SOAR (Security Orchestration, Automation and Response).

Jon explains what SOAR is, and Lucy improves his answer.

We introduce the question of Monitoring, as raised by our Telegram group.

Lucy explains that you need to start by asking “What do you want to monitor”, and the answer shouldn’t be “everything”. We also talk about how you can respond to monitoring events. Lucy makes a sensible point “When you get an alarm from a monitor, it’s just telling you there’s something wrong to be looking at, and it’s up to you to add the intelligence to it”.

We discuss what enterprise monitoring tools we’ve used, including SCOM (System Center Operations Manager – a Microsoft product, part of SCCM) and CA OIM (previously known as “NSM”, “TNG”, “NISM”). We also mention some open source tools, like Zabbix, Nagios, Monit, Grafana and a free/paid product PRTG.

There’s also a conversation about how you can monitor processes running on a machine to reduce the amount of “noise”. Jon mentions about writing content to a log file, and capturing the output, but that won’t capture all the updates, Lucy mentions you can just monitor whether a log file has been touched in X hours!

Jerry talks about Nagios monitoring plugins, and how they would report issues using error codes.

Al mentions the podcast “Self Hosted Show“.

Jerry talks about the difference between metrics and polling. Lucy mentions that she did a Microsoft Statistics and Analytics course, and that your polling tool should be feeding metrics data for later use.

Jon and Lucy draw some information from their pasts about dealing with incidents and about how it’s difficult to pull logs from boxes, especially when there’s a need to resume service as soon as possible. We also discuss the difficulty of having a constant log transfers to other devices, particularly in carrier grade equipment that might be processing many gigabytes per second, a proxy for a large company that might be producing many 10,000’s of log files per 24 hours, collecting logs from cloud providers that charge for egress traffic, or perhaps if there’s someone malicious inside your network that is trying to hide their actions, they might spam the monitoring solution with valid or invalid log entries to frustrate investigators.

Jerry talks about how application developers he’s worked with frequently embed log collection features into their applications so that you have a known API point you can ask for the status of that application, and use that from your polling system.

Jon brings up a point made in the Telegram group from Stuart, who mentions that his workloads are frequently ephemeral, and that he really needs something that handles service discovery, like Prometheus and Consul.

Jon went on a Wireshark Webinar which he’d strongly endorse people watch (he’s waiting on approval to post the link), and ideally get training from the creator of the course!

Jon also is reading “Analogue Network Security” by Winn Schwartau.

Jerry mentions a weekly podcast “The Pod Delusion” which has restarted. Jon mentions “The Coolest Nerds In The Room” podcast. Al talks about the “Lost Connections” audio book and connected podcast – “Uncovering the Real Causes of Depression with Johann Hari“. Lucy mentions the school in Salford who are teaching all their pupils BSL (British Sign Language) to ensure that deaf students at the school are included.

We thank Dave Lee for his continuing work in fixing up our audio. Jerry non-ironically mentions that he hopes our audio will be better this episode. Dave has advised us that he laughed extensively when he heard this.

Dave is also one of our Patreons – if you also want to be a Patreon, please follow this link: https://www.patreon.com/adminadminpodcast.

We have many ways you can talk to us, including email and Telegram. Details for reaching us on these are on our contact page.

Admin Admin Podcast #076 Show Notes – Audience Participation

In this episode, we go through your questions and feedback. Keep it coming! For example via our Telegram group

First question is from meaty:

– Meaty, a sysdmin in education

First a touch of background to add some context: I work as a team lead & sysadmin (+ “hack” of all trades) in education on a fairly large Windows network. Low budget, high demand, and besides some legal stuff and, contrary to what all the teachers and admin staff believe, no overly urgent requirements (no intellectual property, no critical systems, no four-9’s uptime requirements, but we do have lots of personal and sensitive data). We have an old, mostly unchanging network but due to the nature of teaching, many departments change up their location and/or software (which is often cheap, poorly made and has incredibly specific requirements) on a termly or yearly basis. Lots of “last minute this is urgent do it now” stuff, and even more projects where we’re not consulted and have to hack together solutions at the 11th hour after the majority of work has been done without anyone communicating with us.

We’re small enough that we don’t have much available extra capacity people or resource-wise, but complex enough to have a couple dozen servers (mostly VMs) running on old hardware and nearly 100 switches across a dozen buildings on four campuses, on top of other random infrastructure that is becoming digitised, such as boilers, cctv, access control. Small team, too, so time is tight. No overtime and no out-of-hours work (9-5 only) which is nice, but causes problems as we have no maintenance windows to make changes!

q1: in order to make our lives easier I’m beginning to embrace more automation. We’ve got the big stuff out of the way but to proceed we’re looking into using lots of custom powershell scripts for a lot of this given the random requirements and poor quality of our software. We’ve run into a small issue but I’m not sure what the best practice and most practical solution is. We often need to run scripts over night. So far we’ve run them off a random server that also does other things during the day (hosts a few end user applications) but we know there’s a better way. What is it? Dedicated server? Does something exist that’ll manage this for us instead of using task scheduler on a 2016 box?

q2: We deal with a lot of sensitive data across a lot of systems involving many different types of person – students, staff, parents, visitors, governors, contractors, etc. We know that if an incident/breach occurs and we need to investigate, we’ll be on the phone to an expensive third party to come in and investigate for us as we just don’t know what to look for or where to find it. We need some kind of centralised logging, which we can deploy in time. For now, though, what are the essentials to enable and where can we find them? (eg: logging in AD)

Running scripts on machines

Jerry suggests Ansible for Windows, it speaks to WinRM and runs powershell scripts on the node. Jon suggests Ansible Tower/AWX. It’s an Ansible job scheduler and a credential store. He also suggests version controlling those powershell scripts/ansible code in version control e.g. with Gitlab. Advantges include the ability to run config mgmt from a single place – a “single pane of glass”

He warns that running Gitlab and AWX on a machine can be resource heavy. Jon refers to his Vagrant machine for Gitlab and AWX.

Al reckons that on the windows side, SCCM is good and in depth but expensive. He notes that charities or educational institutions can get it cheaper

Centralised logging/data security

On windows – the Auditing Service is something that can be enabled on the Domain Controller. It logs events like user logging, searching can be a challenge due to the amount of data created.

Al mentions that you can enable these with some scripts.

He also mentions Logrhythm, and products from Quest (actually InTrust). Winlogbeat can ship logs to e.g. an ELK stack. Jon mentions Snare

Jerry mentions that good versioned backups help with Ransomware attacks

Make your servers disposable (cattle vs. pets)

Encryption at rest
  • Bitlocker (windows)
  • LUKS (Linux)
  • Veracypt (Cross-platform, but beware that there’s no veracrypt device driver for Win10 install environments, which can cause an issue with quarterly Win10 upgrades)

Vendors:

Next question is from Andy

– Andy, deploying Windows Desktops

“Is there an affordable way to image Windows desktops that is less insanely complex than Microsoft’s deployment thing?”

I’ve already had a few suggestions here on Telegram but perhaps other listeners face the same challenge.

  • MDT with SCCM on top
  • You must have a Volume License Key to even image a Windows machine, though it’s technically possible to do it without one
  • MDT builds a “golden image”, which then gets pushed to the server
  • Initial Setup is a big effort, but makes life easier once its done.
  • Sysprep resets the machine’s SID to make sure the image can be put on different machines
  • PXE (Legacy & UEFI)
Our last question comes from Stuart

– Stuart, wonders about what to do in the case of a significant outage at a cloud provider

AWS/GCP/Azure fall off the face of the planet overnight, and you are now faced with either choosing smaller providers (with probably a much smaller feature set) or moving back to on-prem

In that situation, what would you choose?

If the former, how would you deal with the limitations? Would you mix and match workloads across multiple providers or would you stick with one or two and work with the limitations?

If the latter, would your workflow and choice of infrastructure change based upon how you work with the cloud now? Would you steer more towards hyperconverged and/or private cloud in a box solutions, or would it be VMware/KVM/Hyper-V with config management, or just revert to how it was pre-cloud days?

I suppose in a sense it’s a question partly about reliance on the big clouds, but also how do you think on prem has improved (if at all) to keep up with the cloud providers

Jon thinks losing all the big cloud providers is pretty unlikely, Jerry thinks if that happens, we would have bigger problems.

Do we count DigitalOcean? They don’t have things like autoscaling and key mgmt, but it should be possible to build these yourself and use smaller providers. If the big 3 disappeared, smaller providers might rush to fill that space. Jon points out that there isn’t really a framework for running Functions-as-a-service (e.g. AWS Lambda).

Jerry says that a Lambda function is just a container – if you have an easy way to get those up and running.

Jerry mentions he has been working with on-prem for most of the last year. In that environment it’s still worth thinking in terms of cloud workflows to inform the on-prem work. The other thing is that on-prem environments can be made easier to manage by using the tooling that has grown up around managing infra on cloud providers.

Jon mentions VMware.

– Vmware NSX-T can run in AWS (and others, including bare metal)

Jerry mentions oVirt.

Al is still 50/50 between running on-prem stuff and running stuff in the cloud. He doesn’t think on-prem is going anywhere 🙂 He would also be using modern tooling to get things done.

We got some Feedback from David:

Thank you for your podcast.

In episode 075, you asked about tools to check whether a web page had
changed. You might like to try Silas Brown’s WebCheck program:

http://ssb22.user.srcf.net/setup/webcheck.html [Note: we were contacted by the author of this app to note that the URL had changed. This link is now the accurate one.]

Thank you

David

We also got Feedback from Producer Dave:

Hey chaps,

Just wanted to say thanks for a fantastic episode 75.

I gotta be honest, a lot of what you guys talk about goes over my head as I’ve never used Selenium, Terraform, Ansible, etc… but I still enjoy listening because I can often pick up some utter gems.

This time around I’ve managed to fall in luuurve (wrong podcast?) with SyncThing and Digital Ocean.

I’d heard much talk about SyncThing on t’interwebs, but it wasn’t until I heard about it on this episode and actually looked into it more that I realised how powerful it actually is. I’m currently using it to perform a one-way backup key folders on my phone and tablet to my laptop. But I also have a two-way sync (kinda like a Dropbox or NextCloud shared folder) in place so that I can transfer files to my phone seamlessly.

Having heard about Al’s experiences of spinning up a NextCloud instance on a $5 Digital Ocean droplet, I decided to do the same as a test… and ended up shifting over to it permanently. All I had to do was spin up the droplet, snap install nextcloud, enter some information, run a single command to apply a Let’s Encrypt certificate, and that was it. 5 minutes, tops. And moving all my stuff between instances was really straight forward too. So thanks for the confidence to make the move, Al!

At the moment, I have 3 VPSes (costing over £36/month) that I could quite easily replace with a number of DO droplets. A $5 droplet, with backup, plus VAT is just under £6, so I could theoretically spin up to 6 $5 droplets (or fewer if I spin a $10 one up, which I might do for some of the smaller services I’m running), but I don’t think I’ll need that many, which will save me money in the long run – win!

Again, thanks for a great episode, and congratulations on the audio quality… you should give your producer a pay rise #JustSaying

Cheers,

Dave

We lastly got Feedback from Jason:

As gathered from the Iron Sysadmin Slack:

XenoPhage (Jason) [12:59 AM]
Hey @JonTheNiceGuy … Was listening to AdminAdmin 75 .. (Yeah, I’m behind a bit) .. Tell Al to take a look at webinject.pl .. Works great with monitoring systems like nagios/icinga2/etc. for monitoring versions of software.. I’ve used it for years to let me know when updates come out for things i can’t just add a yum repo for. :slightly_smiling_face:

Al seems to have dropped off the recording!

Consolidating services chat:

Jon is involved with the lug.org.uk infrastructure, where they have the following problems:

  • x86 build – becoming unsupported by modern OSes
  • Too many machines – looking for a way to reduce the number of physical machines.

Jerry’s instinct is to decouple services, Jon is interested in using docker or something similar

Docker has a way to glue the networking of individual containers together. More complex deployments would probably require e.g. Kubernetes – which is much more complicated.

Any suggestions from listeners?

Al is back!

Thanks Dave! 🙂 We agree to a payrise on-air..

Thanks Patreons!

  • Mike
  • Yannick
  • Andomi
  • Dave

Events:

  • Oggcamp – We’re all going – see you there? 🙂

Welcome to new listeners! Give us feedback

Admin Admin Podcast #075 Show Notes – Highly Available Updates

In this show we read your feedback, we answer your questions and talk about what we’ve been up to!

Want to join the community talking about this podcast on Telegram? Join us!

Al’s back!

Jerry has been subcontracting freelance work out to people, and talks a bit about how and why he started doing that. He also talks about how he’s using Selenium to sign up for free train wifi, using the Selenium IDE Firefox plugin and having a headless browser using Selenium Side-Runner.

Al asks about how to update systems after there was a security incident on his email system. Jon mentions a Firefox plugin which watches for page changes. (He credits this find to Steve Gibson, but he couldn’t find the one Steve mentions. He found this one instead: check4change.  Jon describes an Ansible script for running Apt upgrades but notes that it doesn’t perform reboots if they’re required, Jerry also mentions Unattended Upgrades and Yum-Cron.

Jon refers to a Fortigate Playbook policy Ansible script that he’s written (but isn’t endorsed for use by his employer – use at your own risk!) He describes change management boards. Jerry mentions you can convert an XLS to a CSV and that Ansible will handle CSV files.

Jon also mentions about the fact he’s learning to use Terraform for IaaS (Infrastructure as a Service – basically Azure/AWS). He wrote a blog post about how he got started with Terraform on Azure. Jon mentions about a talk with HashiCorp and RedHat talking about how Ansible and Terraform can work together. Jerry explains how you can chain the output of Terraform to start an Ansible task.

Al talks about the Phoenix Project as an Audiobook, and asks for recommendations on further Audiobooks to listen to. Jon recommends the “#CauseAScene Podcast“.

We then answer a question from a member of the Admin Admin Podcast community about High Availability:

Scenario is a CentOS7 guest VM on a single ESXi host (no vCentre/HA) located at customer site.

Users connect to a friendly URL via web browser which is mapped via A record pointing to a static IP within customer block of IP’s associated with customer broadband.

The solution works well but has several points of failure as this single CentOS7 server running on a non-HA ESXi host not to mention customer Internet reliance.

If wanted to make this highly available either onsite or via AWS/Azure how would you go about this whilst also keeping this secure.

Main Components

  • Apache Web Server
  • PHP Code – Customer Bespoke Internal Website
  • MySQL/MariaDB – Database

How would you go about breaking these up into either individual VM’s? IE multiple web servers and separate PHP and Database servers.

I would imagine you would then also require a front-end load balancer or reverse proxy.

Is this something you might look at using Docker for and how would this impact database state and backups to ensure no data is lost.

We discuss feedback we received by email:

Hi chaps

In a recent episode you asked for feedback, so I thought I would drop a quick line to say that I enjoy listening to the show and look forward to it every other month. 60-90 minutes is about the right length and I think having a topic each time (e.g. IPv6) works well.

I can’t think of anything to offer for improvements – just more of thesame please!

Cheers

Jon replied:

Are there any subjects you want us to cover in the next few episodes?

To which the response was:

Anything to do with Ansible or AWS would be of interest to me, as I use Ansible for all my servers and I’d like to get into AWS as it’s becoming something that clients are asking about.

Based on this email, we briefly discuss differences between Azure and AWS networking compared to Physical networking.

We also discuss another email feedback (trimmed on the podcast, but represented in full here):

Hi guys !

I just finished listening to episode 73 where Jon talked about the difficulties he had with certbot and some exotic architecture.

Here at work, we also had to figure out a way to secure many websites, hosted on various kinds of servers, and running on a variety of operating systems.

The best solution we came up with is to use nginx as a reverse proxy. That proxy handles the TLS part, and it’s the only place our certificates are located. Its exposes the .well-known directory and redirects traffic to the proper servers. If we need to add or remove cyphers for security reasons, all of our websites are protected at once.

We also have a wildcard certificate for our company’s domain. I’ll take frenchguy.ch as an example. This cert is actually valid for 2 domains : frenchguy.ch and *.frenchguy.ch

Each time we need a new subdomain, we create it in our DNS and point it to the reverse proxy. We can then forward traffic to a whatever server we want inside our network, even exotic ones.

This has many advantages :

  • there is only one place where the certs needs to be stored – easy to backup, no need to run around the network when time comes to renew the certs ;
  • the cert is a wildcard so we can have as many subdomains as we want ;
  • the only server exposed to the internet is the reverse proxy, not the actual web servers ;
  • the traffic can be forwarded to old servers that do not support the new TLS protocols, or have old vulnerable cyphers – yes, I’m talking about you IIS ! ;
  • the traffic is forwarded over HTTP, so we can reduce the load on the web servers ;
  • there is no need to modify the actual webserver, in particular, no need to expose the .well-known directory ;
  • servers can be moved around, migrated from one architecture to another, etc.. without having to bother about certificates or encryption.

That’s how we did it at work, it works like a charm for us, but that doesn’t mean it’s the only way to do it. If someone as another way to solve that problem I’d be glad to hear it !

Thanks for the great show, it’s always a pleasure to listen to.

Yannick
a.k.a. The frenchguy from Switzerland

We thank Dave Lee (@thelovebug) for doing our audio production, we thank our Patreons and we mention OggCamp (get your tickets now!), and note that we all plan to be there! Hope to see you there!