In this episode, possibly the shortest since Jon joined the team, we have a conversation with Stuart (Mastodon | Twitter), who is a member of our Telegram community. We’re also missing Al.

Stuart talks about Prometheus, and compares it to Nagios. He talks about the differences between how Prometheus collects data, particularly how Prometheus talks to local exporters to collect metrics, rather than polling data every 5 minutes. He lists a collection of exporters from a whole range of products (too many to list here!) and then Jerry and Stuart discuss rewriting native data sources into a format that Prometheus works.

Stuart has linked to some additional sources of information about Prometheus:

• Prometheus Monitoring Experts – posts by Brian Brazil – adds lots of information about how prometheus works and how to make use of it
• Chris’s Wiki :: blog/sysadmin – interesting blogs, talks about their experience with prometheus.
• Grafana are heavily involved in prometheus development now, and frequently blogs about using prometheus.
• cronmanager – Generates cron statistics in prometheus format, eg job status, time to run, outputs to textfile collector directory.
• Textfile collector scripts – generates textfile outputs in Prometheus formats for a number of applications, including smartmon, apt, yum and pacman.
• Thanos and Cortex – two of the prometheus High Availability options.

Moving on with the show, we cover for the fact we’re missing Al by asking two questions on his behalf, the first covers how we believe Al is suffering from Alert Fatigue, and how he can collect results from scripts that run on his servers in a specific way. Stuart explains how he’d use Prometheus for this, Jerry mentions that he’d collect logs for later parsing and only forward logs in the case where the script has failed to run successfully. Jon mentions that he’d consider using Monit to run the tasks, as that will notify if the job fails to run. He also suggests using triggers for bash scripts to send an email on failure, and changing email titles based on the outcome of the task.

He also asks about monitoring disks on a homemade NAS. Jon mentions he’s used Monit with SmartMonTools (similar to this page) to monitor disk statuses in the past. Jerry and Stuart also mention that he could be using Prometheus for this. We also discuss that this may in fact be built into the NAS he’s trying to build. We discussed monitoring with Lucy in Episode 77.

Jon talks about the testing he’s been doing with Nebula, which is a meshed overlay VPN (Virtual Private Network) product, and compares it to a Hub-and-Spoke (or Star) VPN topology. He compares it, briefly, with ZeroTier and mentions that he needs to do more exploration into ZeroTier.

Jerry asks Stuart some questions about SaltStack, and compares it to Ansible.

As always, we’d encourage any listeners to join our Telegram Group, or contact us using the other links! We also have a Patreon which you can use to support the show if you’re so inclined.

In this show we read your feedback, we answer your questions and talk about what we’ve been up to!

Want to join the community talking about this podcast on Telegram? Join us!

Al’s back!

• He’s been using Microsoft Visual Studio Code and PowerShell. Al has been automating creating a spreadsheet from various data sources, and auditing NTFS folder group permissions to a SQL database.
• He’s also used NMap (either the GUI – zenmap, or CLI – nmap) to identify machines on a network at a client, and then got questioned by the security team about why he was running it! He passed them a link to a Darknet Dairies episode about someone else doing the same thing (based on a Twitter Thread).
• He’s set up a NextCloud environment using snap. Snaps autoupdate, which is good. He’s writing documentation using Markdown and synchronising across machines using NextCloud. We get into a conversation about the benefits of using Markdown, especially when using NextCloud’s Markdown Editor. Al mentions a file he’s published using markdown on github about the differences between CentOS and Ubuntu.
• Jon also explains that as well as DropBox and NextCloud (each for specific usecases), he also uses SyncThing to share SSH keys and Keepass files between specific machines since he started using Termux for Android, as he couldn’t find a good sync or webdav client for Termux.

Jerry has been subcontracting freelance work out to people, and talks a bit about how and why he started doing that. He also talks about how he’s using Selenium to sign up for free train wifi, using the Selenium IDE Firefox plugin and having a headless browser using Selenium Side-Runner.

Al asks about how to update systems after there was a security incident on his email system. Jon mentions a Firefox plugin which watches for page changes. (He credits this find to Steve Gibson, but he couldn’t find the one Steve mentions. He found this one instead: check4change.  Jon describes an Ansible script for running Apt upgrades but notes that it doesn’t perform reboots if they’re required, Jerry also mentions Unattended Upgrades and Yum-Cron.

Jon refers to a Fortigate Playbook policy Ansible script that he’s written (but isn’t endorsed for use by his employer – use at your own risk!) He describes change management boards. Jerry mentions you can convert an XLS to a CSV and that Ansible will handle CSV files.

Jon also mentions about the fact he’s learning to use Terraform for IaaS (Infrastructure as a Service – basically Azure/AWS). He wrote a blog post about how he got started with Terraform on Azure. Jon mentions about a talk with HashiCorp and RedHat talking about how Ansible and Terraform can work together. Jerry explains how you can chain the output of Terraform to start an Ansible task.

Al talks about the Phoenix Project as an Audiobook, and asks for recommendations on further Audiobooks to listen to. Jon recommends the “#CauseAScene Podcast“.

We then answer a question from a member of the Admin Admin Podcast community about High Availability:

Scenario is a CentOS7 guest VM on a single ESXi host (no vCentre/HA) located at customer site.

Users connect to a friendly URL via web browser which is mapped via A record pointing to a static IP within customer block of IP’s associated with customer broadband.

The solution works well but has several points of failure as this single CentOS7 server running on a non-HA ESXi host not to mention customer Internet reliance.

If wanted to make this highly available either onsite or via AWS/Azure how would you go about this whilst also keeping this secure.

Main Components

• Apache Web Server
• PHP Code – Customer Bespoke Internal Website
• MySQL/MariaDB – Database

How would you go about breaking these up into either individual VM’s? IE multiple web servers and separate PHP and Database servers.

I would imagine you would then also require a front-end load balancer or reverse proxy.

Is this something you might look at using Docker for and how would this impact database state and backups to ensure no data is lost.

We discuss feedback we received by email:

Hi chaps

In a recent episode you asked for feedback, so I thought I would drop a quick line to say that I enjoy listening to the show and look forward to it every other month. 60-90 minutes is about the right length and I think having a topic each time (e.g. IPv6) works well.

I can’t think of anything to offer for improvements – just more of thesame please!

Cheers

Jon replied:

Are there any subjects you want us to cover in the next few episodes?

To which the response was:

Anything to do with Ansible or AWS would be of interest to me, as I use Ansible for all my servers and I’d like to get into AWS as it’s becoming something that clients are asking about.

Based on this email, we briefly discuss differences between Azure and AWS networking compared to Physical networking.

We also discuss another email feedback (trimmed on the podcast, but represented in full here):

Hi guys !

I just finished listening to episode 73 where Jon talked about the difficulties he had with certbot and some exotic architecture.

Here at work, we also had to figure out a way to secure many websites, hosted on various kinds of servers, and running on a variety of operating systems.

The best solution we came up with is to use nginx as a reverse proxy. That proxy handles the TLS part, and it’s the only place our certificates are located. Its exposes the .well-known directory and redirects traffic to the proper servers. If we need to add or remove cyphers for security reasons, all of our websites are protected at once.

We also have a wildcard certificate for our company’s domain. I’ll take frenchguy.ch as an example. This cert is actually valid for 2 domains : frenchguy.ch and *.frenchguy.ch

Each time we need a new subdomain, we create it in our DNS and point it to the reverse proxy. We can then forward traffic to a whatever server we want inside our network, even exotic ones.

This has many advantages :

• there is only one place where the certs needs to be stored – easy to backup, no need to run around the network when time comes to renew the certs ;
• the cert is a wildcard so we can have as many subdomains as we want ;
• the only server exposed to the internet is the reverse proxy, not the actual web servers ;
• the traffic can be forwarded to old servers that do not support the new TLS protocols, or have old vulnerable cyphers – yes, I’m talking about you IIS ! ;
• the traffic is forwarded over HTTP, so we can reduce the load on the web servers ;
• there is no need to modify the actual webserver, in particular, no need to expose the .well-known directory ;
• servers can be moved around, migrated from one architecture to another, etc.. without having to bother about certificates or encryption.

That’s how we did it at work, it works like a charm for us, but that doesn’t mean it’s the only way to do it. If someone as another way to solve that problem I’d be glad to hear it !

Thanks for the great show, it’s always a pleasure to listen to.

Yannick
a.k.a. The frenchguy from Switzerland

We thank Dave Lee (@thelovebug) for doing our audio production, we thank our Patreons and we mention OggCamp (get your tickets now!), and note that we all plan to be there! Hope to see you there!

Podcast: Play in new window | Download