Ricky Smith

Agentic Tooling Across Multiple Repositories

Sat, 10 Jan 2026 00:00:00 -0500

In my organization, we have a great number of repositories that contain overlapping logic. For example, to manage our large number of cloud environments, we have separate repositories just for Terraform. We've split up our code based on purpose and audience to better organize our work. However, this creates challenges when we need to make a change that spans multiple repositories. This challenge has always existed, but it was manageable when doing work manually – you can just open the multiple repositories in your editor of choice and have multiple terminal tabs open for working with git in each repository.

But as we lean into the use of agentic tooling (AI agents that can act on your behalf) to automate repetitive tasks – especially when we want those tools to perform broader changes that require context across multiple repositories – this becomes a bigger challenge. In this post I’ll walk through how I use git worktrees and a simple directory convention to give those agents the right context to work safely across many repos.

Organizing My Repositories

Before we get into the solution, let's take a look at how I organize my repositories locally. My usual setup is to place all of my repositories in a ~/projects directory structure, usually by organization, then by full project namespace that matches our remote repository.

So, you'd expect to see a structure like this on my laptop:

~/projects/
| - acme-corp/
| - | - infra-team/
| - | - | - terraform/account-provisioning/
| - | - | - infra-team/terraform/networking/
| - | - | - infra-team/terraform/networking-modules/
| - | - | - shared/terraform/service-resources/
| - | - | - shared/terraform/service-modules/
| - | - | - app-devs/frontend-app/
| - | - | - app-devs/backend-api/
| - | - some-product-team/backend-services/
| - public/some-open-source-repo/

A High-Level Solution – Directory Structure with Git Worktrees

My idea was to use git worktrees to create a single directory structure that contains multiple repositories, all synced to the same branch name. This way, I can create a workspace for a specific task that requires context across multiple repositories. The branch-named parent directory provides a space for defining context for the task at hand and provides a workspace for agentic tooling without polluting the individual repositories.

Git Worktrees

Git worktrees let you create additional working directories for the same repository. Each worktree is tied to a branch and shares the same .git data, so you don’t need to reclone. Creating a worktree is fast and uses little disk space, and remote operations (fetch, pull, push) are shared across the main repo and all worktrees.

The common use case is when you need to quickly pivot to working on another problem/feature, but you have a lot of WIP stuff in your working directory. Most developers I know would go about stashing their changes or creating a "WIP" commit, then switching branches. With git worktrees, you can just create a new working directory for the branch you want to work on and leave your main working directory alone. It's very handy, but not a tool that most people reach for often (maybe not often enough).

Git Worktree Directory Structure

In my new scheme, I've started to create a directory structure that looks like ~/projects/worktrees/<branch-name>/<repository-name>.

So, if we think of a really broad task, like "scan all Terraform modules using Trivy and fix any issues found", I might create a worktree structure like this:

~/projects/worktrees/scan-fix-trivy-issues/
| - terraform-account-provisioning/
| - terraform-networking/
| - terraform-networking-modules/
| - terraform-service-resources/
| - terraform-service-modules/

This has some really big benefits:

Each task can include (or not) repositories that are needed
All repositories are checked out to the same branch name, making merge-request/pull-request tracking easier.
The parent directory (scan-fix-trivy-issues) provides a workspace for the task, outside of any one repository
Agentic tools can be run from the parent directory, providing a single context for all repositories involved in the task.

Giving Context

The big "aha" moment was that the parent directory (the branch name) provides a context for the task at hand. This is really powerful when using agentic tooling, because the tooling can be run from the parent directory, and it can operate on all repositories within that context. So far, I'm starting my tasks with two key files: AGENTS.md and PLAN.md.

The AGENTS.md file is already a well-known pattern for providing seed context for agentic tooling. By placing this file in the parent directory, I can give high-level information about the directory structure and instructions for the agents to read AGENTS.md files in each repository, and to reference the PLAN.md file for the overall plan.

This directory contains multiple repositories related to scanning and fixing Trivy issues in our Terraform modules.
Each repository is checked out to the same branch name: `scan-fix-trivy-issues`.
Follow the plan outlined in the `PLAN.md` file to complete the task.
For each repository, please read the `AGENTS.md` file located in the root of the repository for specific instructions related to that repository.
When making changes, please create git commits with clear messages, push changes to the remote repository, and create merge requests as needed.
Track all created merge requests in a `MERGE_REQUESTS.md` file in this parent directory.

In my PLAN.md file, I've found that providing a high-level outline of the steps, and a checklist of the steps to perform, helps the agent stay on track and also provides a way for the agent to keep track of progress. You're likely to pause the overall task execution to focus on a specific issue, to push changes and address CI/CD results, or if you need to reboot your computer. At any time, you can just tell your agent to "continue the plan" and it can pick up where it left off.

# Plan: Scan and fix Trivy issues

- [ ] Scan all Terraform modules with Trivy
- [ ] Triage findings and propose fixes
- [ ] Apply fixes and run `terraform validate`
- [ ] Open merge requests and record them in `MERGE_REQUESTS.md`

Shell Alias for Creating Worktrees

Now, one of the pain points of working with git worktrees is that the commands can be a bit verbose. To make it easier to create these worktree structures, I've created a few ZSH aliases that automate the process, especially focusing on this workflow.

# Create a new worktree for a new branch
# Usage: wt-new <branch-name>
wt-new() {
    if [ -z "$1" ]; then
        echo "Error: Branch name required"
        echo "Usage: wt-new <branch-name>"
        return 1
    fi
    
    local branch_name="$1"
    local repo_name=$(basename $(git rev-parse --show-toplevel 2>/dev/null))
    
    if [ -z "$repo_name" ]; then
        echo "Error: Not in a git repository"
        return 1
    fi
    
    local worktree_path="$HOME/projects/worktrees/$branch_name/$repo_name"
    
    # Create the directory structure if it doesn't exist
    mkdir -p "$HOME/projects/worktrees/$branch_name"
    
    # Create the worktree with a new branch
    if git worktree add -b "$branch_name" "$worktree_path"; then
        echo "✓ Worktree created at: $worktree_path"
        
        # Copy all .envrc files from the source repository
        local source_repo=$(git rev-parse --show-toplevel)
        local envrc_files=$(find "$source_repo" -name ".envrc" -not -path "*/\.git/*" 2>/dev/null)
        
        if [ -n "$envrc_files" ]; then
            echo "$envrc_files" | while read -r envrc_file; do
                local rel_path="${envrc_file#$source_repo/}"
                local target_dir="$worktree_path/$(dirname "$rel_path")"
                mkdir -p "$target_dir"
                cp "$envrc_file" "$worktree_path/$rel_path"
                echo "✓ Copied $rel_path"
            done
        fi
        
        cd "$worktree_path"
    else
        echo "✗ Failed to create worktree"
        return 1
    fi
}

This alias, wt-new, takes a branch name as an argument, creates the necessary directory structure, and sets up a new worktree for the current repository. It also copies any .envrc files from the source repository to the new worktree, ensuring that environment configurations are preserved.

The Real Magic

Now, if you're paying attention, you might think this is a pretty nice way to create a per-task structure, but there's some extra steps that REALLY turn this whole thing into pure magic.

With this directory layout in place, agentic tooling gets really interesting when you add a few conventions:

Instructing the agent to create git commits as it works
Instructing the agent to push changes to the remote repository when it completes a step (while specifying git push options to automatically create a merge-request on the remote)
Instructing the agent to keep track of merge requests in a MERGE_REQUESTS.md file in the parent directory
Giving the agent access to CI/CD pipelines so it can see output of tests, scans, and other jobs (through an MCP server, for example)
Instructing the agent to document instructions I provide in the ./AGENTS.md file, so that any refinements I give it during the chat session are captured (I find this helps to keep important instructions in the AI context)
Have a tool like pre-commit already running on the repositories – the agent will see the pre-commit failures and fix them as it goes

Demonstrating the Workflow

To demonstrate the workflow, let's say I want to create a new worktree for the branch scan-fix-trivy-issues for a few repositories, I would navigate to each repository and run wt-new scan-fix-trivy-issues.

cd ~/projects/acme-corp/infra-team/terraform/account-provisioning
wt-new scan-fix-trivy-issues
cd ~/projects/acme-corp/infra-team/terraform/networking
wt-new scan-fix-trivy-issues
cd ~/projects/acme-corp/infra-team/terraform/networking-modules
wt-new scan-fix-trivy-issues

Next, create my AGENTS.md and PLAN.md files in the scan-fix-trivy-issues directory to provide context for the task.

~/projects/worktrees/scan-fix-trivy-issues/
| - terraform-account-provisioning/
| - terraform-networking/
| - terraform-networking-modules/
| - AGENTS.md
| - PLAN.md

From there, I can just run my favorite AI tool (codex) in the parent directory and prompt it to Get started. It will read the AGENTS.md file by default, which tells it to read the PLAN.md file and follow the steps outlined there.

At some key points, I'll instruct the agent to push changes, which will cause it to create commits, push them to the remote repository, and create merge requests automatically and log them in the MERGE_REQUESTS.md file. I can tell it to Check the status of the merge request pipelines and it'll go off and look at the output of the CI/CD jobs and get to work on troubleshooting failures.

Early Results

I've only been using this workflow for a couple of days, but it's been extremely promising. There have been a few key moments where the agent has (rightly) indicated that it might cause an outage with a change and I've been able to pull in additional repositories (like our repositories that manage helm charts and environment-specific input values) to give it more context to make more "informed decisions".

I also very much like that I have given it a large sandbox where it can play safely. The worktrees are all created with a branch, so there's a low risk that I'll have the wrong branch checked out. I also don't have to worry about a lot of pending changes in my main working directories and it getting in the way of other work. In fact, I've found myself already setting up this context, setting the agent off to do work, and then I'll switch my focus to something else in the main working directory. As someone who has been very hesitant to relinquish control to AI tools, this has been a really nice way to reap the benefits while still having some confidence that it's not going to make a huge mess.

(A human wrote this blog post, but GPT-5.1 was used for editing assistance. Use of emdashes was a human choice. 😄)

Streamlining My Homelab Deployments with Bare Git Repositories and Git Hooks

Fri, 19 Dec 2025 00:00:00 -0500

I finally stopped fighting with my homelab deploy process and turned it into something that feels almost civilized.

The homelab: an old gaming desktop running Ubuntu, hosting a handful of services in Docker Compose. The goal: fast iteration without losing my mind or my work. It took a few generations to get here.

The First Era: SCP, Makefiles, and Madness

I wrote compose files locally, committed to GitHub for backup, SCP'd them over, SSH'd in, and ran the deploy by hand. For quick troubleshooting, that workflow was trash. I inevitably edited files directly on the server in vi/nano, and "often" became "very often."

Cumbersome != sustainable.

During this time I toyed with Github Actions to deploy on push and Gitlab CI/CD pipelines, but the friction of setting up runners, secrets, and debugging remote jobs made it more hassle than it was worth for a homelab box. I spent a number of hours chasing my tail trying to get those systems to work smoothly – this is something I do in my day job … and at-scale – but I could never keep myself focused enough to jump through all the hoops in my spare time. I'd much rather be playing video games, tbh.

The Second Era: Remote VS Code, Local Regret

I gave up for a while and just used VS Code remote editing on the server along with SSH to deploy by hand. Yes, it had better ergonomics, but I kept forgetting to sync changes back to my local repo (or Github), so the risk of losing work was high. And, it started to get in the way of using newer GenAI tools outside VS Code (hi, Codex) as those tools often didn't play well being run inside of an SSH session and I was too lazy to really chase down the causes. (This is a homelab, after all, and the purpose is to learn new things, not fight old ones.)

This is ugly. It's embarrassing. I hated it. It was easy. I loved it. I hated that I loved it. I'm lucky it didn't blow up in my face.

The Third Era: A New Dawn Arises

(Don't say I don't have a flair for the dramatic.)

I don't know why it took me so long to consider getting back to basics and relying on Git over SSH. I suppose I have been trained to think of gitops in a "Push to central repository and from there trigger events" mindset, but for a single homelab server, that came with such a heavy lift. My training for at-scale infrastructure has often blinded me to simpler solutions for smaller scale problems. I've forced myself to think in terms of Kubernetes, Terraform, and CI/CD pipelines when all I really needed was a way to push code and have it deploy. Even today, I'm thinking about some of those command-line driven pipeline systems that I could set up on the server, just to mentally smack my hand because it's more complex than it needs to be. (Let's face it, I'll still probably do that later, just for fun, but it'll be invoked through the git hook).

The simple solution is simple … push directly to the server with git over ssh. And hooks! I can use a hook on the server's repo to do things! And, oh! I see the output of that script in my push response! As soon as that last part clicked, it was obvious.

On the server, I created a bare repo (/srv/git/my-stack.git).
On my desktop, I set that bare repo as a remote for my working copy.
The bare repo has a post-receive hook that:
- updates a checked-out working tree,
- runs docker compose up -d (or whatever deploy command I need),
- mirrors the repo to an external remote (currently GitLab).

Here's the flow end-to-end:

sequenceDiagram
  participant Dev as Dev machine
  participant Bare as Bare repo on homelab
  participant Hook as post-receive hook
  participant WorkTree as Checked-out work tree
  participant Stack as Docker Compose stack
  participant Mirror as External remote (e.g. GitLab)

  Dev->>Bare: git push server main
  Bare-->>Dev: push output (including hook logs)

  Bare->>Hook: trigger post-receive
  Hook->>WorkTree: git checkout -f main
  Hook->>Stack: docker compose up -d --build
  Hook->>Mirror: git push mirror

  Stack-->>Dev: logs, status (via MCP / CLI)

Here's an example /srv/git/my-stack.git/hooks/post-receive hook (if you're trying this yourself, don't forget to chmod +x it):

#!/usr/bin/env bash
# Post-receive hook for the bare remote repo on three.
# Deploys main into /home/ricky/docker-apps/my-app-stack and restarts the stack.
set -euo pipefail

TARGET_REF="refs/heads/main"
# Don't judge me for having my work tree in my home dir ... it's a homelab!
WORK_TREE="/home/ricky/docker-apps/my-app-stack"

HOOK_DIR="$(cd "$(dirname "$0")" && pwd)"
GIT_DIR="${HOOK_DIR%/hooks}"

update_work_tree() {
  git --git-dir="$GIT_DIR" --work-tree="$WORK_TREE" checkout -f main
}

restart_stack() {
  cd "$WORK_TREE"
  docker compose up -d --build
}

while read -r oldrev newrev ref; do
  [ "$ref" = "$TARGET_REF" ] || continue
  echo "Deploying $ref -> $newrev to $WORK_TREE"
  update_work_tree
  restart_stack
done

Push once, deploy, and back up. No more stray edits.

The deploy script lives with the hook, so the server is the source of truth for how to run the stack. My local workflow stays normal Git: edit, git commit, git push server main, watch the hook do the rest. If I want history elsewhere, the hook's mirror keeps GitLab in sync without me thinking about it.

It's a kind of magic

All of this came about in the context of a new service I was creating for my homelab: an MCP server to expose read-only container details and logs. This service was a code-for-sport project. I've created a couple of MCP servers already, but nothing at home that I would consider keeping around for a while. One way my development workflow was failing me was that all these new GenAI tools couldn't be directed to assert control in safe ways. Deploying this new MCP server was the first of my homelab services that I built using this new Git-over-SSH-with-hooks workflow … and it surprised me once I started to get it even half-working.

Suddenly, Codex (my current client of choice) could see the code, understand the context of the deploy process, and suggest changes that I could quickly test by pushing to the server. I could even have Codex inspect logs and running containers through the MCP server. Its access was still limited – it could only use the read-only MCP server tools and then git push when I told it to, but the context this flow provided made the AI assistance much more effective.

Even if it did misbehave somehow (and let's face it, GenAI is still just a slightly-competent-recent-college-graduate-level-intern able to regurgitate some fairly sophisticated code that it saw once in a StackOverflow post … wait, is StackOverflow still a thing?!), the deployment process was still git-based, so I could still roll back changes or fix things manually if needed.

Onward

I've now moved all of my docker compose stacks to use this deployment process. It's already encouraged me to pick up some issues that I pushed off because they required a lot of deployments and the iteration cycle was too painful. I even gave codex a set of really complex instructions to troubleshoot a specific issue, hit enter, and went to dinner with my wife … I haven't trusted a GenAI tool that much before and even called it out to her (and she scoffed because she has a more-than-healthy skepticism of AI tools in general and I like that about her).

Honestly, seeing this whole setup work has rather changed my mind about GenAI. We still have to be careful about the kind of control we give these tools, but with the right context and guardrails, they can be a huge productivity boost.

Avoiding Common Pitfalls in Terraform Module Design

Sat, 27 Sep 2025 00:00:00 -0400

I've been working with Terraform for over 6 years now, and during this time, I've encountered several common mistakes when designing modules that can lead to frustrations, mismanagement, and even mistrust in the tool itself.

I continue to see teams complaining that Terraform is overly complex, hard to maintain, doesn't scale well, and – the most common complaint – that state files are unreliable and need constant manual intervention. That last point, especially, leads teams to fight against Terraform rather than working with it. In some cases, teams abandon the use of CICD processes altogether, opting instead to run Terraform commands manually, which only leads to more problems (not to mention a compliance and security nightmare).

Before I go into more detail, I want to clarify that I will be using the terms "root module" and "child module" throughout this post. A root module is the top-level configuration in a Terraform project, while a child module is a reusable component that can be called from multiple root modules. A root module would define provider and backend configuration, while a child module would not.

Poor Module Naming

One of the most common mistakes I see is poor module naming (especially root modules). Modules should be named by their business purpose, not by the resource type they manage.

For example, a module that manages an S3 bucket for storing user uploads should be named user-uploads-bucket rather than just s3-bucket. This makes it clear what the module is for and helps avoid confusion when multiple modules manage similar resources. It also makes it easier to understand the overall architecture of the infrastructure at a glance when you see the root modules invoked for an environment.

Even when creating a truly reusable child module, the name should reflect its purpose. For example, if I were creating a module that would encapsulate the default configuration for all S3 buckets in my organization that adhered to best practices and compliance requirements, I would name it something like compliant-s3-bucket. This makes it clear that the module is not just a generic S3 bucket but one that meets specific organizational standards. That module would then be invoked by a "static-website-bucket" module or a "user-uploads-bucket" module, etc.

Poor Module Scope

This goes hand-in-hand with poor module naming. Modules should have a clear and focused scope. A module that tries to do too much becomes difficult to understand, maintain, and reuse.

I can't count how many times I've seen an engineer adding a variable to a module to add some new functionality that has already had a dozen engineers do the same before them; each shovelling on more complexity and scope. The result is always the same: a module that nobody understands, nobody wants to use, and nobody wants to maintain. We only really notice these modules once they've reached the point of being unmanageable, and by then, it's often easier to rewrite them from scratch than to try and untangle the mess.

I think of modules a lot like microservices. Each module should have a single responsibility and do it well. If a module is responsible for managing a VPC, it shouldn't also be responsible for managing EC2 instances or RDS databases. Those should be separate modules that can be composed together as needed. If you find yourself adding more and more variables to a module, it's a sign that the module's scope is too broad and needs to be broken down into smaller, more focused modules.

That may sound like I'm advocating for incredibly small modules, and in some cases, I am. But the key is that each module should have a clear purpose and be easy to understand. A module that encapsulates a single cloud resource is likely too small, but a module that encapsulates a single business function (like "user authentication" or "payment processing") is likely just right.

Following this principle will also make naming the module easier – if you can name the purpose, you can define the scope and you can give it a good name.

Modules are too flexible

I see a lot of recommendations and "best practices" that advocate for making modules as flexible as possible. They don't use those words directly, but they recommend that everything should be a variable, and that modules should be designed to handle a wide variety of use cases. And I see engineers taking this to the absolute extreme, creating modules with dozens of variables, many of which are complex objects or maps.

A good example is the creation of an IAM role. I often see a child module that creates an AWS IAM role in a generic way, but then also has variables for the definition of the role's policies, trust relationships, tags, and even the ability to attach managed policies. The result is a module that is so flexible that it can be used for almost any purpose, but it's also incredibly complex and difficult to use. It's also pretty meaningless.

In these cases, I typically create a module that creates a role with a specific purpose, such as service-irsa-role. The module would have a few variables, such as the role name (but probably not even directly given the role name .. I'd probably have other business-relevant variables that would be used to construct the role name, like environment, application, and component), But I would avoid making the module too flexible by allowing for arbitrary policies or managed policy attachments. And I would absolutely NOT allow for the trust relationship to be defined as a variable. The trust relationship should be fixed based on the purpose of the module (in this case for IRSA). When the service using this role needs additional permissions, the root module that invokes this child module can create and attach additional policies as needed. This allows the child module to remain focused and easy to use while still allowing for flexibility in the root module – and the root module can be invoked in different environments for deploying the same application in dev, staging, and production.

Given this opinion, you may not be surprised to hear that I avoid using most modules from the Terraform Registry. I find that most of them are too generic and flexible, making them difficult to use and understand. I'll often use them as a reference to determine which resources should be included in my own module. I prefer to create my own modules that are focused on specific business purposes and have a clear scope. This also allows me to enforce organizational standards and best practices more easily.

Exploiting Terragrunt's hooks

While this isn't exactly a Terraform mistake, I see a lot of teams using Terragrunt's hooks to run custom scripts or commands before or after Terraform commands. These teams often don't agree with the idea of using static and simple Terraform modules, and instead want to run custom logic to modify the state or configuration before applying changes. In my experience, they usually have a coding/scripting approach to Terraform, rather than a declarative "configuration" approach. They'll exploit Terragrunt's hooks to clean up state or modify configuration files.

I see this as an exploitation. They're fighting against the nature of Terraform, trying to bend it to match their mindset and not fully understanding how Terraform is designed to work. The worst part is that they won't see how great Terraform can be when used correctly because they're so focused on trying to make it work their way. These teams usually end up with brittle, hard-to-maintain infrastructure that requires constant manual intervention and breaks frequently and they end up distrusting Terraform as a whole.

Granted, I'm not against Terragrunt hooks (or even Terragrunt's code generation features), but I think they should be used sparingly and only when absolutely necessary.

Death by a thousand cuts

Developer discipline is a big subject and very much applies to Terraform. Developers are constantly weighing the cost of doing something "the right way" versus doing it "the easy way" or "the quick way".

It's so easy to see a module called "elasticache" that is used to create Elasticache Redis clusters and modify it to also be able to create Memcached clusters. Or to add a list(object) variable to a VPC module to allow for the creation of additional subnets. Or even just a list(string) variable to allow for additional tags. Each of these changes may seem small and insignificant on their own, but over time, they add up and lead to a module that is so complex and difficult to use that nobody wants to use it anymore.

The Subtle Differences in Apply-Before-Merge and Apply-After-Merge

Wed, 14 Jun 2023 00:00:00 -0400

My infrastructure team, as part of a larger Terraform Restructure initiative, is moving our Terraform repositories to be apply-after-merge. There are significant changes in process between these two approaches; not all are obvious.

I won’t try to hide: I like Apply-After-Merge more. I will attempt to show the strengths and problems with both processes, but you should know that I have already come to an opinion.

Let’s start by looking at the two processes side-by-side, what humans need to do, and what the CICD pipeline will do.

The Apply-Before-Merge (ABM) process is like this:

The submitter create a Merge Request (MR)
The CICD pipeline:
1. Creates a plan for the required changes and logs it for review (Plan Phase)
2. Waits for manual intervention
The reviewer looks at the code changes and the plan file
The reviewer clicks the "Apply" button to trigger the pipeline to continue
The pipeline fetches the persisted plan file from step 2 and executes the plan (Apply Phase)
The pipeline merges the MR into the default branch

The Apply-After-Merge (AAM) process looks like this:

The submitter creates a MR
The CICD pipeline creates a plan for the required changes and logs it for review
The reviewer looks at the code changes and the plan file
The reviewer clicks the "Merge" button and the code is merged into the default branch
The pipeline – using the default branch – creates and executes a plan (Apply Phase)

Now keep in mind that CICD pipelines are async processes – there will be multiple, conflicting, merge requests in various stages of the process at any given time.

"Everybody has a plan until they get punched in the mouth" – Mike Tyson

In each approach, we need to create a plan file prior to MR review, but in ABM the plan file has another purpose: that plan file will be the exact changes made during the apply step. That means that the plan file needs to be stored somewhere so that it can be retrieved later (during the Apply Phase). It also means that a plan has the opportunity to go stale.

The storage of a process-critical, but still ephemeral, bit of data can be tricky to get right and reliable. In our case, we’re using GitLab CI caching, but that comes with potential pitfalls – the cache entries need a key that’s scoped to the MR, are there potential race conditions that could cause the wrong plan file to occupy the cache? What happens if the cache entry is evicted? Many times these questions go unasked.

"The starting point for all achievement is desire." – Napoleon Hill

A much more subtle difference is in the question where is the desired state? And also, what does the default branch represent?

In Terraform, you are always dealing with two states: the desired state and the state of reality. When Terraform is creating a plan, the first thing it does is investigate the state of reality (what resources exist in the target environment and how are they configured), compare that to the desired state (the terraform code, in totality), and build a series of steps to change reality so that it matches the desired state. The plan file is that set of steps.

So, in ABM, where is the desired state stored? It’s not in the default branch – you’ve already applied a new desired state to the environment before merging the code. Does that mean it’s the unmerged branch? What happens if you have multiple unmerged branches? Do you have multiple desired states?

This also leads you to think that the default branch represents reality when using ABM – but that’s incorrect. It’s common for resources created by Terraform to be modified outside of Terraform. It’s commonplace for reality to morph in ways that’s entirely appropriate. Or, even if your environment shouldn’t change outside of your Terraform code, there’s no guarantee that it hasn’t changed. Your default branch becomes a Schrödinger’s cat – until you run a plan it both does and does not represent reality.

In fact, in ABM, the default branch represents neither reality (reliably), nor the desired state. If you squint and try real hard, you could say that the default branch represents "the last applied and quickly and successfully merged desired state assuming there are no other MRs in or after the Apply Stage that more accurately describe the desired state." That’s a LOT of caveats.

In AAM, the desired state is a lot more clear: it’s the default branch. Full stop. The default branch is your desired state, even if you’ve failed to (yet) reach the desired state and reality doesn’t match it.

"When things go wrong, don't go with them." – Elvis Presley

Let’s consider a common failure state: your terraform apply fails.

I’ve mentioned that in AAM the default branch represents the desired state – and not necessarily the state of reality. So, a failure to reach the desired state (executing your plan), results in an ongoing separation between the desired state and reality. But, that’s pretty much it. At any point, you can re-apply the default branch to again attempt to reach your desired state. In practice, this results in failed pipelines that are easy to remedy and so pretty much not a big deal.

Consider another scenario: the git merge fails.

The merge itself can fail for few reasons, but the most common is that there are multiple merge requests in flight at a time, resulting in a slow-moving race condition: MR1 is merged while you're looking at MR2. In ABM, this can be devistating. If MR1 and MR2 are both approved, and the applies are happening simutaniously, and MR1 is merged, this can – and will – result in MR2 failing to merge because it requires a rebase. You are now in a state where MR2 has been applied, but not merged.

Any other MR could now be applied and REVERT the changes from MR1 that are already applied. Again, this can be devistating to an infrastructure. It's very easy for resources to be deleted because they're not defined in MR3's desired state.

In AAM, any required rebases are found and addressed before the apply – no harm, no foul.

Complex Data Structures Are An Anti-Pattern

Fri, 06 Jan 2023 00:00:00 -0500

In Terraform, I've found that KISS > DRY and that the use of complex data structures as inputs are a gilded foot gun.

I've now created a few large Terraform repositories that have managed sprawling infrastructure spanning multiple environments at once and I've felt pain.

Oh! How I've hurt myself.

I learned Terraform as a software engineer and I applied my opinionated "best practices" to Terraform code – especially DRY. Terraform's primary means of DRYing your code is using Modules and I thought of Modules like do-everything blueprints. If I needed an AWS VPC, I'd create a "network" module. As my project needs changed, so would the module; it would be made to be more flexible.

That single network module would morph. Originally, it would only create a VPC and maybe a couple of subnets. Then public/private subnet pairs in 2 availability zones. Then, the big mistake would happen: I would make the module dynamic.

My project would pass in subnet configuration as an input .. something like:

subnets = {
  us_east_1a_public = {
    availability_zone = "us-east-1a",
    cidr              = "10.0.0.0/20",
  },
  us_east_1a_private = {
    availability_zone = "us-east-1a",
    cidr              = "10.0.16.0/20"
  },
  us_east_1b_public = {
    availability_zone = "us-east-1b",
    cidr              = "10.0.32.0/20",
  },
  us_east_1b_private = {
    availability_zone = "us-east-1b",
    cidr              = "10.0.48.0/20"
  }
  # You know there's more ... you're HA, right?
}

Then, you want to add network routing rules to just your private subnets, but you don't want to refactor a ton, so each entry in your inputs starts to look like …

us_east_1a_public = {
  availability_zone = "us-east-1a",
  cidr              = "10.0.0.0/20",
  routes = {
    "0.0.0.0/0" = aws_internet_gateway.igw.id
  }
},
us_east_1a_private = {
  availability_zone = "us-east-1a",
  cidr              = "10.0.16.0/20"
  routes = {
    "0.0.0.0/0" = modules.all_my_nat_gateways.nat_ids["us_east_1a"]
  }
}

You know that's ugly. It looks right but feels wrong. Eh, it works, right? It's fine.

It wasn't fine.
-Narrator

I think most software developers go down this path. They don't want to repeat themselves and they're told that static values inside code files is wrong. We do everything we can to avoid typing the word resource a 2nd time or 3rd time and we sure as hell don't want to see more than one resource block of the same type … I mean, can you imagine?!

You've done this, too, dear reader. I'm sure of it. Maybe you've noticed why it hurts … maybe not.

This is what you've done:

You've added logic to a framework based on static files.
You've made your resource block complex – it has expressions and loops in it. Maybe you had to create locals to hide the complexity, but that made it worse.
You've stopped defining resources in Terraform code and started defining them in your own language.
You now have an undocumented interface.
You've taken a declarative language and made it imperative.
You think you've made something beautiful.

You've really stepped in it this time.

Terraform is a framework for declaring your infrastructure using a simple, static, block-based interface. You've just stolen all three of its selling points from it. You've undone the very essence of Terraform. You monster.

As your repo of related projects grows, so does your library of widely reused, overly generic, modules. And so do the complexity of each of them as they morph and bend to fit scenarios they were never intended for.

Here comes my hot takes … my lessons learned. My cheat codes to avoid that pain.

More Terraform code is better than less.

Terraform projects and modules should be inflexible, single-purposed, and opinionated for each scenario. Input variables should be avoided unless their purpose is obvious.

So what that you have each subnet defined in its own resource block with the CIDR inline as a static value?

resource "aws_subnet" "us_east_1a_public" {
  vpc_id            = aws_vpc.my_vpc.id
  cidr_block        = "10.0.0.0/20"
  availability_zone = "us-east-1a"

  tags = {
    Name = "us-east-1a Public"
  }
}

resource "aws_subnet" "us_east_1a_private" {
  vpc_id            = aws_vpc.my_vpc.id
  cidr_block        = "10.0.16.0/20"
  availability_zone = "us-east-1a"

  tags = {
    Name = "us-east-1a Private"
  }
}

Nothing. That is readable. It's reasonable. It's easy to reason about. People unfamiliar with Terraform can understand it. It's documented. It's standardized. It's beautiful.

Everything is on fire and it's great

Thu, 29 Dec 2022 00:00:00 -0500

I haven't blogged about it, but a couple of years ago I went through an acquisition. It was a pretty incredible experience and overall pretty positive. I'm sure I'll blog about that at some point, but today I want to discuss a change that I experienced from going through that … a change in me. A change I wish happened a lot sooner.

I figured out how to not concern myself with most problems.

… or, at least I think I did. Something in me changed.

At Rigor (a 30-something-person startup), I was the do-everything-tech-dude. I was hired to be a C# developer, but switched to Ruby, but switched to infrastructure. I also managed our Google Workspace, Slack Workspace, marketing email delivery, office networking … anything and everything. It didn't even stop at the border of the company, either. I ended up managing the Slack Workspace and helping with IT and networking for the startup village where we had our office.

It seems that at some point I acquired an insatiable appetite for responsibility… or control? Or maybe I just saw problems I could solve all around me and there was nobody around to keep my focus on our product. Whatever the reason, it worked out really well … except …

That shit was toxic.

Over the four years at Rigor, I felt an insane amount of stress. Nothing that I had ever felt before. I'm a rescuer by nature. I run into high-stress situations, but this was different. This wasn't a high-stakes infrastructure outage … this was a frog boil. I didn't know it, but I was living in the stew all the while figuring out how to better cut the carrots.

I ended up having anxiety attacks. Once so bad I drove myself to the hospital because I thought I was having a heart attack. Read that again, dear reader. I drove. Myself.

Once acquired, something strange happened. I didn't have to worry anymore. We had money. We weren't going to go out of business. Our 30-something employees weren't going to go without their already-under-market salary. Suddenly, everything was fine. I felt a weight lift from me that I could never fully express. Yeah, there was stress in the new company, but the hot water was comfortable relative to the inferno that I had convinced myself was normal.

It's now been two years since the acquisition. I've moved to a higher-level team that has a ton of responsibilities. And I feel nearly no stress. I'm sitting here during our winter break (Splunk employees are off from Christmas to New Year… amazing) and it struck me that I hadn't thought about work at all in a week. I'm not worried about what my co-workers are doing. I don't bother them with what I'm doing. I can't concern myself with all of that.

There have been some organizational shifts since then … and I see other people being worried about it … but I just can't be bothered to even think about it. Those kinds of changes would be earth-shattering to me at any other point in my career. They would signal the end of whatever organization I was in. A sure death knell.

But, something changed in me.

Everything is a mess and I'm fine. There's messy business all around and I don't care. This is a dumpster fire and it's entirely normal. I've said these zen-like phrases before. It finally sank in.

Sure, my team is crazy understaffed (we should be 40 people and we're a team of 9) and other teams love to blame us for their own shortcomings … but this is great. We're not going anywhere. We're always improving. They treat me really well. Nothing else seems to matter to me anymore – I no longer need the control I can't have. I don't have to prove anything to anybody (I mean it this time). I have little to fear.

Now I'm stuck figuring out why. Why am I so zen-like? Have I gained the confidence I always faked? Have I just figured out how to stay out of business that isn't my own? Or, am I getting complacent?

Monitoring your home internet connection because boredom

Wed, 08 Jul 2020 00:00:00 -0400

Early in the COVID-19 lockdown extravaganza 🎉 my wife and I purchased a new home and moved in. It was a stressful time, filled with uncertainty. In the aftermath, I found myself with pretty terrible internet service and since we are both working in isolation, I am hellbent on finding out why …

So I'll use the tools I know – Telegraf, InfluxDB, Grafana, and Docker – to collect data, analyze it, and still probably not figure it out … but it's something to do.

This post is not aimed to be a step-by-step guide, as much as it is a mental dump. I aim to give you ideas and resources so if you want to replicate this kind of thing yourself you can do your own research to figure it out on your own.

Overview

My main goals were to meaure the quality of my internet connectivity and provide a dashboard or two so I can see short term and long term changes and trends. I set out to do this in two main ways: perform regular tests and pull spectrum analysis data directly from my cable modem.

Performing tests is really common and what you'd expect: send pings and perform http requests. Pulling data out of a cable modem is something I've never seen before and I wasn't sure if it was even possible (it is).

I ended up using a raspberry pi for data collection since it's always online (it's also my pi-hole) and using my Windows PC for storage and reporting. I may move things to a 2nd Raspberry Pi, but since I wanted to store a ton of data, I opted to keep the database local until I can build a better storage system.

Raspberry PI:

Telegraf
A Ruby Script to pull data from my modem

Windows 10 PC:

Docker
InfluxDB
Grafana

Here's what it ended up looking like:

Docker

I'm not going to cover setting up Docker. It's been done and you can find some really great articles about how to go about it. I'll only tell you about my environment.

I'm hosting this on my Windows 10 PC (I am a gamer, after all) with Docker Desktop. However, I do most of my command line work inside of Ubuntu running on the Windows Subsystem for Linux (bonus shoutout to the Windows Terminal Preview for finally making this not awful).

InfluxDB

InfluxDB is my main squeeze for timeseries data. We use it a ton at Rigor for both customer facing data, as well as all of our internal operational metrics storage and reporting.

I'm running this inside of docker, with the following command:

docker run --name influxdb \
  -e "INFLUXDB_ADMIN_PASSWORD=V$5gQ7*dX^z^5EvT8v7N01IXbom*XJBx^f" \
  -e INFLUXDB_ADMIN_USER=admin \
  --restart=always \
  -p 8086:8086 \
  -v "E:\Docker\influxdb-data:/var/lib/influxdb"
  influxdb

This does the following:

Sets environment variables which will instruct the service to create a first user (no, that's not my real password)
Instructs Docker to start the container on boot and always restart it if it fails
Exposes the InfluxDB API port publicly (port 8086)
Mounts a folder as a volume so we persist the delicious data no matter what

Important note: since I'm mounting a volume, I must run this in a Windows shell and not inside Ubuntu

You'll likely want to also start a Chronograf container temporarily, so you can manage your InfluxDB server and set it up however you'd like. Perhaps you want a database just for this data? Perhaps you want to create a write-only use for Telegraf? Chronograf will make it easier.

Telegraf

The 'T' in the TICK stack – Telegraf is a massively powerful data collection application. It's easy to overlook how wonderfully useful this application is because of how simplistic it appears. I'll configure it to perform all my tests and run my own script to pull data from the cable modem, batch it up, and ship it to influx on a fast schedule.

Keep in mind that Telegraf only keeps data in memory. This has two important implications when it cannot send data to its outputs:

it will continue to consume memory to hold data until it OOMs
it will loose all data it hasn't shipped if it crashes These, to me, are Telegraf's only real drawbacks. They aren't deal breakers for most applications, but they're important to note since it's possible you can drop data in some scenarios.

Again, I'm not going to detail how to install Telegraf – let Google be your guide there, but once you have it installed it should be as simple as editing the /etc/telegraf/telegraf.conf file and restarting the telegraf service (sudo service telegraf restart on Ubuntu).

Here's an sample of my telegraf config file. Notice that it has a single output – Influxdb – and a number of inputs – ping, http_response, and my own ruby script.

Ruby Script

Many cable modems have a web interface just like home routers – most people don't know about it, but there it is. Most of the time, you can reach it at http://192.168.100.1 and the login is either on a sticker on your cable modem, or it's something incredibly common. For my Netgear CM1200, it's admin/password.

Once you login, you should be able to get some pretty detailed measurements about your connection. This is an actual spectrum analysis that the cable modem performs and shows you a single snapshot.

I looked into using "real" monitoring protocols, like SNMP, to pull this data but my cable modem doesn't support it and it's unlikely yours will, either. Most consumer level network devices don't expose these kinds of protocols – any why would they, anyway?

So, I had to take a dirtier approach: web scraping. You hate it. I hate it. Let's get it over with.

➡ Look at my fabulous Ruby script here!

This script will connect to the web interface, grab the HTML for the page pictured above, and pull data out using regular expressions. (Un-)Luckily, the web interface doesn't generate html tables server side. Instead, it writes out all the data to a Javascript variable in a pipe-delimited format. This makes for easier regular expression patterns and since the HTML output is very stable (at least until a firmware update changes the interface), I can just grab the exact line number holding the variable I want. Ugly, for sure, but #ShipIt 🚀.

Grafana

Finally, the last step – creating a dashboard to dazzle your friends and blog readers.

Yet, again. I won't spend any real time getting Grafana running. I just use Docker:

docker run --name grafana -p 3000:3000 grafana

Once grafana is running, I configured an InfluxDB datasource, and created my dashboards.

Here's the JSON export of my dashboards – you'll need to tweak it if you're trying to recreate this, for sure.

What did I learn?

Once this whole thing was up and running, I immediately found that I was having pretty serious packet loss.

I also found that there are a number of frequencies on which my cable modem is unable to obtain solid locks and others that vary greatly through the day.

Setting us up the bomb

Tue, 25 Jul 2017 00:00:00 -0400

I've always known that scripts were out there and scouring the internet for exploitable websites. It's why the more widely used blogging applications (I'm looking at you, Wordpress) are so widely hacked.

To make my own little positive impact on the world (and because it's fun), I've decided to set up a few traps on my site.

I don't mind telling you about the traps, since it's unlikely that anybody trying to hack my site would actually read the blog. If they did, they'd know that my site is completely static, hosted on AWS S3, and served by AWS CloudFront. I won't say that my site is unhackable, but I will say that it would be difficult and there would be little to gain. There's no database, no credentials, and no secrets. Also, recovering from a hacked site would take me less than a minute.

So, on to the trap– the gzip bomb.

Gzip, a variant of the zip file you know and love, is very good at compressing repeating data. What if I could take a bunch of data, compress it, and somehow convince script kiddies to decompress the data causing their script to slow down, fill their hard drive, and maybe even crash some things? In fact, it's not terribly difficult.

As part of optimizing your website, you ought to compress your text content, using gzip, and instruct your visitor's browser to decompress the content prior to displaying it. This makes sites load much faster, since they're downloading less data, and it's not very difficult to do– most web servers handle this automatically or with easy configuration. The browser should download a gzip file and the HTTP response should have a Content-Encoding header of gzip. That header is the instruction to the browser to decomress the content.

So, to set up the bomb, I'll create a gzip file containing a bunch of garbage data–repeating zeros–, name it something an exploit script will absolutely look for, and upload it to my hosting.

After taking a look at my CloudFront usage report, I see that I often get scanned and the most common thing people are looking for is the /wp-login.php file. This is the file that processes Wordpress admin logins, so it makes sense that it would be top of my list.

So, with a handy flick of the wrist:

dd if=/dev/zero bs=1024 count=10240000 | gzip > wp-login.php
aws s3 cp wp-login.php s3://my_bucket/ --content-type "text/html" --content-encoding "gzip"

I have created a 10MB file, called wp-login.php that extracts to 10GB of nothing. It only takes about a minute to create the file. It does not extract a file that is 10GB, nor is it repeating zero characters– it's binary zeros … null characters. Trying to extract the data will result in simple nonsense, but the attempt will still be made and that's the important part.

It exists now, on my site. I won't link to it because it's possible that if you clicked that link it would crash your browser (although it seems that Chrome handles it pretty well, albeit slowly).

Am I sure this will inflict pain upon the lower levels of the internet hacker community? No, but it costs me nothing to do and I am an optimist at heart.

Learning

Automating Jekyll deployment to S3 using CircleCI

Mon, 30 Jan 2017 00:00:00 -0500

I've been working a great deal with Jekyll lately. I've used it for this blog for a while, but I hadn't spent much time automating the whole process. Unfortunately, when I set out to hook everything up, there didn't seem to be anybody out there doing exactly what I wanted, so it took me a little more time than I expected.

I'll guide you through the configuration steps I've taken to get my site automatically deploying to S3 (hosted by CloudFront) using CircleCI whenever you push to your master branch.

Overview of the Set Up

When this is all configured, we will be automatically deploying your Jekyll website to an S3 bucket every time you push code to your repository's master branch.

It assumes you already have your site hosted using S3 (and, optionally, Cloudfront).

We will be using the s3_website gem to push code to S3.

I find this a great set up because it will probably set our caching headers on our S3 objects and Gzips them, but also perform CloudFront invalidations on each build. Not to mention it takes out the grunt work of manually deploying my website.

Set Up CircleCI

First, a few steps that I shouldn't need to go into much detail:

Create an account on CircleCI
Add your project Point the new project to your website/blog git repository. This will kick off an initial build that will fail.
Add AWS permissions
1. From the dashboard, select the cog next to your project.
2. Go to "AWS Permissions" in the sidebar.
3. Enter AWS Access Keys that have adequate permissions.

Configure The Build

First, you'll want to add gem s3_website to your Gemfile and $ bundle install. This ruby gem will provide us with an executable that knows how to deploy our Jekyll build to S3 and can even configure our S3 bucket and CloudFront for proper hosting (mostly).

Circle.yml

CircleCI uses a circle.yml file, placed in the root of your project, for configuration. Here is mine:

machine: environment: JEKYLL_ENV: production RUBY_ENV: production ruby: version: 2.4.0 dependencies: pre: - gem install jekyll s3_website jekyll-sitemap jekyll-paginate jekyll-gist override: - bundle install: # note the colon here timeout: 240 # note the double indentation (four spaces) here cache_directories: - "/opt/circleci/.rvm/gems" compile: override: - bundle exec jekyll b -d $CIRCLE_ARTIFACTS test: override: - echo "Oh yeah!" # No way to test Jekyll. Just do something to satisfy CircleCI. deployment: # Only deploy master production: branch: master commands: - s3_website push

bundle exec jekyll b -d $CIRCLE_ARTIFACTS - This will make our jekyll output, usually saved to ./_site, save to a special folder on the CircleCI container that will be saved later and allows you to see the output of each build through their web interface. This is mostly a luxury, but can make debugging builds later easier.

test:
  override:
    - echo "Oh yeah!" # No way to test Jekyll. Just do something to satisfy CircleCI.

Remember that initial build that failed? This will fix it. CircleCI is built on the premise that builds are tested. The initial build failed because there were no tests executed. A build without tests is a failure (generally that's a great policy). As far as I know, Jekyll has no mechanism to test the site that's generated, so this configuration option will simply execute a bash command that outputs some text. Since the command executes successfully, CircleCI sees it as a success and will move on to deploy after build.

 deployment:
  production:
    branch: master
    commands:
      - s3_website push

This block instructs CircleCI to invoke s3_website push for any changes to the master branch.

s3_website.yml

This configuration file will give s3_website the information it needs to deploy to s3 and configure/manipulate your cloudfront distribution.

s3_bucket: ricky-dev.com # Below are examples of all the available configurations. # See README for more detailed info on each of them. site: <%= ENV['CIRCLE_ARTIFACTS'] %> # index_document: index.html # error_document: error.html # Set some good and long optimization values cache_control: "images/*": 'public, max-age=604800, s-max-age=86400' "css/*": 'public, max-age=604800, s-max-age=86400' "*": 'public, max-age=604800, s-max-age=86400' gzip: - .html - .css - .md - .cs - .xml - .json # gzip_zopfli: true # See http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for valid endpoints # s3_endpoint: ap-northeast-1 # ignore_on_server: that_folder_of_stuff_i_dont_keep_locally # Most of these shouldn't be included in the build, but I wouldn't want to be wrong and publish this exclude_from_upload: - Gemfile - _config.yml - s3_website.yml - circle.yml # s3_reduced_redundancy: true cloudfront_distribution_id: YOUR_DISTRIBUTION cloudfront_distribution_config: # default_cache_behavior: # min_TTL: <%= 60 * 60 * 24 %> aliases: quantity: 1 items: - ricky-dev.com - www.ricky-dev.com # cloudfront_invalidate_root: true cloudfront_wildcard_invalidation: true # concurrency_level: 5 # redirects: # index.php: / # about.php: about.html # music-files/promo.mp4: http://www.youtube.com/watch?v=dQw4w9WgXcQ # routing_rules: # - condition: # key_prefix_equals: blog/some_path # redirect: # host_name: blog.example.com # replace_key_prefix_with: some_new_path/ # http_redirect_code: 301

cloudfront_distribution_id - Be sure to set this value, if you're using CloudFront or comment it out, otherwise.

site: <%= ENV['CIRCLE_ARTIFACTS'] %> - This tells s3_website to look for the site in the CircleCI artifacts folder and not the default _site folder.

gzip: - I've included a few very common extensions for s3_website to gzip before uploading. Depending on your website content, you should customize this list.

exclude_from_upload: - In the next section, I'll instruct Jekyll not to include our config files in the site build, so this block should be redundant, but I left it in place to make sure nothing gets uploaded.

_config.yml

One last step is to make sure that our brand new config files don't get deployed to our public hosting. While there's nothing special in these configs, there's no reason to publish them and generally giving any information to would-be bad guys should be avoided.

IMPORTANT: If you decide to skip this step, you will still need to exclude the vendor folder. The build will fail with it because of a pre-build step CircleCI performs.

Just add/merge this block into your Jekyll config:

exclude:
  - vendor
  - Gemfile
  - Gemfile.lock
  - s3_website.yml
  - circle.yml

Be careful what you ask for; a lesson in monitoring

Tue, 21 Jul 2015 00:00:00 -0400

This morning was like any other—wake up late, shower, get stuck in traffic—but with a twist. As soon as I hit traffic, which was really bad even by I-85-in-midtown-Atlanta standards, my smart watch started to buzz every few seconds. Trio's servers are unreachable according to our monitoring service. It's going to be one of those mornings. When our monitoring service tells me something is wrong, I have a mental checklist, and I began working it (more-or-less in order).

Check NewRelic for performance issues.
Check Heroku's status page for outages.
Check AWS' status page.
Submit support ticket to Heroku, if applicable.
Look into any stop-gaps I can put in place to lessen the impact.
Inform users of potential issues.

I was drafting a user notice to pop-up in-app when something I saw hit me: NewRelic also has monitoring turn on and those alarms weren't going off; why not?

Eureka!

There is a fundamental difference in how NewRelic and Pingdom perform their monitoring—Pingdom follows redirects and tests the final destination whereas NewRelic (by default) accepts a redirect as a success. This little detail is important because a while back, we decided to throw away our single-page, teaser, homepage and simply redirect users to our iTunes store page. So, since we made that change our monitoring service hasn't actually been monitoring our website, but we've been monitoring the iTunes store. Oops.

With a quick fix and a flick of the wrist, Pingdom is now fetching our robots.txt file. This has two benefits: it's actually monitoring that our server is alive and accessible and it's fetching a static page, so there is very little overhead on the server.

TL;DR When you're setting up HTTP monitoring, create a static file and monitor that file's url.