My Favourite Platform Engineering Tools

In this post, I'll share my favorite platform engineering tools that have not only saved countless hours but have fundamentally changed how I solve small and complex problems in platform engineering.

Octoherd

Octoherd is a powerful GitHub automation tool that lets you execute scripts across multiple repositories with ease. It provides a simple command-line interface for interacting with GitHub's API and automating repetitive tasks.

Use Case I: I used Octoherd to archive GitHub repositories instead of burdening individual repository owners with this mundane task. With over 1,000 repositories needing to be archived, Octoherd's intuitive API made scripting the solution straightforward. The process worked so well that I contributed the script back to the Octoherd community. If you're facing a similar bulk archiving challenge, check out github.com/octoherd/script-archive-repository.

npx @octoherd/script-archive-repository \
  -T ghp_0123456789abcdefghjklmnopqrstuvwxyzA \
  -R "stefanbuck/repo-a" \
  -R "stefanbuck/repo-b" \
  --octoherd-bypass-confirms

Use Case II: When I needed to analyze CI/CD configuration files across hundreds of repositories to identify patterns, I created octoherd-script-get-files. Rather than cloning every repository (which would have been time-consuming and resource-intensive) this script allowed me to download only the specific files needed for analysis. This approach enabled me to quickly identify inconsistencies across our organization's CI/CD practices.

npx octoherd-script-get-files \
  -T ghp_0123456789abcdefghjklmnopqrstuvwxyzA \
  -R "stefanbuck/*" \
  --source '.github/workflows/*' \
  --output ./out

Special Shout-out to Gregor Martynus for creating Octoherd among countless other invaluable tools that have revolutionized GitHub automation.

Octokit Plugin Create Pull Request

The Octokit Plugin Create Pull Request, another brilliant tool by Gregor Martynus, provides a programmatic way to create pull requests.

Use Case: In our organization, we needed to roll out standardized changes across hundreds of repositories. Previously, such changes had to be painfully coordinated through a combination of Jira tickets and Slack messages leading to delays, and more importantly inconsistent implementation.

By using this Octokit plugin, we completely transformed our change management process. We've created over 2,500 pull requests programmatically, ensuring that teams receive proposed changes directly in their GitHub repositories where they can properly review, discuss, and merge them.

This approach has been a massive success, transforming what was once a coordination nightmare into a streamlined process that both our platform and development teams appreciate.

Kyverno

Kyverno is a powerful Kubernetes-native policy engine. It functions as a Swiss Army knife for Kubernetes management, capable of validating, mutating, and even generating resources dynamically. I'm confident that Kyverno will become an essential component in every cluster in the coming years, similar to how ArgoCD or Flux have become standard tools.

The policies available at kyverno.io/policies/ are maintained by the community and serve as samples that demonstrate the power and flexibility of Kyverno. Often, these policies are all you need for your implementation, and even when they don't match your requirements exactly, they provide an excellent starting point. For those interested in experimenting with Kyverno, their playground offers a great way to test and try out policies without needing to set up your own cluster.

Use Case: We implemented Kyverno to address several challenges in our Kubernetes environment. Beyond enforcing security policies (like requiring specific labels or limiting privileged containers), we leveraged Kyverno's mutation capabilities to automatically inject sidecar containers and modify resource configurations without requiring developers to change their manifests.

In one case, we used Kyverno to automatically generate network policies based on application metadata, ensuring proper network segmentation without burdening our development teams with the complexities of Kubernetes networking.

This dramatically improved our security posture while reducing developer friction.

JMESPath

JMESPath is a query language specifically designed for extracting and transforming JSON data. While jq is often the go-to tool for ad-hoc JSON processing on the command line, JMESPath stands out for its consistent implementation across multiple programming languages and environments.

Use Case: When working on a platform that needs to process JSON data from various sources across different components, I stumpled JMESPath. The key advantage was JMESPath's availability as libraries in multiple languages including Python, Go, JavaScript, and more. This allowed us to write JSON queries once and implement them consistently across different parts of our system—from backend services to automation scripts.

The comprehensive compliance test suite that each JMESPath library passes gave us confidence that our JSON processing logic would work identically regardless of the language or environment. This consistency eliminated subtle bugs that might otherwise occur when implementing similar but slightly different JSON processing logic.

ZX

ZX is a powerful tool developed by Google that modernizes shell scripting by seamlessly integrating JavaScript with command-line operations. It provides a clean, familiar syntax for developers who want the flexibility of JavaScript while maintaining the ability to execute shell commands easily.

Use Case: My journey with ZX typically begins with a simple bash script. However, as requirements grow more complex, I often find myself wrestling with the limitations of traditional shell scripting, especially when dealing with complex data processing, error handling, and maintainability.

Dive

Dive is a powerful tool designed to analyze Docker image layers and their contents. It provides an interactive interface for exploring each layer of a Docker image, making it easier to identify bloat and inefficiencies in your container builds.

Use Case: Although I haven't used it recently, Dive played a crucial role in our container optimization efforts. When our CI/CD pipelines were taking too long and our deployments were slow due to large image sizes, we needed to understand exactly what was contributing to our bloated images.

Dive allowed us to inspect each layer of our Docker images in detail, revealing unnecessary files and inefficient layer caching. By visualizing exactly what changes were happening at each step of our Dockerfiles, we were able to implement targeted optimizations like improved .dockerignore files, better layer ordering, and multi-stage builds. These changes reduced our image sizes by over 60% and significantly improved our deployment times.

Conclusion

The best platform engineering solutions are those that address your team's unique needs while reducing complexity and technical debt. I encourage you to explore these tools and find the ones that help solve your specific challenges. Your perfect toolkit might look different from mine, but the right tools will make all the difference for you and your platform consumers.