But I'll also be the first to admit it: the mental tax is real. Memorizing the subtle differences in flags between Linux distributions, recalling the exact syntax for a netstat command you only use twice a year, or building a complex diagnostic script from scratch under pressure — it’s tough. The real challenge isn't just remembering the commands; it's about composing them into a workflow to solve a problem right now.

That’s why I’ve started evaluating Amazon Q CLI as a potential companion tool for my systems and network administration teams. The pitch is simple: what if your terminal understood plain English and translated that into working shell commands or even fully functioning scripts?

Well, it can. Sometimes it gives you exactly what you need, and other times it offers a starting point that’s about 80% there. Either way, when you're knee-deep in troubleshooting and short on time, that’s a decent head start.

Translate Mode: The Everyday Troubleshooter

My first test? I wanted to see if Amazon Q CLI could handle some of the small-yet-annoying queries that usually require a few minutes of mental gymnastics or a quick Stack Overflow detour.

During a disk space emergency (you know the kind: /var at 99%, panic rising), I tried this:

q translate "find log files in /var/log that are greater than 100 MB and modified in the last 60 mins"

It nailed it with a clean, readable find command. No man pages needed. No -mtime vs -mmin second-guessing.

Later, during a login audit, I threw this at it:

q translate "show failed or preauth ssh login attempts recorded in the journal during the last 180 mins"

Again, solid result. Parsed the right logs, scoped the time range — and importantly, didn't assume a specific distro. That’s one less thing to babysit.

It even handled a security check that I often include during instance validation:

q translate "list all open ports on this instance that are accessible from 0.0.0.0"

It gave me a reliable netstat pipeline that confirmed no unintended exposure. Clean and simple.

Need More Firepower: Enter Amazon Q Chat

Once you start trying to automate entire workflows — beyond single-line commands — Amazon Q Chat becomes the better companion. It’s less of a translator, more of a collaborator. Ask it for a script, and it doesn’t just stop at one-liners. It builds structure, logic, and explains what it's doing.

First I opened up Amazon Q chat on the instance,

Then asked:

“Generate a shell script that monitors CPU usage for a service called myservice and logs anything over 85%.”

Amazon Q returned a script that included ps, threshold logic, timestamps, and log formatting that made the output actually usable. Did I have to tweak it? Sure. But it felt like starting a race at the halfway mark.

Another day, I needed to dig into running processes and open ports related to NGINX. Instead of bouncing between ps, lsof, and a stack of bookmarks, I asked:

“Give me a script to list all processes for nginx and the network ports they’re using.”

It handed me a usable loop with pgrep and lsof, complete with structured echo output and basic validation. Bonus: it commented the sections so even junior admins could follow it confidently.

I also gave it a bit of a stress test:

“Build me a script to check for inode exhaustion, high disk usage, and mounts in read-only state — and flag anything risky.”

While running the script, Amazon Q CLI did more than just list a few commands. It initially flagged some issues as critical—though these turned out to be false alarms—but then automatically corrected the script and provided a cleaner, updated version.

Beyond Bash: Feeding Amazon Q Our Own World with MCP

We’re also exploring how Amazon Q could become even smarter inside our environment using Model Context Protocol (MCP). By plugging in our own runbooks, internal tool references, and wikis, we could reach a point where Amazon Q doesn’t just say “check the logs” — it tells you which logs, what known issues to match against, and what our escalation policy is.

Imagine:

“What’s the fix for a stuck Kafka consumer on our staging cluster?”

And Amazon Q responds with our exact process — or even kicks off an automation run-book. That’s the direction we’re heading.

Installing Amazon Q CLI On the Instance — Not Just Your IDE

Now here’s the part I’m still experimenting with — and maybe where this blog diverges from the usual.

Most documentation and demos show Amazon Q CLI being used from a developer’s machine, a laptop, or an IDE. But here’s the thing: IDEs can’t SSH into an instance mid-incident. And they can’t run netstat or read logs inside /var/log on an EC2 box.

That’s why I’m making a case for something a little unconventional: installing Amazon Q CLI on each EC2 instance — as a kind of assistant for system-level investigation.

Imagine: if you SSH into a problematic instance and Amazon Q CLI is already there, it can help accelerate the triage process. Whether it's a memory leak, a rogue process, or configuration drift, Amazon Q can give you the skeleton commands — and sometimes full scripts — without flipping through wikis or shell history.

This could be especially useful for:

• Instances running COTS products where logs and configs are scattered and vendor tooling is poor

• Legacy app servers with complicated service interdependencies and little documentation

• Security patch verification, where you need to confirm kernel versions or missing updates quickly

The Amazon Q CLI doesn’t act autonomously — it won’t magically fix things. But when you’re in the terminal, troubleshooting a stubborn issue, it’s a powerful ally to have already installed and ready.

⚠️ Important Note: While installing Amazon Q CLI directly on EC2 instances can be valuable for rapid troubleshooting, it’s essential to evaluate your organization's security and compliance requirements before doing so. Production environments may have restrictions on outbound network traffic, IAM role access, or the installation of developer tools. Ensure that any such deployment aligns with your operational and security policies.

Try It Out!

If you want to give this a spin yourself, install Amazon Q CLI:

curl --proto '=https' --tlsv1.2 -sSf "https://desktop-release.q.us-east-1.amazonaws.com/latest/q-x86_64-linux.zip" -o "q.zip"
unzip q.zip
sudo ./install.sh

Then authenticate and open up some of those harder questions, like:

q translate "compare /etc/httpd/conf/httpd.conf with the baseline in /opt/configs/httpd.conf"

Or dig deeper with Chat:

“Generate a script that checks CPU, memory, and open file descriptors for all processes owned by the apache user.”

Final Thoughts

Amazon Q CLI isn’t about dumbing things down. But it’s something worth considering — especially when your team is fighting fires, or when you're trying to codify your troubleshooting instincts into something repeatable.

For admins managing fleets of EC2 instances running complex stacks — like a commercial app that spawns Java, Node.js, and background daemons — having Amazon Q CLI right there inside the instance might be the difference between 30 minutes of rabbit-hole debugging and a 5-minute fix.

If you’re someone who SSHs into servers often and spends half that time remembering arcane command syntax — give it a shot, it might give you a second brain at the terminal — one that knows bash, AWS, and maybe even your own internal rules.

Disclaimer: Please note that AWS is constantly evolving, and new features may be available since the release of this blog post. It's recommended to review the latest documentation to determine the most suitable solutions for your specific needs. This blog is a reference guide only. Ensure that all solutions and tooling — including Amazon Q CLI — comply with your organization's security and compliance policies. Some services may still be evolving and may not yet meet all regulatory or industry-specific standards.

Deploying and managing applications in Kubernetes environments can be challenging, especially when dealing with complex configurations and multiple environments. Imagine a scenario where an operator needs to be deployed across development, staging, and production environments, each requiring unique configurations and settings. Without a structured approach, handling this for each environment can quickly become error-prone and time-consuming.

In this blog, the focus will be on simplifying and streamlining AWS ROSA (RedHat OpenShift Service on AWS) OpenShift Operator deployments using Kustomize and RedHat OpenShift GitOps (ArgoCD). By implementing GitOps methodologies, architects & engineers can alleviate the burden of managing multiple configurations while ensuring that deployments are consistent and reproducible. With GitOps, Git repository becomes the single source of truth for all configuration changes, making deployments predictable and auditable.

The exploration will cover the benefits of Kustomize, highlighting its seamless integration with Kubernetes and OpenShift, including support for both oc and kubectl commands. Additionally, the advantages of using Argo CD for continuous deployment will be discussed, showcasing how it enhances the deployment process by providing automated synchronization with Git repositories.

Why Kustomize?

Kustomize is a tool that enables configuration management through a "base and overlay" model, allowing to manage common and environment-specific configurations without duplicating YAML files.

• Native Kubernetes Integration: Kustomize is built into both kubectl and oc (OpenShift CLI), offering a seamless experience.

• Base and Overlay Structure: Define a common configuration (base) and apply environment-specific customizations (overlays) using patches.

• Declarative Configurations: By adhering to declarative principles, Kustomize simplifies version control and collaboration.

Benefits of GitOps with OpenShift GitOps (ArgoCD)

OpenShift GitOps (based on Argo CD) is a RedHat native continuous deployment tool that synchronizes the cluster's state with the configurations stored in Git. Here’s how it helps:

• Automated Continuous Delivery: Monitor Git repositories for changes, and apply updates automatically to the OpenShift cluster.

• Seamless Kustomize Integration: Manage complex, environment-specific configurations without duplicating YAML files.

• Declarative and Auditable: Track every change made to the cluster through Git, offering full traceability.

High-level flow deploying an Operator using OpenShift GitOps

This section tees up the steps for deploying an operator that is commonly used in the industry. Dynatrace is a well-known observability platform that enables organizations to monitor the performance of their applications and infrastructure effectively. In developing this blog, the decision to showcase the Dynatrace Operator stemmed from its widespread use and recognition within the industry. The focus here is not to provide a comprehensive guide to fully configuring Dynatrace, which involves additional steps like handling subscription keys, but rather to highlight the operator deployment process. This approach emphasizes the importance of automating the deployment in a consistent manner. For those interested in fully configuring Dynatrace, additional information can be found @ https://www.redhat.com/en/blog/partner-showcase-openshift-app-observability-with-dynatrace-operator.

Below are the overarching setup process to deploy the Dynatrace Operator using Kustomize and OpenShift GitOps.

Step 1: Set Up the Folder Structure

Create a structured folder layout for managing the Dynatrace Operator deployment using Kustomize. This structure allows for clear organization and easy management of base and environment-specific configurations.

code├── base
│   ├── kustomization.yaml
│   └── dynatrace-operator-subscription.yaml
├── overlays
│   ├── development
│   │   └── kustomization.yaml
│   ├── staging
│   │   └── kustomization.yaml
│   └── production
│       └── kustomization.yaml
└── argocd
    └── application.yaml
    └── kustomization.yaml

• Base Folder: Contains the base configuration for the Dynatrace Operator subscription. This folder holds the essential deployment YAML that can be reused across different environments.

• Overlays Folder: Each environment (development, staging, production) has its own folder containing its specific Kustomization file. Overlays allow for customization of the base configuration without duplicating it.

• ArgoCD Folder: Contains the Argo CD application manifest and its own Kustomization file that defines how Argo CD will manage the deployment of the Dynatrace Operator using Kustomize.

Step 2: Define the Base Configuration

In the base folder, the configuration files specify the core settings for the Dynatrace Operator. This includes deployment manifests and services that are common across all environments. PATCH-ME placeholders provides a convenient way to replace with environment specific values.

Example: base/dynatrace-operator-subscription.yaml

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/dynatrace-operator.openshift-operators: ""
  name: dynatrace-operator
  namespace: openshift-operators
spec:
  channel: PATCH-ME  # Placeholder for the channel
  installPlanApproval: PATCH-ME  # Placeholder for install plan approval
  name: dynatrace-operator
  source: certified-operators
  sourceNamespace: openshift-marketplace

Step 2: Create the base Kustomization configuration

This file specifies that the Dynatrace Operator subscription defined in dynatrace-operator-subscription.yaml is a resource that Kustomize should apply.

Example: base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - dynatrace-operator-subscription.yaml

Step 3: Create Environment-Specific Overlays

This file references the base configuration and applies the necessary changes specific to the development environment. The inline patch specifies what values will replace the PATCH-ME placeholders in the base YAML.

Example: overlays/development/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

patches:
  - target:
      version: v1
      kind: Subscription
      name: dynatrace-operator
     patch: |
      - op: replace
        path: /spec/channel
        value: alpha  # Dev-specific channel
      - op: replace
        path: /spec/installPlanApproval
        value: Automatic  # Dev-specific install plan approval

Step 4: Deploy using OpenShift GitOps(ArgoCD)

Create an Argo CD Application manifest that points to the appropriate overlay for deployment. This configuration informs Argo CD to sync the specified environment's configuration to the OpenShift cluster automatically.

Example: Argo CD Application Manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dynatrace-operator
  namespace: openshift-gitops
spec:
  project: default
  source:
    repoURL: 'https://github.com/your-repo/dynatrace-operator'
    targetRevision: HEAD
    path: overlays/development
    kustomize:
      namePrefix: dev-
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: dynatrace-operators
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

• Source Path: By setting path: overlays/development, Argo CD knows to apply the development overlay and deploy the development-specific configuration.

• Destination Namespace: The namespace: dynatrace-operators defines where the operator will be deployed. This can be different for each environment.

• Automated Sync: The syncPolicy ensures that Argo CD automatically applies changes when they are pushed and merge to Git repo, keeping the cluster state in sync with the desired state in Git.

Why GitOps (Argo CD) Simplifies Management

GitOps, particularly via OpenShift GitOps (Argo CD), offers a superior method of managing operator deployments for several reasons:

• Centralized Source of Truth: The Git repository serves as the single source of truth for all configurations. This ensures consistency across environments and makes it easy to audit changes.

• Automated Deployments: Argo CD automatically syncs any changes in the Git repository to the Kubernetes cluster, ensuring that the cluster state always matches the desired state in Git.

• Environment Control: By using overlays, we can easily customize deployments for different environments (dev, staging, production), while still maintaining a common base configuration.

• Operator-Driven Approach: Since GitOps itself is managed by an operator in ROSA, the operational burden is reduced. However, deploying GitOps operator to the cluster must be done manually or via automation (e.g., Terraform) to begin with, ensuring the operator is in place to manage future deployments.

Enabling OpenShift GitOps

To enable OpenShift GitOps (Argo CD) in the AWS ROSA cluster, we need to deploy the OpenShift GitOps Operator. This can be done manually through the OpenShift web console or automated using tools like Terraform or Helm. We recommend installing the OpenShift GitOps operator as soon as the cluster is provisioned, treating it as a day 0 activity. From that point forward, GitOps(ArgoCD) should manage all operators and application deployments.

Alternatives to Kustomize and GitOps

While Kustomize and GitOps provide a streamlined approach to deploying OpenShift Operators, other methods also exist for managing configurations and deployments. Terraform is a powerful tool for provisioning and managing infrastructure, making it an excellent choice for cloud resources. However, when it comes to managing Kubernetes configurations, Terraform might not be the best fit due to its complexity in handling Kubernetes manifests and the potential for configuration drift in dynamic environments.

Another option is Helm, which is a popular package manager for Kubernetes. Helm works well with OpenShift GitOps and can simplify the deployment of complex applications through templating. However, for the purposes of this blog, Kustomize was chosen for its straightforward approach to managing configurations directly within Kubernetes without the need for additional templating.

Conclusion

Using Kustomize and GitOps with the OpenShift GitOps Operator for deploying OpenShift Operators on AWS ROSA provides a scalable, maintainable, and consistent approach to managing Kubernetes configurations. Leveraging the power of Kustomize's native Kubernetes support along with OpenShift GitOps' continuous deployment capabilities facilitates efficient and reliable application deployments. This integration ensures that all changes are tracked in version control, allowing for better collaboration and traceability within teams. By automating deployment processes and managing configurations effectively, DevOps practices become smoother and more robust, leading to improved productivity and operational excellence. With the Red Hat OpenShift GitOps Operator, organizations can confidently streamline their workflows while ensuring consistency across different environments, ultimately enhancing the overall performance and reliability of their Kubernetes applications.

\The code snippets provided in this blog server as a framework for understanding the deployment logic. They have been sanitized of real data and should be used for basic guidelines only.*

AI in Action: A Sneak Peek at What's Already Here

Even today, AI is silently working behind the scenes in many of your favorite AWS services. Here are just a few examples:

• Amazon EC2 Auto Scaling: This service leverages machine learning to automatically adjust the number of EC2 instances you have running based on real-time demand. No more scrambling during peak hours – AI ensures you have the resources you need, when you need them, while optimizing costs.

• Amazon DynamoDB: This NoSQL database service utilizes AI for adaptive capacity management. AI analyzes usage patterns and automatically scales storage and throughput to meet fluctuating workloads. Say goodbye to performance bottlenecks and hello to a responsive database.

• Amazon Comprehend: This natural language processing (NLP) service uses AI to extract insights from your text data. Imagine using Comprehend to automatically tag and categorize customer support tickets, enabling faster resolution times.

These are just a taste of the many ways AI is already embedded within AWS. But the future holds even more exciting possibilities.

The AI-Powered Cloud Architect: A Glimpse into Tomorrow

As AI continues to evolve, cloud architects can expect to see a significant shift in their approach:

• Proactive Problem Solving: AI-powered tools will proactively identify potential issues before they become outages. Imagine an AI system that detects anomalies in resource utilization and recommends corrective actions, preventing downtime before it even starts. AWS's upcoming AI Ops services promise to revolutionize incident management by leveraging AI-driven analytics to detect, diagnose, and resolve issues in real-time.

• Self-Optimizing Infrastructure: Cloud infrastructure will become self-aware, automatically scaling and optimizing resources based on real-time demands and workload patterns. This frees you from manual configuration tasks, allowing you to focus on higher-level strategic initiatives. AWS's vision for autonomous cloud infrastructure includes AI-driven resource optimization, dynamic workload management, and predictive capacity planning, ensuring optimal performance and cost-efficiency at all times.

• Improved Security Posture: AI will play a crucial role in safeguarding your cloud environment. AI-powered security solutions can analyze network traffic and user behavior to identify and thwart cyber threats in real-time. AWS's security offerings, such as Amazon GuardDuty and AWS Security Hub, leverage AI and machine learning to provide continuous threat detection, automated remediation, and actionable insights, enhancing the overall security posture of your cloud infrastructure.

Why This Matters: The Power of an Intelligent Cloud

The adoption of AI in cloud infrastructure isn't just about bells and whistles; it's about unlocking a new level of efficiency, security, and agility. Here's how:

• Reduced Costs: AI-driven automation helps eliminate wasteful resource allocation, leading to significant cost savings. By rightsizing instances, optimizing storage usage, and minimizing downtime, AI empowers organizations to maximize their cloud investments and achieve greater ROI.

• Enhanced Agility: The ability to automatically scale resources based on demand allows you to respond quickly to changing business needs. With AI-powered elasticity, you can seamlessly accommodate spikes in traffic, launch new services faster, and experiment with innovative ideas without the fear of resource constraints.

• Improved Security: AI-powered threat detection and prevention systems provide an extra layer of protection for your sensitive data. By continuously analyzing vast amounts of telemetry data and identifying suspicious patterns, AI enhances threat visibility, reduces response times, and strengthens overall resilience against cyber threats.

• Focus on Innovation: By automating routine tasks, AI frees up your valuable time to focus on strategic initiatives and innovation. Whether it's developing new features, exploring emerging technologies, or driving digital transformation initiatives, AI empowers cloud architects to become agents of change and innovation within their organizations.

Embrace the Future: Becoming an AI-Ready Cloud Architect

The future of cloud infrastructure is intelligent, and AWS is at the forefront of this revolution. Here are some steps you can take to become an AI-ready cloud architect:

• Stay Curious: Keep yourself updated on the latest advancements in AI and how they can be applied to cloud infrastructure. Explore resources like the AWS AI Blog, the Amazon YouTube Channel, and the APN Partner Blog to delve into the latest advancements and discover how experts are leveraging AI to build intelligent cloud solutions.

• Embrace Experimentation: Don't be afraid to experiment with new AI-powered AWS services. Leverage AWS's free tier and sandbox environments to explore AI capabilities, test use cases, and gain hands-on experience without incurring additional costs. There are also free Demo and hands on tutorial available for service such as Amazon Bedrock and Amazon SageMaker.

• Develop New Skills: Start building your knowledge of AI concepts and machine learning principles. Enroll in AWS training courses, explore online tutorials and resources, and participate in hands-on labs to develop your AI expertise and position yourself as a leader in AI-driven cloud architecture.

The future of cloud infrastructure is intelligent, and with the help of AI, we can build more robust, efficient, and scalable cloud environments. So, buckle up, cloud architects, the future is intelligent, and it's going to be an amazing ride!

Ever since AWS announced the alpha release for Mounpoint back on Mar 14, 2023, I have been eagerly waiting for the integration to be generally available. The idea of seamlessly mounting S3 buckets as if they were local drives holds immense potential for transforming the way we handle data storage and access. And now, with the feature finally becoming generally available, the moment I've been waiting for has arrived. In this article, I'll be sharing my initial thoughts and experience as I dive into testing the Mountpoint tool for Amazon S3.

Mountpoint, an open-source project for Amazon S3 is a testament to the dynamic evolution of cloud storage solutions that we have been witnessing in the past few years. Conceived to address the growing demand for simplified access to data storage, the AWSLABS developer community has done a tremendous job in making sure that the tool is easy to use and is an enterprise-ready client that supports performant access to S3 at scale.

What is Tool Fit for:

• An Amazon S3 mounted EC2 instance allows the use of native commands, shell commands & library functions like 'ls', 'cat', 'cp', 'touch', 'grep', 'open' etc. to list, read and interact with files.

• It can be a great tool for data lake architectures and use cases, enabling seamless access to read large objects stored in S3 to multiple instances concurrently without the need of downloading them to local storage first.

• It could be a great tool to simplify uploading and downloading files from S3, sharing and transferring files across local and cloud storage while taking advantage of S3 scale and durability.

• It could facilitate collaborative workflows by providing a unified platform for storing, accessing and interacting with shared objects directly from local environments.

What the Tool is Not:

• While Mountpoint offers seamless access to S3 files, it's not optimized for real-time collaborative editing scenarios. It only supports writing only to new files, and writes to new files must be made sequentially.

• While Mountpoint provides a local storage feel, it's not a replacement for traditional local storage solutions. It compliments local storage by harnessing the benefits of cloud resources.

• Mountpoint does not implement all POSIX file system features. It doesn't support advanced file operations such as locking, file permissions and ownership.

Installing the tool

Installing the Mountpoint for Amazon S3 was simple and straightforward. On my Amazon Linux EC2 instance, I made use of the RPM package and installed it using the 'yum' command.

$ wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
$ sudo yum install ./mount-s3.rpm

While the Mountpoint client is designed to automatically pick up the credentials from an IAM role associated with the instance, I am using the AWS credentials from environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY since I am testing on an AWS sandbox playground.

$ mkdir mountp-test-fs
$ mount-s3 mountp-test-bucket mountp-test-fs
bucket mountp-test-bucket is mounted at mountp-test-fs

Changing the directory to the newly mounted folder and creating new files:

$ cd mountp-test-fs
$ mkdir test-folder
$ ls -l
total 0
drwxr-xr-x. 2 ec2-user ec2-user 0 Aug 11 17:50 test-folder
# Create new file
$ echo "Hello Mounted S3 Bucket" > TestFile_Write.txt
$ ls
TestFile_Write.txt
#View the file
$ cat TestFile_Write.txt
Hello Mounted S3 Bucket
#Find the line number for the word 'S3' in the file using grep 
$ grep -n 'S3' TestFile_Write.txt | wc -l
1
#Find the line number for the word 'S3' in the file using sed
$ sed -n '/S3/=' TestFile_Write.txt 
1

S3 bucket

As you can it's pretty simple to integrate with S3 and use the native shell/bash commands to interact with the objects within the bucket. However, as mentioned before one of the limitations is that we cannot edit or update an existing file using mountpoint (Maybe a feature that could be enabled in the future).

#Error during updates to the file objects 
$ echo "Hello, I am trying to update the file with new content." > TestFile_Write.txt
-bash: TestFile_Write.txt: Operation not permitted

Logging

As an Infrastructure Architect, I am constantly committed to maintaining operations and analyzing errors. So after simulating various error scenarios, such as directory deletions and removing files from the S3 bucket, I was able to view the logs via syslog. I found that Mountpoint only logs high-severity events and verbose logging can be enabled with the --debug option.

#Sample of error logs from syslog during my testing 
$ journalctl -e SYSLOG_IDENTIFIER=mount-s3
Aug 11 19:53:09 mount-s3[58999]: [WARN] lookup{req=1408 ino=1 name="test-folder"}: mountpoint_s3::fuse: lookup failed: inode error: file does not exist
Aug 11 19:53:09 mount-s3[58999]: [WARN] lookup{req=1410 ino=1 name="test-folder"}: mountpoint_s3::fuse: lookup failed: inode error: file does not exist
Aug 11 19:53:46 mount-s3[58999]: [WARN] lookup{req=1454 ino=1 name="test-folder"}: mountpoint_s3::fuse: lookup failed: inode error: file does not exist
Aug 11 19:53:51 mount-s3[58999]: [WARN] lookup{req=1460 ino=1 name="TestFile_Write.txt"}: mountpoint_s3::fuse: lookup failed: inode error: file does not exist

Summary

Mountpoint integration simplifies data collaboration, provides a local storage feel, and is easy to set up and use. It will prove to be an amazing tool to seamlessly transfer and manage large datasets from Amazon S3 to a local environment for data analysis, without the need for time-consuming downloads or compromising on storage capacity. However, it has to be noted that it does not facilitate real-time editing capabilities and offline access to the data is not supported since it requires an active internet connection to access S3 resources. I also wished that it can be integrated with CloudWatch natively so that we can easily monitor and analyze traffic and errors. Overall, it's a valuable asset for cloud workflows, bridging the gap between cloud and local storage with convenience, flexibility and scalability.

• Ajith Joseph, Manager, Deloitte;

• Noel Arzadon, Specialist Master, Deloitte;

Introduction

Most organizations that distribute content over the internet want to restrict access to the documents, data and content in their Amazon S3 buckets. While Amazon S3 provides REST APIs and associated IAM roles/policies provide easy access to the files and documents within the bucket, sometimes there is a need to distribute these objects within the Amazon S3 bucket without exposing the S3 bucket names that are embedded in the direct access URLs.

For example, Amazon S3 Pre-signed URLs are great when it comes to distributing content with temporary access to specific documents. One of the most common use cases for S3 pre-signing is distributing documents like correspondence or decisions to the customers where they will receive an email with a link to specific documents which can be viewed and downloaded via a web browser. However, a drawback of this approach is that the S3 bucket name will be visible to the customer since S3 pre-signing process utilizes the same direct S3 REST APIs to generate temporary access to the objects within the bucket.

The below figure shows an example of an S3 pre-signed URL with temporary access to a document. This method will expose the bucket name where it is stored.

This could be a deal breaker for organizations that have stringent security rules. While there are many ways to overcome such situations, utilizing a Amazon CloudFront distribution to distribute S3 content could be a possible solution that can be implemented swiftly while adhering to all security needs.

Amazon CloudFront for signing and custom domain names

Amazon CloudFront being one of the most popular CDN services is a great way to distribute content from S3 buckets, also there are additional features that one can take advantage of such as caching, edge processing, geographic restrictions etc. The idea of restricting the users to access S3 private content by requiring viewers to use CloudFront signed URLs and ensuring that the bucket can be accessed only via CloudFront Origin Access Control (OAC) can greatly enhance the security posture of any organization.

The below figure Illustrates how OAC and bucket policies can prevent access to S3 buckets using direct URLs. (Ideally, WAF can also be utilized to enhance CloudFront origin security but since this blog is a beginner's guide to CloudFront signed URLs, we won't be deploying WAF).

Objectives:

In this walk-through, you will see examples of using Amazon CloudFront for:

• Signed URLs to restrict access to S3 content and mask S3 bucket name in the URL

• Sample Python code to generate signed URLs

• Testing of signed URLs to download/view a test document from S3

Walk-through Steps:

Pre-requisites:

• Understanding of OpenSSL, IaC such as CloudFormation, CloudFront Distribution & DNS

• Basic python programming

Step 1:Create the key pair to be associated with the CloudFront signers

The signer will use the private key to sign the URL and the CloudFront utilizes the public key to verify the signature. In this blog(Step 4) we will explain how to use a simple Python module to sign the URLs using private keys and generate signed URLs that are valid for a specified expiry time. First, let's create the key pairs required.

Note: The key pair must be SSH-2 RSA, base64 encoded PEM format and 2048 bit.

Using the OpenSSL, we can generate an RSA key pair of 2048 bits and save it as private_key.pem

openssl genrsa -out private_key.pem 2048

Now extract the public key out of the private key using the following command

openssl rsa -pubout -in private_key.pem -out public_key.pem

Step 2:Provisioning CloudFront distribution and adjust S3 bucket permissions

Below is a code snippet to create Origin Access Control via Cloudfromation to securely access the S3 content

  CloudblogCloudFrontOAC:
    Type: AWS::CloudFront::OriginAccessControl
    Properties:
      OriginAccessControlConfig:
        Description: Origin access control for signed urls to S3
        Name: CBOAC-S3
        OriginAccessControlOriginType: s3
        SigningBehavior: always
        SigningProtocol: sigv4

After the OAC is created, associate it with a CloudFront Distribution and also if available we will highly recommend using a custom domain name and associating it with the distribution. (This blog doesn't have steps on how a domain can be created, I have used a popular domain service provider to create the domain used in this blog)

  DSCloudFrontDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        WebACLId: "arn:aws:wafv2:us-east-1:XXXXXXXXXXX:global/webacl/CB-WebACL-ForBlog/"
        Aliases: 
          - cftest.cloudblog.ajosephlive.com
        ViewerCertificate: 
          AcmCertificateArn: "arn:aws:acm:us-east-1:XXXXXXXXXXXXX:certificate/XXXXXXXXXXXXX"
          MinimumProtocolVersion: TLSv1.2_2021
          SslSupportMethod: sni-only  
        Origins:
          - Id: CBS3Origin
            DomainName: "    test-bucket-in-us-east-2-cloudblog.s3.us-east-2.amazonaws.com"
            OriginAccessControlId: !Ref CloudblogCloudFrontOAC
            S3OriginConfig:
              OriginAccessIdentity: ""

        Enabled: 'true'
        Comment: S3 CloudFront
        DefaultCacheBehavior:
          TargetOriginId: CBS3Origin
          AllowedMethods:
            - GET
            - HEAD
            - OPTIONS
            - PATCH
          ViewerProtocolPolicy: redirect-to-https
          CachePolicyId: !Ref CBCloudFrontCachePolicy

        IPV6Enabled: false
        CacheBehaviors:
          - AllowedMethods:
              - GET
              - HEAD
            TargetOriginId: DSS3Origin
            PathPattern: /media/*
            ViewerProtocolPolicy: redirect-to-https
            CachePolicyId: !Ref CBCloudFrontCachePolicy

        CustomErrorResponses:
          - ErrorCode: '404'
            ResponsePagePath: "/error-pages/404.html"
            ResponseCode: '200'
            ErrorCachingMinTTL: '30'

You should see the CloudFront distribution deployed similar to the below figure.

Update the bucket permission to allow access only from CloudFront using OAC. A sample JSON policy is below:

{
    "Version": "2012-10-17",
    "Statement": {
        "Sid": "AllowCloudFrontServicePrincipalReadOnly",
        "Effect": "Allow",
        "Principal": {
            "Service": "cloudfront.amazonaws.com"
        },
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::test-bucket-in-us-east-2-cloudblog/*",
        "Condition": {
            "StringEquals": {
                "AWS:SourceArn": "arn:aws:cloudfront::<AWS account ID>:distribution/<CloudFront distribution ID>"
            }
        }
    }
}

Step 3: Uploading public key to CloudFront

Now upload the public key that was created in Step 1 to CloudFront public keys. Also, create a key group for this public key.

Associate the distribution created to use this key group and most importantly restrict viewer access to use signed URLs. This will ensure that CloudFront origin is only accessible via signed URLs.

Below updates to the CloudFormation template for the TrustedKeyGroups parameter should do the trick.

        CacheBehaviors:
          - AllowedMethods:
              - GET
              - HEAD
            TargetOriginId: DSS3Origin
            TrustedKeyGroups: 
              - cloudblog-ajoseph-test-pub-key-group

Verifying the update via the CloudFront console should look like this:

Step 4: Code for generating signed URLs

For testing the CloudFront signing process, we created a lambda function written in Python which utilizes the private key generated in Step 1 and produces a Signed URL.

Note: Since we are importing modules from RSA in the below code snippet, we had to deploy a Lambda Layer with the rsa package.

from botocore.signers import CloudFrontSigner
import rsa
import base64
import json
import os
import boto3
from datetime import datetime, timedelta

ssm_client = boto3.client('ssm')

#In this example we are using parameter store to store the key pair values; for better secuirty pls store them in Secrets Manager or similar.
public_key_id = ssm_client.get_parameter(Name="/cloudblog/ajoseph/signed-url/public-key-id",WithDecryption=True)
private_key_data_dict = ssm_client.get_parameter(Name="/cloudblog/ajoseph/signed-url/private-key",WithDecryption=True)
private_key_data=private_key_data_dict['Parameter']['Value']  

def rsa_signer(message):
    private_key = private_key_data
    return rsa.sign(
        message,
        rsa.PrivateKey.load_pkcs1(private_key.encode('utf8')),'SHA-1')

def lambda_handler(event, context):
    try:
        query_string=event['queryStringParameters']["filename"]
    except:
        query_string=None 
    key_id = public_key_id
    url = "https://cftest.cloudblog.ajosephlive.com/"+query_string
    cf_signer = CloudFrontSigner(key_id , rsa_signer)
    expire_date = datetime.utcnow() + timedelta(hours=1) # expires in 1 hour
    # Signing with a canned policy::
    signed_url = cf_signer.generate_presigned_url(url, date_less_than=expire_date)

    return {
       'statusCode': 200,
       'body': signed_url
    }

Step 5: Testing the signed URLs

Executing the Python code from above provides a CloudFront signed URL that has temporary access to the file TestFile.pdf stored in test-bucket-in-us-east-2-cloudblog. From the below screen print, it is evident that the bucket name is no longer visible in the signed URL.

This signed URL can now be used in any web browser to download/view the file until the expiry time.

Conclusion

In this brief walk-through, we have learned that configuring CloudFront Signed URLs to require user access files in Amazon S3 is a great way of enhancing security, masking bucket names and controlling access to contents stored in AWS. We followed a very simple example and did not attempt to build a production-grade architecture. Utilize the basic concepts of Signed URLs taught here along with KMS keys for encryption, AWS WAF for inspection, custom headers validation, and lambda@edge functions to create a robust architecture to distribute Amazon S3 contents.