How to Track AWS Resources with a Bash Script

TL;DR

Use a Bash script with the AWS CLI to collect AWS resource usage into a report, then schedule it with a Linux cron job for automatic daily execution. The project tracks resources such as EC2 instances, S3 buckets, Lambda functions, and IAM users, helping teams identify unused infrastructure and support cloud cost control.

Transcript

hello everyone my name is Abhishek and welcome back to my channel so today is day 7 of our full devops course and in this class we'll be talking about a real-time shell script project that usually most of the devops engineers use uh when they are on the cloud infrastructure so firstly let me explain what this project is and why is it useful and the... Read More

Key Insights

• Cloud adoption is motivated partly by reduced maintenance overhead because organizations do not need to create data centers, manage physical servers, and maintain a dedicated team for constant server patching, updating, and upgrading.
• Pay-as-you-use billing is a major cloud benefit because organizations are not committed to purchased physical infrastructure regardless of usage. However, AWS still charges for provisioned resources, so unused infrastructure must be identified and managed.
• Resource tracking is a DevOps and AWS administration responsibility because developers may create infrastructure that remains unused. Examples in the lesson include unused EC2 instances and EBS volumes that are provisioned without being attached to an EC2 instance.
• The AWS resource tracker is a Bash script that uses AWS CLI commands to gather cloud inventory information. The example focuses on EC2 instances, S3 buckets, Lambda functions, and IAM users rather than attempting to demonstrate every available AWS resource.
• A resource report can record how many EC2 instances are active, how many S3 buckets exist, how many Lambda functions exist, and how many IAM users exist. Organizations can supply this information to a reporting dashboard for ongoing visibility.
• Cron is a Linux scheduling mechanism that can execute a script automatically at a specified time. Scheduling removes the need for an engineer to log in and manually run the resource tracker every day at the required reporting time.
• AWS CLI configuration requires an access key, secret access key, default region, and output format. Running aws configure supplies these settings so the Linux environment can authenticate and communicate with the relevant AWS account.
• An explicit Bash shebang is important because sh can be a symbolic link to Bash or Dash. Since those shells have syntax differences, using /bin/bash directly helps ensure that a script written for Bash runs with the intended interpreter.

Questions & Answers

Q: How do you track AWS resources with a Bash script?

Create a Bash script that calls the AWS CLI to retrieve information about the resources used by the organization. The example tracker gathers details for EC2 instances, S3 buckets, Lambda functions, and IAM users, then places the resulting resource usage information in a file. The same information can later be supplied to a reporting dashboard for regular monitoring.

Q: Why should DevOps engineers monitor unused AWS resources?

DevOps engineers should monitor unused resources because AWS can continue charging for infrastructure that has been created even when nobody actively uses it. The lesson gives examples of a developer creating many EC2 instances or leaving EBS volumes unattached. Tracking usage supports cost effectiveness, which is identified as one of the primary reasons organizations move to cloud infrastructure.

Q: How can an AWS resource report run automatically every day?

A Linux cron job can schedule the shell script to execute every day at a specified time. After the resource tracker is integrated with cron, a Linux process waits for the scheduled time and runs the script automatically. This prevents missed reports when the responsible engineer is unavailable or cannot log in and execute the script manually.

Q: What AWS services does the resource tracker monitor?

The demonstrated project focuses on commonly used resources: EC2 instances, S3 buckets, Lambda functions, and IAM users. The resulting file can show information such as how many EC2 instances are active and how many buckets, functions, and users exist. The instructor limits the demonstration to selected services because covering every AWS resource would take considerably more time.

Q: What must be configured before running the AWS tracker?

The Linux environment needs Bash and an installed AWS CLI before the tracker can run. The AWS CLI must also be configured to authenticate with the AWS account. Running aws configure prompts for an access key, secret access key, default region, and output format, which can be JSON. Once this setup is complete, the script can communicate with AWS.

Q: Why does the script use /bin/bash instead of sh?

The script uses an explicit /bin/bash shebang because sh may only be a symbolic link. Depending on the Linux environment, that link can point to Bash or Dash. Bash and Dash have slight syntax differences, so a script written with Bash features might fail under Dash. Selecting /bin/bash directly makes the intended script interpreter clear.

Q: Is shell scripting the only way to track AWS resources?

Shell scripting is one way to build the resource tracker, but it is not presented as the only or optimal approach. The lesson says teams can use Lambda functions, Python scripts, AWS SDKs, or CDK-related approaches. The chosen solution combines shell scripting and the AWS CLI because those are the technologies introduced in the preceding classes.

Q: Why do organizations move infrastructure to the cloud?

The lesson identifies manageability and cost effectiveness as two primary reasons for moving to cloud infrastructure. Cloud use can reduce the burden of creating a data center, maintaining servers, applying security patches, and performing updates or upgrades. Its pay-as-you-use model also avoids purchasing physical infrastructure that must be paid for whether it is actively used or not.

Summary & Key Takeaways

• Cloud infrastructure can reduce server maintenance overhead and support pay-as-you-use billing, but organizations must still monitor resources they create. Unused EC2 instances or unattached EBS volumes can continue generating costs, making resource tracking an important responsibility for DevOps engineers and AWS administrators who support cost-effective cloud operations.

• The project combines Bash shell scripting with the AWS CLI to collect information about commonly used AWS resources. Its intended report can include active EC2 instances, S3 buckets, Lambda functions, and IAM users. In a production setting, the collected information would usually be supplied to a reporting dashboard rather than manually delivered.

• A Linux cron job can execute the resource-tracking script automatically at a specified time each day. Before writing the script, the environment needs Bash, an installed AWS CLI, and configured AWS credentials, region, and output format. The instructor recommends an explicit /bin/bash shebang to avoid Bash and Dash syntax differences.