LocalStack

A local AWS cloud stack for development and testing

🚀 About

In this HashiQube DevOps lab, you'll get hands-on experience with LocalStack and Terraform.

LocalStack provides an easy-to-use test/mocking framework for developing cloud applications. It spins up a testing environment on your local machine that provides the same functionality and APIs as the real AWS cloud environment, allowing you to develop and test your cloud applications without incurring AWS costs.

📋 Provision

bash docker/docker.sh
bash localstack/localstack.sh
bash terraform/terraform.shCopy to clipboardErrorCopied

vagrant up --provision-with basetools,localstack,terraformCopy to clipboardErrorCopied

docker compose exec hashiqube /bin/bash
bash hashiqube/basetools.sh
bash docker/docker.sh
bash docsify/docsify.sh
bash localstack/localstack.sh
bash terraform/terraform.shCopy to clipboardErrorCopied

🛠️ Using Terraform with LocalStack

Terraform Plan

To perform a Terraform plan:

1. Change directory to the LocalStack directory:

   cd /vagrant/localstackCopy to clipboardErrorCopied

2. Initialize Terraform and run the plan:

   terraform init
   terraform planCopy to clipboardErrorCopied

Terraform Apply

To apply your Terraform configuration:

1. Change directory to the LocalStack directory:

   cd /vagrant/localstackCopy to clipboardErrorCopied

2. Initialize Terraform, run a plan, and apply the changes:

   terraform init
   terraform plan
   terraform applyCopy to clipboardErrorCopied

🔄 Terraform and Vault Integration

Terraform has many providers, and you can use it to create resources across various platforms like GitHub, AWS, Azure, Cloudflare, and many others. You can also manage HashiCorp Vault with Terraform.

Setting Up Terraform with Vault

1. Get Terraform and LocalStack running:

   • GitHub Codespace: bash localstack/localstack.sh
   • Vagrant: vagrant up --provision-with basetools,localstack

2. Bring Vault up:

   • GitHub Codespace: bash vault/vault.sh
   • Vagrant: vagrant up --provision-with basetools,vault

3. Run Terraform plan with the Vault module enabled:

   VAULT_TOKEN="YOUR_VAULT_TOKEN" TF_VAR_vault_enabled=true terraform planCopy to clipboardErrorCopied

   Example output:

   # module.hashicorp-vault[0].vault_kv_secret_v2.example will be created
   + resource "vault_kv_secret_v2" "example" {
       + cas                 = 1
       + data                = (sensitive value)
       + data_json           = (sensitive value)
       + delete_all_versions = true
       + disable_read        = false
       + id                  = (known after apply)
       + metadata            = (known after apply)
       + mount               = "kvv2"
       + name                = "secret"
       + path                = (known after apply)
   
       + custom_metadata {
           + data         = {
               + "bar" = "12345"
               + "foo" = "vault@example.com"
             }
           + max_versions = 5
         }
     }
   
   # module.hashicorp-vault[0].vault_mount.kvv2 will be created
   + resource "vault_mount" "kvv2" {
       + accessor                     = (known after apply)
       + audit_non_hmac_request_keys  = (known after apply)
       + audit_non_hmac_response_keys = (known after apply)
       + default_lease_ttl_seconds    = (known after apply)
       + description                  = "KV Version 2 secret engine mount"
       + external_entropy_access      = false
       + id                           = (known after apply)
       + max_lease_ttl_seconds        = (known after apply)
       + options                      = {
           + "version" = "2"
         }
       + path                         = "kvv2"
       + seal_wrap                    = (known after apply)
       + type                         = "kv"
     }Copy to clipboardErrorCopied

4. Run Terraform apply with the Vault module enabled:

   VAULT_TOKEN="YOUR_VAULT_TOKEN" TF_VAR_vault_enabled=true terraform applyCopy to clipboardErrorCopied

   Example output:

   module.hashicorp-vault[0].vault_mount.kvv2: Creation complete after 1s [id=kvv2]
   module.hashicorp-vault[0].vault_kv_secret_v2.example: Creating...
   module.hashicorp-vault[0].vault_kv_secret_v2.example: Creation complete after 0s [id=kvv2/data/secret]Copy to clipboardErrorCopied

5. Access Vault to see the secret engine enabled:

KV2 secret engine enabled in Vault

Secrets stored in the KV2 secret engine

For further details, look at the code in /vagrant/localstack:

# modules.tf
module "hashicorp-vault" {
  source = "../modules/vault-kv-secret"
  count  = var.vault_enabled ? 1 : 0
}Copy to clipboardErrorCopied

🌐 LocalStack Web Interface

After running the provisioner, you can create an account at LocalStack Cloud to view your resources.

Creating a LocalStack Account

1. Go to https://www.localstack.cloud/ and register, or sign up directly at https://app.localstack.cloud/sign-up
2. You can register using SSO with GitHub credentials

Once logged in, you'll see the LocalStack Dashboard:

LocalStack Cloud dashboard

Viewing LocalStack Instances

Scroll down in the left-hand menu to see your running instances:

LocalStack running instances

If you click on S3, you'll see the bucket created by Terraform:

S3 service in LocalStack

S3 bucket details

💻 Running Terraform Locally

You can also run Terraform commands on your local machine:

1. Install Terraform on your laptop:

   • Follow the instructions at https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli

2. Verify Terraform is installed:

   terraform -versionCopy to clipboardErrorCopied

   Example output:

   Terraform v1.5.7
   on darwin_arm64
   + provider registry.terraform.io/hashicorp/aws v5.55.0
   + provider registry.terraform.io/hashicorp/null v3.2.2
   + provider registry.terraform.io/hashicorp/random v3.6.2
   
   Your version of Terraform is out of date! The latest version
   is 1.8.5. You can update by downloading from https://www.terraform.io/downloads.htmlCopy to clipboardErrorCopied

3. Navigate to the LocalStack directory:

   cd localstack
   pwdCopy to clipboardErrorCopied

   Output:

   /Users/riaan/workspace/personal/hashiqube/localstackCopy to clipboardErrorCopied

4. Initialize Terraform:

   terraform initCopy to clipboardErrorCopied

   Output:

   Initializing the backend...
   Initializing provider plugins...
   - Checking for available provider plugins...
   - Downloading plugin for provider "aws" (hashicorp/aws) 2.33.0...
   The following providers do not have any version constraints in configuration,
   so the latest version was installed.
   To prevent automatic upgrades to new major versions that may contain breaking
   changes, it is recommended to add version = "..." constraints to the
   corresponding provider blocks in configuration, with the constraint strings
   suggested below.
   * provider.aws: version = "~> 2.33"
   Terraform has been successfully initialized!
   You may now begin working with Terraform. Try running "terraform plan" to see
   any changes that are required for your infrastructure. All Terraform commands
   should now work.
   If you ever set or change modules or backend configuration for Terraform,
   rerun this command to reinitialize your working directory. If you forget, other
   commands will detect it and remind you to do so if necessary.Copy to clipboardErrorCopied

5. Run Terraform plan:

   terraform planCopy to clipboardErrorCopied

6. Apply the Terraform configuration:

   terraform applyCopy to clipboardErrorCopied

7. Verify resources in LocalStack:

   vagrant ssh -c "awslocal s3 ls"Copy to clipboardErrorCopied

   Output:

   2024-06-25 17:42:18 my-bucketCopy to clipboardErrorCopied

📚 Further Learning

Thanks to the folks at LocalStack for publishing examples for learning Terraform. You can explore more examples from their GitHub repository:

1. Clone the samples repository:

   git clone git@github.com:localstack-samples/localstack-terraform-samples.git
   cd localstack-terraform-samplesCopy to clipboardErrorCopied

2. Try the demo-deploy example:

   cd demo-deploy
   cp ../../provider.tf .Copy to clipboardErrorCopied

3. Initialize and apply the configuration:

   terraform init
   terraform plan
   terraform applyCopy to clipboardErrorCopied

🌐 Provision HashiQube on AWS, GCP, or Azure

HashiQube is a DevOps lab that runs all the HashiCorp products and popular Open Source Integrations. It can also help you learn Terraform!

Head over to the HashiQube Cloud Tutorial to learn more.

🔧 LocalStack Terraform Examples

Here are some example Terraform configuration files for LocalStack:

variables.tf

variable "vault_enabled" {
  description = "Enable the vault module"
  type        = bool
  default     = false
}Copy to clipboardErrorCopied

main.tf

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    apigateway     = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    es             = "http://localhost:4566"
    firehose       = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    route53        = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    s3             = "http://localhost:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"
  }
}

resource "aws_s3_bucket" "localstack-s3-bucket" {
  bucket = "localstack-s3-bucket"
  acl    = "public-read"
}Copy to clipboardErrorCopied

outputs.tf

output "s3_bucket_name" {
  value = aws_s3_bucket.localstack-s3-bucket.bucket
}Copy to clipboardErrorCopied

🔗 Additional Resources

• LocalStack Official Website
• LocalStack Terraform Samples
• LocalStack Pro Samples
• Terraform AWS Provider - Custom Service Endpoints
• Terraform Documentation
• LocalStack GitHub Repository
• AWS CLI Local

#!/bin/bash
# https://docs.localstack.cloud/get-started/

sudo usermod -aG docker vagrant
export PATH=$PATH:/root/.local/bin

arch=$(lscpu | grep "Architecture" | awk '{print $NF}')
if [[ $arch == x86_64* ]]; then
  ARCH="amd64"
elif  [[ $arch == aarch64 ]]; then
  ARCH="arm64"
fi
echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ CPU is $ARCH"
echo -e '\e[38;5;198m'"++++ "

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Ensure Docker Daemon is running (Dependency)"
echo -e '\e[38;5;198m'"++++ "
if pgrep -x "dockerd" >/dev/null
then
  echo -e '\e[38;5;198m'"++++ Docker is running"
else
  echo -e '\e[38;5;198m'"++++ Ensure Docker is running.."
  sudo bash /vagrant/docker/docker.sh
fi

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Ensure Terraform is installed (Dependency)"
echo -e '\e[38;5;198m'"++++ "
if [[ ! -f /usr/local/bin/terraform ]];
then
  echo -e '\e[38;5;198m'"++++ Terraform is not installed, installing"
  sudo bash /vagrant/terraform/terraform.sh
else
  echo -e '\e[38;5;198m'"++++ Terraform is installed"
fi

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Bring up Localstack"
echo -e '\e[38;5;198m'"++++ "
pip3 install --upgrade awscli-local --break-system-packages
sudo rm -rf awscliv2.zip
# https://aws.amazon.com/blogs/developer/aws-cli-v2-now-available-for-linux-arm/ aarch64
curl -s "https://awscli.amazonaws.com/awscli-exe-linux-${arch}.zip" -o "awscliv2.zip"
sudo rm -rf aws
sudo unzip -q awscliv2.zip
yes | sudo ./aws/install --update
echo -e '\e[38;5;198m'"aws --version"
aws --version
python3 -m pip install awscli-local --break-system-packages --quiet
python3 -m pip install flask-cors --break-system-packages --quiet
sudo -E docker stop localstack_main
yes | sudo docker system prune --volumes
sudo docker run --rm -it -d -p 4566:4566 -p 4571:4571 --rm --privileged --name localstack_main localstack/localstack
sudo docker ps | grep localstack

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Running Terraform Init, Plan and Apply in Localstack directory"
echo -e '\e[38;5;198m'"++++ "
cd /vagrant/localstack/
export PATH=$HOME/.local/bin:$PATH
echo -e '\e[38;5;198m'"++++ Removing previous Terraform state files.."
rm -rf ./terraform.tfstate*
echo -e '\e[38;5;198m'"++++ Terraform init.."
terraform init
echo -e '\e[38;5;198m'"++++ Terraform fmt.."
terraform fmt
echo -e '\e[38;5;198m'"++++ Terraform validate.."
terraform validate
echo -e '\e[38;5;198m'"++++ Terraform plan.."
terraform plan
echo -e '\e[38;5;198m'"++++ Terraform apply.."
terraform apply --auto-approve
echo -e '\e[38;5;198m'"++++ Awslocal s3 ls.."
awslocal s3 ls || true
# echo -e '\e[38;5;198m'"++++ Terraform destroy.."
# terraform destroy --auto-approve
Copy to clipboardErrorCopied

terraform {
  # The configuration for this backend will be filled in by Terragrunt or via a backend.hcl file. See
  # https://www.terraform.io/docs/backends/config.html#partial-configuration
  #  backend "s3" {}

  # Only allow this Terraform version. Note that if you upgrade to a newer version, Terraform won't allow you to use an
  # older version, so when you upgrade, you should upgrade everyone on your team and your CI servers all at once.
  required_version = "~> 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "~> 4.0"
    }
    consul = {
      source  = "hashicorp/consul"
      version = "~> 2.0"
    }
    nomad = {
      source  = "hashicorp/nomad"
      version = "~> 2.0"
    }
    boundary = {
      source  = "hashicorp/boundary"
      version = "~> 1.0"
    }
  }
}

provider "aws" {
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"
  region                      = "us-east-1"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true


  endpoints {
    acm                      = "http://localhost:4566"
    amplify                  = "http://localhost:4566"
    apigateway               = "http://localhost:4566"
    apigatewayv2             = "http://localhost:4566"
    appconfig                = "http://localhost:4566"
    applicationautoscaling   = "http://localhost:4566"
    appsync                  = "http://localhost:4566"
    athena                   = "http://localhost:4566"
    autoscaling              = "http://localhost:4566"
    backup                   = "http://localhost:4566"
    batch                    = "http://localhost:4566"
    cloudformation           = "http://localhost:4566"
    cloudfront               = "http://localhost:4566"
    cloudsearch              = "http://localhost:4566"
    cloudtrail               = "http://localhost:4566"
    cloudwatch               = "http://localhost:4566"
    cloudwatchlogs           = "http://localhost:4566"
    codecommit               = "http://localhost:4566"
    cognitoidentity          = "http://localhost:4566"
    cognitoidp               = "http://localhost:4566"
    config                   = "http://localhost:4566"
    costexplorer             = "http://localhost:4566"
    docdb                    = "http://localhost:4566"
    dynamodb                 = "http://localhost:4566"
    ec2                      = "http://localhost:4566"
    ecr                      = "http://localhost:4566"
    ecs                      = "http://localhost:4566"
    efs                      = "http://localhost:4566"
    eks                      = "http://localhost:4566"
    elasticache              = "http://localhost:4566"
    elasticbeanstalk         = "http://localhost:4566"
    elasticsearch            = "http://localhost:4566"
    elb                      = "http://localhost:4566"
    elbv2                    = "http://localhost:4566"
    emr                      = "http://localhost:4566"
    events                   = "http://localhost:4566"
    firehose                 = "http://localhost:4566"
    glacier                  = "http://localhost:4566"
    glue                     = "http://localhost:4566"
    iam                      = "http://localhost:4566"
    iot                      = "http://localhost:4566"
    iotanalytics             = "http://localhost:4566"
    iotevents                = "http://localhost:4566"
    kafka                    = "http://localhost:4566"
    kinesis                  = "http://localhost:4566"
    kinesisanalytics         = "http://localhost:4566"
    kinesisanalyticsv2       = "http://localhost:4566"
    kms                      = "http://localhost:4566"
    lakeformation            = "http://localhost:4566"
    lambda                   = "http://localhost:4566"
    mediaconvert             = "http://localhost:4566"
    mediastore               = "http://localhost:4566"
    neptune                  = "http://localhost:4566"
    organizations            = "http://localhost:4566"
    qldb                     = "http://localhost:4566"
    rds                      = "http://localhost:4566"
    redshift                 = "http://localhost:4566"
    redshiftdata             = "http://localhost:4566"
    resourcegroups           = "http://localhost:4566"
    resourcegroupstaggingapi = "http://localhost:4566"
    route53                  = "http://localhost:4566"
    route53resolver          = "http://localhost:4566"
    s3                       = "http://localhost:4566"
    s3control                = "http://localhost:4566"
    sagemaker                = "http://localhost:4566"
    secretsmanager           = "http://localhost:4566"
    serverlessrepo           = "http://localhost:4566"
    servicediscovery         = "http://localhost:4566"
    ses                      = "http://localhost:4566"
    sesv2                    = "http://localhost:4566"
    sns                      = "http://localhost:4566"
    sqs                      = "http://localhost:4566"
    ssm                      = "http://localhost:4566"
    stepfunctions            = "http://localhost:4566"
    sts                      = "http://localhost:4566"
    swf                      = "http://localhost:4566"
    timestreamwrite          = "http://localhost:4566"
    transfer                 = "http://localhost:4566"
    waf                      = "http://localhost:4566"
    wafv2                    = "http://localhost:4566"
    xray                     = "http://localhost:4566"
  }

  default_tags {
    tags = {
      Environment = "Local"
      Service     = "LocalStack"
    }
  }
}

# https://registry.terraform.io/providers/hashicorp/vault/latest/docs
provider "vault" {
  address = "http://127.0.0.1:8200"
  # # https://registry.terraform.io/providers/hashicorp/vault/latest/docs#example-auth_login-usage
  # auth_login {
  #   path = "auth/aws/login"
  #   method = "aws"
  #   parameters = {
  #     role = "dev-role-iam"
  #   }
  # }
}

# https://registry.terraform.io/providers/hashicorp/consul/latest/docs
provider "consul" {
  address    = "http://127.0.0.1:8500"
  datacenter = "dc1"
}

# https://registry.terraform.io/providers/hashicorp/nomad/latest/docs
provider "nomad" {
  address = "http://127.0.0.1:4646"
  region  = ""
}

# https://registry.terraform.io/providers/hashicorp/boundary/latest/docs
provider "boundary" {
  addr                            = "http://127.0.0.1:19200"
  password_auth_method_login_name = "admin"
  password_auth_method_password   = "password"
}
Copy to clipboardErrorCopied