Use Cloider buckets with the Terraform AWS provider

View as Markdown Open in Claude Open in ChatGPT

A cloider_bucket speaks the S3 API, so the HashiCorp aws provider, the AWS CLI, every AWS SDK and rclone work with it. The Cloider provider creates the bucket and its access key. This guide shows how to then upload files from Terraform with aws_s3_object, and how to hand the same four settings to the AWS CLI, SDKs, rclone and workloads in a cluster.

At a glance

Every S3 client needs four settings:

Setting	Value	From Terraform
Endpoint	The region's S3 endpoint, such as `https://s3.eu-central-2.cloider.com`	`cloider_bucket.<name>.endpoint`
Region	The bucket's region code, such as `eu-central-2`	`cloider_bucket.<name>.region`
Access key	The key's access key id	`cloider_bucket_key.<name>.access_key`
Secret key	The key's secret	`cloider_bucket_key.<name>.secret_key` (sensitive)

The regions and their endpoints are also listed by cloider_bucket_regions. See S3 compatibility for what the S3 API supports.

Step 1: Create a bucket and a key

storage/main.tf
provider "cloider" {}

resource "cloider_bucket" "assets" {
  name   = "acme-assets"
  public = true
}

resource "cloider_bucket_key" "deploy" {
  label   = "assets deploy"
  buckets = [cloider_bucket.assets.id]
}

storage/outputs.tf
output "s3_endpoint" {
  value = cloider_bucket.assets.endpoint
}

output "s3_region" {
  value = cloider_bucket.assets.region
}

output "s3_access_key" {
  value = cloider_bucket_key.deploy.access_key
}

output "s3_secret_key" {
  value     = cloider_bucket_key.deploy.secret_key
  sensitive = true
}

cd storage
terraform init && terraform apply

The secret key is only shown once

Cloider shows a key's secret only when the key is created. Terraform keeps it in state, so terraform output -raw s3_secret_key keeps working. If the state is lost, create a new key.

Step 2: Upload files with the aws provider

Keep the files in a separate configuration from the one that creates the key, with the key in that configuration's environment. A provider can't be configured from a key created in the same plan, and keeping them apart means a change to the site never touches the bucket.

export AWS_ACCESS_KEY_ID="$(terraform -chdir=storage output -raw s3_access_key)"
export AWS_SECRET_ACCESS_KEY="$(terraform -chdir=storage output -raw s3_secret_key)"

site/main.tf
terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

# The skip_ settings stop the provider from calling AWS's own account and
# metadata APIs, which a Cloider key can't reach
provider "aws" {
  region = "eu-central-2"

  skip_credentials_validation = true
  skip_requesting_account_id  = true
  skip_metadata_api_check     = true

  endpoints {
    s3 = "https://s3.eu-central-2.cloider.com"
  }
}

# One file with inline content
resource "aws_s3_object" "robots" {
  bucket       = "acme-assets"
  key          = "robots.txt"
  content      = "User-agent: *\nDisallow:\n"
  content_type = "text/plain"
}

# Every file under ./site, uploaded again when it changes
locals {
  content_types = {
    html = "text/html"
    css  = "text/css"
    js   = "text/javascript"
    png  = "image/png"
    svg  = "image/svg+xml"
  }
}

resource "aws_s3_object" "site" {
  for_each = fileset("${path.module}/site", "**")

  bucket       = "acme-assets"
  key          = each.value
  source       = "${path.module}/site/${each.value}"
  source_hash  = filemd5("${path.module}/site/${each.value}")
  content_type = lookup(local.content_types, reverse(split(".", each.value))[0], "application/octet-stream")
}

cd site
terraform init && terraform apply

Because the bucket is public, each file is then readable at https://acme-assets.s3.eu-central-2.cloider.com/<key>, for example https://acme-assets.s3.eu-central-2.cloider.com/robots.txt. Listing the bucket and changing files still need a key.

Large sites

Terraform tracks every uploaded file in state, which gets slow with thousands of files. For large or frequently changing sites, sync with the AWS CLI or rclone from CI instead, see below and Static assets on object storage.

Use the key with the AWS CLI

Create a cloider profile from the outputs:

aws configure set aws_access_key_id "$(terraform output -raw s3_access_key)" --profile cloider
aws configure set aws_secret_access_key "$(terraform output -raw s3_secret_key)" --profile cloider
aws configure set region eu-central-2 --profile cloider
aws configure set endpoint_url https://s3.eu-central-2.cloider.com --profile cloider

aws s3 cp ./photo.jpg s3://acme-assets/ --profile cloider
aws s3 ls s3://acme-assets --profile cloider

Use the key with AWS SDKs and other clients

The AWS SDKs read these environment variables, AWS_ENDPOINT_URL_S3 included:

export AWS_ACCESS_KEY_ID="$(terraform output -raw s3_access_key)"
export AWS_SECRET_ACCESS_KEY="$(terraform output -raw s3_secret_key)"
export AWS_REGION=eu-central-2
export AWS_ENDPOINT_URL_S3=https://s3.eu-central-2.cloider.com

See SDKs for code in JavaScript, Python, Go and other languages.

Use the key with rclone

~/.config/rclone/rclone.conf
[cloider]
type = s3
provider = Other
access_key_id = <access key>
secret_access_key = <secret key>
endpoint = https://s3.eu-central-2.cloider.com
region = eu-central-2

rclone sync ./public cloider:acme-assets

Or have Terraform write the file with the local provider, from the same outputs. More tools are on S3 tools.

Hand the key to a workload in a cluster

With the kubernetes provider configured from a cloider_cluster, store the settings in a Secret and load it with envFrom:

main.tf
resource "kubernetes_secret_v1" "s3" {
  metadata {
    name      = "s3-credentials"
    namespace = "default"
  }

  data = {
    AWS_ACCESS_KEY_ID     = cloider_bucket_key.deploy.access_key
    AWS_SECRET_ACCESS_KEY = cloider_bucket_key.deploy.secret_key
    AWS_REGION            = cloider_bucket.assets.region
    AWS_ENDPOINT_URL_S3   = cloider_bucket.assets.endpoint
  }
}

See Use buckets with apps and clusters.

Store Terraform's own state in a bucket

The same kind of bucket and key can hold Terraform state, with the s3 backend. See Store state in a bucket.

Troubleshooting

Problem	Cause	Fix
The aws provider tries to reach `sts.amazonaws.com` or the metadata service	The `skip_` settings are missing	Add `skip_credentials_validation`, `skip_requesting_account_id` and `skip_metadata_api_check`
`NoSuchBucket`	Wrong bucket name, or `endpoints.s3` missing so requests go to AWS	Set `endpoints { s3 = ... }` to the bucket's `endpoint`
`AccessDenied` when uploading	The key is `read_only`, or doesn't list this bucket	Use a read and write key whose `buckets` includes the bucket, or `all_buckets = true`
`s3_secret_key` output is null	The key was imported, not created by Terraform	Create a new key with Terraform

Next steps

Frequently asked questions

Can I use the Terraform AWS provider with Cloider object storage?

Yes, for S3 objects. Configure the aws provider with endpoints { s3 = "https://s3.eu-central-2.cloider.com" }, the bucket's region, a Cloider bucket key in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and the skip_credentials_validation, skip_requesting_account_id and skip_metadata_api_check settings.

Should I create the Cloider bucket with the AWS provider?

No. Create buckets and keys with the Cloider provider's cloider_bucket and cloider_bucket_key, which also set the region, visibility and versioning. Use the aws provider only for objects in the bucket.

What S3 endpoint does a Cloider bucket use?

The endpoint of its region, such as https://s3.eu-central-2.cloider.com, available as the cloider_bucket resource's endpoint attribute.

At a glance​

Step 1: Create a bucket and a key​

Step 2: Upload files with the aws provider​

Use the key with the AWS CLI​

Use the key with AWS SDKs and other clients​

Use the key with rclone​

Hand the key to a workload in a cluster​

Store Terraform's own state in a bucket​

Troubleshooting​

Next steps​