Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions .gcb/builds/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Terraform for Integration Test Resources

This directory contains Terraform configurations to set up the resources needed
for integration tests (Firestore, KMS, Buckets, etc.).

## Safe Usage for Personal Test Projects

By default, this configuration uses a remote GCS backend which tracks the state
of the shared `rust-sdk-testing` project. **Do not run `terraform apply`
directly with the remote backend if you are targeting a personal test project,
as it may interfere with the shared state or disable services in the shared
project.**
Comment on lines +9 to +12
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels like we should be able to prevent accidental deletion of / creation of things.

Maybe we can have some kind of sigil resource that must exist (and is created outside terraform) before anything runs? Or maybe something with the prevent_destroy flag?

Or maybe we just don't have permissions to delete these things, and only a robot can.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created #5462 to track that idea.


To safely use these Terraform scripts to set up resources in your personal test
project:

1. **Set your project variable**:

```bash
PROJECT=$(gcloud config get project)
```

1. **Override the backend to use local state**: Create a `backend_override.tf`
file. This tells Terraform to store the state locally on your machine instead
of the shared GCS bucket.

```bash
cat > backend_override.tf <<EOF
terraform {
backend "local" {
path = "${HOME}/${PROJECT}-rust-sdk.tfstate"
}
}
EOF
```

1. **Initialize Terraform**:

```bash
terraform init -reconfigure
```

1. **Plan and Apply**:

### Option A: For Local Testing (Recommended)

If you only need the resources for running tests locally (and do not need to
set up GCB triggers), target only the `services` and `resources` modules.
This avoids the need to create CI-specific service accounts:

```bash
terraform plan -var="project=${PROJECT}" -target=module.services -target=module.resources -out="/tmp/builds.plan"
terraform apply "/tmp/builds.plan"
```

## Known Issues

- **Services not enabled error**: If Terraform fails with an error that services
like Firestore or KMS are not enabled, you may need to enable them manually
using
`gcloud services enable firestore.googleapis.com cloudkms.googleapis.com` and
retry. This can happen because Terraform checks if the service is active
before the enablement has fully propagated in the Google Cloud backend.
109 changes: 15 additions & 94 deletions doc/contributor/howto-guide-set-up-development-environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,15 @@ defaults in your `settings.json` file:

```json
{
"rust-analyzer.cargo.buildScripts.overrideCommand": [
"cargo",
"check",
"--quiet",
"--profile=test",
"--message-format=json",
"--keep-going"
],
"rust-analyzer.check.workspace": false
"rust-analyzer.cargo.buildScripts.overrideCommand": [
"cargo",
"check",
"--quiet",
"--profile=test",
"--message-format=json",
"--keep-going"
],
"rust-analyzer.check.workspace": false
}
```

Expand Down Expand Up @@ -173,85 +173,13 @@ project.

### One time set up

We use [Secret Manager], [Workflows], [Firestore], [Speech-to-Text], and [KMS]
to run integration tests. Follow the [Enable the Secret Manager API] guide to,
as it says, enable the API and make sure that billing is enabled in your
projects. To enable the APIs you can run this command:
To run integration tests, you need to set up resources like Firestore, KMS keys,
and service accounts in a Google Cloud Project.

```bash
gcloud services enable workflows.googleapis.com firestore.googleapis.com speech.googleapis.com cloudkms.googleapis.com
```

Verify this is working with something like:

```bash
gcloud firestore databases list
gcloud secrets list
gcloud workflows list
```

It is fine if the list is empty, you just don't want an error.

### Create a service account

The integration tests need a service account (SA) in your project. This service
account is used to:

- Run tests that perform IAM operations, temporarily granting this service
account some permissions.
- Configure the service account used for test workflows.

For a test project, just create the SA using the CLI:

```bash
gcloud iam service-accounts create rust-sdk-test \
--display-name="Used in SA testing" \
--description="This SA gets assigned to roles on short-lived resources during integration tests"
```

For extra safety, disable the service account:

```bash
GOOGLE_CLOUD_PROJECT="$(gcloud config get project)"
gcloud iam service-accounts disable rust-sdk-test@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com
```

### Create a database

The integration tests need the default Firestore database in your project. You
can create this database using:

```bash
gcloud firestore databases create --location=us-central1
```

If the database already exists you should verify that this is a Firestore native
database:

```bash
gcloud firestore databases describe --format='value(type)'
# Expected output:
# FIRESTORE_NATIVE
```

### Create a KMS key ring and a crypto key

We use KMS keys with storage. The [Use customer-managed encryption keys] guide
covers how to create and configure a crypto key for use with Cloud Storage. We
recommend you name the key ring after its location, but feel free to use a
different naming convention:

```bash
GOOGLE_CLOUD_PROJECT="$(gcloud config get project)"
gcloud kms keyrings create us-central1 --location=us-central1
gcloud kms keys create storage-examples \
--keyring=us-central1 \
--location=us-central1 \
--purpose=encryption \
--rotation-period=10d --next-rotation-time=+p10d
gcloud storage service-agent --project=${GOOGLE_CLOUD_PROJECT} \
--authorize-cmek=projects/${GOOGLE_CLOUD_PROJECT}/locations/us-central1/keyRings/us-central1/cryptoKeys/storage-examples
```
We recommend using Terraform to automate this setup. See the instructions in
[.gcb/builds/README.md](../../.gcb/builds/README.md) for a safe way to set up
these resources in your personal test project without affecting the shared
project.

### Running tests

Expand Down Expand Up @@ -356,15 +284,8 @@ git ls-files -z --
xargs -0 terraform fmt
```

[enable the secret manager api]: https://cloud.google.com/secret-manager/docs/configuring-secret-manager
[firestore]: https://cloud.google.com/firestore/
[getting-started-rust]: https://www.rust-lang.org/learn/get-started
[golang-install]: https://go.dev/doc/install
[google cloud cli]: https://cloud.google.com/cli
[install terraform]: https://developer.hashicorp.com/terraform/install
[kms]: https://cloud.google.com/kms/
[mdbook]: https://rust-lang.github.io/mdBook/
[secret manager]: https://cloud.google.com/secret-manager/
[speech-to-text]: https://cloud.google.com/speech-to-text
[use customer-managed encryption keys]: https://cloud.google.com/storage/docs/encryption/using-customer-managed-keys
[workflows]: https://cloud.google.com/workflows/
Loading