Skip to main content

Google Cloud Service Account Guide for NoETL

This comprehensive guide covers everything you need to know about working with Google Cloud service accounts for NoETL, including:

  1. Creating a service account
  2. Granting necessary permissions
  3. Entering service account impersonation
  4. Exiting service account impersonation
  5. Best practices

Creating a Google Cloud Service Account​

This section explains how to create a Google Cloud service account specifically for use with NoETL.

The instructions create a service account with the name noetl-service-account in the project noetl-demo-19700101.

Prerequisites​

  1. Google Cloud SDK (gcloud) installed on the machine
  2. Permissions to create service accounts in the Google Cloud project
  3. The project noetl-demo-19700101 already created in Google Cloud

Step 1: Set the Active Project​

List all authenticated accounts

gcloud auth list

Authenticate with google account

gcloud auth login [email protected] --force

Set the active account

gcloud config set account [email protected]

Set active Google Cloud project:

gcloud config set project noetl-demo-19700101

Verify that the project is set correctly:

gcloud config get-value project

Update application default credentials

gcloud auth application-default login

Verify the active account

gcloud config get-value account

Step 2: Create the Service Account​

Create the service account with the specified name:

gcloud iam service-accounts create noetl-service-account \
  --project=noetl-demo-19700101 \
  --display-name="NoETL Service Account"

This creates a service account with the email address: noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com

Step 3: Grant User Permission to Impersonate the Service Account​

To impersonate the service account from the user account, run:

gcloud iam service-accounts add-iam-policy-binding noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com \
  --member="user:[email protected]" \
  --role="roles/iam.serviceAccountTokenCreator"

Step 4: Create a Service Account Key​

Create a key file for the service account:

mkdir -p .secrets
gcloud iam service-accounts keys create .secrets/noetl-service-account.json \
  --iam-account=noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com

Important: Keep this key file secure and never commit it to source control.

Step 5: Create HMAC Keys for Google Cloud Storage​

Create HMAC keys for S3-compatible access to Google Cloud Storage:

gcloud storage hmac create noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com \
  --project=noetl-demo-19700101

This will output an Access Key ID and Secret that will be used for S3-compatible access.

Step 6: Using the Service Account with NoETL​

Option 1: Set Environment Variables​

export GOOGLE_APPLICATION_CREDENTIALS="$(pwd)/.secrets/noetl-service-account.json"
export GOOGLE_CLOUD_PROJECT="noetl-demo-19700101"

Verification​

To verify that the service account was created successfully:

gcloud iam service-accounts describe noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com

To list all service accounts in the project:

gcloud iam service-accounts list --project=noetl-demo-19700101

To check if the service account key file is valid and contains the expected type:

cat $GOOGLE_APPLICATION_CREDENTIALS | jq -r '.type == "service_account"'

Granting Required Permissions to Service Accounts​

This section explains how to grant the necessary permissions to a service account for use with NoETL, focusing on Google Cloud Storage and Secret Manager access.

Required Permissions​

Based on common error messages, the service account needs the following permissions:

  1. For Google Cloud Storage access:

    • storage.objects.list - To list objects in a bucket
    • storage.buckets.get - To access bucket metadata

  2. For Secret Manager access:

    • secretmanager.secrets.list - To list secrets in a project
    • secretmanager.secrets.get - To get secret metadata
    • secretmanager.versions.access - To access secret versions

Granting the Required Roles​

The easiest way to grant these permissions is by assigning predefined IAM roles that include these permissions.

For Google Cloud Storage​

To grant full access to Google Cloud Storage:

gcloud projects add-iam-policy-binding noetl-demo-19700101 \
  --member="serviceAccount:noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com" \
  --role="roles/storage.admin"

For read-only access to objects:

gcloud projects add-iam-policy-binding noetl-demo-19700101 \
  --member="serviceAccount:noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

For Secret Manager​

To grant permission to list and access secrets:

gcloud projects add-iam-policy-binding noetl-demo-19700101 \
  --member="serviceAccount:noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

To grant full access to Secret Manager:

gcloud projects add-iam-policy-binding noetl-demo-19700101 \
  --member="serviceAccount:noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com" \
  --role="roles/secretmanager.admin"

Example for NoETL Service Account​

For the NoETL service account, run:

gcloud projects add-iam-policy-binding noetl-demo-19700101 \
  --member="serviceAccount:noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com" \
  --role="roles/storage.admin"

gcloud projects add-iam-policy-binding noetl-demo-19700101 \
  --member="serviceAccount:noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com" \
  --role="roles/secretmanager.admin"

Verifying Permissions​

After granting the roles, verify the permissions:

gcloud projects get-iam-policy noetl-demo-19700101 \
  --flatten="bindings[].members" \
  --format="table(bindings.role,bindings.members)" \
  --filter="bindings.role:storage"

gcloud projects get-iam-policy noetl-demo-19700101 \
  --flatten="bindings[].members" \
  --format="table(bindings.role,bindings.members)" \
  --filter="bindings.role:secretmanager"

Entering Service Account Impersonation​

This section explains how to enter Google Cloud service account impersonation mode, allowing it to act as a service account without needing its key file.

Why Use Service Account Impersonation?​

Service account impersonation allows:

  1. Temporarily assume a service account's identity without managing key files
  2. Test permissions and access before deploying applications
  3. Perform administrative tasks that require service account permissions
  4. Follow the principle of least privilege by using temporary credentials

Prerequisites​

Before you can impersonate a service account, ensure:

  1. Google Cloud user account exists with sufficient permissions
  2. The service account exists in Google Cloud project
  3. User account has been granted the roles/iam.serviceAccountTokenCreator role for the service account
  4. The service account has all the necessary permissions for the operations are intended to perform

Methods for Entering Impersonation Mode​

There are two main methods to enter service account impersonation mode:

This method allows you to impersonate a service account without needing its key file:

gcloud auth print-access-token --impersonate-service-account=SERVICE_ACCOUNT_EMAIL

For a more persistent impersonation session:

gcloud config set auth/impersonate_service_account SERVICE_ACCOUNT_EMAIL

To verify the impersonation is active:

gcloud config get-value account

This should show the service account email.

Method 2: Using a Service Account Key File​

This method requires a service account key file:

gcloud auth activate-service-account --key-file=PATH_TO_KEY_FILE

For example:

gcloud auth activate-service-account --key-file=.secrets/noetl-service-account.json

Setting Application Default Credentials (ADC)​

For applications that use Application Default Credentials:

With Impersonation​

export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=SERVICE_ACCOUNT_EMAIL

With Key File​

export GOOGLE_APPLICATION_CREDENTIALS="$(pwd)/PATH_TO_KEY_FILE"

Example: Testing Service Account Permissions​

Here's an example of how to impersonate a service account to test its permissions:

  1. Enter impersonation mode:

    gcloud config set auth/impersonate_service_account noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com

  2. Test access to a Google Cloud Storage bucket:

    gsutil ls gs://noetl-samples/

  3. Test access to Secret Manager:

    gcloud secrets list --project=noetl-demo-19700101

Exiting Service Account Impersonation​

This section explains how to exit from Google Cloud service account impersonation and return to your regular user account.

Why Exit from Impersonation?​

When you're impersonating a service account, you're acting as that service account. However, some operations, such as granting permissions to a service account, require to be authenticated as a user with sufficient permissions, and not the service account itself. In these cases, exit impersonation and return to your regular user account.

If you encounter permission errors while using an impersonated service account, such as "Permission denied" for Google Cloud Storage or Secret Manager operations, exit impersonation, grant the necessary permissions to the service account, and then re-enter impersonation.

Steps to Exit from Service Account Impersonation​

1. Check Your Current Active Account​

First, check which account is currently in use:

gcloud config get-value account

If this shows a service account email (ending with .iam.gserviceaccount.com), it is an impersonated service account.

2. Exit from Impersonation and Return to User Account​

To exit from impersonation and return to regular user account, run:

gcloud auth login

This will open a browser window to authenticate with regular Google account.

3. Verify the Switch​

After logging in, verify that a regular user account is in use:

gcloud config get-value account

This should now show the regular user email address.

4. For Application Default Credentials (ADC)​

If the Application Default Credentials (ADC) are in use in your application, it is required to reset those:

gcloud auth application-default login

This will set up ADC for the user account.

Example: Granting Permissions to a Service Account​

Here's an example of how to exit from impersonation to grant permissions to a service account:

  1. Exit from impersonation:

    gcloud auth login

  2. Grant permissions to the service account:

    gcloud projects add-iam-policy-binding noetl-demo-19700101 \
      --member="serviceAccount:noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com" \
      --role="roles/secretmanager.secretAccessor"

  3. Verify the permissions were granted:

    gcloud projects get-iam-policy noetl-demo-19700101 \
      --flatten="bindings[].members" \
      --format="table(bindings.role,bindings.members)" \
      --filter="bindings.members:noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com"

Important Note About Impersonation​

When using service account impersonation, you need to ensure:

  1. User account has the roles/iam.serviceAccountTokenCreator role for the service account
  2. The service account itself has the necessary permissions for the operations

If you're still encountering permission issues after granting the roles, you may need to exit impersonation mode, grant the permissions, and then re-enter impersonation mode:

gcloud config unset auth/impersonate_service_account

gcloud config set auth/impersonate_service_account noetl-service-account@noetl-demo-19700101.iam.gserviceaccount.com

Best Practices​

Security Best Practices​

  1. Never commit service account keys to source control
  2. Use impersonation over key files when possible for better security
  3. Limit the duration of your impersonation sessions
  4. Verify which account you're using before performing sensitive operations
  5. Never share impersonation credentials with unauthorized users

Permission Best Practices​

  1. Least Privilege: Grant only the permissions that are only necessary
  2. Ensure the service account has the necessary permissions before attempting to use it
  3. Return to your regular user account for administrative tasks like granting permissions

Operational Best Practices​

  1. Key Rotation: Regularly rotate service account keys
  2. Monitoring: Set up monitoring and auditing for service account usage
  3. Keep your authentication contexts separate to avoid confusion and potential security issues
  4. Always be aware of which account you're currently using, especially when performing sensitive operations
  5. Use service accounts only for the specific tasks they're designed for