Skip to main content

Exporting Workflow Event History to GCS

Prerequisites​

Before configuring the Export sink, please complete the following steps in Google Cloud.

  1. Create a GCS bucket and take note of its bucket name, for example, "test-export"
  • Enable customer-managed encryption keys (CMEK) if you need additional security for your GCS bucket.
  • For now we only support single region (or “Region” option when creating the bucket in GCS, not “Multi-Region” or “Dual-Region”)
  • The region of the bucket has to be the same as the region of your Temporal Cloud Namespace.
  1. Record the GCP Project ID that owns the bucket.
  2. Create a service account in the same project that grants Temporal permission to write to your GCS bucket.
  3. Follow the instructions in the Temporal Cloud UI, there are two ways to set up this service account:
    • Manual Setup:
      • Input the service account ID, GCP project ID and GCS bucket name.
      • Follow the instructions, manually set up a new service account.
    • Automated Setup:

Configure Workflow History Export​

You can use either the Temporal Cloud UI or tcld to configure the Workflow History Export.

Using Temporal Cloud UI​

The following steps guide you through setting up Workflow History Export using the Temporal Cloud UI.

  1. In the Cloud UI, navigate to the Namespaces section. Confirm that the Export feature is visible and properly displayed.
  2. Configure the Export sink for a Namespace:
    1. Choose GCS as the Sink type.
    2. Provide the following information:
      1. Name
      2. Service account ID
      3. GCP Project ID
      4. GCS bucket name
  3. After inputting the necessary values, click on Verify. You should be able to write to the sink successfully. If not, please fix any errors or reach out to support for help.
    • If you just created the GCS bucket and granted the permission for your service account, it may take some time for it to populate the permission, thus, you may need to wait for several minutes (up to 5 minutes) then you can click the Verify button to verify the connection.
  4. Clicking Create will complete the Export sink set up
  5. The page will auto-refresh and you should see the status “Enabled” on the Export screen. You are now ready to export Workflow histories.
  6. You can toggle the enable button if you want to stop export and resume in the future. Note: when you re-enable the feature, it will start from the current point in time, and not from the time when you disabled export.
  7. You can also delete export by clicking Delete.
tip

Don't forget to click Create at the end of your setup to confirm your export.

Using tcld​

To access export-related commands in tcld, please follow these steps:

  1. Download the latest version of tcld. [https://docs.temporal.io/cloud/tcld/#install-tcld]

  2. Make sure tcld version is v0.35.0 or above.

  3. Run the command: tcld n export gcs:

    NAME:
       tcld namespace export gcs - Manage GCS export sink

    USAGE:
       tcld namespace export gcs command [command options] [arguments...]

    COMMANDS:
       create, c    Create export sink
       update, u    Update export sink
       validate, v  Validate export sink
       get, g       Get export sink
       delete, d    Delete export sink
       list, l      List export sinks
       help, h      Shows a list of commands or help for one command

    OPTIONS:
       --help, -h  show help

  4. Run the tcld n export gcs create command and provide the following information:

    • —namespace
    • —sink-name
    • —service-account-email
    • —gcs-bucket
    • For Example:
    tcld n export gcs create -n test.ns --sink-name test-sink --service-account-email test-sink@test-export-sink.iam.gserviceaccount.com --gcs-bucket test-export-validation`

  5. Check the status of this command by either viewing the Namespace Export status in the Temporal Cloud UI or using the following command and looking for the state of “Active”.

tcld n export gcs g -n test.ns --sink-name test-sink
{
	"name": "test.ns",
	"resourceVersion": "b954de0c-c6ae-4dcc-90bd-3918b52c3f28",
	"state": "Active",
	"spec": {
		"name": "test-sink",
		"enabled": true,
		"destinationType": "Gcs",
		"s3Sink": null,
		"gcsSink": {
			"saId": "test-sink",
			"bucketName": "test-export-validationn",
			"gcpProjectId": "test-export-sink",
		}
	},
	"health": "Ok",
	"errorMessage": "",
	"latestDataExportTime": "0001-01-01T00:00:00Z",
	"lastHealthCheckTime": "2024-01-23T06:40:02Z"
}

Next Steps​