Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Cloud connections

The [cloud] settings and [[cloud.connections]] tables decide which stores the home screen lists and how each one logs in. For a guide to logging in, see Connect to cloud storage. Named lists of datasets are catalogs. Settings lists every [cloud] key and its default.

[cloud]
env_files = [".env"]
discover = ["s3", "gcs"]
list_on_start = false

An S3-compatible endpoint, its keys and its region come from AWS_* variables or a connection (below), never from keys in this file.

Connections

Add one [[cloud.connections]] table per account or endpoint. Each appears as a row under CLOUD, and a dataset in a catalog can name one with connection. Replace <ENDPOINT> with your server’s URL and <BUCKET> with a bucket the keys can read. The keys come from the environment variables the entry names:

[[cloud.connections]]
name = "onprem"
label = "On-prem MinIO"
kind = "s3"
endpoint_url = "<ENDPOINT>"
region = "us-east-1"
addressing = "path"
access_key_id_env = "ONPREM_KEY"
secret_access_key_env = "ONPREM_SECRET"
buckets = ["<BUCKET>"]
FieldKindsMeaning
nameallRequired. Lowercase letters, digits and -, at most 40 characters. Used in s3://<name>@bucket/key
labelallShown instead of the name
kindallRequired. s3, gcs or azure
bucketss3, gcsBucket names to show when the keys can read but not list
endpoint_urls3An S3-compatible server. Without it, the source is AWS
regions3Region to sign for
addressings3path or virtual. Default: path with an endpoint, virtual without
access_key_id_env, secret_access_key_env, session_token_envs3Names of the environment variables holding the keys
profiles3An AWS profile to take the keys, endpoint and region from, instead of the *_env keys
accountazureRequired, except with connection_string_env. The storage account
account_key_env, sas_env, connection_string_envazureThe environment variable holding the account key, a SAS token, or a connection string. Set at most one. With none, az or Azure PowerShell signs in
secret_commands3, azureA program that prints the secret: the S3 secret access key (with access_key_id_env), or the Azure account key (with account). Use it instead of secret_access_key_env or account_key_env
credentials_filegcsA service account key or application-default login file, absolute or under ~. Its project is listed first
configurationgcsA gcloud configuration whose login to use. Without it, the application-default login
projectgcsThe project listed first, and the one listed when projects cannot be searched

Secrets from files and commands

A password manager or vault can supply a secret without it touching the config or the environment. Replace <ENDPOINT> with your server, <SECRET_COMMAND> with the command that prints the secret (pass show minio/onprem, op read op://vault/minio/secret) and <KEY_FILE> with the path of a service account key:

[[cloud.connections]]
name = "onprem"
kind = "s3"
endpoint_url = "<ENDPOINT>"
access_key_id_env = "ONPREM_KEY"
secret_command = "<SECRET_COMMAND>"

[[cloud.connections]]
name = "analytics"
kind = "gcs"
credentials_file = "<KEY_FILE>"

secret_command splits the command into arguments as a shell would, but runs the program directly with no shell, so |, $VAR and globs have no special meaning. It runs once, the first time the source is used, with a 30-second limit. What it prints is kept in memory for the session and never written, logged or shown. When it fails, only its error output is reported. On Windows, a .cmd or .bat wrapper works.

env_files reads variables from files such as a project’s .env. A path is relative to the directory datui starts in, or can start with ~. Only cloud variables are read: the AWS_*, GOOGLE_* and AZURE_* ones datui uses, MC_HOST_<alias>, and the names [[cloud.connections]] entries point at with *_env. Anything else in the file, such as a database password, is ignored. A variable already set in the environment wins. Nothing is exported, so programs datui starts do not see these variables. No .env file is read unless it is listed in env_files.

The home screen says what discover and list_on_start change there. -c cloud.discover=none overrides discover for one run.

instance_identity = true lets datui ask the cloud VM it runs on for credentials: an EC2 instance role, a GCE service account, an Azure VM’s managed identity. It is off by default because outside those VMs the metadata request waits for a timeout. Cloud Run and Cloud Functions, and Azure App Service, Functions and Container Apps, set variables that show an identity is available (K_SERVICE, IDENTITY_ENDPOINT, MSI_ENDPOINT), so datui uses those identities without the setting.

A secret written directly into a source (secret_access_key = "...") is refused, and so is any key datui does not recognize; the error names the key. A variable that is named but not set is reported when the source is used. The source never falls back to other keys in the environment.

Detected sources

Sources appear when datui finds a supported login or an explicit configuration. Listing a bucket does not guarantee permission to read every object inside it.

SourceIDAppears when
Amazon S3, or the AWS_ENDPOINT_URL endpoints3-defaultAWS_ACCESS_KEY_ID, an ECS or Fargate task role, an EKS web identity, AWS_PROFILE, or a ~/.aws directory
Each other AWS profile that can log inaws-<profile>Keys, credential_process, SSO or a role in the profile
Each MinIO client aliasmc-<alias>An alias with keys in mc’s config.json (~/.mc/, ~/.mcli/, or MC_CONFIG_DIR), or MC_HOST_<alias> in the environment, which wins
s3cmd’s servers3cfgKeys in the [default] section of ~/.s3cfg (%APPDATA%\s3cmd.ini on Windows, or S3CMD_CONFIG)
AzureazThe Azure CLI has been used (~/.azure, or AZURE_CONFIG_DIR), or Azure PowerShell signed in (~/.Azure/AzureRmContext.json). Its rows are storage accounts, found across your subscriptions. With az on PATH or the Az.Accounts module installed and neither signed in, the row says not signed in
Azure from the environmentazure-envAZURE_STORAGE_CONNECTION_STRING, AZURE_STORAGE_ACCOUNT_NAME with a key or SAS token, or a service principal (AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET or AZURE_FEDERATED_TOKEN_FILE)
Google Cloudgcs-defaultGOOGLE_SERVICE_ACCOUNT, GOOGLE_SERVICE_ACCOUNT_PATH, GOOGLE_SERVICE_ACCOUNT_KEY, GOOGLE_APPLICATION_CREDENTIALS, the file written by gcloud auth application-default login, or else the active gcloud configuration’s login. Its rows are projects
Each other gcloud configuration with a different accountgcloud-<configuration>An account in configurations/config_<name> under ~/.config/gcloud (%APPDATA%\gcloud on Windows, or CLOUDSDK_CONFIG)
Each [[cloud.connections]] entryits nameAlways

To show only some kinds of login found on the machine, or none:

[cloud] discover-c cloud.discover=Shows
unset, true or "all"allEvery login found
false or "none"noneNone
["gcs"], "s3,azure"gcs, s3,azureThose kinds. s3 covers AWS profiles, mc, s3cmd, and s3-default with its keys from AWS_*

[[cloud.connections]] entries appear whatever discover says.

A connection in the config with the same name as a detected source replaces it. When the same server and key are found in several places, they share one row, and its note lists every place. mc’s placeholder aliases and its public play server are left out. Connect to cloud storage has worked examples of [[cloud.connections]].

Google Cloud lists every project the login can find. The project named in the environment or the active gcloud configuration comes first, and is the only one listed when searching for projects is refused. A profile that needs the AWS CLI shows needs the AWS CLI when it is not installed, and an expired SSO login shows the CLI’s message; see AWS profiles. A cloud VM’s identity is discovered only with [cloud] instance_identity = true (above).