Skip to main content

Snapshot Storage and Fetching

This page explains where snapshot files come from, how the Go SDK fetches them, and how local versus S3-backed snapshot paths behave today.

Fetcher Model​

The client-level entry point is:

  • Client.SnapshotFetcher()

If you do not inject a custom fetcher, the SDK builds a default fetcher from client.Config.SnapshotStorage.

Default Scheme Support​

Today the built-in fetcher supports:

  • s3://...
  • file://...
  • plain local filesystem paths with no URI scheme

That behavior comes from internal/snapshot/fetcher.go.

Local Temporary Directory Behavior​

The fetcher always materializes the snapshot files into a local temporary directory first:

localDir, err := os.MkdirTemp("", "fluss-kv-snapshot-*")

That local directory is then passed to the read-only snapshot reader and removed afterward.

So even when snapshot files are remote, the current flow is:

  • remote fetch
  • local temp directory
  • local read-only snapshot iteration

Local File Path Behavior​

If Fluss snapshot metadata points at local files, the built-in fetcher can work with:

  • file:///absolute/path/to/file
  • plain filesystem paths

In that case the SDK copies those files into its own temporary directory before opening the snapshot reader.

S3-Compatible Storage Behavior​

When SnapshotStorageConfig is supplied, the SDK enables S3 fetching. The important fields are:

  • S3Endpoint
  • S3AccessKey
  • S3SecretKey
  • S3Region
  • S3UseSSL
  • S3PathStyle
  • S3BucketOverride

The canonical E2E harness configures:

SnapshotStorage: client.SnapshotStorageConfig{
	S3Endpoint:  "http://rustfs:9000",
	S3AccessKey: "flussadmin",
	S3SecretKey: "flussadmin",
	S3Region:    "us-east-1",
	S3UseSSL:    false,
	S3PathStyle: true,
},

The built-in S3 fetcher normalizes bare endpoints like host:port into http:// or https:// as needed, then downloads objects using the AWS SDK for Go v2.

Local Default vs Explicit S3 Configuration​

This distinction matters:

  • If you do not configure S3 and the metadata points at local files, the SDK can still fetch snapshots through the local-file path.
  • If the metadata points at s3://... and you have not configured S3 access correctly, snapshot fetching will fail.
  • The canonical demo/fluss-paimon run deliberately configures explicit S3-compatible storage so the real support contract validates the remote-object-store path rather than silently relying on local files.

What the Docs Should Promise​

The docs should be explicit:

  • local file:// and plain-path fetch is supported by the built-in default fetcher
  • S3-compatible fetching is supported when you configure SnapshotStorageConfig
  • the strongest real validation today is the RustFS-backed demo path
  • the feature is not yet documented as universal proof across every possible object store, warehouse layout, or cluster environment