Skip to main content

Snapshots and Lakehouse

This is one of the most important areas to document carefully, because it is real and working but still narrower than a general production-ready story.

What Is Supported Today​

The Go SDK currently supports:

  • KV snapshot metadata calls
  • lake snapshot metadata calls
  • current public SnapshotScanRows(...) flow
  • validation against the canonical Fluss + Paimon demo environment

Snapshot Scan​

The current public helper is:

  • TableClient.SnapshotScanRows(...)

Real example:

schema := client.NewSchema(
	client.Int64Type(),
	client.StringType(),
	client.StringType(),
)

rows, err := cli.Table(env.kvTable).SnapshotScanRows(ctx, schema, client.SnapshotScanOptions{
	BucketID:   0,
	SnapshotID: &snapshotID,
})
if err != nil {
	log.Fatal(err)
}

This is already exercised by the E2E harness, but the docs should frame it honestly:

  • it is validated on the current RustFS/Paimon demo path
  • it already supports schema-id-aware remapping for historical snapshot rows
  • broader portability across more filesystem backends and cluster layouts still deserves more validation
  • it materializes snapshot files into a local temporary directory before local read-only iteration

How the Current Implementation Works​

At a high level:

  1. resolve the latest snapshot ID if the caller does not provide one
  2. request snapshot metadata for the chosen bucket
  3. fetch the listed remote files into a local temporary directory
  4. open that directory through the read-only snapshot reader
  5. decode rows back into the requested Go schema

The current local reader is backed by rockyardkv over the fetched local snapshot directory.

Continue to:

Lake Snapshot Metadata​

The admin surface currently includes:

  • GetLatestLakeSnapshot(...)
  • GetLakeSnapshot(...)
  • GetReadableLakeSnapshot(...)

The E2E harness uses these calls as part of the validated support contract.

How To Position This​

These features should be documented as:

  • implemented
  • real-cluster validated in the canonical demo path
  • still not a blanket claim of mature lakehouse portability or a separate Paimon-native reader API