From eff30ded4f6cd00e26b3bf4452285833f0a33d66 Mon Sep 17 00:00:00 2001 From: ejseqera Date: Tue, 19 May 2026 17:28:37 -0400 Subject: [PATCH 1/5] docs(studios): add S3 versioning guidance for checkpoint storage costs Studios writes a checkpoint every five minutes to the same S3 key. When S3 versioning is enabled on the work bucket, each write creates a new object version rather than an overwrite, producing up to 96 non-current versions per day per active session. Add a new subsection under "Studio session checkpoints" that: - Explains the versioning interaction and its cost implications - Recommends an S3 Lifecycle rule (NoncurrentVersionExpiration: 1 day) with a ready-to-use JSON policy block and aws s3api CLI command - Provides a bulk-delete shell command for clearing existing accumulated non-current versions - Clarifies that non-current versions are safe to delete, while the current version and checkpoint directories must not be removed Changes applied to both platform-cloud and platform-enterprise docs. --- platform-cloud/docs/studios/managing.md | 51 ++++++++++++++++++++ platform-enterprise_docs/studios/managing.md | 51 ++++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/platform-cloud/docs/studios/managing.md b/platform-cloud/docs/studios/managing.md index a9cd8fbb06..54699c81de 100644 --- a/platform-cloud/docs/studios/managing.md +++ b/platform-cloud/docs/studios/managing.md @@ -186,6 +186,57 @@ When starting a Studio session, a *checkpoint* is automatically created. A check Checkpoints vary in size depending on libraries installed in your session environment. This can potentially result in many large files stored in the compute environment's pipeline work directory and saved to cloud storage. This storage will incur costs based on the cloud provider. Due to the architecture of Studios, you cannot delete any checkpoint files to save on storage costs. Deleting a Studio session's checkpoints will result in a corrupted Studio session that cannot be started nor recovered. ::: +### S3 versioning and checkpoint storage costs + +If your compute environment work directory uses an S3 bucket with **versioning enabled**, checkpoint writes create a new S3 object version every five minutes rather than overwriting the previous one. For an active Studio session, this produces up to 96 new object versions per day per session. Over time, these non-current versions accumulate and can significantly increase storage costs. + +:::warning +Only the latest version of each checkpoint file is read by Platform. However, non-current S3 object versions are not automatically removed and will continue to accrue storage costs until explicitly deleted or expired. +::: + +**Recommended mitigation:** Apply an S3 Lifecycle rule to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes: + +```json +{ + "Rules": [ + { + "ID": "expire-studios-checkpoint-noncurrent-versions", + "Filter": { "Prefix": ".studios/checkpoints/" }, + "Status": "Enabled", + "NoncurrentVersionExpiration": { "NoncurrentDays": 1 } + } + ] +} +``` + +To apply this rule using the AWS CLI: + +```bash +aws s3api put-bucket-lifecycle-configuration \ + --bucket \ + --lifecycle-configuration file://lifecycle.json +``` + +To remove existing accumulated non-current versions on the `.studios/checkpoints/` prefix: + +```bash +aws s3api list-object-versions \ + --bucket \ + --prefix ".studios/checkpoints/" \ + --query 'Versions[?IsLatest==`false`].[Key,VersionId]' \ + --output text | \ +while IFS=$'\t' read -r key version_id; do + aws s3api delete-object \ + --bucket \ + --key "$key" \ + --version-id "$version_id" +done +``` + +:::note +Non-current object versions (intermediate checkpoint writes) are safe to delete. Do **not** delete the current (latest) version of any checkpoint file or the checkpoint directory itself — doing so will corrupt the Studio session and it cannot be recovered. +::: + When you stop and start a session, or start a new session from a previously created checkpoint, changes such as installed software packages and configuration files are restored and made available. Changes made to mounted data are not included in a checkpoint. Checkpoints can be renamed and the name has to be unique per Studio. Spaces in checkpoint names are converted to underscores automatically. diff --git a/platform-enterprise_docs/studios/managing.md b/platform-enterprise_docs/studios/managing.md index a30c6785a4..8b83116fd2 100644 --- a/platform-enterprise_docs/studios/managing.md +++ b/platform-enterprise_docs/studios/managing.md @@ -277,6 +277,57 @@ When starting a Studio session, a *checkpoint* is automatically created. A check Checkpoints vary in size depending on libraries installed in your session environment. This can potentially result in many large files stored in the compute environment's pipeline work directory and saved to cloud storage. This storage will incur costs based on the cloud provider. Due to the architecture of Studios, you cannot delete any checkpoint files to save on storage costs. Deleting a Studio session's checkpoints will result in a corrupted Studio session that cannot be started nor recovered. ::: +### S3 versioning and checkpoint storage costs + +If your compute environment work directory uses an S3 bucket with **versioning enabled**, checkpoint writes create a new S3 object version every five minutes rather than overwriting the previous one. For an active Studio session, this produces up to 96 new object versions per day per session. Over time, these non-current versions accumulate and can significantly increase storage costs. + +:::warning +Only the latest version of each checkpoint file is read by Platform. However, non-current S3 object versions are not automatically removed and will continue to accrue storage costs until explicitly deleted or expired. +::: + +**Recommended mitigation:** Apply an S3 Lifecycle rule to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes: + +```json +{ + "Rules": [ + { + "ID": "expire-studios-checkpoint-noncurrent-versions", + "Filter": { "Prefix": ".studios/checkpoints/" }, + "Status": "Enabled", + "NoncurrentVersionExpiration": { "NoncurrentDays": 1 } + } + ] +} +``` + +To apply this rule using the AWS CLI: + +```bash +aws s3api put-bucket-lifecycle-configuration \ + --bucket \ + --lifecycle-configuration file://lifecycle.json +``` + +To remove existing accumulated non-current versions on the `.studios/checkpoints/` prefix: + +```bash +aws s3api list-object-versions \ + --bucket \ + --prefix ".studios/checkpoints/" \ + --query 'Versions[?IsLatest==`false`].[Key,VersionId]' \ + --output text | \ +while IFS=$'\t' read -r key version_id; do + aws s3api delete-object \ + --bucket \ + --key "$key" \ + --version-id "$version_id" +done +``` + +:::note +Non-current object versions (intermediate checkpoint writes) are safe to delete. Do **not** delete the current (latest) version of any checkpoint file or the checkpoint directory itself — doing so will corrupt the Studio session and it cannot be recovered. +::: + When you stop and start a session, or start a new session from a previously created checkpoint, changes such as installed software packages and configuration files are restored and made available. Changes made to mounted data are not included in a checkpoint. Checkpoints can be renamed and the name has to be unique per Studio. Spaces in checkpoint names are converted to underscores automatically. From 67c9e3ac5e9c5ffc368adc3956eb243cd8a5d02c Mon Sep 17 00:00:00 2001 From: ejseqera Date: Tue, 19 May 2026 17:29:45 -0400 Subject: [PATCH 2/5] docs(studios): remove AWS CLI commands from checkpoint versioning guidance --- platform-cloud/docs/studios/managing.md | 39 +------------------- platform-enterprise_docs/studios/managing.md | 39 +------------------- 2 files changed, 2 insertions(+), 76 deletions(-) diff --git a/platform-cloud/docs/studios/managing.md b/platform-cloud/docs/studios/managing.md index 54699c81de..da703689ca 100644 --- a/platform-cloud/docs/studios/managing.md +++ b/platform-cloud/docs/studios/managing.md @@ -194,44 +194,7 @@ If your compute environment work directory uses an S3 bucket with **versioning e Only the latest version of each checkpoint file is read by Platform. However, non-current S3 object versions are not automatically removed and will continue to accrue storage costs until explicitly deleted or expired. ::: -**Recommended mitigation:** Apply an S3 Lifecycle rule to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes: - -```json -{ - "Rules": [ - { - "ID": "expire-studios-checkpoint-noncurrent-versions", - "Filter": { "Prefix": ".studios/checkpoints/" }, - "Status": "Enabled", - "NoncurrentVersionExpiration": { "NoncurrentDays": 1 } - } - ] -} -``` - -To apply this rule using the AWS CLI: - -```bash -aws s3api put-bucket-lifecycle-configuration \ - --bucket \ - --lifecycle-configuration file://lifecycle.json -``` - -To remove existing accumulated non-current versions on the `.studios/checkpoints/` prefix: - -```bash -aws s3api list-object-versions \ - --bucket \ - --prefix ".studios/checkpoints/" \ - --query 'Versions[?IsLatest==`false`].[Key,VersionId]' \ - --output text | \ -while IFS=$'\t' read -r key version_id; do - aws s3api delete-object \ - --bucket \ - --key "$key" \ - --version-id "$version_id" -done -``` +**Recommended mitigation:** Apply an S3 Lifecycle rule to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes. You can also delete existing accumulated non-current versions manually using your cloud provider's console or CLI. :::note Non-current object versions (intermediate checkpoint writes) are safe to delete. Do **not** delete the current (latest) version of any checkpoint file or the checkpoint directory itself — doing so will corrupt the Studio session and it cannot be recovered. diff --git a/platform-enterprise_docs/studios/managing.md b/platform-enterprise_docs/studios/managing.md index 8b83116fd2..ba2108207d 100644 --- a/platform-enterprise_docs/studios/managing.md +++ b/platform-enterprise_docs/studios/managing.md @@ -285,44 +285,7 @@ If your compute environment work directory uses an S3 bucket with **versioning e Only the latest version of each checkpoint file is read by Platform. However, non-current S3 object versions are not automatically removed and will continue to accrue storage costs until explicitly deleted or expired. ::: -**Recommended mitigation:** Apply an S3 Lifecycle rule to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes: - -```json -{ - "Rules": [ - { - "ID": "expire-studios-checkpoint-noncurrent-versions", - "Filter": { "Prefix": ".studios/checkpoints/" }, - "Status": "Enabled", - "NoncurrentVersionExpiration": { "NoncurrentDays": 1 } - } - ] -} -``` - -To apply this rule using the AWS CLI: - -```bash -aws s3api put-bucket-lifecycle-configuration \ - --bucket \ - --lifecycle-configuration file://lifecycle.json -``` - -To remove existing accumulated non-current versions on the `.studios/checkpoints/` prefix: - -```bash -aws s3api list-object-versions \ - --bucket \ - --prefix ".studios/checkpoints/" \ - --query 'Versions[?IsLatest==`false`].[Key,VersionId]' \ - --output text | \ -while IFS=$'\t' read -r key version_id; do - aws s3api delete-object \ - --bucket \ - --key "$key" \ - --version-id "$version_id" -done -``` +**Recommended mitigation:** Apply an S3 Lifecycle rule to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes. You can also delete existing accumulated non-current versions manually using your cloud provider's console or CLI. :::note Non-current object versions (intermediate checkpoint writes) are safe to delete. Do **not** delete the current (latest) version of any checkpoint file or the checkpoint directory itself — doing so will corrupt the Studio session and it cannot be recovered. From 0c4a2538567caa3f7e085484f01a57e95aada547 Mon Sep 17 00:00:00 2001 From: Justine Geffen Date: Wed, 10 Jun 2026 11:35:14 +0200 Subject: [PATCH 3/5] Update platform-cloud/docs/studios/managing.md Co-authored-by: Rob Newman <61608+robnewman@users.noreply.github.com> Signed-off-by: Justine Geffen --- platform-cloud/docs/studios/managing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform-cloud/docs/studios/managing.md b/platform-cloud/docs/studios/managing.md index da703689ca..abcb3c1dd8 100644 --- a/platform-cloud/docs/studios/managing.md +++ b/platform-cloud/docs/studios/managing.md @@ -188,7 +188,7 @@ Checkpoints vary in size depending on libraries installed in your session enviro ### S3 versioning and checkpoint storage costs -If your compute environment work directory uses an S3 bucket with **versioning enabled**, checkpoint writes create a new S3 object version every five minutes rather than overwriting the previous one. For an active Studio session, this produces up to 96 new object versions per day per session. Over time, these non-current versions accumulate and can significantly increase storage costs. +If your compute environment work directory uses an object storage bucket with **versioning enabled**, checkpoint writes create a new object version rather than overwriting the previous one. For an active Studio session, this produces many object versions per session. Over time, these non-current versions accumulate and can significantly increase storage costs. :::warning Only the latest version of each checkpoint file is read by Platform. However, non-current S3 object versions are not automatically removed and will continue to accrue storage costs until explicitly deleted or expired. From abc94bda9f3a5e8bcce19a506f8a4d717640e474 Mon Sep 17 00:00:00 2001 From: Justine Geffen Date: Wed, 10 Jun 2026 11:35:24 +0200 Subject: [PATCH 4/5] Update platform-cloud/docs/studios/managing.md Co-authored-by: Rob Newman <61608+robnewman@users.noreply.github.com> Signed-off-by: Justine Geffen --- platform-cloud/docs/studios/managing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform-cloud/docs/studios/managing.md b/platform-cloud/docs/studios/managing.md index abcb3c1dd8..52d73de3b3 100644 --- a/platform-cloud/docs/studios/managing.md +++ b/platform-cloud/docs/studios/managing.md @@ -191,7 +191,7 @@ Checkpoints vary in size depending on libraries installed in your session enviro If your compute environment work directory uses an object storage bucket with **versioning enabled**, checkpoint writes create a new object version rather than overwriting the previous one. For an active Studio session, this produces many object versions per session. Over time, these non-current versions accumulate and can significantly increase storage costs. :::warning -Only the latest version of each checkpoint file is read by Platform. However, non-current S3 object versions are not automatically removed and will continue to accrue storage costs until explicitly deleted or expired. +Only the latest version of each checkpoint file is read by Platform. However, non-current object versions are not automatically removed and will continue to accrue storage costs until explicitly deleted or expired. ::: **Recommended mitigation:** Apply an S3 Lifecycle rule to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes. You can also delete existing accumulated non-current versions manually using your cloud provider's console or CLI. From dd8008959baeb69e4824878e1c9732f5cc90544d Mon Sep 17 00:00:00 2001 From: Justine Geffen Date: Wed, 10 Jun 2026 11:37:16 +0200 Subject: [PATCH 5/5] Update platform-cloud/docs/studios/managing.md Co-authored-by: Rob Newman <61608+robnewman@users.noreply.github.com> Signed-off-by: Justine Geffen --- platform-cloud/docs/studios/managing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform-cloud/docs/studios/managing.md b/platform-cloud/docs/studios/managing.md index c79cf71c74..ed57924ac0 100644 --- a/platform-cloud/docs/studios/managing.md +++ b/platform-cloud/docs/studios/managing.md @@ -271,7 +271,7 @@ If your compute environment work directory uses an object storage bucket with ** Only the latest version of each checkpoint file is read by Platform. However, non-current object versions are not automatically removed and will continue to accrue storage costs until explicitly deleted or expired. ::: -**Recommended mitigation:** Apply an S3 Lifecycle rule to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes. You can also delete existing accumulated non-current versions manually using your cloud provider's console or CLI. +**Recommended mitigation:** Apply lifecycle rules to expire non-current object versions on the `.studios/checkpoints/` prefix. A one-day expiry retains the current version while removing intermediate five-minute writes. You can also delete existing accumulated non-current versions manually using your cloud provider's console or CLI. :::note Non-current object versions (intermediate checkpoint writes) are safe to delete. Do **not** delete the current (latest) version of any checkpoint file or the checkpoint directory itself — doing so will corrupt the Studio session and it cannot be recovered.