From 1c0b1497478473fa0f273db1907c6aa2d7114165 Mon Sep 17 00:00:00 2001 From: DWarez Date: Fri, 10 Jul 2026 11:09:15 +0200 Subject: [PATCH 1/4] add: serverless inference tutorial using sagemaker sdk Signed-off-by: DWarez --- .../sagemaker-notebook.ipynb | 1090 +++++++++++++++++ 1 file changed, 1090 insertions(+) create mode 100644 docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb diff --git a/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb new file mode 100644 index 0000000000..a4c0e05540 --- /dev/null +++ b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb @@ -0,0 +1,1090 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cell-00", + "metadata": {}, + "source": [ + "# Serve embeddings on demand with SageMaker serverless inference\n", + "\n", + "## What serverless inference is\n", + "\n", + "A real-time SageMaker endpoint runs on instances you pick and pay for by the\n", + "hour, whether or not any requests arrive. You choose the instance type and\n", + "count, attach an autoscaling policy, and keep at least one instance warm so the\n", + "endpoint can answer immediately. That is the right shape for steady, predictable\n", + "traffic, and wasteful for everything else.\n", + "\n", + "**Serverless inference** takes the instances out of the picture. You give\n", + "SageMaker two numbers (how much memory a copy of the model needs and how many\n", + "requests it may handle at once) and it provisions compute on demand, scales it\n", + "with traffic, and scales it to **zero** when the endpoint is idle. You pay per\n", + "millisecond of request processing and for the data processed, and nothing while\n", + "no requests are in flight.\n", + "\n", + "![SageMaker serverless inference workflow](https://docs.aws.amazon.com/images/sagemaker/latest/dg/images/serverless-endpoints-how-it-works.png)\n", + "\n", + "The trade-off is the **cold start**. When a request arrives and no compute is\n", + "warm (right after deployment, or after an idle stretch), SageMaker has to start\n", + "a worker, pull the container, and load the model before it can answer. That\n", + "first request is slow; the ones behind it, while the worker stays warm, are\n", + "fast. If you cannot tolerate the occasional slow request, *provisioned\n", + "concurrency* keeps a set number of workers warm at all times, at the cost of\n", + "paying for them whether or not they are used.\n", + "\n", + "## When it fits, and when it doesn't\n", + "\n", + "Serverless is the cost-effective choice when traffic is **intermittent or\n", + "unpredictable** and the workload can absorb an occasional cold start:\n", + "\n", + "- Internal tools, dashboards, and low-QPS APIs that sit idle most of the day.\n", + "- Dev, test, and staging endpoints you do not want billed around the clock.\n", + "- New models whose traffic you cannot forecast yet.\n", + "\n", + "Reach for a provisioned real-time endpoint instead when traffic is **steady and\n", + "high** (a busy always-on instance is cheaper per request and has no cold starts),\n", + "or when a strict latency SLA leaves no room for one. And mind the hard limits:\n", + "serverless is **CPU-only** (no GPUs), memory is capped at **6 GB**, and a single\n", + "endpoint handles at most **200 concurrent requests**. Large or GPU-bound models\n", + "do not belong here, and long-running or large-payload jobs belong on\n", + "[asynchronous inference](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html)\n", + "instead.\n", + "\n", + "## The use case we picked\n", + "\n", + "This tutorial serves an **embedding model for a low-traffic semantic-search\n", + "box**. Embedding a user's live query is small, synchronous, and latency-shaped,\n", + "but on an internal search tool the queries arrive in bursts with long idle gaps\n", + "between them, so paying for an always-on instance is hard to justify. That is\n", + "exactly the shape serverless was built for. (The offline other half, embedding\n", + "a whole corpus in one large batch, is a job for\n", + "[asynchronous inference](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html);\n", + "this notebook is only the online query side.)\n", + "\n", + "The model is `BAAI/bge-small-en-v1.5`: 384-dimensional vectors and a ~130 MB\n", + "download that loads comfortably inside the smallest serverless memory tier. It\n", + "is served with\n", + "[Text Embeddings Inference (TEI)](https://huggingface.co/docs/text-embeddings-inference),\n", + "Hugging Face's container for embedding models.\n", + "\n", + "References:\n", + "\n", + "- [SageMaker serverless inference](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html)\n", + "- [Serverless endpoint operations](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-create-invoke-update-delete.html)\n", + "- [Minimizing cold starts with provisioned concurrency](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-autoscale.html)\n" + ] + }, + { + "cell_type": "markdown", + "id": "cell-01", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "Run the next cell before importing the SDK. It installs the SageMaker Python\n", + "SDK into the active kernel.\n", + "\n", + "You also need an existing SageMaker execution role with access to SageMaker,\n", + "S3, CloudWatch, and the ECR repository that hosts the TEI container. Serverless\n", + "inference is available in a\n", + "[subset of AWS regions](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html).\n", + "Run this in one of them.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "cell-02", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install \"sagemaker>=3.0.0\" --upgrade --quiet" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "cell-03", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "sagemaker.config INFO - Not applying SDK defaults from location: /Library/Application Support/sagemaker/config.yaml\n", + "sagemaker.config INFO - Not applying SDK defaults from location: /Users/dwarez/Library/Application Support/sagemaker/config.yaml\n" + ] + } + ], + "source": [ + "import datetime as dt\n", + "import json\n", + "import math\n", + "import os\n", + "import time\n", + "\n", + "import boto3\n", + "from botocore.exceptions import ClientError\n", + "\n", + "from sagemaker.core import image_uris\n", + "from sagemaker.core.helper.session_helper import Session, get_execution_role\n", + "from sagemaker.core.inference_config import ServerlessInferenceConfig\n", + "from sagemaker.serve import ModelBuilder, ModelServer\n", + "from sagemaker.serve.builder.schema_builder import SchemaBuilder" + ] + }, + { + "cell_type": "markdown", + "id": "cell-04", + "metadata": {}, + "source": [ + "## The model and how serverless is sized\n", + "\n", + "A serverless endpoint is configured by two numbers instead of an instance type:\n", + "\n", + "- **Memory** (`MEMORY_SIZE_IN_MB`) must be at least as large as one copy of the\n", + " model plus its container. It has to be one of 1024, 2048, 3072, 4096, 5120, or\n", + " 6144 MB, and SageMaker gives the container more vCPUs as you go up.\n", + " `BAAI/bge-small-en-v1.5` is tiny, so 2 GB is ample; a larger embedding model\n", + " would need a larger tier.\n", + "- **Max concurrency** (`MAX_CONCURRENCY`) caps how many requests the endpoint\n", + " processes at once, up to 200. Invocations beyond it are throttled rather than\n", + " queued, which keeps one endpoint from consuming your whole account quota.\n", + "\n", + "Leaving `PROVISIONED_CONCURRENCY` unset keeps the endpoint fully on-demand: it\n", + "scales to zero when idle and you accept cold starts. Set it to a small integer\n", + "(no greater than `MAX_CONCURRENCY`) to keep that many workers permanently warm:\n", + "no cold starts, but you pay for the reserved capacity around the clock.\n", + "\n", + "To serve a different model, set `HF_MODEL_ID`, set `EMBEDDING_DIM` to its output\n", + "dimension, and raise `MEMORY_SIZE_IN_MB` if the model is larger.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "cell-05", + "metadata": {}, + "outputs": [], + "source": [ + "PROJECT = \"hf-serverless-embed\"\n", + "RUN_ID = dt.datetime.now(dt.timezone.utc).strftime(\"%Y%m%d%H%M%S\")\n", + "\n", + "MODEL_ID = os.getenv(\"HF_MODEL_ID\", \"BAAI/bge-small-en-v1.5\")\n", + "EMBEDDING_DIM = int(os.getenv(\"EMBEDDING_DIM\", \"384\"))\n", + "TEI_VERSION = os.getenv(\"TEI_VERSION\", \"1.8.2\")\n", + "ENDPOINT_NAME = os.getenv(\"SAGEMAKER_ENDPOINT_NAME\", f\"{PROJECT}-{RUN_ID}\")\n", + "\n", + "# Serverless sizing. Memory must be one of 1024/2048/3072/4096/5120/6144 MB and\n", + "# at least the size of one model copy; max concurrency is capped at 200.\n", + "MEMORY_SIZE_IN_MB = int(os.getenv(\"MEMORY_SIZE_IN_MB\", \"2048\"))\n", + "MAX_CONCURRENCY = int(os.getenv(\"MAX_CONCURRENCY\", \"5\"))\n", + "\n", + "# Leave unset for a pure scale-to-zero endpoint; set it to keep workers warm.\n", + "_provisioned = os.getenv(\"PROVISIONED_CONCURRENCY\")\n", + "PROVISIONED_CONCURRENCY = int(_provisioned) if _provisioned else None\n", + "\n", + "# Set CLEANUP=false to inspect the endpoint after the tutorial finishes.\n", + "CLEANUP = os.getenv(\"CLEANUP\", \"true\").lower() not in {\"0\", \"false\", \"no\"}\n", + "\n", + "assert MEMORY_SIZE_IN_MB in {1024, 2048, 3072, 4096, 5120, 6144}, MEMORY_SIZE_IN_MB\n", + "assert 1 <= MAX_CONCURRENCY <= 200, MAX_CONCURRENCY\n", + "assert PROVISIONED_CONCURRENCY is None or PROVISIONED_CONCURRENCY <= MAX_CONCURRENCY" + ] + }, + { + "cell_type": "markdown", + "id": "cell-06", + "metadata": {}, + "source": [ + "## Set up the SageMaker session\n", + "\n", + "The endpoint runs under a SageMaker execution role: an IAM role that grants\n", + "access to S3, ECR, and CloudWatch. Set `SAGEMAKER_EXECUTION_ROLE_ARN` to the\n", + "role you want to use, or `SAGEMAKER_EXECUTION_ROLE_NAME` if you only have its\n", + "name. Inside SageMaker Studio or a notebook instance you can leave both unset\n", + "and the role is detected automatically.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "cell-07", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "region: us-east-1\n", + "role: arn:aws:iam::754289655784:role/sagemaker_execution_role\n", + "endpoint: hf-serverless-embed-20260710084432\n" + ] + } + ], + "source": [ + "requested_region = os.getenv(\"AWS_REGION\") or os.getenv(\"AWS_DEFAULT_REGION\")\n", + "boto_session = boto3.Session(region_name=requested_region) if requested_region else boto3.Session()\n", + "sess = Session(boto_session=boto_session)\n", + "region = sess.boto_region_name\n", + "\n", + "sm = boto_session.client(\"sagemaker\")\n", + "\n", + "\n", + "def resolve_role(session, sagemaker_session):\n", + " role_arn = os.getenv(\"SAGEMAKER_EXECUTION_ROLE_ARN\")\n", + " if role_arn:\n", + " return role_arn\n", + "\n", + " role_name = os.getenv(\"SAGEMAKER_EXECUTION_ROLE_NAME\")\n", + " if role_name:\n", + " iam = session.client(\"iam\")\n", + " return iam.get_role(RoleName=role_name)[\"Role\"][\"Arn\"]\n", + "\n", + " return get_execution_role(sagemaker_session=sagemaker_session)\n", + "\n", + "\n", + "role = resolve_role(boto_session, sess)\n", + "\n", + "print(f\"region: {region}\")\n", + "print(f\"role: {role}\")\n", + "print(f\"endpoint: {ENDPOINT_NAME}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-08", + "metadata": {}, + "source": [ + "## Select the TEI serving container\n", + "\n", + "Serverless inference is CPU-only, so we always use the CPU build of Text\n", + "Embeddings Inference, `huggingface-tei-cpu`. `image_uris.retrieve` resolves the\n", + "image URI for the current region. No instance type is involved. Serverless has\n", + "no instances to size.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "cell-09", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
[07/10/26 10:44:48] INFO     Defaulting to only available Python version: py310                   image_uris.py:615\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:44:48]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Defaulting to only available Python version: py310 \u001b]8;id=3481485;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/image_uris.py\u001b\\\u001b[2mimage_uris.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481486;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/image_uris.py#615\u001b\\\u001b[2m615\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     Defaulting to only supported image scope: cpu.                       image_uris.py:539\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Defaulting to only supported image scope: cpu. \u001b]8;id=3481492;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/image_uris.py\u001b\\\u001b[2mimage_uris.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481493;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/image_uris.py#539\u001b\\\u001b[2m539\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "683313688378.dkr.ecr.us-east-1.amazonaws.com/tei-cpu:2.0.1-tei1.8.2-cpu-py310-ubuntu22.04\n" + ] + } + ], + "source": [ + "image_uri = image_uris.retrieve(\n", + " framework=\"huggingface-tei-cpu\",\n", + " region=region,\n", + " version=TEI_VERSION,\n", + " image_scope=\"inference\",\n", + ")\n", + "print(image_uri)" + ] + }, + { + "cell_type": "markdown", + "id": "cell-10", + "metadata": {}, + "source": [ + "## Build the model\n", + "\n", + "`ModelBuilder` describes the SageMaker model: the Hub model ID, the TEI\n", + "container, the model server, and a small input/output example. The container\n", + "downloads the model from the Hub when the worker starts. We do not pass an\n", + "instance type (serverless provisions its own compute), and the explicit\n", + "`image_uri` above is what pins the container. For gated or private models, set\n", + "`HF_TOKEN` (or `HUGGING_FACE_HUB_TOKEN`) before running the notebook.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "cell-11", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
[07/10/26 10:44:52] DEBUG    Auto-detecting optimal instance type for model...           model_builder_utils.py:341\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:44:52]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG \u001b[0m Auto-detecting optimal instance type for model\u001b[33m...\u001b[0m \u001b]8;id=3481500;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481501;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#341\u001b\\\u001b[2m341\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     Loading cached SSO token for hf                                          tokens.py:312\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Loading cached SSO token for hf \u001b]8;id=3481508;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py\u001b\\\u001b[2mtokens.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481509;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py#312\u001b\\\u001b[2m312\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[07/10/26 10:44:55] DEBUG    HuggingFace JumpStart Model ID not detected. Building for  model_builder_utils.py:2905\n",
+       "                             HuggingFace Model ID.                                                                 \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:44:55]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG \u001b[0m HuggingFace JumpStart Model ID not detected. Building for \u001b]8;id=3481515;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481516;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#2905\u001b\\\u001b[2m2905\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m HuggingFace Model ID. \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[07/10/26 10:44:56] DEBUG    Using default CPU instance type: ml.m5.large                model_builder_utils.py:375\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:44:56]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG \u001b[0m Using default CPU instance type: ml.m5.large \u001b]8;id=3481522;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481523;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#375\u001b\\\u001b[2m375\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     SageMaker Python SDK will collect telemetry to help us better telemetry_logging.py:110\n",
+       "                             understand our user's needs, diagnose issues, and deliver                             \n",
+       "                             additional features.                                                                  \n",
+       "                             To opt out of telemetry, please disable via TelemetryOptOut                           \n",
+       "                             parameter in SDK defaults config. For more information, refer                         \n",
+       "                             to                                                                                    \n",
+       "                             https://sagemaker.readthedocs.io/en/stable/overview.html#conf                         \n",
+       "                             iguring-and-using-defaults-with-the-sagemaker-python-sdk.                             \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m SageMaker Python SDK will collect telemetry to help us better \u001b]8;id=3481530;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481531;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/telemetry/telemetry_logging.py#110\u001b\\\u001b[2m110\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m understand our user's needs, diagnose issues, and deliver \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m additional features. \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m To opt out of telemetry, please disable via TelemetryOptOut \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m parameter in SDK defaults config. For more information, refer \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m to \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#conf\u001b[0m \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m \u001b[4;38;2;0;105;255miguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     Loading cached SSO token for hf                                          tokens.py:312\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Loading cached SSO token for hf \u001b]8;id=3481536;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py\u001b\\\u001b[2mtokens.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481537;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py#312\u001b\\\u001b[2m312\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[07/10/26 10:44:59] DEBUG    Either inference spec or model is provided. ModelBuilder   model_builder_utils.py:1375\n",
+       "                             is not handling MLflow model input                                                    \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:44:59]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG \u001b[0m Either inference spec or model is provided. ModelBuilder \u001b]8;id=3481543;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481544;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#1375\u001b\\\u001b[2m1375\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m is not handling MLflow model input \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    WARNING  CUDA is not enabled on your device. [Errno 2] No such file or    local_hardware.py:111\n",
+       "                             directory: 'nvidia-smi'. Please run ModelBuilder on CUDA enabled                      \n",
+       "                             hardware to deploy locally.                                                           \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m CUDA is not enabled on your device. \u001b[1m[\u001b[0mErrno \u001b[1;36m2\u001b[0m\u001b[1m]\u001b[0m No such file or \u001b]8;id=3481551;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/utils/local_hardware.py\u001b\\\u001b[2mlocal_hardware.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481552;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/utils/local_hardware.py#111\u001b\\\u001b[2m111\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m directory: \u001b[38;2;0;135;0m'nvidia-smi'\u001b[0m. Please run ModelBuilder on CUDA enabled \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m hardware to deploy locally. \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     45.11316258151532 percent of docker disk space at                local_hardware.py:200\n",
+       "                             /Users/dwarez/Library/Containers/com.docker.docker/Data/vms/0/                        \n",
+       "                             is used.                                                                              \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m \u001b[1;36m45.11316258151532\u001b[0m percent of docker disk space at \u001b]8;id=3481558;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/utils/local_hardware.py\u001b\\\u001b[2mlocal_hardware.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481559;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/utils/local_hardware.py#200\u001b\\\u001b[2m200\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m \u001b[38;2;225;0;225m/Users/dwarez/Library/Containers/com.docker.docker/Data/vms/0/\u001b[0m \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m is used. \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    DEBUG    Skipping auto-detection as image_uri is provided:           model_builder_utils.py:894\n",
+       "                             683313688378.dkr.ecr.us-east-1.amazonaws.com/tei-cpu:2.0.1-                           \n",
+       "                             tei1.8.2-cpu-py310-ubuntu22.04                                                        \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG \u001b[0m Skipping auto-detection as image_uri is provided: \u001b]8;id=3481565;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481566;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#894\u001b\\\u001b[2m894\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m \u001b[1;36m683313688378.\u001b[0mdkr.ecr.us-east-\u001b[1;36m1.\u001b[0mamazonaws.com/tei-cpu:\u001b[1;36m2.0\u001b[0m.\u001b[1;36m1\u001b[0m- \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m tei1.\u001b[1;36m8.2\u001b[0m-cpu-py310-ubuntu22.\u001b[1;36m04\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[07/10/26 10:45:00] INFO     Role                                                          iam_role_resolver.py:598\n",
+       "                             'arn:aws:iam::754289655784:role/sagemaker_execution_role'                             \n",
+       "                             validated for serving. Using it.                                                      \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:45:00]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Role \u001b]8;id=3481573;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/iam_role_resolver.py\u001b\\\u001b[2miam_role_resolver.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481574;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/iam_role_resolver.py#598\u001b\\\u001b[2m598\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m \u001b[38;2;0;135;0m'arn:aws:iam::754289655784:role/sagemaker_execution_role'\u001b[0m \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m validated for serving. Using it. \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     Role                                                          iam_role_resolver.py:598\n",
+       "                             'arn:aws:iam::754289655784:role/sagemaker_execution_role'                             \n",
+       "                             validated for serving. Using it.                                                      \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Role \u001b]8;id=3481579;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/iam_role_resolver.py\u001b\\\u001b[2miam_role_resolver.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481580;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/iam_role_resolver.py#598\u001b\\\u001b[2m598\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m \u001b[38;2;0;135;0m'arn:aws:iam::754289655784:role/sagemaker_execution_role'\u001b[0m \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m validated for serving. Using it. \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     Creating model with name:                                       session_helper.py:1922\n",
+       "                             hf-serverless-embed-model-20260710084432                                              \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Creating model with name: \u001b]8;id=3481587;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py\u001b\\\u001b[2msession_helper.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481588;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py#1922\u001b\\\u001b[2m1922\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m hf-serverless-embed-model-\u001b[1;36m20260710084432\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[07/10/26 10:45:01] DEBUG    No boto3 session provided. Creating a new session.                        utils.py:357\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:45:01]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG \u001b[0m No boto3 session provided. Creating a new session. \u001b]8;id=3481595;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481596;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py#357\u001b\\\u001b[2m357\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    DEBUG    No config provided. Using default config.                                 utils.py:365\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG \u001b[0m No config provided. Using default config. \u001b]8;id=3481602;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481603;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py#365\u001b\\\u001b[2m365\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     Runs on sagemaker prod, region:us-east-1                                  utils.py:375\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Runs on sagemaker prod, region:us-east-\u001b[1;36m1\u001b[0m \u001b]8;id=3481609;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481610;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py#375\u001b\\\u001b[2m375\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[07/10/26 10:45:02] INFO     Loading cached SSO token for hf                                          tokens.py:312\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:45:02]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Loading cached SSO token for hf \u001b]8;id=3481615;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py\u001b\\\u001b[2mtokens.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481616;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py#312\u001b\\\u001b[2m312\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[07/10/26 10:45:03] INFO     ✅ Model has been created:                                       model_builder.py:3641\n",
+       "                             'hf-serverless-embed-model-20260710084432' using server TEI in                        \n",
+       "                             SAGEMAKER_ENDPOINT mode (ARN:                                                         \n",
+       "                             arn:aws:sagemaker:us-east-1:754289655784:model/hf-serverless-emb                      \n",
+       "                             ed-model-20260710084432)                                                              \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:45:03]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m ✅ Model has been created: \u001b]8;id=3481623;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder.py\u001b\\\u001b[2mmodel_builder.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481624;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder.py#3641\u001b\\\u001b[2m3641\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m \u001b[38;2;0;135;0m'hf-serverless-embed-model-20260710084432'\u001b[0m using server TEI in \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m SAGEMAKER_ENDPOINT mode \u001b[1m(\u001b[0mARN: \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m arn:aws:sagemaker:us-east-1:754289655784:model/hf-serverless-emb \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m ed-model-\u001b[1;36m20260710084432\u001b[0m\u001b[1m)\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "hf_token = os.getenv(\"HF_TOKEN\") or os.getenv(\"HUGGING_FACE_HUB_TOKEN\")\n", + "\n", + "env_vars = {\n", + " \"HF_MODEL_ID\": MODEL_ID,\n", + " \"MAX_BATCH_TOKENS\": os.getenv(\"MAX_BATCH_TOKENS\", \"16384\"),\n", + " \"MAX_CLIENT_BATCH_SIZE\": os.getenv(\"MAX_CLIENT_BATCH_SIZE\", \"32\"),\n", + "}\n", + "\n", + "if hf_token:\n", + " env_vars[\"HF_TOKEN\"] = hf_token\n", + " env_vars[\"HUGGING_FACE_HUB_TOKEN\"] = hf_token\n", + "\n", + "resource_tags = [\n", + " {\"Key\": \"Project\", \"Value\": PROJECT},\n", + " {\"Key\": \"ModelId\", \"Value\": MODEL_ID},\n", + " {\"Key\": \"CreatedBy\", \"Value\": \"hf-sagemaker-docs\"},\n", + "]\n", + "\n", + "model_builder = ModelBuilder(\n", + " model=MODEL_ID,\n", + " role_arn=role,\n", + " sagemaker_session=sess,\n", + " image_uri=image_uri,\n", + " model_server=ModelServer.TEI,\n", + " env_vars=env_vars,\n", + " schema_builder=SchemaBuilder(\n", + " sample_input={\"inputs\": [\"who wrote the origin of species\"]},\n", + " sample_output=[[0.0] * EMBEDDING_DIM],\n", + " ),\n", + ")\n", + "\n", + "tei_model = model_builder.build(model_name=f\"{PROJECT}-model-{RUN_ID}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cell-12", + "metadata": {}, + "source": [ + "## Deploy the serverless endpoint\n", + "\n", + "`ServerlessInferenceConfig` carries the two sizing numbers and, optionally,\n", + "provisioned concurrency. Passing it to `deploy` as the `inference_config` is\n", + "what makes the endpoint serverless; there is no `instance_type` or\n", + "`initial_instance_count` to set. Deployment still takes a few minutes while\n", + "SageMaker registers the endpoint; the first *invocation* is where the cold start\n", + "shows up.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "cell-13", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
[07/10/26 10:45:11] INFO     SageMaker Python SDK will collect telemetry to help us better telemetry_logging.py:110\n",
+       "                             understand our user's needs, diagnose issues, and deliver                             \n",
+       "                             additional features.                                                                  \n",
+       "                             To opt out of telemetry, please disable via TelemetryOptOut                           \n",
+       "                             parameter in SDK defaults config. For more information, refer                         \n",
+       "                             to                                                                                    \n",
+       "                             https://sagemaker.readthedocs.io/en/stable/overview.html#conf                         \n",
+       "                             iguring-and-using-defaults-with-the-sagemaker-python-sdk.                             \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:45:11]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m SageMaker Python SDK will collect telemetry to help us better \u001b]8;id=3481629;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481630;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/telemetry/telemetry_logging.py#110\u001b\\\u001b[2m110\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m understand our user's needs, diagnose issues, and deliver \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m additional features. \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m To opt out of telemetry, please disable via TelemetryOptOut \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m parameter in SDK defaults config. For more information, refer \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m to \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#conf\u001b[0m \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m \u001b[4;38;2;0;105;255miguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     Creating endpoint-config with name                              session_helper.py:1093\n",
+       "                             hf-serverless-embed-20260710084432                                                    \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Creating endpoint-config with name \u001b]8;id=3481636;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py\u001b\\\u001b[2msession_helper.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481637;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py#1093\u001b\\\u001b[2m1093\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m hf-serverless-embed-\u001b[1;36m20260710084432\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
                    INFO     Creating endpoint with name hf-serverless-embed-20260710084432  session_helper.py:1125\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO \u001b[0m Creating endpoint with name hf-serverless-embed-\u001b[1;36m20260710084432\u001b[0m \u001b]8;id=3481643;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py\u001b\\\u001b[2msession_helper.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481644;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py#1125\u001b\\\u001b[2m1125\u001b[0m\u001b]8;;\u001b\\\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "bd8db3947a3c4a6c9e2e296012473d4f", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Output()" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
[07/10/26 10:45:13] WARNING  Failed to enable live logging: An error occurred                session_helper.py:2776\n",
+       "                             (AccessDeniedException) when calling the FilterLogEvents                              \n",
+       "                             operation: User:                                                                      \n",
+       "                             arn:aws:sts::754289655784:assumed-role/AWSReservedSSO_HF-Sandbo                       \n",
+       "                             x-access_a9a3037b77bf6782/dario.salvati@huggingface.co is not                         \n",
+       "                             authorized to perform: logs:FilterLogEvents on resource:                              \n",
+       "                             arn:aws:logs:us-east-1:754289655784:log-group:/aws/sagemaker/En                       \n",
+       "                             dpoints/hf-serverless-embed-20260710084432:log-stream: because                        \n",
+       "                             no identity-based policy allows the logs:FilterLogEvents                              \n",
+       "                             action. Fallback to default logging...                                                \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m[07/10/26 10:45:13]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m Failed to enable live logging: An error occurred \u001b]8;id=3481650;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py\u001b\\\u001b[2msession_helper.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481651;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py#2776\u001b\\\u001b[2m2776\u001b[0m\u001b]8;;\u001b\\\n", + "\u001b[2;36m \u001b[0m \u001b[1m(\u001b[0mAccessDeniedException\u001b[1m)\u001b[0m when calling the FilterLogEvents \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m operation: User: \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m arn:aws:sts::\u001b[1;36m754289655784\u001b[0m:assumed-role/AWSReservedSSO_HF-Sandbo \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m x-access_a9a3037b77bf6782/dario.salvati@huggingface.co is not \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m authorized to perform: logs:FilterLogEvents on resource: \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m arn:aws:logs:us-east-1:754289655784:log-group:\u001b[38;2;225;0;225m/aws/sagemaker/En\u001b[0m \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m \u001b[38;2;225;0;225mdpoints/\u001b[0m\u001b[38;2;225;0;225mhf-serverless-embed-20260710084432\u001b[0m:log-stream: because \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m no identity-based policy allows the logs:FilterLogEvents \u001b[2m \u001b[0m\n", + "\u001b[2;36m \u001b[0m action. Fallback to default logging\u001b[33m...\u001b[0m \u001b[2m \u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n"
+      ],
+      "text/plain": []
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [
+       "
[07/10/26 10:47:14] INFO     ✅ Deployment successful: Endpoint                               model_builder.py:2971\n",
+       "                             'hf-serverless-embed-20260710084432' using TEI in                                     \n",
+       "                             SAGEMAKER_ENDPOINT mode (ARN:                                                         \n",
+       "                             arn:aws:sagemaker:us-east-1:754289655784:endpoint/hf-serverless-                      \n",
+       "                             embed-20260710084432)                                                                 \n",
+       "
\n"
+      ],
+      "text/plain": [
+       "\u001b[2;36m[07/10/26 10:47:14]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m ✅ Deployment successful: Endpoint                               \u001b]8;id=3481657;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder.py\u001b\\\u001b[2mmodel_builder.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481658;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder.py#2971\u001b\\\u001b[2m2971\u001b[0m\u001b]8;;\u001b\\\n",
+       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'hf-serverless-embed-20260710084432'\u001b[0m using TEI in                \u001b[2m                     \u001b[0m\n",
+       "\u001b[2;36m                    \u001b[0m         SAGEMAKER_ENDPOINT mode \u001b[1m(\u001b[0mARN:                                    \u001b[2m                     \u001b[0m\n",
+       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-east-1:754289655784:endpoint/hf-serverless- \u001b[2m                     \u001b[0m\n",
+       "\u001b[2;36m                    \u001b[0m         embed-\u001b[1;36m20260710084432\u001b[0m\u001b[1m)\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "endpoint status: InService\n",
+      "endpoint config: hf-serverless-embed-20260710084432\n",
+      "model: hf-serverless-embed-model-20260710084432\n"
+     ]
+    }
+   ],
+   "source": [
+    "serverless_config = ServerlessInferenceConfig(\n",
+    "    memory_size_in_mb=MEMORY_SIZE_IN_MB,\n",
+    "    max_concurrency=MAX_CONCURRENCY,\n",
+    "    provisioned_concurrency=PROVISIONED_CONCURRENCY,\n",
+    ")\n",
+    "\n",
+    "endpoint = model_builder.deploy(\n",
+    "    endpoint_name=ENDPOINT_NAME,\n",
+    "    inference_config=serverless_config,\n",
+    "    container_timeout_in_seconds=900,\n",
+    "    tags=resource_tags,\n",
+    "    wait=True,\n",
+    ")\n",
+    "\n",
+    "endpoint_description = sm.describe_endpoint(EndpointName=ENDPOINT_NAME)\n",
+    "endpoint_config_name = endpoint_description[\"EndpointConfigName\"]\n",
+    "model_name = tei_model.model_name\n",
+    "\n",
+    "print(f\"endpoint status: {endpoint_description['EndpointStatus']}\")\n",
+    "print(f\"endpoint config: {endpoint_config_name}\")\n",
+    "print(f\"model: {model_name}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-14",
+   "metadata": {},
+   "source": [
+    "## Invoke the endpoint and watch the cold start\n",
+    "\n",
+    "TEI takes `{\"inputs\": [...]}` and returns one vector per input. The first call\n",
+    "below lands on a cold endpoint: SageMaker starts a worker and loads the model,\n",
+    "so it is slow. The calls after it reuse the warm worker and return in a fraction\n",
+    "of the time. The gap between them *is* the cold start: the cost you accept in\n",
+    "exchange for scaling to zero.\n",
+    "\n",
+    "The numbers vary from run to run, and CloudWatch reports the cold-start portion\n",
+    "separately as the `OverheadLatency` metric. To force another cold start, leave\n",
+    "the endpoint idle for a while and invoke again.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "cell-15",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "
[07/10/26 10:47:42] INFO     Loading cached SSO token for hf                                          tokens.py:312\n",
+       "
\n"
+      ],
+      "text/plain": [
+       "\u001b[2;36m[07/10/26 10:47:42]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Loading cached SSO token for hf                                          \u001b]8;id=3481663;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py\u001b\\\u001b[2mtokens.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481664;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py#312\u001b\\\u001b[2m312\u001b[0m\u001b]8;;\u001b\\\n"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "cold-start invocation: 1.50 s\n",
+      "warm invocations: 0.15 s, 0.15 s, 0.16 s\n",
+      "embedding dimensions: 384\n"
+     ]
+    }
+   ],
+   "source": [
+    "def embed(texts):\n",
+    "    response = endpoint.invoke(\n",
+    "        body=json.dumps({\"inputs\": texts}),\n",
+    "        content_type=\"application/json\",\n",
+    "        accept=\"application/json\",\n",
+    "    )\n",
+    "    return json.loads(response.body.read())\n",
+    "\n",
+    "\n",
+    "def timed_embed(texts):\n",
+    "    start = time.perf_counter()\n",
+    "    vectors = embed(texts)\n",
+    "    return vectors, time.perf_counter() - start\n",
+    "\n",
+    "\n",
+    "cold_vectors, cold_latency = timed_embed([\"who wrote the origin of species\"])\n",
+    "assert len(cold_vectors) == 1 and len(cold_vectors[0]) == EMBEDDING_DIM\n",
+    "\n",
+    "warm_latencies = []\n",
+    "for _ in range(3):\n",
+    "    _, warm_latency = timed_embed([\"a warm follow-up request\"])\n",
+    "    warm_latencies.append(warm_latency)\n",
+    "\n",
+    "print(f\"cold-start invocation: {cold_latency:.2f} s\")\n",
+    "print(f\"warm invocations: {', '.join(f'{value:.2f} s' for value in warm_latencies)}\")\n",
+    "print(f\"embedding dimensions: {len(cold_vectors[0])}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-16",
+   "metadata": {},
+   "source": [
+    "## Check the embeddings are usable\n",
+    "\n",
+    "Latency is only half the story: the vectors also have to be good enough to rank\n",
+    "text by meaning. We embed a query together with a handful of candidate sentences\n",
+    "in one call, score each candidate against the query with cosine similarity, and\n",
+    "confirm the passage that actually answers the query comes out on top.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "cell-17",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "0.815  Charles Darwin published On the Origin of Species in 1859, laying out his theory of evolution by natural selection.\n",
+      "0.551  Python is a high-level programming language known for its readable syntax.\n",
+      "0.517  The Great Barrier Reef is the world's largest coral reef system, off the coast of Queensland, Australia.\n",
+      "0.507  Mount Kilimanjaro is the highest mountain in Africa.\n",
+      "\n",
+      "retrieval smoke test passed: the Darwin passage ranks first\n"
+     ]
+    }
+   ],
+   "source": [
+    "def cosine(a, b):\n",
+    "    dot = sum(x * y for x, y in zip(a, b))\n",
+    "    norm_a = math.sqrt(sum(x * x for x in a))\n",
+    "    norm_b = math.sqrt(sum(y * y for y in b))\n",
+    "    if norm_a == 0 or norm_b == 0:\n",
+    "        raise ValueError(\"cosine similarity is undefined for a zero vector\")\n",
+    "    return dot / (norm_a * norm_b)\n",
+    "\n",
+    "\n",
+    "assert cosine([1.0, 0.0], [0.0, 1.0]) == 0.0\n",
+    "assert abs(cosine([1.0, 2.0, 3.0], [1.0, 2.0, 3.0]) - 1.0) < 1e-9\n",
+    "\n",
+    "query = \"who wrote the origin of species\"\n",
+    "candidates = [\n",
+    "    \"Charles Darwin published On the Origin of Species in 1859, laying out his theory of evolution by natural selection.\",\n",
+    "    \"The Great Barrier Reef is the world's largest coral reef system, off the coast of Queensland, Australia.\",\n",
+    "    \"Python is a high-level programming language known for its readable syntax.\",\n",
+    "    \"Mount Kilimanjaro is the highest mountain in Africa.\",\n",
+    "]\n",
+    "\n",
+    "vectors = embed([query] + candidates)\n",
+    "query_vector, candidate_vectors = vectors[0], vectors[1:]\n",
+    "\n",
+    "ranked = sorted(\n",
+    "    (\n",
+    "        {\"score\": cosine(query_vector, vector), \"text\": text}\n",
+    "        for text, vector in zip(candidates, candidate_vectors)\n",
+    "    ),\n",
+    "    key=lambda item: item[\"score\"],\n",
+    "    reverse=True,\n",
+    ")\n",
+    "\n",
+    "for hit in ranked:\n",
+    "    print(f\"{hit['score']:.3f}  {hit['text']}\")\n",
+    "\n",
+    "assert ranked[0][\"text\"] == candidates[0]\n",
+    "print(\"\\nretrieval smoke test passed: the Darwin passage ranks first\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-18",
+   "metadata": {},
+   "source": [
+    "## Clean up\n",
+    "\n",
+    "Serverless endpoints cost nothing while idle, but the endpoint, its\n",
+    "configuration, and the model resource linger until you delete them, and a\n",
+    "provisioned-concurrency endpoint keeps billing for the reserved workers.\n",
+    "Deleting the endpoint also returns its share of your account's serverless\n",
+    "concurrency quota.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "cell-19",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "deleting endpoint\n",
+      "deleting endpoint config\n",
+      "deleting model\n"
+     ]
+    }
+   ],
+   "source": [
+    "def ignore_not_found(error):\n",
+    "    code = error.response.get(\"Error\", {}).get(\"Code\", \"\")\n",
+    "    message = error.response.get(\"Error\", {}).get(\"Message\", \"\")\n",
+    "    return code in {\"ResourceNotFound\", \"ResourceNotFoundException\"} or \"not exist\" in message\n",
+    "\n",
+    "\n",
+    "def cleanup_resources():\n",
+    "    print(\"deleting endpoint\")\n",
+    "    try:\n",
+    "        sm.delete_endpoint(EndpointName=ENDPOINT_NAME)\n",
+    "        sm.get_waiter(\"endpoint_deleted\").wait(\n",
+    "            EndpointName=ENDPOINT_NAME,\n",
+    "            WaiterConfig={\"Delay\": 15, \"MaxAttempts\": 60},\n",
+    "        )\n",
+    "    except ClientError as error:\n",
+    "        if not ignore_not_found(error):\n",
+    "            raise\n",
+    "\n",
+    "    print(\"deleting endpoint config\")\n",
+    "    try:\n",
+    "        sm.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n",
+    "    except ClientError as error:\n",
+    "        if not ignore_not_found(error):\n",
+    "            raise\n",
+    "\n",
+    "    print(\"deleting model\")\n",
+    "    try:\n",
+    "        sm.delete_model(ModelName=model_name)\n",
+    "    except ClientError as error:\n",
+    "        if not ignore_not_found(error):\n",
+    "            raise\n",
+    "\n",
+    "\n",
+    "if CLEANUP:\n",
+    "    cleanup_resources()\n",
+    "else:\n",
+    "    print(f\"left endpoint running: {ENDPOINT_NAME}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "SageMaker v3 (.venv-sm-v3)",
+   "language": "python",
+   "name": "sagemaker-v3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

From 1680d6546df8b66a7c297f3d9c10e786895f6b64 Mon Sep 17 00:00:00 2001
From: DWarez 
Date: Fri, 10 Jul 2026 11:24:46 +0200
Subject: [PATCH 2/4] fix: removing cell outputs

Signed-off-by: DWarez 
---
 .../sagemaker-notebook.ipynb                  | 569 +-----------------
 1 file changed, 19 insertions(+), 550 deletions(-)

diff --git a/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
index a4c0e05540..e286c68bdb 100644
--- a/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
+++ b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
@@ -93,37 +93,20 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "cell-02",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Note: you may need to restart the kernel to use updated packages.\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "%pip install \"sagemaker>=3.0.0\" --upgrade --quiet"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "cell-03",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "sagemaker.config INFO - Not applying SDK defaults from location: /Library/Application Support/sagemaker/config.yaml\n",
-      "sagemaker.config INFO - Not applying SDK defaults from location: /Users/dwarez/Library/Application Support/sagemaker/config.yaml\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "import datetime as dt\n",
     "import json\n",
@@ -170,7 +153,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "id": "cell-05",
    "metadata": {},
    "outputs": [],
@@ -216,20 +199,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "cell-07",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "region: us-east-1\n",
-      "role: arn:aws:iam::754289655784:role/sagemaker_execution_role\n",
-      "endpoint: hf-serverless-embed-20260710084432\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "requested_region = os.getenv(\"AWS_REGION\") or os.getenv(\"AWS_DEFAULT_REGION\")\n",
     "boto_session = boto3.Session(region_name=requested_region) if requested_region else boto3.Session()\n",
@@ -274,44 +247,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": null,
    "id": "cell-09",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:44:48] INFO     Defaulting to only available Python version: py310                   image_uris.py:615\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:44:48]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Defaulting to only available Python version: py310                   \u001b]8;id=3481485;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/image_uris.py\u001b\\\u001b[2mimage_uris.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481486;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/image_uris.py#615\u001b\\\u001b[2m615\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     Defaulting to only supported image scope: cpu.                       image_uris.py:539\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Defaulting to only supported image scope: cpu.                       \u001b]8;id=3481492;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/image_uris.py\u001b\\\u001b[2mimage_uris.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481493;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/image_uris.py#539\u001b\\\u001b[2m539\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "683313688378.dkr.ecr.us-east-1.amazonaws.com/tei-cpu:2.0.1-tei1.8.2-cpu-py310-ubuntu22.04\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "image_uri = image_uris.retrieve(\n",
     "    framework=\"huggingface-tei-cpu\",\n",
@@ -339,293 +278,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": null,
    "id": "cell-11",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:44:52] DEBUG    Auto-detecting optimal instance type for model...           model_builder_utils.py:341\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:44:52]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG   \u001b[0m Auto-detecting optimal instance type for model\u001b[33m...\u001b[0m           \u001b]8;id=3481500;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481501;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#341\u001b\\\u001b[2m341\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     Loading cached SSO token for hf                                          tokens.py:312\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Loading cached SSO token for hf                                          \u001b]8;id=3481508;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py\u001b\\\u001b[2mtokens.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481509;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py#312\u001b\\\u001b[2m312\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:44:55] DEBUG    HuggingFace JumpStart Model ID not detected. Building for  model_builder_utils.py:2905\n",
-       "                             HuggingFace Model ID.                                                                 \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:44:55]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG   \u001b[0m HuggingFace JumpStart Model ID not detected. Building for  \u001b]8;id=3481515;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481516;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#2905\u001b\\\u001b[2m2905\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         HuggingFace Model ID.                                      \u001b[2m                           \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:44:56] DEBUG    Using default CPU instance type: ml.m5.large                model_builder_utils.py:375\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:44:56]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG   \u001b[0m Using default CPU instance type: ml.m5.large                \u001b]8;id=3481522;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481523;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#375\u001b\\\u001b[2m375\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     SageMaker Python SDK will collect telemetry to help us better telemetry_logging.py:110\n",
-       "                             understand our user's needs, diagnose issues, and deliver                             \n",
-       "                             additional features.                                                                  \n",
-       "                             To opt out of telemetry, please disable via TelemetryOptOut                           \n",
-       "                             parameter in SDK defaults config. For more information, refer                         \n",
-       "                             to                                                                                    \n",
-       "                             https://sagemaker.readthedocs.io/en/stable/overview.html#conf                         \n",
-       "                             iguring-and-using-defaults-with-the-sagemaker-python-sdk.                             \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better \u001b]8;id=3481530;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481531;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/telemetry/telemetry_logging.py#110\u001b\\\u001b[2m110\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver     \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         additional features.                                          \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut   \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         to                                                            \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#conf\u001b[0m \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255miguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m     \u001b[2m                        \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     Loading cached SSO token for hf                                          tokens.py:312\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Loading cached SSO token for hf                                          \u001b]8;id=3481536;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py\u001b\\\u001b[2mtokens.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481537;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py#312\u001b\\\u001b[2m312\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:44:59] DEBUG    Either inference spec or model is provided. ModelBuilder   model_builder_utils.py:1375\n",
-       "                             is not handling MLflow model input                                                    \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:44:59]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG   \u001b[0m Either inference spec or model is provided. ModelBuilder   \u001b]8;id=3481543;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481544;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#1375\u001b\\\u001b[2m1375\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         is not handling MLflow model input                         \u001b[2m                           \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    WARNING  CUDA is not enabled on your device. [Errno 2] No such file or    local_hardware.py:111\n",
-       "                             directory: 'nvidia-smi'. Please run ModelBuilder on CUDA enabled                      \n",
-       "                             hardware to deploy locally.                                                           \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m CUDA is not enabled on your device. \u001b[1m[\u001b[0mErrno \u001b[1;36m2\u001b[0m\u001b[1m]\u001b[0m No such file or    \u001b]8;id=3481551;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/utils/local_hardware.py\u001b\\\u001b[2mlocal_hardware.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481552;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/utils/local_hardware.py#111\u001b\\\u001b[2m111\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         directory: \u001b[38;2;0;135;0m'nvidia-smi'\u001b[0m. Please run ModelBuilder on CUDA enabled \u001b[2m                     \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         hardware to deploy locally.                                      \u001b[2m                     \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     45.11316258151532 percent of docker disk space at                local_hardware.py:200\n",
-       "                             /Users/dwarez/Library/Containers/com.docker.docker/Data/vms/0/                        \n",
-       "                             is used.                                                                              \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m \u001b[1;36m45.11316258151532\u001b[0m percent of docker disk space at                \u001b]8;id=3481558;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/utils/local_hardware.py\u001b\\\u001b[2mlocal_hardware.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481559;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/utils/local_hardware.py#200\u001b\\\u001b[2m200\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225m/Users/dwarez/Library/Containers/com.docker.docker/Data/vms/0/\u001b[0m   \u001b[2m                     \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         is used.                                                         \u001b[2m                     \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    DEBUG    Skipping auto-detection as image_uri is provided:           model_builder_utils.py:894\n",
-       "                             683313688378.dkr.ecr.us-east-1.amazonaws.com/tei-cpu:2.0.1-                           \n",
-       "                             tei1.8.2-cpu-py310-ubuntu22.04                                                        \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG   \u001b[0m Skipping auto-detection as image_uri is provided:           \u001b]8;id=3481565;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py\u001b\\\u001b[2mmodel_builder_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481566;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder_utils.py#894\u001b\\\u001b[2m894\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[1;36m683313688378.\u001b[0mdkr.ecr.us-east-\u001b[1;36m1.\u001b[0mamazonaws.com/tei-cpu:\u001b[1;36m2.0\u001b[0m.\u001b[1;36m1\u001b[0m- \u001b[2m                          \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         tei1.\u001b[1;36m8.2\u001b[0m-cpu-py310-ubuntu22.\u001b[1;36m04\u001b[0m                              \u001b[2m                          \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:45:00] INFO     Role                                                          iam_role_resolver.py:598\n",
-       "                             'arn:aws:iam::754289655784:role/sagemaker_execution_role'                             \n",
-       "                             validated for serving. Using it.                                                      \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:45:00]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Role                                                          \u001b]8;id=3481573;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/iam_role_resolver.py\u001b\\\u001b[2miam_role_resolver.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481574;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/iam_role_resolver.py#598\u001b\\\u001b[2m598\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:iam::754289655784:role/sagemaker_execution_role'\u001b[0m     \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         validated for serving. Using it.                              \u001b[2m                        \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     Role                                                          iam_role_resolver.py:598\n",
-       "                             'arn:aws:iam::754289655784:role/sagemaker_execution_role'                             \n",
-       "                             validated for serving. Using it.                                                      \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Role                                                          \u001b]8;id=3481579;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/iam_role_resolver.py\u001b\\\u001b[2miam_role_resolver.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481580;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/iam_role_resolver.py#598\u001b\\\u001b[2m598\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:iam::754289655784:role/sagemaker_execution_role'\u001b[0m     \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         validated for serving. Using it.                              \u001b[2m                        \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     Creating model with name:                                       session_helper.py:1922\n",
-       "                             hf-serverless-embed-model-20260710084432                                              \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Creating model with name:                                       \u001b]8;id=3481587;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py\u001b\\\u001b[2msession_helper.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481588;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py#1922\u001b\\\u001b[2m1922\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         hf-serverless-embed-model-\u001b[1;36m20260710084432\u001b[0m                        \u001b[2m                      \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:45:01] DEBUG    No boto3 session provided. Creating a new session.                        utils.py:357\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:45:01]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG   \u001b[0m No boto3 session provided. Creating a new session.                        \u001b]8;id=3481595;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481596;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py#357\u001b\\\u001b[2m357\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    DEBUG    No config provided. Using default config.                                 utils.py:365\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;135;0mDEBUG   \u001b[0m No config provided. Using default config.                                 \u001b]8;id=3481602;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481603;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py#365\u001b\\\u001b[2m365\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     Runs on sagemaker prod, region:us-east-1                                  utils.py:375\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Runs on sagemaker prod, region:us-east-\u001b[1;36m1\u001b[0m                                  \u001b]8;id=3481609;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481610;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/utils/utils.py#375\u001b\\\u001b[2m375\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:45:02] INFO     Loading cached SSO token for hf                                          tokens.py:312\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:45:02]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Loading cached SSO token for hf                                          \u001b]8;id=3481615;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py\u001b\\\u001b[2mtokens.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481616;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py#312\u001b\\\u001b[2m312\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:45:03] INFO     ✅ Model has been created:                                       model_builder.py:3641\n",
-       "                             'hf-serverless-embed-model-20260710084432' using server TEI in                        \n",
-       "                             SAGEMAKER_ENDPOINT mode (ARN:                                                         \n",
-       "                             arn:aws:sagemaker:us-east-1:754289655784:model/hf-serverless-emb                      \n",
-       "                             ed-model-20260710084432)                                                              \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:45:03]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m ✅ Model has been created:                                       \u001b]8;id=3481623;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder.py\u001b\\\u001b[2mmodel_builder.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481624;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder.py#3641\u001b\\\u001b[2m3641\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'hf-serverless-embed-model-20260710084432'\u001b[0m using server TEI in   \u001b[2m                     \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         SAGEMAKER_ENDPOINT mode \u001b[1m(\u001b[0mARN:                                    \u001b[2m                     \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-east-1:754289655784:model/hf-serverless-emb \u001b[2m                     \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         ed-model-\u001b[1;36m20260710084432\u001b[0m\u001b[1m)\u001b[0m                                         \u001b[2m                     \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
+   "outputs": [],
    "source": [
     "hf_token = os.getenv(\"HF_TOKEN\") or os.getenv(\"HUGGING_FACE_HUB_TOKEN\")\n",
     "\n",
@@ -678,151 +334,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": null,
    "id": "cell-13",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:45:11] INFO     SageMaker Python SDK will collect telemetry to help us better telemetry_logging.py:110\n",
-       "                             understand our user's needs, diagnose issues, and deliver                             \n",
-       "                             additional features.                                                                  \n",
-       "                             To opt out of telemetry, please disable via TelemetryOptOut                           \n",
-       "                             parameter in SDK defaults config. For more information, refer                         \n",
-       "                             to                                                                                    \n",
-       "                             https://sagemaker.readthedocs.io/en/stable/overview.html#conf                         \n",
-       "                             iguring-and-using-defaults-with-the-sagemaker-python-sdk.                             \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:45:11]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better \u001b]8;id=3481629;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481630;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/telemetry/telemetry_logging.py#110\u001b\\\u001b[2m110\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver     \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         additional features.                                          \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut   \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         to                                                            \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#conf\u001b[0m \u001b[2m                        \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255miguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m     \u001b[2m                        \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     Creating endpoint-config with name                              session_helper.py:1093\n",
-       "                             hf-serverless-embed-20260710084432                                                    \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Creating endpoint-config with name                              \u001b]8;id=3481636;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py\u001b\\\u001b[2msession_helper.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481637;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py#1093\u001b\\\u001b[2m1093\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         hf-serverless-embed-\u001b[1;36m20260710084432\u001b[0m                              \u001b[2m                      \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
                    INFO     Creating endpoint with name hf-serverless-embed-20260710084432  session_helper.py:1125\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Creating endpoint with name hf-serverless-embed-\u001b[1;36m20260710084432\u001b[0m  \u001b]8;id=3481643;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py\u001b\\\u001b[2msession_helper.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481644;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py#1125\u001b\\\u001b[2m1125\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "bd8db3947a3c4a6c9e2e296012473d4f",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Output()"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:45:13] WARNING  Failed to enable live logging: An error occurred                session_helper.py:2776\n",
-       "                             (AccessDeniedException) when calling the FilterLogEvents                              \n",
-       "                             operation: User:                                                                      \n",
-       "                             arn:aws:sts::754289655784:assumed-role/AWSReservedSSO_HF-Sandbo                       \n",
-       "                             x-access_a9a3037b77bf6782/dario.salvati@huggingface.co is not                         \n",
-       "                             authorized to perform: logs:FilterLogEvents on resource:                              \n",
-       "                             arn:aws:logs:us-east-1:754289655784:log-group:/aws/sagemaker/En                       \n",
-       "                             dpoints/hf-serverless-embed-20260710084432:log-stream: because                        \n",
-       "                             no identity-based policy allows the logs:FilterLogEvents                              \n",
-       "                             action. Fallback to default logging...                                                \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:45:13]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m Failed to enable live logging: An error occurred                \u001b]8;id=3481650;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py\u001b\\\u001b[2msession_helper.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481651;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/core/helper/session_helper.py#2776\u001b\\\u001b[2m2776\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[1m(\u001b[0mAccessDeniedException\u001b[1m)\u001b[0m when calling the FilterLogEvents        \u001b[2m                      \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         operation: User:                                                \u001b[2m                      \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         arn:aws:sts::\u001b[1;36m754289655784\u001b[0m:assumed-role/AWSReservedSSO_HF-Sandbo \u001b[2m                      \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         x-access_a9a3037b77bf6782/dario.salvati@huggingface.co is not   \u001b[2m                      \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         authorized to perform: logs:FilterLogEvents on resource:        \u001b[2m                      \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         arn:aws:logs:us-east-1:754289655784:log-group:\u001b[38;2;225;0;225m/aws/sagemaker/En\u001b[0m \u001b[2m                      \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mdpoints/\u001b[0m\u001b[38;2;225;0;225mhf-serverless-embed-20260710084432\u001b[0m:log-stream: because  \u001b[2m                      \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         no identity-based policy allows the logs:FilterLogEvents        \u001b[2m                      \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         action. Fallback to default logging\u001b[33m...\u001b[0m                          \u001b[2m                      \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
\n"
-      ],
-      "text/plain": []
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:47:14] INFO     ✅ Deployment successful: Endpoint                               model_builder.py:2971\n",
-       "                             'hf-serverless-embed-20260710084432' using TEI in                                     \n",
-       "                             SAGEMAKER_ENDPOINT mode (ARN:                                                         \n",
-       "                             arn:aws:sagemaker:us-east-1:754289655784:endpoint/hf-serverless-                      \n",
-       "                             embed-20260710084432)                                                                 \n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:47:14]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m ✅ Deployment successful: Endpoint                               \u001b]8;id=3481657;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder.py\u001b\\\u001b[2mmodel_builder.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481658;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/sagemaker/serve/model_builder.py#2971\u001b\\\u001b[2m2971\u001b[0m\u001b]8;;\u001b\\\n",
-       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'hf-serverless-embed-20260710084432'\u001b[0m using TEI in                \u001b[2m                     \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         SAGEMAKER_ENDPOINT mode \u001b[1m(\u001b[0mARN:                                    \u001b[2m                     \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-east-1:754289655784:endpoint/hf-serverless- \u001b[2m                     \u001b[0m\n",
-       "\u001b[2;36m                    \u001b[0m         embed-\u001b[1;36m20260710084432\u001b[0m\u001b[1m)\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "endpoint status: InService\n",
-      "endpoint config: hf-serverless-embed-20260710084432\n",
-      "model: hf-serverless-embed-model-20260710084432\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "serverless_config = ServerlessInferenceConfig(\n",
     "    memory_size_in_mb=MEMORY_SIZE_IN_MB,\n",
@@ -867,33 +382,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "id": "cell-15",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/html": [
-       "
[07/10/26 10:47:42] INFO     Loading cached SSO token for hf                                          tokens.py:312\n",
-       "
\n"
-      ],
-      "text/plain": [
-       "\u001b[2;36m[07/10/26 10:47:42]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Loading cached SSO token for hf                                          \u001b]8;id=3481663;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py\u001b\\\u001b[2mtokens.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=3481664;file:///Users/dwarez/hf/repos/hub-docs/docs/sagemaker/.venv-sm-v3/lib/python3.12/site-packages/botocore/tokens.py#312\u001b\\\u001b[2m312\u001b[0m\u001b]8;;\u001b\\\n"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "cold-start invocation: 1.50 s\n",
-      "warm invocations: 0.15 s, 0.15 s, 0.16 s\n",
-      "embedding dimensions: 384\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "def embed(texts):\n",
     "    response = endpoint.invoke(\n",
@@ -938,23 +430,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
    "id": "cell-17",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "0.815  Charles Darwin published On the Origin of Species in 1859, laying out his theory of evolution by natural selection.\n",
-      "0.551  Python is a high-level programming language known for its readable syntax.\n",
-      "0.517  The Great Barrier Reef is the world's largest coral reef system, off the coast of Queensland, Australia.\n",
-      "0.507  Mount Kilimanjaro is the highest mountain in Africa.\n",
-      "\n",
-      "retrieval smoke test passed: the Darwin passage ranks first\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "def cosine(a, b):\n",
     "    dot = sum(x * y for x, y in zip(a, b))\n",
@@ -1011,20 +490,10 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
    "id": "cell-19",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "deleting endpoint\n",
-      "deleting endpoint config\n",
-      "deleting model\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "def ignore_not_found(error):\n",
     "    code = error.response.get(\"Error\", {}).get(\"Code\", \"\")\n",

From b8571084e8d26670c5dac515ef68d0c71640da54 Mon Sep 17 00:00:00 2001
From: DWarez 
Date: Fri, 10 Jul 2026 11:38:00 +0200
Subject: [PATCH 3/4] fix: PROVISIONED_CONCURRENCY handling 0 or negative
 values

Signed-off-by: DWarez 
---
 .../serverless-inference-embedding-tei/sagemaker-notebook.ipynb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
index e286c68bdb..ee9c52827d 100644
--- a/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
+++ b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
@@ -173,7 +173,7 @@
     "\n",
     "# Leave unset for a pure scale-to-zero endpoint; set it to keep workers warm.\n",
     "_provisioned = os.getenv(\"PROVISIONED_CONCURRENCY\")\n",
-    "PROVISIONED_CONCURRENCY = int(_provisioned) if _provisioned else None\n",
+    "PROVISIONED_CONCURRENCY = int(_provisioned) if _provisioned and int(_provisioned) > 0 else None\n",
     "\n",
     "# Set CLEANUP=false to inspect the endpoint after the tutorial finishes.\n",
     "CLEANUP = os.getenv(\"CLEANUP\", \"true\").lower() not in {\"0\", \"false\", \"no\"}\n",

From 059a64e36c60ee40cceef7282ed468c5d00a11e0 Mon Sep 17 00:00:00 2001
From: DWarez 
Date: Mon, 13 Jul 2026 12:05:45 +0200
Subject: [PATCH 4/4] add: cold start timing overhead plot

change: more precise phrasing regarding costs

Signed-off-by: DWarez 
---
 .../sagemaker-notebook.ipynb                  | 56 ++++++++++++++++++-
 1 file changed, 54 insertions(+), 2 deletions(-)

diff --git a/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
index ee9c52827d..fc69c01f92 100644
--- a/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
+++ b/docs/sagemaker/notebooks/sagemaker-sdk/serverless-inference-embedding-tei/sagemaker-notebook.ipynb
@@ -20,7 +20,10 @@
     "requests it may handle at once) and it provisions compute on demand, scales it\n",
     "with traffic, and scales it to **zero** when the endpoint is idle. You pay per\n",
     "millisecond of request processing and for the data processed, and nothing while\n",
-    "no requests are in flight.\n",
+    "no requests are in flight, so an idle on-demand endpoint is free. (The exception\n",
+    "is provisioned concurrency, described below: reserving warm workers bills around\n",
+    "the clock, so with it set the endpoint keeps costing money even while completely\n",
+    "idle.)\n",
     "\n",
     "![SageMaker serverless inference workflow](https://docs.aws.amazon.com/images/sagemaker/latest/dg/images/serverless-endpoints-how-it-works.png)\n",
     "\n",
@@ -98,7 +101,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%pip install \"sagemaker>=3.0.0\" --upgrade --quiet"
+    "%pip install \"sagemaker>=3.0.0\" matplotlib --upgrade --quiet"
    ]
   },
   {
@@ -415,6 +418,55 @@
     "print(f\"embedding dimensions: {len(cold_vectors[0])}\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "cell-15a",
+   "metadata": {},
+   "source": [
+    "### Visualize the cold-start overhead\n",
+    "\n",
+    "The gap between the first (cold) call and the warm calls that follow is the\n",
+    "cold-start overhead: the time SageMaker spends starting a worker and loading\n",
+    "the model before it can serve a response. Plotting the latencies measured\n",
+    "above makes that gap concrete. After an idle stretch the endpoint scales back\n",
+    "to zero, so a later call pays the cold start again."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-15b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ.setdefault(\"MPLCONFIGDIR\", \"/tmp/matplotlib\")\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "warm_mean = sum(warm_latencies) / len(warm_latencies)\n",
+    "overhead = cold_latency - warm_mean\n",
+    "\n",
+    "labels = [\"cold start\\n(worker load + inference)\", \"warm\\n(inference only)\"]\n",
+    "values = [cold_latency, warm_mean]\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(5, 4))\n",
+    "bars = ax.bar(labels, values, color=[\"#e07a5f\", \"#3d9a8b\"])\n",
+    "ax.set_ylabel(\"latency (seconds)\")\n",
+    "ax.set_title(f\"Cold start adds ~{overhead:.2f}s of overhead\")\n",
+    "ax.margins(y=0.15)\n",
+    "for bar, value in zip(bars, values):\n",
+    "    ax.text(\n",
+    "        bar.get_x() + bar.get_width() / 2,\n",
+    "        value,\n",
+    "        f\"{value:.2f}s\",\n",
+    "        ha=\"center\",\n",
+    "        va=\"bottom\",\n",
+    "    )\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "cell-16",