Sunday, March 3, 2024
No menu items!
HomeArtificial Intelligence and Machine LearningAutomate vending Amazon SageMaker notebooks with Amazon EventBridge and AWS Lambda

Automate vending Amazon SageMaker notebooks with Amazon EventBridge and AWS Lambda

Having an environment capable of delivering Amazon SageMaker notebook instances quickly allows data scientists and business analysts to efficiently respond to organizational needs. Data is the lifeblood of an organization, and analyzing that data efficiently provides useful insights for businesses. A common issue that organizations encounter is creating an automated pattern that enables development teams to launch AWS services. Organizations want to enable their developers to launch resources as they need them, but in a centralized and secure fashion.

This post demonstrates how to centralize the management of SageMaker instance notebooks using AWS services including AWS CloudFormation, AWS Serverless Application Model (AWS SAM), AWS Service Catalog, Amazon EventBridge, AWS Systems Manager Parameter Store, Amazon API Gateway, and AWS Lambda. We walk through how to use these AWS services to automate the process of vending SageMaker notebooks to end-users.

Solution overview

In our solution, a notebook user requests a notebook instance using AWS Service Catalog. The request is processed by AWS CloudFormation, which delivers the notebook instance. EventBridge monitors the AWS Service Catalog API for completion of the notebook instance resource provisioning. An event-based rule in EventBridge calls the Lambda event processor, which runs a Lambda function returning the presigned URL.

The following architectural diagram illustrates the infrastructure state as defined in the CloudFormation templates.

The process consists of the following steps:

A user requests a new notebook via the AWS Service Catalog console.
AWS Service Catalog launches a CloudFormation stack.
AWS CloudFormation launches the SageMaker notebook.
A SageMaker notebook is now running.
An EventBridge function is triggered when a new AWS Service Catalog product is launched.
The Amazon CloudWatch event invokes a Lambda function that generates the presigned URL and a user-specific SSM parameter.
A user requests a new presigned URL.
A Lambda function generates a new presigned URL and updates the user’s SSM parameter with the new URL.


To implement this solution, you must have the following prerequisites:

An AWS account with local credentials properly configured (typically under ~/.aws/credentials).
An AWS Identity and Access Management (IAM) role configured with administrative privileges.
The latest version of the AWS Command Line Interface (AWS CLI). For more information, refer to Installing or updating the latest version of the AWS CLI.
Git client to clone the source code located here.
The latest version of the AWS SAM CLI.

Deploy resources with AWS CloudFormation

To create your resources with AWS CloudFormation, complete the following steps:

Deploy the s3-iam-config CloudFormation template:

aws cloudformation create-stack
–stack-name s3-iam-config
–template-body file://templates/s3-iam-config.yml
–parameters file://parameters/s3-iam-params.json

The output should look like the following code:

“StackId”: “arn:aws:cloudformation:us-east-1:123456789012:stack/s3-iam-config/9be9f990-0909-11eb-811c-0a78092beb51”

The template creates an Amazon Simple Storage Service (Amazon S3) bucket.

Run the following command to get the S3 bucket name generated in the previous step:

aws cloudformation describe-stacks
–stack-name s3-iam-config
–query “Stacks[0].Outputs[?OutputKey==’S3BucketName’].OutputValue”
–output text

The output should look like the following:


Run the following command using the output from the previous step (update the bucket name):

aws s3 cp templates/sm-notebook.yml s3://<bucket_name>/sm-notebook.yml

The output should look like the following:

upload: templates/sm-notebook.yml to s3://s3-iam-config-s3bucket-1p85zr5051d86/sm-notebook.yml

Open the parameters/service-catalog-params.json file and update the S3BucketName parameter to the bucket name from the previous step. Update the UserIAMPrincipal with the ARN of the IAM role you’re using for this demo.

“ParameterKey” : “NotebookInstanceType”,
“ParameterValue” : “ml.t2.medium”
“ParameterKey” : “S3IAMConfigStackName”,
“ParameterValue” : “s3-iam-config”
“ParameterKey” : “ServiceCatalogTemplateName”,
“ParameterValue” : “sm-notebook.yml”
“ParameterKey” : “S3BucketName”,
“ParameterValue” : “<input_your_bucket_name>
“ParameterKey” : “UserIAMPrincipal”,
“ParameterValue” : “<input_your_iam_principal_arn>

Deploy the service-catalog CloudFormation template:

aws cloudformation create-stack
–stack-name service-catalog-config
–template-body file://templates/service-catalog.yml
–parameters file://parameters/service-catalog-params.json

The output should look like the following:

“StackId”: “arn:aws:cloudformation:us-east-1:123456789012:stack/service-catalog-config/fb29c5e0-28a0-11ec-8337-123f746ae8a3”

Deploy resources with AWS SAM

To deploy resources with AWS SAM, complete the following steps:

Change your directory to the lambda directory:

cd lambda/

Build the application:

sam build

The output should look like the following:

Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml

Commands you can use next
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync –stack-name {stack-name} –watch
[*] Deploy: sam deploy –guided

Deploy the application:

sam deploy –guided

Respond to the questions in the CLI as shown in the following code:

Configuring SAM deploy

Looking for config file [samconfig.toml] : Found
Reading default arguments : Success

Setting default arguments for ‘sam deploy’
Stack Name [sam-app]: sam-app
AWS Region [us-east-1]: us-east-1
Parameter EventBridgeFunctionName [EventBridgeFunction]: EventBridgeFunction
Parameter EventRuleName [SvcCatalogEventRule]: SvcCatalogEventRule
Parameter RefreshFunctionName [RefreshURLFunction]: RefreshURLFunction
#Shows you resources changes to be deployed and require a ‘Y’ to initiate deploy
Confirm changes before deploy [y/N]: N
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]: Y
#Preserves the state of previously provisioned resources when an operation fails
Disable rollback [y/N]: N
EventBridgeFunction may not have authorization defined, Is this okay? [y/N]: Y
RefreshURLFunction may not have authorization defined, Is this okay? [y/N]: Y
Save arguments to configuration file [Y/n]: Y
SAM configuration file [samconfig.toml]: samconfig.toml
SAM configuration environment [default]: dev

The output should look like the following:

Looking for resources needed for deployment:
Creating the required resources…
Successfully created!
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1f4i68wsmouhw
A different default S3 bucket can be set in samconfig.toml

Saved arguments to config file
Running ‘sam deploy’ for future deployments will use the parameters saved above.
The above parameters can be changed by modifying samconfig.toml
Learn more about samconfig.toml syntax at

Uploading to sam-app/6f3e2f13cfdca08133238f77fc2c667b 9425988 / 9425988 (100.00%)
Uploading to sam-app/b153fd4be66b581361f7d46efae25f18 9425968 / 9425968 (100.00%)

Deploying with following values
Stack name : sam-app
Region : us-east-1
Confirm changeset : False
Disable rollback : False
Deployment s3 bucket : aws-sam-cli-managed-default-samclisourcebucket-1f4i68wsmouhw
Capabilities : [“CAPABILITY_IAM”]
Parameter overrides : {“EventBridgeFunctionName”: “EventBridgeFunction”, “EventRuleName”: “SvcCatalogEventRule”, “RefreshFunctionName”: “RefreshURLFunction”}
Signing Profiles : {}

Initiating deployment
Uploading to sam-app/c82cdea2bfbc2abc6520a97fce4c8a8b.template 6754 / 6754 (100.00%)

Waiting for changeset to be created..

CloudFormation stack changeset
Operation LogicalResourceId ResourceType Replacement
+ Add EventBridgeFunctionHelloWorldPermissio AWS::Lambda::Permission N/A
+ Add EventBridgeFunctionRole AWS::IAM::Role N/A
+ Add EventBridgeFunction AWS::Lambda::Function N/A
+ Add PermissionForEventsToInvokeLambda AWS::Lambda::Permission N/A
+ Add RefreshURLFunctionHelloWorldPermission AWS::Lambda::Permission N/A
+ Add RefreshURLFunctionRole AWS::IAM::Role N/A
+ Add RefreshURLFunction AWS::Lambda::Function N/A
+ Add ServerlessRestApiDeploymentb762875163 AWS::ApiGateway::Deployment N/A
+ Add ServerlessRestApiProdStage AWS::ApiGateway::Stage N/A
+ Add ServerlessRestApi AWS::ApiGateway::RestApi N/A
+ Add SvcCatalogEventRule AWS::Events::Rule N/A

Changeset created successfully. arn:aws:cloudformation:us-east-1:123456789012:changeSet/samcli-deploy1641934511/763fe89c-9c6a-4cef-a1a6-90986d7decfd

2022-01-11 15:55:22 – Waiting for stack create/update to complete

CloudFormation events from stack operations
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
CREATE_IN_PROGRESS AWS::IAM::Role RefreshURLFunctionRole –
CREATE_IN_PROGRESS AWS::IAM::Role EventBridgeFunctionRole –
CREATE_IN_PROGRESS AWS::IAM::Role EventBridgeFunctionRole Resource creation Initiated
CREATE_IN_PROGRESS AWS::IAM::Role RefreshURLFunctionRole Resource creation Initiated
CREATE_COMPLETE AWS::IAM::Role EventBridgeFunctionRole –
CREATE_IN_PROGRESS AWS::Lambda::Function EventBridgeFunction –
CREATE_IN_PROGRESS AWS::Lambda::Function EventBridgeFunction Resource creation Initiated
CREATE_COMPLETE AWS::IAM::Role RefreshURLFunctionRole –
CREATE_COMPLETE AWS::Lambda::Function EventBridgeFunction –
CREATE_IN_PROGRESS AWS::Lambda::Function RefreshURLFunction –
CREATE_IN_PROGRESS AWS::Lambda::Function RefreshURLFunction Resource creation Initiated
CREATE_IN_PROGRESS AWS::Events::Rule SvcCatalogEventRule –
CREATE_IN_PROGRESS AWS::Events::Rule SvcCatalogEventRule Resource creation Initiated
CREATE_COMPLETE AWS::Lambda::Function RefreshURLFunction –
CREATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi –
CREATE_COMPLETE AWS::ApiGateway::RestApi ServerlessRestApi –
CREATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi Resource creation Initiated
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeploymentb762875163 –
CREATE_IN_PROGRESS AWS::Lambda::Permission EventBridgeFunctionHelloWorldPermissio –
CREATE_IN_PROGRESS AWS::Lambda::Permission RefreshURLFunctionHelloWorldPermission Resource creation Initiated
CREATE_IN_PROGRESS AWS::Lambda::Permission EventBridgeFunctionHelloWorldPermissio Resource creation Initiated
CREATE_IN_PROGRESS AWS::Lambda::Permission RefreshURLFunctionHelloWorldPermission –
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeploymentb762875163 Resource creation Initiated
CREATE_COMPLETE AWS::ApiGateway::Deployment ServerlessRestApiDeploymentb762875163 –
CREATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage –
CREATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage Resource creation Initiated
CREATE_COMPLETE AWS::Lambda::Permission RefreshURLFunctionHelloWorldPermission –
CREATE_COMPLETE AWS::Lambda::Permission EventBridgeFunctionHelloWorldPermissio –
CREATE_COMPLETE AWS::ApiGateway::Stage ServerlessRestApiProdStage –
CREATE_COMPLETE AWS::Events::Rule SvcCatalogEventRule –
CREATE_IN_PROGRESS AWS::Lambda::Permission PermissionForEventsToInvokeLambda –
CREATE_IN_PROGRESS AWS::Lambda::Permission PermissionForEventsToInvokeLambda Resource creation Initiated
CREATE_COMPLETE AWS::Lambda::Permission PermissionForEventsToInvokeLambda –
CREATE_COMPLETE AWS::CloudFormation::Stack sam-app –

CloudFormation outputs from deployed stack
Key RefreshURLFunctionIamRole
Description Implicit IAM Role created for Hello World function
Value arn:aws:lambda:us-east-1:123456789012:function:RefreshURLFunction

Key RefreshURLFunctionAPI
Description API Gateway endpoint URL for Prod stage for Hello World function

Key RefreshURLFunction
Description Hello World Lambda Function ARN
Value arn:aws:lambda:us-east-1:123456789012:function:RefreshURLFunction

Successfully created/updated stack – sam-app in us-east-1

Test the solution

Now that you have deployed the solution, let’s test the workflow.

On the AWS Service Catalog console, under Administration in the navigation pane, choose Portfolios.

Choose your SageMaker notebook.

Choose Launch product.

At the bottom of the page, choose Launch product.

You should see a page similar to the following screenshot.

Wait a few moments for the status to show as Available.

Open your terminal and run the following command to get the presigned URL from Parameter Store:

aws ssm get-parameter
–name “/SageMaker/Notebooks/Demo-User-Notebook”
–query Parameter.Value

The output should look like the following:


EventBridge rule

EventBridge is configured with an event rule to process an API response for the AWS Service Catalog API. This rule is configured to pass the notebook instance state so that you can use Lambda to return a presigned URL response as a triggered action. The event rule is configured as follows:

“detail-type”: [“AWS API Call via CloudTrail”],
“detail”: {
“eventSource”: [“”],
“eventName”: [“ProvisionProduct”]

The following screenshot of the EventBridge console shows your event rule.

The AWS CloudTrail API is being monitored using the event source for The monitored event name is ProvisionProduct. Monitoring this event allows you to take effective action in response to AWS Service Catalog reporting back the successful delivery state of the notebook instance. When a ProvisionProduct event occurs, a Lambda function called DemoEventBridgeFunction is invoked, which returns a presigned URL to the end-user.

Lambda function for returning presigned notebook instance URLs

To ensure secure access to user-requested notebooks via AWS Service Catalog, a presigned URL is created and returned to the user. This provides a secure method of accessing the notebook instance and performing business critical functions. For this purpose, we use the EventBridgeServiceCatalogFunction function, which uses a waiter for the notebook instance state to become available. Waiters provide a means of polling a service and suspending the execution of a task until a specific condition is met. When it’s ready, the function generates a presigned URL. Finally, the function creates an SSM parameter with the generated presigned URL. The SSM parameter uses the following pattern: /SageMaker/Notebooks/%s-Notebook”%user_name/. This allows us to create a common namespace for all our SageMaker notebook SSM parameters while keeping them unique based off of user_name.

Presigned URLs have a defined expiration. The Lambda function deploys notebooks with a session expiration of 12 hours. Because of this, developers need to generate a new presigned URL when their existing presigned URL expires. The RefreshURLFunction accomplishes this by allowing users to invoke the function from calling the API Gateway. Developers can invoke this function and pass their notebook name, and it returns a presigned URL. When the RefreshURLFunction is complete, a user can make a call to Parameter Store, get the new presigned URL, and then access their notebook.

Get the RefreshURLFunction API Gateway URL with the following code:

aws cloudformation describe-stacks
–stack-name sam-app
–query “Stacks[0].Outputs[?OutputKey==’RefreshURLFunctionAPI’].OutputValue”
–output text
–region us-east-1

The output should look like the following:

Invoke the function RefreshURLFunction by calling the API Gateway. Update input_url with the URL from the previous step:

curl -X POST <input_url> -d ‘{“notebook_user_name”: “Demo-User”}’

The output should look like the following:

{“PreSignedURL”: “”}%

Open a browser and navigate to the PreSignedURL from the previous step.

The webpage should look like the following screenshot.


In this post, we demonstrated how to deploy the infrastructure components for a SageMaker notebook instance environment using AWS CloudFormation. We then illustrated how to use EventBridge to return the notebook instance state from the AWS Service Catalog API. Lastly, we showed how to use a Lambda function to return the presigned notebook instance URL for accessing the delivered resource. For more information, see the Amazon SageMaker Developer Guide. Thank you for reading!

About the Authors

Joe Keating is a Senior Customer Delivery Architect in Professional Services at Amazon Web Services. He works with AWS customers to design and implement a variety of solutions in the AWS Cloud. Joe enjoys cooking with a glass or two of wine and achieving mediocrity on the golf course.

Matt Hedges is a Cloud Application Architect at Amazon Web Services. He works closely with customers to align technology needs with business drivers to deliver their applications on AWS. With a focus on migrations and modernization, Matt works with enterprise customers around the world to pioneer changes that unlock the full potential of the cloud. Matt enjoys spending time with family, playing musical instruments, cooking, playing video games, fixing old cars, and learning new things.

Virginia Chu is a Senior DevSecOps Architect in Professional Services at Amazon Web Services. She works with enterprise-scale customers around the globe to design and implement a variety of solutions in the AWS Cloud.

Read MoreAWS Machine Learning Blog



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments