How do you put in a Python package deal in your setting in case you don’t have any web entry? I just lately got here throughout this concern when creating an AWS Sagemaker Studio setting for my workforce on AWS.

Constructing an AWS personal setting for Sagemaker

For this specific venture, I arrange Sagemaker in VPC Solely mode with the constraint of protecting the structure personal, which implies making a VPC and personal subnets, however no entry to the web.

So all community communications, together with software communication with AWS APIs, should undergo VPC Endpoint interfaces. This permits for protecting connection secured as knowledge despatched and acquired won’t ever undergo the web utilizing the AWS community spine as an alternative.

It’s significantly suited to limiting publicity to safety dangers, extra significantly once you’re processing private info, or should adjust to some safety requirements.

Accessing the Pypi package deal repository from AWS Sagemaker

In my workforce, Knowledge Scientists use Python as a major language and generally want Python packages that aren’t offered in Sagemaker’s pre-built Python pictures, so I’ll concentrate on this use case. Fortuitously, the answer can be working for different languages and repositories like npm.

Your customers will sometimes attempt to set up no matter package deal they want through pip command. However, as no web entry is allowed, this command will fail as a result of pip received’t be capable to contact Pypi.org servers.

Opening web

One possibility is to open entry to the web and permit outbound HTTP connections to Fastly CDN IPs utilized by Pypi.org servers. However, this isn’t viable in our case as we don’t need any web connection within the structure.

Utilizing a devoted Pypi server

AWS weblog additionally gives an instance of utilizing a Python package deal named Bandersnatch. This text describes the right way to arrange a server, appearing like a bastion host, which can mirror Pypi and might be accessible solely to your personal subnets.

This isn’t a viable possibility as you’ve to know upfront which Python packages it’s good to present, and also you’ll someway should create public subnets and provides the Pypi server mirror entry to the web.

Utilizing AWS Cordeartifact

That is finally the answer I got here up with and which works in my case.

AWS Codeartifact is the artifact administration resolution offered by AWS. It’s appropriate with different AWS companies like AWS Service Catalog to manage entry to sources inside a company.

To make use of it, you’ll should create a “area” which serves as an umbrella to handle entry and apply insurance policies throughout your group. Then, you’ll should create a repository that may serve your artifacts to your totally different purposes.

Additionally, one repository can have upstream repositories. So, if a Python package deal shouldn’t be out there within the goal repository, the demand might be transmitted to the upstream repository to be fulfilled.

Extra exactly, this workflow takes under consideration package deal variations. Official documentation gives an in depth workflow:

If my_repo accommodates the requested package deal model, it’s returned to the consumer.

If my_repo doesn’t include the requested package deal model, CodeArtifact seems for it in my_repo‘s upstream repositories. If the package deal model is discovered, a reference to it’s copied to my_repo, and the package deal model is returned to the consumer.

If neither my_repo nor its upstream repositories include the package deal model, an HTTP 404 Not Discovered response is returned to the consumer.

Cool proper? It can even cache the package deal model for future requests.

That is exactly the technique we’re going to use, as AWS Codeartifact permits us to outline a repository that has an exterior connection like Pypi as an upstream repository.

Creating AWS Codeartifact sources with Terraform

As AWS Codeartifact is an AWS service, you may simply create a VPC endpoint in your setting VPC to connect with it.

Observe: I’m utilizing Terraform v1.6.4 and aws supplier v5.38.0

locals {
area = "us-east-1"
}
useful resource "aws_security_group" "vpce_sg" {
identify        = "AllowTLS"
description = "Enable TLS inbound visitors and all outbound visitors"
vpc_id      = aws_vpc.your_vpc.id
tags = {
Identify = "allow_tls_for_vpce"
}
}
useful resource "aws_vpc_security_group_ingress_rule" "allow_tls_ipv4" {
security_group_id = aws_security_group.allow_tls.id
cidr_ipv4         = aws_vpc.your_vpc.cidr_block
from_port         = 443
ip_protocol       = "tcp"
to_port           = 443
}
knowledge "aws_iam_policy_document" "codeartifact_vpce_base_policy" {
assertion {
sid    = "EnableRoles"
impact = "Enable"
actions = [
"codeartifact:GetAuthorizationToken",
"codeartifact:GetRepositoryEndpoint",
"codeartifact:ReadFromRepository",
"sts:GetServiceBearerToken"
]
sources = [
"*",
]
principals {
kind = "AWS"
identifiers = [
aws_iam_role.your_sagemaker_execution_role.arn
]
}
}
}
useful resource "aws_vpc_endpoint" "codeartifact_api_vpce" {
vpc_id            = aws_vpc.your_vpc.id
service_name      = "com.amazonaws.${native.area}.codeartifact.api"
vpc_endpoint_type = "Interface"
subnet_ids        = aws_subnets.your_private_subnets.ids
security_group_ids = [
aws_security_group.vpce_sg.id,
]
private_dns_enabled = true
coverage              = knowledge.aws_iam_policy_document.codeartifact_vpce_base_policy.json
tags = { Identify = "codeartifact-api-vpc-endpoint" }
}

Then, you’ll should create the totally different sources wanted for Codeartifact to deal with your requests for brand spanking new Python packages by mirroring Pypi: a site, a Pypi repository with an exterior connection, and a repository that defines Pypi as an upstream repository.

useful resource "aws_codeartifact_domain" "my_domain" {
area = "my-domain"
encryption_key = ""
tags = { Identify = "my-codeartifact-domain" }
}
useful resource "aws_codeartifact_repository" "public_pypi" {
repository = "pypi-store"
area     = aws_codeartifact_domain.my_domain.area
external_connections {
external_connection_name = "public:pypi"
}
tags = { Identify = "pypi-store-repository" }
}
useful resource "aws_codeartifact_repository" "my_repository" {
repository = "my_repository"
area     = aws_codeartifact_domain.my_domain.area
upstream {
repository_name = aws_codeartifact_repository.public_pypi.repository
}
tags = { Identify = "my-codeartifact-repository" }
}
knowledge "aws_iam_policy_document" "my_repository_policy_document" {
assertion {
impact = "Enable"
principals {
kind        = "AWS"
identifiers = [aws_iam_role.your_sagemaker_execution_role.arn]
}
actions   = ["codeartifact:ReadFromRepository"]
sources = [aws_codeartifact_repository.my_repository.arn]
}
}
useful resource "aws_codeartifact_repository_permissions_policy" "my_repository_policy" {
repository      = aws_codeartifact_repository.my_repository.repository
area          = aws_codeartifact_domain.my_domain.area
policy_document = knowledge.aws_iam_policy_document.my_repository_policy_document.json
}

Right here it’s! Now you can arrange a Pypi mirror on your personal setting simply.

To make issues usable, you’ll even have to inform pip instructions to direct requests to a selected index. Fortuitously, AWS created an API to do the heavy lifting for you. Simply add this to your code to make it work:

aws codeartifact login --tool pip --repository $CODE_ARTIFACT_REPOSITOR_ARN --domain $CODE_ARTIFACT_DOMAIN_ID --domain-owner $ACCOUNT_ID --region $REGION

Final however not least, add a VPC Endpoint for AWS Codeartifact in your VPC.

knowledge "aws_iam_policy_document" "codeartifact_vpce_base_policy" {
assertion {
sid    = "EnableRoles"
impact = "Enable"
actions = [
"codeartifact:GetAuthorizationToken",
"codeartifact:GetRepositoryEndpoint",
"codeartifact:ReadFromRepository",
"sts:GetServiceBearerToken"
]
sources = [
"*",
]
principals {
kind = "AWS"
identifiers = [
aws_iam_role.your_sagemaker_execution_role.arn
]
}
}
}
useful resource "aws_vpc_endpoint" "codeartifact_api_vpce" {
vpc_id            = aws_vpc.your_vpc.id
service_name      = "com.amazonaws.${native.area}.codeartifact.api"
vpc_endpoint_type = "Interface"
subnet_ids        = aws_subnets.your_private_subnets.ids
security_group_ids = [
aws_security_group.vpce_sg.id,
]
private_dns_enabled = true
coverage              = knowledge.aws_iam_policy_document.codeartifact_vpce_base_policy.json
tags = { Identify = "codeartifact-api-vpc-endpoint" }
}

If you want to obtain notifications for my upcoming posts relating to AWS and extra, please subscribe right here.

Do you know you may clap a number of instances?