SSL For RDS With Glue Python Job and AWS SDK For Pandas

AWS SDK For Pandas

Recently AWS renamed the AWS data wrangler python library to AWS SDK for Pandas. This is an AWS Professional Service open source python initiative that extends the power of Pandas library to AWS connecting DataFrames and AWS data related services.

import sys
import awswrangler as wr
import pandas as pd
con_postgresql = wr.postgresql.connect(connection=”My-RDS-PostgreSQL-Connection”)
con_postgresql.close()

Amazon RDS Ready — Encryption Requirements

The purpose of the Amazon Relational Database Service (RDS) Ready Program is to recognise AWS Partner products that support the use of Amazon RDS database as a backend for business applications deployed within a customer’s AWS account or provided as SaaS deployed in APN Partner’s AWS Account.

Attempting an SSL Connection From Glue To RDS

A connection in Glue is created to a RDS database that has rds.force_ssl set.

Glue Connection
Glue Connection OK

The Problem

Next we try an use that connection in a AWS Glue Python Job utilising the AWS SDK For Pandas.

import sys
import awswrangler as wr
import pandas as pd
con_postgresql = wr.postgresql.connect(connection=”My-RDS-PostgreSQL-Connection”)con_postgresql.close()
SSL Error
SSL Error
import sys
import awswrangler as wr
import pandas as pd
import ssl
import os
import urllib.request
def download_rds_root_ca(filename: str):
print(“Downloading RDS CA root cert…”)
urllib.request.urlretrieve(‘https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem', filename=filename)
print(“Downloaded RDS CA root cert.”)
def create_rds_ssl_context():
cafile = ‘/tmp/rds-ca-2019-root.pem’
download_rds_root_ca(cafile)
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS);
ssl_context.verify_mode = ssl.CERT_REQUIRED;
ssl_context.load_verify_locations(cafile=cafile, capath=None, cadata=None)
return ssl_context
print(“Connecting to RDS database…”)
rds_ssl_context = create_rds_ssl_context()
con_postgresql = wr.postgresql.connect(connection=”My-RDS-PostgreSQL-Connection”, ssl_context=rds_ssl_context)
print(“Successfully connected to RDS database.”)

Run With SSL

Running the job again with the correct SSL certificate in place we get a successful execution.

Job Run OK
Job Run Logs

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Matt Houghton

Matt Houghton

20 Followers

Data Architect @CDL_Software , AWS Community Builder, 13 x AWS Certified. Qlik Global Luminary 50.