When upgrading minor versions of RDS Multi-AZ clusters the connections are switched from the current writer to a newly upgraded reader.
Clients have the option to connect to either the cluster writer endpoint or the cluster reader endpoint. Normally, they would connect to the writer endpoint. This endpoint is directed to the current writer instance. When a restart of the cluster occurs due to either a change in the parameters of the cluster or a failover, one of the readers is promoted to be the new writer. When this occurs, the cluster writer endpoint is updated to point to the new writer by updating the instance endpoint in the DNS. Due to inherent delays in DNS propagation, this can take up to 30 seconds to resolve. Normally, the application would require logic to reconnect after the temporary lack of availability of the database.
The Advanced JDBC Wrapper Driver can automatically detect and switch the connection to the new writer. By utilizing topology information inside the RDS Multi-AZ database, the driver can switch over the connection in around one second or less when there is no replica lag.
In the post Introducing the Advanced JDBC Wrapper Driver for Amazon Aurora, we showed you how to use the driver to handle the failover of an Amazon Aurora cluster. In this post, we show how to use the AWS Advanced JDBC Wrapper Driver to do a reduce downtime to one second or less when doing a minor version upgrade on either Multi-AZ Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Multi-AZ Amazon RDS for MySQL with two readable standbys. We also discuss the new API introduced in Amazon RDS that enables this.
In order to achieve one second or less downtime you need to provision an RDS for PostgreSQL Multi-AZ DB cluster running at least PostgreSQL 15.4 with rds_tools extension version 1.4 or above. If you have an existing cluster on this version, upgrade to at least the R3 release. Additionally, you need to install the rds_tools extension using the following DDL:
RDS for PostgreSQL cluster metadata
To enable rapid switchovers, we’ve implemented a topology function within Amazon RDS for PostgreSQL. To access this function, make sure you have installed the latest rds_tools extension, with a version of 1.4 or higher, and meet the supported engine version requirements as outlined in the prerequisites. After successfully installing the rds_tools extension, you can inspect topology metadata with the following command:
Here’s a breakdown of each column:
id – Displays the DBI_RESOURCE_ID of each instance within an RDS Multi-AZ cluster
endpoint – This lists the Amazon Route 53 CNAME for each node in the cluster.
port – Displays the port numbers linked to each instance in the RDS Multi-AZ cluster
The parameter client_identifier in the show_topology function is optional but recommended. Passing a client identifier allows Amazon RDS to track which client libraries or proxies are actively querying the topology. This provides useful telemetry for Amazon RDS developers and helps guide support for additional clients in the future.
Additionally, the rds_tools extension provides the function rds_tools.multi_az_db_cluster_source_dbi_resource_id() to indicate the resource ID of the writer.
The Advanced JDBC Wrapper Driver uses this topology information to connect directly to all three instances within an RDS Multi-AZ cluster.
The new topology information is available in Amazon RDS for PostgreSQL versions 15.4, 14.9, 13.12, or higher. If you have an existing cluster on these versions, upgrade to at least the R3 release.
RDS for MySQL cluster metadata
Similar to Amazon RDS for PostgreSQL, there is a table in the mysql database called rds_topology with the same columns: id, endpoint and port.
The following SQL can be used to inspect the topology:
Using the Advanced JDBC Wrapper Driver in your application
The maven coordinates are as follows:
Using the driver is straightforward. The connection schema for Amazon RDS for PostgreSQL is as follows:
The connection schema for Amazon RDS for MySQL is as follows:
Some changes in your code are required to handle three conditions:
FailoverFailed – The connection to the new writer could not be established and the application will have to open a new connection
FailoverSuccess – The connection was reestablished and the application will have to set any session state that was added after the connection was opened
TransactionStateUnknown – The application was in the middle of a transaction when the failover occurred and the connection was reestablished but the transaction will have to be retried
It is important to note that when setting up the connection you modify the failoverClusterTopologyRefreshRate. This setting determines the rate at which the driver reads the topology, by default it is 2000ms. Setting it to 100ms is required to ensure discovery of the new writer as soon as possible
You can use the following code to test the fast switchover capabilities of the driver:
Let’s see how this code works. Below is an example of how to run the test and the output of the logs. The downtime is around 300ms in this case.
Create a test cluster using the following AWS CLI command:
Configure the properties with actual database user and password.
Replace connString with actual cluster endpoint.
Run the demo application above in an EC2 instance within the same VPC as RDS MultiAZ DB cluster
While the demo application is running, perform DB engine minor version upgrade via AWS console or AWS CLI
The demo application automatically reconnects to the new writer with downtime less than one second. For this test, the switchover took 331ms which you can see in the following logs:
Make sure you clean up your Multi-AZ DB Cluster you created if you do not plan on using them in the future.
Using the improved logic in the Advanced JDBC Wrapper Driver, you can improve the availability of writes from your application while performing a minor version upgrade on an RDS Multi-AZ cluster without having to worry about DNS propagation delays
Add this to your project using the Maven coordinates above and integrate the code into your project(s). Let us know how it works in the comments below.
About the Author
Dave Cramer is a Senior Software Engineer for Amazon Web Services. He is also a major contributor to PostgreSQL as the maintainer of the PostgreSQL JDBC driver. His passion is client interfaces and working with clients
Read MoreAWS Database Blog