One of the steps in migrating a database to the cloud involves transferring existing data from the source (on premises) to the target (cloud). For medium and large databases, such as those ranging from hundreds of gigabytes to 5 terabytes, speed of data transfer matters. One of the challenges is to minimize the downtime of the application during migration. When migrating databases supporting critical business processes, the impact of delayed data transfer can be significant.
AWS provides a variety of data transfer mechanisms, such as AWS Snow Family, AWS Storage Gateway, and AWS DataSync, to aid in cloud migration. As an alternative, you can achieve efficient transfer of your data during migration using Tsunami UDP, an open-source and free software that is simple to install and use. Tsunami UDP achieves high-speed file transfer by combining UDP and TCP protocols, making it optimal for moving data across long-distance networks. Tsunami UDP achieves more throughput than most TCP based tools like SCP and SFTP by using UDP to transfer data at high speeds and TCP to maintain communication.
In this post, we show you how to use Tsunami UDP to speed up the migration of a large Oracle database to Amazon RDS Custom for Oracle. You can apply the steps in this post to a managed service such as Amazon RDS Custom for Oracle or a self-managed Oracle database on Amazon Elastic Compute Cloud (Amazon EC2).
Solution walkthrough
The following diagram illustrates the solution architecture.
This solution installs Tsunami UDP on the source and destination database servers to enable data transfer. To simulate data transfer over a long-distance network, we use a source RDS Custom for Oracle database running in us-west-2 to represent our on-premises database, and an RDS Custom for Oracle database running in us-east-1 as the destination.
The high-level steps are as follows:
Install Tsunami UDP on the source (on-premises) database instance and on the destination RDS Custom for Oracle DB instance.
Create Oracle database backups on the source RDS Custom for Oracle instance.
Transfer backup files from source to destination using Tsunami UDP.
Restore the transferred backup file on the destination instance.
Prerequisites
For this walkthrough, the following prerequisites are necessary:
Background knowledge about Oracle backup and restore
An active AWS account
Secure network connectivity between source and destination, for example AWS Direct Connect or AWS VPN
A source Oracle database instance running on an on-premises environment or Amazon EC2, or an RDS Custom for Oracle instance. See AWS documentation for Setting up your environment for Amazon RDS Custom for Oracle.
Amazon RDS Custom for Oracle or Oracle on Amazon EC2 as the destination
Git command to download the Tsunami UDP source code, if not already installed. If using RDS Custom, this already comes preinstalled
Tsunami UDP software
Make command to compile Tsunami UDP from source code. If using RDS Custom, if not already installed
Install Tsunami UDP on the source and destination Oracle servers
Tsunami UDP binaries are not distributed; only the source code is available and should be compiled on the source and destination server. The binaries consume 10 MB of storage space. You don’t need root privileges to compile and run Tsunami UDP; a normal user account is sufficient. Make sure that the Tsunami user has permissions to read the file so it can be transferred.
As a best practice, install third-party software in new directories under the /rdsdbata mount point. Doing so ensures that your software is protected since /rdsdbata is included in RDS automated and manual snapshots. If installing TsunamiUDP as a non-root user, you need an admin user to create the folder and make your operating system user its owner.
Follow these steps to install Tsunami UDP – first on the server hosting your source Oracle database server, and then on the destination Amazon RDS Custom for Oracle.
Download the source code
Use the following Git command to download the source code:
Alternatively, download the source code directly from the Tsunami UDP repository source. You will see an output similar to the following code when you download Tsunami UDP.
Compile and install Tsunami UDP software
Once downloaded, install the Tsunami UDP software by issuing the make command in the directory holding the source You will receive an output similar to the following.
You can add the Tsunami UDP server and client to the path in the profile of your operating system user to avoid having to specify the whole path when executing the programs to transfer files.
Do not forget that the above steps need to be followed on both source and destination servers.
Configure network connectivity between source and destination
On both servers, open TCP ports 22 and 46224, and UDP port 46224 to allow seamless flow of Tsunami UDP traffic.
In this example, we open the required TCP and UDP ports by configuring security group rules at the source and destination. Work with your network administrator to configure the required rules if you are following this walkthrough using an on-premises source.
Create an Oracle database backup on the source
We used the following Oracle Data Pump command to unload data and metadata from our sample database into an operating system dump file:
The following screenshot shows part of the output of the preceding command, with a successful Data Pump export.
In this example, the size of the export dump backup file is 463 GB.
Transfer the files using Tsunami UDP
Tsunami UDP uses a two-step process to transfer files. The first step is to start the Tsunami server on the source server using the tsunamid command. The second step is to start the Tsunami client on the destination using the tsunami command.
To transfer the files, complete the following steps:
On the source server, identify the files to transfer and start the Tsunami server. In this example, we transfer the 463 GB source.dmp dump file.
Tsunami UDP allows you to transfer one or more files with a single command. For a detailed guide of Tsunami UDP’s command, refer to the following usage guide.
Navigate to the path where the export dump files are located and start the Tsunami UDP server process. In our example, we transfer one backup file with the following command:
The default behavior is for the client not to be asked for a password when connecting to the server. This can be overwritten by specifying the –secret option when starting the Tsunami UDP server. For our example, the command would be:
Read MoreAWS Database Blog
