How to Get Data To/From AWS Databases

Written by
CloverDX
CloverDX

Let’s begin with a hard truth. Many cloud providers, including AWS, make it their aim to get your data into their databases and keep it there. As far as Amazon are concerned, they’re the only cloud database provider on the market. Of course, this is far from the truth.

As such, migrating some or all of your data to an AWS database is actually quite simple – and there are plenty of databases to choose from.

Amazon offer transactional databases that capture and store data, as well as a range of data warehouses that can analyse and spot trends in large quantities of data. These databases make up AWS’s ‘collect-store-analyze’ ecosystem.

diagram of the AWS 'collect, store, analyze' databases ecosystem

But, although it’s easy enough to migrate your workloads to an AWS database, it’s a painful process attempting to get your data back or integrate it with your multi-cloud environment.

In the following blog, we’ll be laying out how you can better manage your AWS database import, store and retrieval journey.

Deliver a successful data project - get a demo

Step #1: Get your data to an AWS database

When migrating an existing small-medium database to or from an AWS database, use AWS Database Migration Service (DMS). This service allows you to move your existing data, full load, from its source point to the target database instance in a one-time migration.

Alternatively, you can import and export larger quantities of data using AWS Import/Export Disk or AWS Snowball.

Things become trickier once you need to upload daily batches of data regularly. Although AWS’s tooling is suitable for managing AWS data tasks, it doesn’t account for more universal jobs (i.e. uploading data from Azure or other competitor databases). As a result, your organization will need to orchestrate a process for daily uploads that isn’t AWS specific. This will usually involve using third-party ‘connectors’.

Step #2: Store your data

So, you’re planning a data migration to an AWS database. But which one do you choose?

We’ve listed some quick pros and cons of the main players:

AWS Database

Pros

Cons

S3

  • Developer friendly
  • Easy to integrate within an AWS ecosystem
  • Capable of storing large quantities of data
  • Decently priced
  • Downloading data is expensive
  • For further support, you’ll need to purchase the ‘AWS support plan’
  • Not easily integrated in a multi-cloud environment

Glacier

  • Built for housing large data backups cheaply ($0.01 per GB per month)
  • It’s secure and simple
  •  It’s expensive to retrieve your data

Aurora

  • It harnesses Amazon’s RDS database, a service that simplifies and automates time-consuming database management tasks
  • Provides continuous scalability
  • For data that requires write updates against tables, Aurora’s performance may not be to standard

RedShift

  • Fast performance
  • A similar interface to PostgreSQL
  • A range of integrated security features
  • Parallel processing/uploading is only supported by Amazon databases
  • If you’re writing data from distributed, non-AWS systems, you have to handle the process yourself

RDS

  • Automatic backups, security and management
  • High performance and capacity
  • Only compatible with Oracle or AWS relational databases

DynamoDB

  • Limitless scalability
  • Integrated monitoring and management
  • Predictable performance
  • Deployable only on AWS
  • Cannot manage complex queries
  • Expensive and low latency

 

Step #3: Get your data out

Now, here’s the tricky part. Once your data’s in, you’ll have to pry it from AWS’s hands to get it out.

Understandably, AWS don’t want you to migrate your data outside of their databases, nor do they make it easy to move your data between different databases. As such, extracting your data can be a difficult task.

For one-time large migrations, it’s likely you’ll need to move your data out slowly, piece-by-piece as to not incur large costs. Alternatively, for daily data chores and integrating your data between databases, you’ll need to find your own unique solution. This is a complex task that your data team might not be fit to perform.

Similarly to moving batches of data into AWS, you can simplify the process by using connectors that orchestrate your multi-cloud environment (and the data that moves in between your databases).

Info Sheet: CloverDX on AWS

Align your multi-cloud universe

Although getting your data to your AWS databases of choice is relatively simple, ongoing uploads, exports and orchestrations between different databases is difficult.

In order to negotiate your databases smoothly, you’ll need to find the right specialist data experts or invest in connectors that can piece together your multi-cloud ecosystem.

At CloverDX, we’re not interested in the ‘Azure vs. AWS’ argument, but rather how you can make your cloud databases work together in harmony. If you’d like to understand how to simplify your data migration processes, download our thorough guide.

New call-to-action

 

AWS
Posted on December 03, 2019
DISCOVER  CloverDX Data Integration Platform  Design, automate and operate data jobs at scale. Learn more

Related Content

Subscribe to our blog


Where to go next