AWS Athena
The Amazon Web Services (AWS) Athena data-source connector allows you to sync data to AWS. The AWS Athena data-source connector does the following:
- writes data as JSON files in AWS S3
- makes data available through a Lake Formation Governed Table in the Glue Data Catalog so that the data is available throughout other AWS services, such as EMR and Redshift.
Prerequisites
To use this connector, you will need the following:
-
An AWS account.
-
An S3 bucket where the data will be written.
-
An AWS Lake Formation database where tables will be created (one per stream)
-
AWS credentials:
- Access Key Id
- Secret Access Key
🐕🦺 Setup guide
⚙️ Step 1: create an Athena data source in Zenskar
Set up data source
- Log into your Zenskar account.
- In the left navigation bar, click Metering > Data Sources. In the top-right corner, click + ADD DATA SOURCE.
- In the Set Up Source section of the Add New Data Source page, enter a name for the Athena data-source connection.
- Select Athena from the Source Type dropdown.
Configure data source
In the Source Config section of the Add New Data Source page, add the following details:
- Database Name: the database in which the tables will be created. You will find the instructions to create a new Lake Formation database here.
- Bucket: the AWS S3 bucket in which the data will be written. You will find the instructions to create a new S3 bucket here.
- AWS Region: the region in which your resources are deployed.
- AWS Access Key Id: You will find the instructions to create a new user here. Select Programmatic Access to generate Access Key Id-AWS Secret Access Key pair.
- AWS Secret Access Key
Assigning proper permissions
The policy used by the user or the role must have access to the following services:
- AWS Lake Formation
- AWS Glue
- AWS S3
You can use the AWS policy generator to help you generate an appropriate policy.
Ensure that the role or user you will use has appropriate permissions on the database in AWS Lake Formation. You will find more information about Lake Formation permissions in the AWS Lake Formation Developer Guide.
Supported sync modes
Feature | Supported? |
---|---|
Full Refresh Sync | Yes |
Incremental - Append Sync | Yes |
Namespaces | No |
Data type map
The Glue tables will be created with schema information provided by the source, i.e : You will find the same columns and types in the destination table as in the source except for the following types which will be translated for compatibility with the Glue Data Catalog:
Type in the source | Type in the destination |
---|---|
number | float |
integer | int |
Updated about 1 month ago