AWS Data Engineer

Role: AWS Data Engineer

Location: Remote

What is in it for you?

As an AWS Data Engineer, you will be a part of an Agile team to build healthcare applications and implement new features while adhering to the best coding development standards

Responsibilities: -

1. AWS (Amazon Web Services):

• Core Services: Proficiency with core AWS services like EC2, S3, RDS, Lambda, and VPC.
• Data Services: Experience with AWS data services Glue and EMR.
• AWS DMS: Knowledge of AWS Database Migration Service (DMS) for migrating databases to AWS.
• CDC: Understanding of Change Data Capture (CDC) techniques to capture and replicate changes from source databases to target databases.
• Security: Understanding of AWS security best practices, IAM, and encryption.

2. Databricks:

• PySpark & Spark SQL: Strong analytical skills in PySpark & Spark SQL for big data processing and analysis.
• Delta Live Tables: Expertise in using Delta Live Tables for building reliable and scalable data pipelines.
• Notebooks: Strong utilization of Databricks Notebooks for data analysis.
• Workflows : Setting up and monitoring Databricks Workflows.
• Data Integration: Experience integrating Databricks with AWS services.

3. DevOps Principles:

• CI/CD Pipelines: CI/CD pipelines using Azure Pipelines.
• Version Control: Proficiency with Azure Repos and Git for version control.
• Automation: Scripting and automation using PowerShell, Bash, or Python. Automating the build, test, and deployment processes

4. Infrastructure as Code (IaC):

• Terraform: Experience with Terraform for managing AWS and Azure infrastructure.

5. On Prem integration with AWS

• Integrating on prem data with AWS and Databricks.
• Thoroughly test and validate the data to ensure it has been transferred correctly and is fully functional.

6. Optimization and Monitoring:

• Optimize AWS services and Databricks for performance and cost-efficiency.
• Proficiency in setting up monitoring and logging using tools like AWS CloudWatch to track the performance and health of the complete data flow.