AWS Data Engineer with Apache Iceberg Expertise

Location: Bengaluru Experience: 6 - 12 years of Experience Role Type: Banking Mode of Work: Hybrid

6/26/20241 min read

Job Description:

We are seeking a seasoned AWS Data Engineer with experience in OpenTable formats (e.g. Apache Iceberg), to join our team. The successful candidate will be responsible for designing and implementing scalable and efficient data storage and retrieval systems, optimizing data lake performance, and ensuring the smooth handling of our data workloads on AWS.

Key Responsibilities:

  • Design, develop, and maintain scalable data pipelines and architectures leveraging AWS cloud services and Apache Iceberg.

  • Implement and manage data lake solutions using Apache Iceberg to handle time-travel, schema evolution, and transaction support.

  • Optimize data storage and querying by integrating Iceberg tables into AWS services like S3, Glue, Redshift, and EMR.

  • Work closely with data scientists and analysts to ensure seamless data access and efficient data processing workflows.

  • Monitor, troubleshoot, and optimize performance of data solutions, managing scalability and cost-effectiveness.

  • Keep up to date with the latest AWS offerings and Apache Iceberg features, continuously improving our data infrastructure.

  • Develop best practices for data governance, security, and compliance within our AWS and Iceberg environments.

  • Document system designs, architectures, and operational manuals for data solutions.

Qualifications:

  • Extensive experience with AWS data services and a strong understanding of OpenTable formats, specifically Apache Iceberg.

  • Proven track record in building and optimizing data lakes and data warehouse solutions.

  • Proficiency in working with big data processing tools and frameworks.

  • Strong coding skills in languages such as Python, Java, or Scala.

  • Experience with infrastructure as code (IaC), such as AWS CloudFormation or Terraform.

  • Familiarity with data orchestration tools like Apache Airflow.

  • Excellent analytical, problem-solving, and teamwork skills.

  • Strong communication abilities for both technical and non-technical audiences.

Join our innovative team to advance your career in AWS data engineering and play a pivotal role in shaping our next-generation data platform.