AWS Senior Data Engineer

5 days ago

Vizag, Andhra Pradesh, India InfoCepts Full time

Key Result Areas and Activities:

  • Study existing technology landscape and understand current data integration framework and do impact assessment for the requirements.
  • Develop spark jobs using Scala for new project requirements.
  • Enhance existing spark jobs for any ongoing product enhancement.
  • Performance tuning of spark jobs, stress testing etc.
  • Create new data pipelines for developed/enhanced spark jobs using AWS Lambda or Apache Airflow
  • Responsible for database design process, logical design, physical design, star schema, snowflake schema etc.
  • Analyze data processing, integration, modelling and reporting requirements & Define data loading strategies considering volume, data types, frequency and analytics specifications.
  • Ensure optimal balance between cost and performance.
  • Project documentation, Adheres to Quality guidelines & Schedules.
  • Works hand in hand with PM for successful delivery of project and provide Estimation, scoping, scheduling assistance.
  • Manage build phase and quality assure code to ensure fulfilling requirements and adhering to Cloud architecture. Resolve difficult design and develop issues.

Work and Technical Experience:

Must-Have:

  • Overall, 7-9 years of IT Exp
  • 5+ years of AWS related project
  • Good to have Associate Level and Professional Level AWS Certification
  • In depth knowledge of following AWS Services required: S3, EC2, EMR, Severless, Athena, AWS Glue, Lambda, Step Functions
  • Cloud Databases (Must have) – AWS Aurora, Singlestore, RedShift, Snowflake
  • Big Data (Must have) - Hadoop, Hive, Spark, YARN
  • Programming Language (Must have) – Scala, Python, Shell Scripts, PySpark
  • Operating System (Must have) - Any flavor of Linux, Windows
  • Must have very strong SQL Skills
  • Orchestration Tools (Must have): Apache Airflow
  • Expertise in developing ETL workflows comprising complex transformations like SCD, deduplications, aggregations etc.
  • Should have thorough conceptual understanding of AWS VPC, Subnets, Security Groups, & Route Tables
  • Should be a quick and self-learner and be ready to adapt to new AWS Services or new Big Data Technologies as and when required

Qualifications:

  • Bachelor's degree in computer science, engineering, or related field (Master's degree is a plus)
  • Demonstrated continued learning through one or more technical certifications or related methods
  • Minimum 5 years of experience on Cloud related projects

Qualities:

  • Hold strong technical knowledge and experience
  • Should have the capability to deep dive and research in various technical related fields
  • Self-motivated and focused on delivering outcomes for a fast-growing team and firm
  • Able to communicate persuasively through speaking, writing, and client presentations
  • Able to consult, write, and present persuasively
  • Able to work in a self-organized and cross-functional team
  • Able to iterate based on new information, peer reviews, and feedback
  • Prior experience of working in a large media company would be added advantage

  • AWS Data Engineer Position

    2 days ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Key Data Engineer RoleAs a highly skilled and experienced data engineer, you will be responsible for designing, developing, and implementing end-to-end data engineering solutions on the AWS platform.You will design and implement scalable databases to optimize performance and ensure data consistency, throughput, and storage efficiency.Collaborate with...

  • AWS Data Engineering Specialist

    2 days ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 1,60,00,000 - ₹ 2,20,00,000

    Dear Candidate,We are seeking an experienced professional to lead our Data Engineering efforts.The ideal candidate will have expertise in designing, developing, and maintaining robust ETL/ELT pipelines on AWS using Spark and PySpark.Key Responsibilities:Design and implement scalable data pipelinesOptimize large-scale data processing workflows for performance...

  • AWS Data Engineer

    2 days ago

    Vizag, Andhra Pradesh, India beBeeDataEngineer Full time US$ 1,50,000 - US$ 2,00,000

    As a key member of our team, you will be responsible for designing and developing data pipelines and applications using AWS services.Key responsibilities include:Designing and implementing data pipelines to process and transform large datasets.Developing and deploying data applications using AWS services such as Sagemaker.Supporting the development and...

  • AWS Data Integration Specialist

    1 week ago

    Vizag, Andhra Pradesh, India beBeeDataIntegration Full time US$ 12,00,000 - US$ 15,00,000

    Job TitleSenior AWS Glue Developer is responsible for designing and implementing data ingestion and transformation solutions using serverless AWS services.Role SummaryThis position involves working on cutting-edge data integration projects, leveraging the power of AWS to drive business growth through actionable insights. You will also introduce...

  • Data Engineering Lead

    4 days ago

    Vizag, Andhra Pradesh, India beBeeEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior Data Engineer">The primary objective of this role is to spearhead data engineering initiatives, laying the foundation for data-driven projects. To succeed in this position, you must possess a strong technical skillset and demonstrate leadership abilities.">This involves leading best practices in design, development, and implementation while...

  • Senior PostgreSQL Data Engineer

    7 days ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Title:PostgreSQL Data ArchitectJob DescriptionWe are seeking a seasoned PostgreSQL data architect to design and implement scalable data solutions.Key ResponsibilitiesDesign, build, and maintain a robust PostgreSQL-based data warehouse for analytics and reporting.Develop and manage data pipelines (batch and streaming) for ingestion, transformation, and...

  • Senior Data Engineering Specialist

    2 weeks ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data Engineering SpecialistKey Responsibilities:Design and develop high-performance data pipelines and ETL processes on Databricks.Lead the architecture and implementation of scalable, secure, and efficient data solutions.Collaborate with stakeholders to gather requirements and deliver actionable data insights.Optimize data workflows for...

  • Chief Data Engineering Specialist

    1 week ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,40,00,000

    Senior Databricks Data Engineer Position We are seeking an experienced Senior Databricks Data Engineer to join our team and play a key role in designing, developing, and maintaining scalable and high-performance data pipelines using Databricks and PySpark. Key Responsibilities: Design and develop efficient data pipelines that cater to the organization's data...

  • Senior Data Engineering Specialist

    2 days ago

    Vizag, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior Data Engineering SpecialistSummaryA highly experienced professional with 6+ years of expertise in development and maintenance in data engineering projects.Key Responsibilities:Collaborate with stakeholders to understand data requirements and translate them into technical solutions, ensuring seamless integration and optimal...

  • Leading Cloud Data Engineer

    1 week ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Title: Senior Data EngineerWe are seeking a skilled data engineer with expertise in cloud services and big data technologies to design scalable, high-performance data pipelines.Main Responsibilities:Develop and maintain data pipelines using AWS and Databricks.Create ETL/ELT processes for large-scale datasets.Apply Databricks for advanced analytics and...