Data Engineer Python, AWS

7 days ago


Chennai, Tamil Nadu, India Virtusa Full time
Data Engineer Python, AWS - CREQ188952 Description Skillset required for this project : Python, AWS, Starbust, Neo4j

Roles and Responsibilities :
Data Engineer Role

Data Pipeline Development:

Design, build, and maintain scalable data pipelines in AWS Glue using Python code language to ingest, transform, and load data from sources (AWS S3 bucket) into data lakes (AWS S3 bucket).


Data Integration:

Integrate data from multiple sources and systems (AWS S3 bucket) to enable unified and comprehensive views of clinical data.

Implement ETL (Extract, Transform, Load) processes to prepare raw data for analysis.

Database Management:
Manage and optimize databases (, Starbust) for performance, scalability, and reliability.
Implement database schemas and indexes to support efficient data querying and retrieval.

Data Governance and Security:
Implement data governance policies and procedures to ensure data security, compliance, and privacy using AZ GxP process.
Establish access controls, encryption, and auditing mechanisms to protect sensitive data.

Monitoring and Optimization:
Monitor data pipelines and systems for performance issues, bottlenecks, and anomalies.
Implement optimizations to improve data processing efficiency and reduce latency.

Collaboration and Documentation:
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and support analytics initiatives.
Document data engineering processes, workflows, and architectures for knowledge sharing and future reference.
Collaborate with testers while testing the developed pipeline and after which promoting to higher environment.

Deployment and Jira Test Execution:
Deploying the code from Github repository from the master branch to higher environments like SIT, PPT and PROD.
While deploying we capture evidence and execute the deployment scripts with different steps and corresponding evidence in Jira.

Automation:
Automation to trigger the code automatically using AWS resources without manual intervention.
Primary Location Chennai, Tamil Nadu, India Job Type Experienced Primary Skills IT Service Management, AWS Native Data Services, Data Modelling Years of Experience 9 Travel No
  • AWS Data Engineer

    4 weeks ago


    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).-...

  • AWS Data Engineer

    3 weeks ago


    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).-...

  • AWS Data Engineer

    2 months ago


    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • AWS Data Engineer

    3 weeks ago


    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...


  • Chennai, Tamil Nadu, India Cortex Consulting Pvt. Ltd. Full time

    Job Description :We are seeking a highly experienced and accomplished Senior AWS Data Engineer to join our growing team in Chennai. You will play a pivotal role in designing, developing, and managing large-scale data pipelines on the AWS cloud platform, leveraging your expertise in Python and data engineering best practices. You will be responsible for the...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns Work closely with DevOps engineer to support...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India RELX Full time

    About the role The role in focus involves designing and implementing robust data pipelines utilizing AWS, Python, and SQL to underpin product analytics and decision-making. As a data engineer, you'll ensure the integrity and performance of data processing and storage systems, optimize data flows within AWS for enhanced performance, and develop efficient...

  • Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India NTT DATA Full time

    Job Description Req ID: NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Chennai, Tamil Nādu (IN-TN), India (IN).Data EngineerJob SummaryWe...

  • Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India NTT DATA Services Full time

    Req ID:282638NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Chennai, Tamil Nādu (IN-TN), India (IN).Data EngineerJob SummaryWe are looking...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India LexisNexis Full time

    Software Engineer IIIDo you enjoy designing analytic solutions at an enterprise scale for global customers?Do you have experience in working on complex problems?About our TeamOur product analytics team focuses on using data to understand and enhance the user experience of our customer facing products. We gather and analyze user interaction data from various...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India HTC Global Services Full time

    Exp : 5+ yearsJob Summary:We are seeking a Senior Developer to design, develop, and maintain scalable and efficient cloud-based solutions. The ideal candidate will have a strong background in AWS CDK, Python, and SQL, with additional experience in data migration and AWS Glue considered a plus.Key Responsibilities:Design and implement cloud infrastructure...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India LexisNexis Full time

    Software Engineer III Do you enjoy designing analytic solutions at an enterprise scale for global customers? Do you have experience in working on complex problems? About our Team Our product analytics team focuses on using data to understand and enhance the user experience of our customer facing products. We gather and analyze user interaction data...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India RELX Full time

    Software Engineer III Do you enjoy designing analytic solutions at an enterprise scale for global customers? Do you have experience in working on complex problems? About our Team Our product analytics team focuses on using data to understand and enhance the user experience of our customer facing products. We gather and analyze user...

  • Data Engineer

    2 months ago


    Bangalore/Chennai, Tamil Nadu, India Notus Full time

    AWS Data EngineerJob Title: AWS Data EngineerLocation: Bangalore / ChennaiExperience: 8+ YearsJob Description:We are seeking an experienced AWS Data Engineer to join our dynamic team. The ideal candidate will be proficient in designing, implementing, and maintaining data solutions on AWS cloud platform. This role requires expertise in AWS services, Kafka,...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :Join Our Team as a Data Engineer at XYZ Company!We are looking for a skilled Data Engineer to join our team. If you have a Bachelor's degree in Computer Science, Information Technology, or a related field, along with at least 4 years of experience in data engineering, this might be the perfect opportunity for you! Bachelor's degree in...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India Manvision Technologies Full time

    Role Summary: Senior Data EngineerPune/Bangalore/ChennaiAbout the RoleAs a Senior Data Engineer, we are looking for someone passionate about working as part of a team to deliver high-quality, professional dashboards that have a business impact.Experience in years:At least 5 years of hands-on experience using AWS to deliver complex data pipelines for large or...

  • AWS Data Engineer

    7 days ago


    Chennai, Tamil Nadu, India CliqHR Full time

    JOB TITLE: Data Engineer at Innovative Tech SolutionsRequired Skills: AWS, Python, SQL, Spark, Airflow, SnowflakeJob Description: Create and manage cloud resources in AWS Data ingestion from various sources like RDBMS, REST HTTP API, flat files, Streams, and Time series data. Implement data ingestion and processing using Big Data technologies Data...


  • Chennai, Tamil Nadu, India Virtusa Full time

    Data Engineer(Pyspark, AWS and SQL) - CREQ186019 Description 4-6 years work experience in Software Engineering especially in professional software product development.Expertise in AWS and python coding.Flexible to learn and work for any programming language based on need.Proficient in pipeline for file loading from multiple different sources having different...

  • AWS, Python

    7 days ago


    Chennai, Tamil Nadu, India Fulcrum Digital Full time

    Job DescriptionWho are weFulcrum Digital is an agile and next-generation digital accelerating company providing digital transformation and technology services right from ideation to implementation. These services have applicability across a variety of industries, including banking & financial services, insurance, retail, higher education, food, healthcare,...

  • AWS Data Engineer

    2 months ago


    Chennai, Tamil Nadu, India CliqHR Full time

    Mandatory Skills : AWS, Python, SQL, spark, Airflow, SnowflakeResponsibilities :- Create and manage cloud resources in AWS- Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and...