Data Engineer Python, AWS

2 weeks ago

chennai, India Virtusa Full time
Data Engineer Python, AWS - CREQ188952 Description Skillset required for this project : Python, AWS, Starbust, Neo4j
Roles and Responsibilities : Data Engineer Role
1. Data Pipeline Development:
Design, build, and maintain scalable data pipelines in AWS Glue using Python code language to ingest, transform, and load data from sources (AWS S3 bucket) into data lakes (AWS S3 bucket).
2. Data Integration:
Integrate data from multiple sources and systems (AWS S3 bucket) to enable unified and comprehensive views of clinical data.
Implement ETL (Extract, Transform, Load) processes to prepare raw data for analysis.
3. Database Management:
Manage and optimize databases (, Starbust) for performance, scalability, and reliability.
Implement database schemas and indexes to support efficient data querying and retrieval.
4. Data Governance and Security:
Implement data governance policies and procedures to ensure data security, compliance, and privacy using AZ GxP process.
Establish access controls, encryption, and auditing mechanisms to protect sensitive data.
5. Monitoring and Optimization:
Monitor data pipelines and systems for performance issues, bottlenecks, and anomalies.
Implement optimizations to improve data processing efficiency and reduce latency.
6. Collaboration and Documentation:
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and support analytics initiatives.
Document data engineering processes, workflows, and architectures for knowledge sharing and future reference.
Collaborate with testers while testing the developed pipeline and after which promoting to higher environment.
7. Deployment and Jira Test Execution:
Deploying the code from Github repository from the master branch to higher environments like SIT, PPT and PROD.
While deploying we capture evidence and execute the deployment scripts with different steps and corresponding evidence in Jira.
8. Automation:
Automation to trigger the code automatically using AWS resources without manual intervention.
Primary Location Chennai, Tamil Nadu, India Job Type Experienced Primary Skills IT Service Management, AWS Native Data Services, Data Modelling Years of Experience 9 Travel No

  • Data Engineer Python, AWS

    18 hours ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Data Engineer Python, AWS - CREQ188952 Description Skillset required for this project : Python, AWS, Starbust, Neo4jRoles and Responsibilities : Data Engineer RoleData Pipeline Development: Design, build, and maintain scalable data pipelines in AWS Glue using Python code language to ingest, transform, and load data from sources (AWS S3 bucket) into data...

  • AWS Data Engineer

    2 weeks ago

    Chennai, India CliqHR Full time

    Job Description :- AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).-...

  • AWS Data Engineer

    4 weeks ago

    Chennai, India CliqHR Full time

    Job Description :- AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).-...

  • AWS Data Engineer

    2 weeks ago

    Chennai, India CliqHR Full time

    Job Description : - AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).-...

  • AWS Data Engineer

    5 days ago

    chennai, India CliqHR Full time

    Job Description : - AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).- Proficient...

  • AWS Data Engineer

    3 weeks ago

    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).-...

  • AWS Data Engineer

    4 weeks ago

    Chennai, India CliqHR Full time

    Job Description : - AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).-...

  • AWS Data Engineer

    4 weeks ago

    chennai, India CliqHR Full time

    Job Description : - AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).- Proficient...

  • AWS Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- AWS, Python, SQL, Spark (with and with snowflake and databricks) AWS DE- AWS, Python, SQL, Spark- Bachelor's degree in Computer Science, Information Technology, or a related field.- 4-8 years of professional experience in data engineering or a related role.- Strong expertise in AWS services (e.g., S3, Redshift, Glue, Lambda).-...

  • AWS Data Engineer

    2 months ago

    Chennai, India CliqHR Full time

    Job Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • AWS Data Engineer

    2 weeks ago

    Chennai, India CliqHR Full time

    Job Description : - Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • AWS Data Engineer

    2 months ago

    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • AWS Data Engineer

    2 months ago

    Chennai, India CliqHR Full time

    Job Description : - Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • AWS Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :- Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification- Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns- Work closely with DevOps engineer to...

  • AWS Data Engineer

    1 day ago

    Chennai, Tamil Nadu, India CliqHR Full time

    Job Description :Must have minimum 5+ years of experience as AWS Data EngineerNice to have : AWS Certified Data Analytics Specialty certification Experience in designing and implementing data pipelines using AWS Glue and orchestration using Airflow for batch, streaming (Kafka/Kinesis) and API ingestion patterns Work closely with DevOps engineer to support...

  • AWS Data Engineer

    2 weeks ago

    Chennai, India RELX India (Pvt) Ltd L&P div Company Full time

    About the roleThe role in focus involves designing and implementing robust data pipelines utilizing AWS, Python, and SQL to underpin product analytics and decision-making. As a data engineer, you'll ensure the integrity and performance of data processing and storage systems, optimize data flows within AWS for enhanced performance, and develop efficient ETL...

  • AWS Data Engineer

    3 weeks ago

    Chennai, India RELX India (Pvt) Ltd L&P div Company Full time

    About the roleThe role in focus involves designing and implementing robust data pipelines utilizing AWS, Python, and SQL to underpin product analytics and decision-making. As a data engineer, you'll ensure the integrity and performance of data processing and storage systems, optimize data flows within AWS for enhanced performance, and develop efficient ETL...

  • Aerial Telecom

    2 weeks ago

    Chennai, India Aerial Telecom Solutions Full time

    Job Title : Data EngineerLocation : Chennai (Work from Office)Experience : 5 to 8 yearsJob Description : We are seeking an experienced Data Engineer to join our team in Chennai. The ideal candidate will have 5 to 8 years of experience in data engineering, with a strong background in building data pipelines, data warehousing, and data management...

  • Aerial Telecom

    2 months ago

    Chennai, India Aerial Telecom Solutions Full time

    Job Title : Data EngineerLocation : Chennai (Work from Office)Experience : 5 to 8 yearsJob Description : We are seeking an experienced Data Engineer to join our team in Chennai. The ideal candidate will have 5 to 8 years of experience in data engineering, with a strong background in building data pipelines, data warehousing, and data management...

  • Aerial Telecom

    2 months ago

    Chennai, India Aerial Telecom Solutions Full time

    Job Title : Data EngineerLocation : Chennai (Work from Office)Experience : 5 to 8 yearsJob Description : We are seeking an experienced Data Engineer to join our team in Chennai. The ideal candidate will have 5 to 8 years of experience in data engineering, with a strong background in building data pipelines, data warehousing, and data management...