Senior Data Engineer

2 days ago

Thiruvananthapuram, India Zeno Health Full time

Job Description:

We are seeking a highly skilled Lead Data Engineer with a strong background in Python, SQL, and modern big data technologies. The ideal candidate will have a proven track record in designing and implementing scalable data pipelines, managing ETL processes, and leveraging diverse data storage and processing frameworks, including relational and non-relational databases such as MySQL, PostgreSQL, and MongoDB. This role is crucial for optimizing our data infrastructure, enhancing data quality, and ensuring system observability. Collaboration with cross-functional teams to deliver reliable and cost-efficient data solutions is a key part of this position.


Required Skill sets ( including the qualitative skills)

Technical Expertise:

  • Proficiency in Python for data manipulation, transformation, and pipeline development.
  • Core Python, PySpark, Data Processing Libraries in Python
  • Strong command of SQL for data analysis and performance optimization.
  • Deep understanding of OOP and SOLID principles for building maintainable code.
  • Experience with both relational (SQL) and non-relational (NoSQL) databases.
  • Knowledge of OLAP and OLTP systems, understanding use cases and implementations.
  • Hands-on experience with Big Data technologies like Apache Spark and Hadoop (HDFS).

Data Warehousing and ETL:

  • Strong knowledge of data warehouse design (Star and Snowflake schemas).
  • Experience with ETL tools, preferably AWS Glue or Apache Airflow.
  • Proficient in managing data warehouses like Amazon Redshift.
  • Cloud and DevOps:
  • Expertise in AWS services for data storage, processing, and deployment.
  • DMS, Glue, Athena, Redshift
  • Familiarity with DevOps practices and CI/CD for data workflows.
  • Experience with BI tools like Tableau and Retool, including configuration and maintenance.

Data Quality and Performance:

  • Proven ability to enhance data quality and integrity through automated validation.
  • Experience in setting up real-time data system observability and monitoring.
  • Skills in optimizing system performance, managing data access, and maintaining high system availability.
  • Ability to design cost-efficient and secure data architecture with strong access management.

Soft Skills:

  • Strong problem-solving skills and analytical mindset.
  • Excellent communication skills, able to convey technical concepts to diverse audiences.
  • Highly organized, proactive, and capable of managing multiple projects efficiently.

Preferred Qualifications:

  • Experience with Apache Kafka or AWS Kinesis for data streaming.
  • Knowledge of Data Governance and data quality best practices.
  • Familiarity with performance optimization and cost analysis tools.


OKR's

  • Design, build, and maintain scalable data pipelines to support analytics and operational needs while ensuring data quality and integrity.
  • Enhance data quality and integrity through automated validation, error detection, and efficient data cleaning processes.
  • Manage, monitor, and optimize large-scale data warehouses (e.g., Amazon Redshift or Google BigQuery), implementing schemas like Star and Snowflake.
  • Develop, deploy, and enhance ETL processes using tools such as AWS Glue or Apache Airflow, ensuring data consistency and high performance.
  • Work with SQL and NoSQL databases, employing the optimal approach for each use case.
  • Ensure real-time data system observability by implementing monitoring solutions to provide visibility into data pipeline health and performance.
  • Optimize data workflows and storage using Big Data frameworks
  • Strengthen system performance monitoring through proactive diagnostics and efficiency assessments.
  • Apply OOP principles and SOLID design patterns to create maintainable and testable data processing code.
  • Manage data access and system availability, ensuring uptime and fast response times for data retrieval.
  • Collaborate closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver timely data solutions.
  • Use AWS or other cloud platforms for deploying, scaling, and monitoring data infrastructure, focusing on cost visibility and accountability.
  • Enhance data access management and control to secure sensitive data and ensure compliance with data governance standards.
  • Strengthen data security architecture by implementing robust data encryption, access controls, and threat detection mechanisms.
  • Optimize BI tools like Tableau and Retool, ensuring smooth deployment and functionality.
  • Enhance data system performance through regular audits, performance tuning, and scalability enhancements.
  • Design cost-efficient data architecture and maintain infrastructure that balances performance with cost optimization.

  • Senior Data Engineer

    4 weeks ago

    Thiruvananthapuram, Kerala, India Techvantage Analytics Full time

    Job DescriptionAt Techvantage Analytics, we are seeking a highly skilled Senior Data Engineer to join our team. The ideal candidate will have expertise in designing, implementing, and maintaining data solutions on the Microsoft Azure Data platform and SQL Server.The Senior Data Engineer will be responsible for collaborating with various stakeholders,...

  • Senior Data Engineer – Azure

    4 weeks ago

    Thiruvananthapuram, Kerala, India Claidroid Full time

    Job Title: Senior Data Engineer – AzureAbout the Role:We are seeking a highly skilled Senior Data Engineer to join our team at Claidroid Technologies. As a Senior Data Engineer, you will be responsible for designing, developing, and implementing efficient ETL/ELT pipelines using Azure Data Factory, Azure Synapse Analytics, or other relevant tools.Key...

  • Data Engineer

    1 day ago

    Thiruvananthapuram, India Alp Consulting Ltd. Full time

    Job Summary: As a Senior DBT/Azure Data Engineer, you will play a pivotal role in designing, implementing, and optimizing data transformation pipelines within the Azure cloud environment or GCP Cloud environment using DBT (Data Build Tool). Your extensive experience in Azure services and data warehousing will enable you to deliver robust and scalable data...

  • Data Engineer

    2 days ago

    Thiruvananthapuram, India Alp Consulting Ltd. Full time

    Job Summary: As a Senior DBT/Azure Data Engineer, you will play a pivotal role in designing, implementing, and optimizing data transformation pipelines within the Azure cloud environment or GCP Cloud environment using DBT (Data Build Tool). Your extensive experience in Azure services and data warehousing will enable you to deliver robust and scalable data...

  • Data Engineer

    1 day ago

    Thiruvananthapuram, India Alp Consulting Ltd. Full time

    Job Summary: As a Senior DBT/Azure Data Engineer, you will play a pivotal role in designing, implementing, and optimizing data transformation pipelines within the Azure cloud environment or GCP Cloud environment using DBT (Data Build Tool). Your extensive experience in Azure services and data warehousing will enable you to deliver robust and scalable data...

  • Data Engineer

    18 hours ago

    Thiruvananthapuram, India Alp Consulting Ltd. Full time

    Job Summary: As a Senior DBT/Azure Data Engineer, you will play a pivotal role in designing, implementing, and optimizing data transformation pipelines within the Azure cloud environment or GCP Cloud environment using DBT (Data Build Tool). Your extensive experience in Azure services and data warehousing will enable you to deliver robust and scalable data...

  • Senior Data Lead

    4 weeks ago

    Thiruvananthapuram, Kerala, India Scipy Technologies Full time

    We are seeking a Senior Data Lead to spearhead our data analytics team at Scipy Technologies. As a key member of our team, you will leverage your expertise in data science to drive business growth through data-driven insights. Your role will involve leading the development of predictive models and machine learning algorithms, designing and implementing data...

  • Senior Data Engineer

    4 weeks ago

    Thiruvananthapuram, India NeoITO Full time

    We are seeking an experienced Data Engineer with 3+ years of hands-on experience in designing, building, and maintaining scalable data infrastructure. The ideal candidate will have deep expertise in the Python stack and be skilled in building robust data pipelines, working with large datasets, and optimizing data architectures to support advanced...

  • Senior data engineer

    4 weeks ago

    Thiruvananthapuram, India NeoITO Full time

    We are seeking an experienced Data Engineer with 3+ years of hands-on experience in designing, building, and maintaining scalable data infrastructure. The ideal candidate will have deep expertise in the Python stack and be skilled in building robust data pipelines, working with large datasets, and optimizing data architectures to support advanced...

  • Senior Data Engineer

    2 weeks ago

    thiruvananthapuram, India NeoITO Full time

    We are seeking an experienced Data Engineer with 3+ years of hands-on experience in designing, building, and maintaining scalable data infrastructure. The ideal candidate will have deep expertise in the Python stack and be skilled in building robust data pipelines, working with large datasets, and optimizing data architectures to support advanced analytics...

  • Senior Data Analyst

    3 weeks ago

    Thiruvananthapuram, Kerala, India Scipy Technologies Full time

    We are looking for a Senior Data Scientist to lead our data analytics team at Scipy Technologies. In this role, you will apply your expertise in data science to solve challenging problems in the healthcare sector.You will work on projects that involve predictive modeling, data mining, and algorithm development, with a focus on improving patient outcomes and...

  • Senior Data Engineer

    4 weeks ago

    Thiruvananthapuram, India NeoITO Full time

    We are seeking an experienced Data Engineer with 3+ years of hands-on experience in designing, building, and maintaining scalable data infrastructure. The ideal candidate will have deep expertise in the Python stack and be skilled in building robust data pipelines, working with large datasets, and optimizing data architectures to support advanced analytics...

  • Senior Data Engineer

    4 weeks ago

    Thiruvananthapuram, India NeoITO Full time

    We are seeking an experienced Data Engineer with 3+ years of hands-on experience in designing, building, and maintaining scalable data infrastructure. The ideal candidate will have deep expertise in the Python stack and be skilled in building robust data pipelines, working with large datasets, and optimizing data architectures to support advanced...

  • Senior Data Engineer

    4 weeks ago

    Thiruvananthapuram, India NeoITO Full time

    We are seeking an experienced Data Engineer with 3+ years of hands-on experience in designing, building, and maintaining scalable data infrastructure. The ideal candidate will have deep expertise in the Python stack and be skilled in building robust data pipelines, working with large datasets, and optimizing data architectures to support advanced analytics...

  • Senior Data Engineer

    4 weeks ago

    thiruvananthapuram, India NeoITO Full time

    We are seeking an experienced Data Engineer with 3+ years of hands-on experience in designing, building, and maintaining scalable data infrastructure. The ideal candidate will have deep expertise in the Python stack and be skilled in building robust data pipelines, working with large datasets, and optimizing data architectures to support advanced analytics...

  • Databricks Senior Data Engineer

    3 weeks ago

    Thiruvananthapuram, India Luxoft Full time

    Project descriptionWe are hiring for projects in the Middle East. The team comprises front-end and back-end developers, data analysts and data scientists, architects, analysts, and project managers. You will work with modern technologies and be an integral part of continuing to evolve our high-performing platforms. If you are serious about being a part of...

  • Senior Software Engineer

    4 weeks ago

    Thiruvananthapuram, Kerala, India QBurst Full time

    Job Title: Senior Software EngineerAbout the Role:We are seeking a skilled Senior Software Engineer to join our team at QBurst. As a Senior Software Engineer, you will be responsible for designing and developing cloud-based applications using Java and microservices architecture.Key Responsibilities:Design and develop RESTful APIs with a focus on scalability,...

  • Senior Software Engineer

    1 month ago

    Thiruvananthapuram, Kerala, India QBurst Full time

    Job Title: Senior EngineerWe are seeking a highly skilled Senior Engineer to join our team at QBurst. As a Senior Engineer, you will play a key role in designing and developing cloud-based applications using Java and Microservices architecture.Key Responsibilities:Design and develop RESTful APIs with a focus on scalability, security, and performance...

  • Senior Cybersecurity Engineer

    4 weeks ago

    Thiruvananthapuram, Kerala, India Guidehouse Full time

    Job Summary:Guidehouse is seeking a highly skilled Senior Cybersecurity Engineer to join our team. As a Senior Cybersecurity Engineer, you will be responsible for designing and implementing secure cloud solutions using AWS. You will work closely with our team to identify and mitigate security risks, ensuring the confidentiality, integrity, and availability...

  • Alp Consulting Ltd. | Data Engineer | thiruvananthapuram

    7 hours ago

    thiruvananthapuram, India Alp Consulting Ltd. Full time

    Job Summary: As a Senior DBT/Azure Data Engineer, you will play a pivotal role in designing, implementing, and optimizing data transformation pipelines within the Azure cloud environment or GCP Cloud environment using DBT (Data Build Tool). Your extensive experience in Azure services and data warehousing will enable you to deliver robust and scalable data...