Data Engineer for Scalable Data Pipelines

2 weeks ago


Chennai, Tamil Nadu, India Colan Infotech Private Limited Full time
Job Description

We are seeking an experienced Data Engineer to join our team at Colan Infotech Private Limited. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining scalable data pipelines using Pyspark / Scala Spark, Databricks, Python, and SQL.

The ideal candidate will have strong programming skills with the ability to write optimized and reusable high-quality code. You will work closely with cross-functional teams to understand business requirements and collaborate with them to implement best practices for data pipelines optimization, ETL, and query performance.

In this role, you will also be responsible for troubleshooting and resolving issues in data pipelines, ensuring minimal downtime. Additionally, you will create and maintain documentation for data pipelines and workflows.

Key Responsibilities:

  • Design and develop scalable data pipelines using Pyspark / Scala Spark, Databricks, Python, and SQL
  • Optimize SQL queries for efficient data extraction and manipulation
  • Demonstrate expertise in Databricks including workflow management and data exploration
  • Collaborate with cross-functional teams to understand business requirements
  • Implement best practices for data pipelines optimization, ETL, and query performance
  • Troubleshoot and resolve issues in data pipelines, ensuring minimal downtime
  • Create and maintain documentation for data pipelines and workflows

Requirements:

  • Pyspark / Scala Spark experience
  • Databricks experience
  • Python experience
  • SQL experience
  • Strong programming skills
  • Able to write optimized and reusable high-quality code

Salary: $120,000 - $180,000 per year

Benefits: We offer a comprehensive benefits package, including health insurance, retirement plan, and paid time off.


  • Senior Data Engineer

    4 weeks ago


    Chennai, Tamil Nadu, India Wavicle Data Solutions Full time

    We are seeking an Experienced AWS Data Architect with a strong background in designing, building, and maintaining scalable data pipelines and solutions on AWS. The ideal candidate will have a deep understanding of AWS services, data engineering best practices, and cloud architecture.Key Responsibilities:Design, develop, and maintain robust ETL pipelines on...


  • Chennai, Tamil Nadu, India NTT DATA Full time

    Job SummaryWe are seeking an experienced Chief Data Science Engineer to join our team at NTT DATA. As a key member of our organization, you will be responsible for designing and implementing machine learning systems, researching and developing ML algorithms and tools, and selecting appropriate data sets.About the RoleYou will work closely with...


  • Chennai, Tamil Nadu, India Truetech Full time

    We are seeking an experienced Data Engineer to join our team at Truetech.**Job Summary:**We are looking for a skilled Big Data Engineer to design, build, and optimize data pipelines using technologies like Apache Spark, Python, Apache Airflow, Kafka, and GCP (BigQuery, DataProc, Cloud Composer). You will work on scalable data solutions, real-time streaming,...


  • Chennai, Tamil Nadu, India m2p fintech Full time

    **Job Description**We are seeking a skilled Data Engineer to join our team at M2P Fintech. As a Data Engineer, you will play a key role in building and maintaining scalable data solutions that drive business growth.**Responsibilities**Develop and maintain highly scalable data processing applications.Write efficient, reusable, and well-documented code.Deliver...


  • Chennai, Tamil Nadu, India Colan Infotech Private Limited Full time

    Greetings from Colan Infotech Private LimitedAs a Senior Data Engineer, you will be responsible for designing and developing scalable data pipelines using Pyspark / Scala Spark, Azure Databricks, Azure Data Factory, Azure Datalake, Python, and SQL.We are looking for an experienced professional who can optimize SQL queries for efficient data extraction and...

  • Engineering Team Lead

    2 weeks ago


    Chennai, Tamil Nadu, India UnitedHealth Group Full time

    At UnitedHealth Group, we're committed to helping people live healthier lives and making the health system work better for everyone.We're seeking an experienced Engineering Team Lead - Data Pipelines to join our team. This role is responsible for leading a team of engineers in designing, developing, and maintaining highly scalable and reliable data...


  • Chennai, Tamil Nadu, India Walking Tree Full time

    Job Title: Data Pipeline Architect">About the Role:">We are seeking an experienced ETL Test Engineer to join our team at Walking Tree. As a key member of our data engineering team, you will design, implement and continuously expand data pipelines by performing extraction, transformation, and loading activities.">Your Responsibilities:">">Gather requirements...


  • Chennai, Tamil Nadu, India m2p fintech Full time

    Job Description:M2P Fintech is looking for a skilled Data Engineer to build and set up advanced data solutions. As part of a collaborative cross-functional team, you will work in a fast-paced and agile work environment, delivering high-quality and innovative solutions that have a significant business impact.Key Responsibilities: - Develop and maintain highly...

  • GCP Data Engineer

    1 month ago


    Chennai, Tamil Nadu, India Saaki Argus & Averil Consulting Full time

    About Saaki Argus & Averil Consulting:We are a key player in the electric vehicle industry, specializing in delivering sustainable mobility solutions for the commercial transportation sector.Our team is seeking a skilled GCP Data Engineer to join us in Chennai.The ideal candidate will have 7 to 12 years of experience in designing, developing, and maintaining...


  • Chennai, Tamil Nadu, India System Soft Technologies Full time

    Job OverviewSystem Soft Technologies is a premier technology company providing exceptional consulting services and solutions that drive innovation, enhance business value, and boost competitiveness. With over 25 years of experience in building trusted partnerships with clients, we have grown into a leading enterprise with a collective resources of over 1,200...


  • Chennai, Tamil Nadu, India Tata Consultancy Services Full time

    About This RoleWe are seeking an experienced Data Engineer to join our team at Tata Consultancy Services. As a Data Engineer, you will be responsible for designing and developing scalable data pipelines and data lakes/warehouses in a cloud environment, preferably Azure.ResponsibilitiesBuilding and optimizing data pipelines and data lakes/warehouses in a...


  • Chennai, Tamil Nadu, India Mastech InfoTrellis Full time

    Mastech InfoTrellis seeks a seasoned Data Engineer to design and implement large-scale data pipelines on Google Cloud Platform (GCP).Estimated Salary: $120,000 - $180,000 per year.About the Role:We are looking for an experienced professional with expertise in GCP services, particularly BigQuery, Dataflow, Pub/Sub, and Cloud Storage. The ideal candidate will...


  • Chennai, Tamil Nadu, India Hirexa Solutions Full time

    Job OverviewHirexa Solutions is seeking a highly skilled Data Pipeline Architect to design and implement efficient data pipelines using Scala and Spark.About the RoleThis is an exceptional opportunity for experienced professionals with strong expertise in big data technologies, scalable data processing, and API development.The ideal candidate will have 6+...


  • Chennai, Tamil Nadu, India RiskInsight Consulting Pvt Ltd Full time

    We are seeking an experienced Data Engineer to join our team at RiskInsight Consulting Pvt Ltd. This role involves designing, developing, and deploying scalable Big Data applications using Apache Spark.ResponsibilitiesCollaborate with data scientists and business analysts to understand requirements and translate them into technical solutions.Write efficient...


  • Chennai, Tamil Nadu, India SM IT SERVICES Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join SM IT SERVICES. The ideal candidate will have a strong background in data engineering and a deep understanding of Google Cloud Platform (GCP) services.Key Responsibilities:Design, develop, and maintain scalable and robust data pipelines using GCP services.Work with BigQuery to design and...


  • Chennai, Tamil Nadu, India Prodapt Full time

    About Prodapt: As a leading consulting and managed services provider, we empower our clients to excel in the digital era.Job Summary: We are seeking an experienced Cloud Engineering Specialist to design, develop, and deploy robust data pipelines using Google Cloud Platform (GCP) services. This role involves collaborating with cross-functional teams to...


  • Chennai, Tamil Nadu, India Crayon Data Full time

    We are seeking a highly skilled Senior Machine Learning Engineer to join our team in Chennai. This is a fantastic opportunity to work on large scale data solutions using machine learning, operation research and pattern searching methods.About Crayon DataCrayon Data is a leading provider of AI-led revenue acceleration solutions, headquartered in Singapore...


  • Chennai, Tamil Nadu, India Tata Consultancy Services Full time

    Tata Consultancy Services is seeking an experienced AWS Data Engineer Lead to join our team. This is a challenging role that requires strong technical expertise in designing and implementing scalable, secure, and efficient data architectures using AWS services.About the RoleIn this role, you will be responsible for leading the design, implementation, and...


  • Chennai, Tamil Nadu, India NTT DATA Full time

    Company OverviewNTT DATA is a $30+ billion trusted global innovator of business and technology services. We serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize and transform for long-term success.We invest over $3.6 billion each year in R&D to help organizations and society move confidently and sustainably into the...

  • Senior Data Engineer

    2 weeks ago


    Chennai, Tamil Nadu, India American Express Full time

    Data Engineer I Job DescriptionCompany OverviewAmerican Express is a global leader in the financial services industry, providing innovative payment solutions to customers around the world. Our company values diversity, equity, and inclusion, and we strive to create an environment where every colleague feels valued and empowered to succeed.Estimated Salary...