Data Engineer I

1 week ago

Chennai, Tamil Nadu, India Clear Demand Inc Full time ₹ 12,00,000 - ₹ 24,00,000 per year
Job Summary:Building on the foundation of the SDE-I role, the DE- I position takes on a greater level of responsibility and leadership.You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.Key Responsibilities:
  • Lead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more.
  • Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO).
  • Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka.
  • Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet/Avro file formats for efficient querying and cost management.
  • Build and maintain high-quality ETL pipelines, ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance.
  • Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices.
  • Mentor junior engineers (SDE-I) and foster a collaborative, growth-oriented environment.
  • Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency.
  • Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, New Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health.
  • Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase.
  • Stay current with advancements in data engineering, orchestration tools, and emerging cloud technologies, continually enhancing the platform's capabilities
Qualifications & Experience:
  • 3+ years of hands-on experience with Apache Airflow and other orchestration tools for managing large-scale workflows and data pipelines.
  • Expertise in AWS technologies, Athena, AWS Glue, DynamoDB, Apache Spark, PySpark, SQL, and NoSQL databases.
  • Experience in designing and managing distributed data processing systems that scale to terabyte and billion-scale datasets using cloud platforms like AWS, GCP, or Digital Ocean.
  • Proficiency in web crawling frameworks, including , HTTP protocols, Puppeteer, Playwright, and Chromium for large-scale data extraction.
  • Experience with monitoring and observability tools such as Grafana, Prometheus, Elasticsearch, and familiarity with monitoring and optimizing resource utilization in distributed systems.
  • Strong understanding of infrastructure as code using Terraform, automated CI/CD pipelines with Jenkins, and event-driven architecture with Kafka.
  • Experience with data lake architectures and optimizing storage using formats such as Parquet, Avro, or ORC.
  • Strong background in optimizing query performance and data processing frameworks (Spark, Flink, or Hadoop) for efficient data processing at scale.
  • Knowledge of containerization (Docker, Kubernetes) and orchestration for distributed system deployments.
  • Deep experience in designing resilient data systems with a focus on fault tolerance, data replication, and disaster recovery strategies in distributed environments.
  • Strong data engineering skills, including ETL pipeline development, stream processing, and distributed systems.
  • Excellent problem-solving abilities, with a collaborative mindset and strong communication skills.

  • Sales Engineer

    3 days ago

    Chennai, Tamil Nadu, India Illumine-i Full time

    The Big PictureAtIllumine Industries, we're on a mission to expand our global reach by partnering with innovative clients in theUS, UK, Canada, Australia, Europe, and MENA regions. As aBusiness Development Specialist, you'll be at the forefront of this mission—bridging the gap between our world-class engineering services and international businesses...

  • Senior DemandGen Marketer

    1 week ago

    Chennai, Tamil Nadu, India Illumine-i Full time

    Company DescriptionIllumine International Inc. (Illumine-i) is a full suite engineering and sustainability consultant specializing in energy infrastructure. With expertise in structural, electrical, and civil solutions, Illumine-i optimizes cost, time, and quality for energy projects. Our founders have extensive experience in renewable energy projects and...

  • GCP Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India CustomerLabs ❤️ 1P Data OPs Full time ₹ 12,00,000 - ₹ 30,00,000 per year

    Position Overview Yesterday is history, tomorrow is a mystery, but today is a gift. That's why we call it the present. - Master OogwayJoin CustomerLabs' dynamic data team as a Data Engineer and play a pivotal role in transforming raw marketing data into actionable insights that power our digital marketing platform. As a key member of our data infrastructure...

  • Sales Engineer

    4 days ago

    Chennai, Tamil Nadu, India, Tamil Nadu Illumine-i Full time

    The Big PictureAt Illumine Industries, we’re on a mission to expand our global reach by partnering with innovative clients in the US, UK, Canada, Australia, Europe, and MENA regions. As a Business Development Specialist, you’ll be at the forefront of this mission—bridging the gap between our world-class engineering services and international businesses...

  • Engineer I

    3 days ago

    Chennai, Tamil Nadu, India Xylem Full time

    Xylem is a Fortune 500 global water solutions company dedicated to advancing sustainable impact and empowering the people who make water work every day. As a leading water technology company with 23,000 employees operating in over 150 countries, Xylem is at the forefront of addressing the world's most critical water challenges. We invite passionate...

  • Senior / Lead I&C Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India BW Design Group Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    Job Summary:Seeking an experienced I&C Engineer who is highly motivated to join our team. The successful candidate should have experience in executing projects for the Oil & Gas/FMCG/Food & Beverage and Advanced tech industries with the highest level of quality, cost effectiveness, and a focus on safety. The successful candidate will work with licensed...

  • Senior Data Engineer I

    1 week ago

    Chennai, Tamil Nadu, India American Express Full time ₹ 15,00,000 - ₹ 30,00,000 per year

    At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new...

  • Senior Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Eucloid Data Solutions Full time

    Eucloid is looking for a skilled Data Engineer to join our team and contribute to the design, development, and optimization of data frameworks and pipelines using Python, Elasticsearch, SQL DB, and Queue System. The ideal candidate will have a strong technical background, a passion for solving complex problems, and experience in building scalable and...

  • Senior Data Engineer

    1 week ago

    Chennai, Tamil Nadu, India AABM Cloud Data Solution Full time

    Senior Data Engineer (Remote, India)10+ Years Data Engineer Experience RequiredSQL, Python, Snowflake - 8+ years Mandatory hands-on experienceExperience with any ETL Tools (SSIS Preferred), Cloud Environments, and DB2/DBT is a plus.

  • Engineer QMS

    2 days ago

    Chennai, Tamil Nadu, India ILJIN Electronics (I) Pvt Ltd Full time

    Company DescriptionILJIN Electronics (I) Pvt Ltd, a subsidiary of Amber Group, is a prominent leader in the Electronics Manufacturing Services (EMS) sector. Recognized for its solutions across various industries such as consumer durables, automotive, telecom, industrials, hearable/wearable, and defense, the company has a track record spanning over two...