Data Engineer I

2 days ago

Chennai, Tamil Nadu, India Clear Demand Inc Full time ₹ 12,00,000 - ₹ 24,00,000 per year
Job Summary:Building on the foundation of the SDE-I role, the DE- I position takes on a greater level of responsibility and leadership.You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.Key Responsibilities:
  • Lead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more.
  • Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO).
  • Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka.
  • Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet/Avro file formats for efficient querying and cost management.
  • Build and maintain high-quality ETL pipelines, ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance.
  • Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices.
  • Mentor junior engineers (SDE-I) and foster a collaborative, growth-oriented environment.
  • Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency.
  • Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, New Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health.
  • Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase.
  • Stay current with advancements in data engineering, orchestration tools, and emerging cloud technologies, continually enhancing the platform's capabilities
Qualifications & Experience:
  • 3+ years of hands-on experience with Apache Airflow and other orchestration tools for managing large-scale workflows and data pipelines.
  • Expertise in AWS technologies, Athena, AWS Glue, DynamoDB, Apache Spark, PySpark, SQL, and NoSQL databases.
  • Experience in designing and managing distributed data processing systems that scale to terabyte and billion-scale datasets using cloud platforms like AWS, GCP, or Digital Ocean.
  • Proficiency in web crawling frameworks, including , HTTP protocols, Puppeteer, Playwright, and Chromium for large-scale data extraction.
  • Experience with monitoring and observability tools such as Grafana, Prometheus, Elasticsearch, and familiarity with monitoring and optimizing resource utilization in distributed systems.
  • Strong understanding of infrastructure as code using Terraform, automated CI/CD pipelines with Jenkins, and event-driven architecture with Kafka.
  • Experience with data lake architectures and optimizing storage using formats such as Parquet, Avro, or ORC.
  • Strong background in optimizing query performance and data processing frameworks (Spark, Flink, or Hadoop) for efficient data processing at scale.
  • Knowledge of containerization (Docker, Kubernetes) and orchestration for distributed system deployments.
  • Deep experience in designing resilient data systems with a focus on fault tolerance, data replication, and disaster recovery strategies in distributed environments.
  • Strong data engineering skills, including ETL pipeline development, stream processing, and distributed systems.
  • Excellent problem-solving abilities, with a collaborative mindset and strong communication skills.

  • Data Annotations Operator I

    1 week ago

    Chennai, Tamil Nadu, India ACV Full time ₹ 4,00,000 - ₹ 8,00,000 per year

    Job Summary: We are seeking a meticulous and detail-oriented Data Annotations Operator I to join our team. As a Data Annotations Operator I, you will be responsible for accurately labeling and annotating datasets, which are critical for training machine learning models. You will work closely with data scientists and machine learning engineers to ensure the...

  • Data Engineer

    1 week ago

    Chennai, Tamil Nadu, India Crayon Data Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Data Engineer – Chennai Build the foundation of data intelligence — design, develop, and optimize data pipelines that power AI-driven insights. Location: Chennai, India Experience: 2–3 years in data engineering or data-driven product development Role Overview As a Data Engineer, you will be responsible for building and...

  • Senior Data Scientist

    3 weeks ago

    Chennai, Tamil Nadu, India, Tamil Nadu Crayon Data Full time

    Role: Sr Data ScientistExperience level: 5 to 7 yearsLocation: ChennaiWhy Crayon? Why now?Crayon is transforming into an AI first company, and every Crayon (that’s what we call ourselves!) is undergoing a journey of upskilling and expanding their capabilities in the AI space.We're building an organization where AI is not a department—it’s a way of...

  • Data Engineer

    1 week ago

    Chennai, Tamil Nadu, India CX Data Labs Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    HiRole - Azure Data EngineerLocation - Chennai, Gurugram (5days Office)Full timeShift timing 2pm to 11pmKey Responsibilities:· Develop, maintain, and optimize scalable ETL/ELT pipelines using PySpark and Databricks.· Collaborate with cross-functional teams to design and implement data models and data integration solutions.· Create and maintain robust SQL...

  • Senior DemandGen Marketer

    2 days ago

    Chennai, Tamil Nadu, India Illumine-i Full time

    Company DescriptionIllumine International Inc. (Illumine-i) is a full suite engineering and sustainability consultant specializing in energy infrastructure. With expertise in structural, electrical, and civil solutions, Illumine-i optimizes cost, time, and quality for energy projects. Our founders have extensive experience in renewable energy projects and...

  • Engineer – E&I

    2 weeks ago

    Chennai, Tamil Nadu, India PROSERV Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Role SynopsisResponsible and dedicated to serve E& I scope of Control business of Proserv.Key Accountabilities (includes, but not limited to the following)Review the contract document and scope of work for Electrical engineering deliverables.Follow the engineering centre process, procedures while engineering all products or projects.Detailed engineering...

  • Lead Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Eucloid Data Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Eucloid is looking for a Lead Data Engineer to join our Data Platform team supporting various business applications. The ideal candidate will support the development of data infrastructure for our clients by participating in activities that may include starting from upstream and downstream technology selection to designing and building different components....

  • Data Engineer

    1 week ago

    Chennai, Tamil Nadu, India AABM Cloud Data Solution Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    We are seeking a Senior Data Engineer with10+ yearsof experience to join our team.Required Qualifications10+ years of experience in data engineering or similar role.Hands on experience in snowflake, SSIS, Python ( including NumPy and Pandas) is mandatory.Expertise in SQL and any cloud environment (AWS & Azure, etc.) is mandatory.Proficiency in mordern BI...

  • Data Engineer I A

    1 week ago

    Chennai, Tamil Nadu, India Bank of America Full time ₹ 40,00,000 - ₹ 1,20,00,000 per year

    Job Description:About UsAt Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. Responsible Growth is how we run our company and how we deliver for our clients, teammates, communities, and shareholders every day.One of the keys to driving Responsible Growth is being a great place to...

  • Rf System Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India Data Patterns Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Greeting from Data Patterns India Ltd Data Patterns (India) Ltd. (DP), is a Core company in Design, Development & Manufacture of Radar (Low frequency to High Frequency), EW systems, Nano Satellites, Avionics, Rugged Avionic Displays and ATE Test Systems for Defence and Aerospace applications. We are an AS9100 Rev 'D' certified company with more than 4...