Data Engineer II

1 day ago

Chennai, Tamil Nadu, India Clear Demand Inc Full time ₹ 12,00,000 - ₹ 24,00,000 per year
Job Summary:Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership. You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.Key Responsibilities
  • Lead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more.
  • Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO).
  • Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka.
  • Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet/Avro file formats for efficient querying and cost management.
  • Build and maintain high-quality ETL pipelines, ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance.
  • Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices.
  • Mentor junior engineers (SDE-I) and foster a collaborative, growth-oriented environment.
  • Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency.
  • Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, New Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health.
  • Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase.
  • Stay current with advancements in data engineering, orchestration tools, and emerging cloud technologies, continually enhancing the platform's capabilities
Qualifications & Experience:
  • 4-5+ years of hands-on experience with Apache Airflow and other orchestration tools for managing large-scale workflows and data pipelines.
  • Expertise in AWS technologies, Athena, AWS Glue, DynamoDB, Apache Spark, PySpark, SQL, and NoSQL databases.
  • Experience in designing and managing distributed data processing systems that scale to terabyte and billion-scale datasets using cloud platforms like AWS, GCP, or Digital Ocean.
  • Proficiency in web crawling frameworks, including , HTTP protocols, Puppeteer, Playwright, and Chromium for large-scale data extraction.
  • Experience with monitoring and observability tools such as Grafana, Prometheus, Elasticsearch, and familiarity with monitoring and optimizing resource utilization in distributed systems.
  • Strong understanding of infrastructure as code using Terraform, automated CI/CD pipelines with Jenkins, and event-driven architecture with Kafka.
  • Experience with data lake architectures and optimizing storage using formats such as Parquet, Avro, or ORC.
  • Strong background in optimizing query performance and data processing frameworks (Spark, Flink, or Hadoop) for efficient data processing at scale.
  • Knowledge of containerization (Docker, Kubernetes) and orchestration for distributed system deployments.
  • Deep experience in designing resilient data systems with a focus on fault tolerance, data replication, and disaster recovery strategies in distributed environments.
  • Strong data engineering skills, including ETL pipeline development, stream processing, and distributed systems.
  • Excellent problem-solving abilities, with a collaborative mindset and strong communication skills.

  • Senior Data Engineer II

    2 weeks ago

    Chennai, Tamil Nadu, India Cirium Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Are you passionate about making a positive impact through technology?Do you approach complex challenges with curiosity and a collaborative spirit?About Our TeamAt Cirium, our vision is to keep the world connected. As industry leaders in aviation analytics, we empower customers to understand the past, interpret the present, and anticipate the future. Our...

  • Senior Data Engineer II

    2 weeks ago

    Chennai, Tamil Nadu, India LexisNexis Risk Solutions Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Senior Data Engineer IIAre you passionate about making a positive impact through technology?Do you approach complex challenges with curiosity and a collaborative spirit?About Our Team At Cirium, our vision is to keep the world connected. As industry leaders in aviation analytics, we empower customers to understand the past, interpret the present, and...

  • Data Engineer II B

    3 days ago

    Chennai, Tamil Nadu, India Bank of America Full time ₹ 1,20,000 - ₹ 3,00,000 per year

    Job Description Data Engineer II B - GBS IND Location: Chennai, India; Zone 1, Gift SezAbout Us At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. Responsible Growth is how we run our company and how we deliver for our clients, teammates, communities, and shareholders every...

  • Data Engineer

    2 weeks ago

    Chennai, Tamil Nadu, India NTT DATA Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Req ID: 336026NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer - Azure to join our team in Chennai, Tamil Nādu (IN-TN), India (IN). "Job Duties: Key Responsibilities:...

  • Data Engineer

    1 week ago

    Chennai, Tamil Nadu, India CX Data Labs Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    HiRole - Azure Data engineerLocation - Chennai, Gurugram (5days Onsite)Full timeImmediate joiner Only (15days Notice period only)Technical Skills:o Proficiency in SQL for complex queries, database design, and optimization.o Strong experience with PySpark for data transformation and processing.o Hands-on experience with Databricks for building and managing...

  • Software Engineer II

    2 weeks ago

    Chennai, Tamil Nadu, India Rocket Software Full time ₹ 40,00,000 - ₹ 1,20,00,000 per year

    Job Description It's fun to work in a company where people truly BELIEVE in what they're doingSummary: The Software Engineer-II designs, develops, troubleshoots, and debugs software programs for software enhancements and new products. In this job, you will join a team maintaining a proprietary database management system running on z/OS having batch,...

  • Senior Data Scientist II

    2 weeks ago

    Chennai, Tamil Nadu, India Poshmark Full time ₹ 8,00,000 - ₹ 20,00,000 per year

    Confidence can sometimes hold us back from applying for a job. Here's a secret: there's no such thing as a "perfect" candidate. Poshmark is looking for exceptional people who want to make a positive impact through their work and help create an organization where everyone can thrive. So whatever background you bring with you, please apply if this role would...

  • Data Engineer II

    2 weeks ago

    Chennai, Tamil Nadu, India American Express Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new...

  • Associate, Back-End Engineer II

    1 week ago

    Chennai, Tamil Nadu, India BNY External Career Site Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Associate, Back-End Engineer IIWe are seeking a skilled Collibra Developer with 3 to 5 years of experience to join our team. The ideal candidate will have strong expertise in Collibra platform development, along with hands-on experience in Python, Java, and ETL processes. You will play a key role in implementing and enhancing data governance solutions to...

  • Associate, Back-End Engineer II

    1 week ago

    Chennai, Tamil Nadu, India BNY Full time ₹ 5,00,000 - ₹ 12,00,000 per year

    Associate, Back-End Engineer IIWe are seeking a skilled Collibra Developer with 3 to 5 years of experience to join our team. The ideal candidate will have strong expertise in Collibra platform development, along with hands-on experience in Python, Java, and ETL processes. You will play a key role in implementing and enhancing data governance solutions to...