Senior Data Engineer

2 days ago

Dindigul, Tamil Nadu, India Recro Full time

Role - Data Engineer

Experience - 3+ yrs

Location - Bangalore

Roles and Responsibilities:

  • Create and maintain optimal data pipeline architecture
  • Create and maintain events/streaming based architecture/design
  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data' technologies.
  • Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
  • Work with data scientists to strive for greater functionality in our data systems.

Mandatory Qualifications:

Proficiency in either Scalar or Python, expertise in Spark, and experience with performance tuning and optimization. DBT is also Mandatory.

We are looking for a candidate with 2+ years of experience in a Data Engineer role.

Experience with big data tools: HDFS/S3, Spark/Flink,Hive,Hbase, Kafka/Kinesis, etc.

Experience with relational SQL and NoSQL databases, including Elasticsearch and Cassandra/Mongodb.

Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.

Experience with AWS /GCP cloud services

Experience with stream-processing systems: Spark-Streaming/Flink etc.

Experience with object-oriented/object function scripting languages: Java, Scala, etc.

Experience building and optimizing 'big data' data pipelines, architectures and data sets.

Strong analytic skills related to working with structured/unstructured datasets.

Build processes supporting data transformation, data structures, dimensional modeling, metadata, dependency, schema registration/evolution and workload management.

Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores.

Experience supporting and working with cross-functional teams in a dynamic environment

Have a few weekend side projects up on GitHub

Have contributed to an open-source project

Have worked at a product company

Have a working knowledge of a backend programming language

Programming Language (Mandatory) → Candidate should be really good in either Scala OR Python (not necessarily both).

Used mainly for writing data pipelines and transformations.

Big Data Framework (Mandatory) → Strong hands-on expertise in Apache Spark.

Must know how to write Spark jobs and also tune/optimize them for better performance (not just basic Spark usage).

Performance Tuning & Optimization (Mandatory) →

Able to handle large datasets efficiently.

Optimize Spark jobs (partitions, shuffles, caching, memory usage, etc.)

DBT (Mandatory) →

Must have hands-on experience with dbt (Data Build Tool) for data transformations and data modeling.

This is often used on top of warehouses like BigQuery, Snowflake, Redshift.

  • Senior Data Engineering Leader

    5 days ago

    Dindigul, Tamil Nadu, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job OverviewThe role of a Senior Data Engineer is to lead and manage the development of data-intensive applications. This involves proposing architecture, designing data warehouses, implementing workflow management systems, and developing high-quality features.

  • Senior Data Engineer

    21 hours ago

    Dindigul, Tamil Nadu, India beBeeDataEngineeringLead Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Title: Data Engineering LeadWe are seeking a highly skilled Data Engineering Lead to join our team.The successful candidate will have a strong background in data engineering, with expertise in designing, building, and maintaining large-scale data systems.About the RoleThis is a senior leadership position that requires a deep understanding of data...

  • Senior Data Engineering Position

    4 days ago

    Dindigul, Tamil Nadu, India beBeeDataIngestion Full time ₹ 80,00,000 - ₹ 1,20,00,000

    Unlock Data Ingestion Mastery: A Senior Engineer PositionWe are seeking a seasoned Senior Engineer with expertise in CI/CD and Kubernetes to design, develop, and optimize data ingestion pipelines for integrating multiple sources into Databricks.Key Responsibilities:Design and implement efficient data ingestion pipelines using Databricks and Spark.Develop and...

  • Senior Data Engineering Specialist

    1 week ago

    Dindigul, Tamil Nadu, India beBeeDataEngineering Full time ₹ 25,00,000 - ₹ 35,00,000

    Job Title: Senior Data Engineering SpecialistWe are seeking a seasoned professional to lead our data engineering initiatives.About Us:We are a boutique consulting firm specializing in data and AI. We have deep expertise in the Databricks ecosystem, helping clients build scalable data platforms and AI solutions that drive real business impact.Data...

  • Senior Data Engineering Specialist

    1 week ago

    Dindigul, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 1,50,00,000

    Job Overview:We are seeking a seasoned professional to join our team as a Senior Data Engineer. This role requires hands-on coding expertise in Python and PySpark, along with strong skills in Telemetry, Big Data, and backend engineering.Key Responsibilities:Design, develop, and deploy large-scale data processing systems using Python and PySparkCollaborate...

  • High-Performance Data Engineer

    3 days ago

    Dindigul, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,50,000 - ₹ 25,50,000

    Senior Data Engineer PositionWe are seeking an experienced Senior Data Engineer to join our team and lead the design, build, and maintenance of high-performance data pipelines.Key Responsibilities:Design and develop scalable data pipelines using industry-leading tools and technologies.Collaborate with cross-functional teams to implement data models, ETL...

  • Senior Data Engineer

    4 days ago

    Dindigul, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 19,00,000

    Unlock the power of data pipeline management with this exciting opportunity!About the RoleWe are seeking a Senior Data Engineer to join our team, responsible for designing and developing complex data pipelines using Airflow, Python, and Snowflake.Key Responsibilities:Design and develop robust data pipelines using Airflow and PythonIntegrate with various data...

  • Senior Data Scraping Engineer Lead

    1 week ago

    Dindigul, Tamil Nadu, India beBeeDataScraper Full time ₹ 12,00,000 - ₹ 15,00,000

    Senior Data Scraping Engineer LeadWe are looking for a highly skilled and experienced Senior Data Scraping Engineer Lead to join our team. The ideal candidate will have a minimum of 4 years of hands-on experience in IT scraping, with at least 2 years leading a team of 5+ developers.The successful candidate will be responsible for designing and developing...

  • Senior Data Architect

    1 day ago

    Dindigul, Tamil Nadu, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Opportunity: Senior Data EngineerData Engineering Expertise:A highly experienced and skilled data engineer with a proven track record of building and managing large-scale data solutions using Snowflake, Databricks, SQL databases, and PySpark.Data Warehousing Specialist:Develop ETL processes, data modeling, and warehousing solutions using AWS Services...

  • Senior Data Engineer

    16 hours ago

    Dindigul, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 1,60,00,000

    Job Description">Were seeking highly skilled data engineers to join our team in a contract capacity. As a data engineer, you will be responsible for building and maintaining ETL pipelines, collaborating with data scientists and product managers to support model development and experimentation, and developing APIs or backend logic to implement dynamic pricing...