Big Data Engineer

1 week ago

Bengaluru, Karnataka, India BRISKWIN IT SOLUTIONS PRIVATE LIMITED Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Description :

About the Role :

We are looking for a skilled and motivated Big Data Engineer with strong experience in Apache NiFi, Snowflake, Databricks, and Python (PySpark).

The ideal candidate will design, develop, and optimize data pipelines, ensuring efficient data ingestion, transformation, and delivery across cloud and analytics platforms.

You will work closely with data architects, analysts, and business teams to enable scalable, high-performance, and reliable data solutions supporting enterprise analytics and reporting.

Key Responsibilities :

- Design, build, and maintain end-to-end data pipelines using Apache NiFi for ingestion, processing, and movement of large-scale data.

- Develop ETL/ELT workflows and data integration processes from multiple structured and unstructured sources into Snowflake.

- Create data transformation logic using Databricks (PySpark) for data cleansing, enrichment, and aggregation.

- Automate data flows, monitoring, and alerting mechanisms to ensure robust pipeline performance.

- Implement parameterized and reusable NiFi templates for consistent data integration.

- Develop and optimize Snowflake SQL scripts, stored procedures, UDFs, and tasks for high-performance data processing.

- Build and manage data models, schemas, and data warehouses within Snowflake.

- Implement data loading strategies (Snowpipe, COPY commands, stages) and ensure performance tuning and cost optimization.

- Monitor Snowflake resources (warehouses, storage, and queries) for efficiency and scalability.

- Integrate Snowflake with NiFi, Databricks, and BI tools to support analytical workloads.

- Develop and optimize PySpark-based transformation jobs within Databricks notebooks and workflows.

- Perform data wrangling, cleansing, and feature engineering for analytics or machine learning models.

- Integrate Databricks workflows with NiFi and Snowflake for seamless end-to-end data processing.

- Apply performance tuning techniques in Spark jobs to optimize resource utilization and execution time.

- Ensure data accuracy, consistency, and integrity across ingestion and transformation layers.

- Implement error handling, logging, and exception management in NiFi and PySpark processes.

- Participate in data validation, reconciliation, and testing for quality assurance.

- Collaborate with data governance teams to define and enforce metadata management and security policies.

- Work with cross-functional teams including business analysts, data scientists, and platform engineers to deliver scalable data solutions.

- Participate in Agile/Scrum ceremonies, providing timely status updates and technical inputs.

- Document data flows, mappings, and pipeline configurations clearly in Confluence or equivalent tools.

- Support production deployments, monitor pipelines, and resolve operational issues proactively.

Education & Certifications :

- Bachelors or Masters degree in Computer Science, Information Technology, Data Engineering, or related discipline.

Certifications (preferred) :

- Snowflake SnowPro Certification

- Databricks Certified Data Engineer Associate/Professional

- Apache NiFi Certification

  • Big Data Engineer

    4 days ago

    Bengaluru, Karnataka, India Etaclam Techno Services Full time ₹ 12,00,000 - ₹ 14,40,000 per year

    Job Title: Big Data EngineerLocation: BengaluruExperience Required: 5 to 8 yearsJob Description:We are looking for a skilled Big Data Engineer with strong hands-on experience in PySpark and SQL. This is a hybrid role, requiring the candidate to work both remotely and from the office as needed. The ideal candidate should be comfortable working on large-scale...

  • Big Data Engineer

    7 days ago

    Bengaluru, Karnataka, India Deqode Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    Profile: Big Data Engineer (System Design)Experience:5+ yearsLocation:BangaloreWork Mode:HybridAbout The RoleWe're looking for an experiencedBig Data Engineerwith system design expertise to architect and build scalable data pipelines and optimize big data solutions.Key ResponsibilitiesDesign, develop, and maintain data pipelines and ETL processes using...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Deqode Full time ₹ 5,00,000 - ₹ 25,00,000 per year

    Big Data Engineer Tech Stack: Python, Hive & Apache Spark Java & Advanced SQL Hadoop & Big Data Ecosystem Apache Airflow & ETL Pipelines Docker & Containerization Data Modelling & Warehousing Experience- 5 years Share your updated resume now.

  • Big Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Brillio Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Key skills - Hive, SQL, Python, Spark, Hadoop and kafka , apache airflowJob Title Big Data EngineerRole OverviewWe are seeking a highly skilled Big Data Engineer to join our team. The ideal candidate will have strong experience in building, maintaining, and optimizing large-scale data pipelines and distributed data processing systems. This role involves...

  • Big Data Engineer

    1 week ago

    Bengaluru, Karnataka, India Suzva Software Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Hiring: Big Data Engineer (PySpark)Client: [Details will be shared upon shortlisting]Location: Bangalore (Onsite)Experience: 6-8 years (Relevant)Joining: Immediate to 15 days Key Skills: Strong hands-on experience in PySpark Expertise in Big Data technologies Solid understanding of SQL, Data Pipelines, and ETLInterested candidates can share their resume...

  • Big Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Innova ESI Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role: Big Data EngineerExperience: 7+ YearsLocation: BangaloreNotice: Immediate Joiners OnlyJob Description:Azure Databricks, Azure Data Factory, Azure Function apps, Apache Spark, Scala, Java, Apache Kafka, event stream and Big DataOptional Skills: Airflow, PythonRoles & ResponsibilitiesOverall, 12 years of experience in IT industry, including 5 years in...

  • Big Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India, Karnataka Tata Consultancy Services Full time

    Job Title :- Data Engineer - PysparkExperience: 5 to 8 YearsLocation: Pune/HyderabadJob DescriptionRequired Skills:5+ years of experience in Big data and pysparkMust-HaveGood work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQLGood-to-HaveGood Spark, Pyspark,Big Data experienceSpark UI/Optimization/debugging techniquesGood...

  • Big Data Engineers/Senior Engineers

    1 week ago

    Bengaluru, Karnataka, India Peoplelogic Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Role: Big Data Engineers/Senior Engineers About the Role:We are seeking highly skilled Big Data Engineers/Senior Engineers with a strong background inbuilding and supporting big data solutions. The ideal candidate will have expertise in big datatechnologies, data pipeline development, and performance optimization within a collaborativeand agile...

  • Big Data Engineer

    6 days ago

    Bengaluru, Karnataka, India Talent Worx Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Location – Pune/ChennaiExp - 6 to 9RequirementsTechnology: - Big Data (Hadoop, Hive, Spark, Scala, PLSQL/ No SQL Database)Knowledge/Experience (Must):· 6-9 years of experience in Hadoop/big data technologies.· Experience with Spark/Storm/Kafka or equivalent streaming/batch processing and eventbased messaging.· Relational and NoSQL database integration...

  • Big Data Engineer

    3 days ago

    Bengaluru, Karnataka, India Saransh Software Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job Title: Data EngineerExperience: 8+ yearsLocation: BengaluruNotice Period: Immediate to 20 DaysEmployment Type: Full-timeAbout the RoleWe are looking for an experienced Data Engineer with strong expertise in Python, Spark, and PySpark to design, develop, and optimize large-scale data pipelines and distributed systems. The ideal candidate will possess a...