Senior Data Engineer

3 months ago

Bengaluru, India Expian Technologies Pvt Ltd Full time


SALARY : 30LPA - 40LPA

PEI Group is a subscriber-focused business intelligence company that focuses on private investment markets in real estate, infrastructure, private equity, private debt, and specialist sector-specific activities within private asset classes. We provide industry-leading journalism, data, and market insight to subscribing clients via a wide portfolio of specialist brands supported by our robust and scalable digital publishing, analytics, and database platform.


Since its inception in 2001, we have grown into a subscriber-focused business intelligence company with our multi-talented global team of over 400 people, spread across EMEA, USA & Asia, our purpose is to inform and connect investment professionals across global, specialised markets.

As a Senior Data Engineer, you will be responsible for designing, implementing, and maintaining data processing pipelines and workflows using Databricks on the Azure platform. Your expertise in PySpark, SQL, Databricks, test-driven development, and Docker will be essential to the success of our data engineering initiatives.


Roles and responsibilities :


- Collaborate with cross-functional teams to understand data requirements and design scalable and efficient data processing solutions.


- Develop and maintain data pipelines using PySpark and SQL on the Databricks platform.


- Optimize and tune data processing jobs for performance and reliability.


- Implement automated testing and monitoring processes to ensure data quality and reliability.


- Work closely with data scientists, data analysts, and other stakeholders to understand their data needs and provide effective solutions.


- Troubleshoot and resolve data-related issues, including performance bottlenecks and data quality problems.


- Stay up to date with industry trends and best practices in data engineering and Databricks.


Key Requirements :


- 5+ years of experience as a Data Engineer, with a focus on Databricks and cloud-based data platforms with a minimum of 2 years of latest experience with TDD on Databricks.


- Hands-on experience in PySpark programming for data manipulation, transformation, and analysis.


- Strong experience in SQL and writing complex queries for data retrieval and manipulation.


- Experience in developing and implementing test cases for data processing pipelines using a test-driven development approach.


- Experience in Docker for containerising and deploying data engineering applications.


- Experience in the scripting language Python is mandatory.


- Strong knowledge of Databricks platform and its components, including Databricks notebooks, clusters, and jobs.


- Experience in designing and implementing data models to support analytical and reporting needs will be an added advantage.


- Strong Knowledge of Azure Data Factory for Data orchestration, ETL workflows, and data integration is good to have.


- Good to have knowledge of cloud-based storage such as Amazon S3 and Azure Blob Storage.


- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.


- Strong analytical and problem-solving skills.


- Strong English communication skills, both written and spoken, are crucial.


- Capability to solve complex technical issues and comprehend risks prior to the circumstance.

(ref:hirist.tech)

  • Data Engineering Lead with Scalable Platforms

    4 weeks ago

    Bengaluru, Karnataka, India HEVO DATA Full time

    Unlock Career Growth as a Data Engineering LeadWe're seeking an exceptional Data Engineering Lead to join our team at Hevo Data. In this role, you'll have the opportunity to own and build scalable platform services that support thousands of customers' data pipelines and ensure load data reliability to cloud warehouses.The Hevo platform core services will...

  • Senior Analytics Engineer

    1 month ago

    Bengaluru, Karnataka, India Data Labs - LANDMARK GROUP Full time

    Job Title:Senior Analytics EngineerAbout the Role:We are seeking a highly skilled Senior Analytics Engineer to join our team at Data Labs - LANDMARK GROUP. As a key member of our engineering team, you will play a crucial role in designing, building, and deploying cutting-edge analytics products for consumption by different stakeholders.The ideal candidate...

  • Senior Data Engineer

    2 months ago

    Bengaluru, India ThoughtFocus Full time

    Job Description: Senior Data EngineerPosition: Senior Data Engineer Experience: 8-13 Years Location: BangaloreEmployment Type: Full-TimeMust have experience in Capital Market/Investment banking/BFSI domain.ONLY IMMEDIATE JOINERS (0-15 DAYS)Summary:We are looking for a highly skilled Senior Data Engineer with 8-13 years of experience in SQL, Python, Pyspark...

  • Senior Data Engineer

    2 months ago

    Bengaluru, India ThoughtFocus Full time

    Job Description: Senior Data EngineerPosition: Senior Data Engineer Experience: 8-13 Years Location: BangaloreEmployment Type: Full-TimeMust have experience in Capital Market/Investment banking/BFSI domain.ONLY IMMEDIATE JOINERS (0-15 DAYS)Summary:We are looking for a highly skilled Senior Data Engineer with 8-13 years of experience inSQL, Python, Pyspark...

  • Data Engineering Director

    4 weeks ago

    Bengaluru, Karnataka, India HEVO DATA Full time

    Engineering Leadership OpportunityThis is a unique chance to own and build scalable platform services that support thousands of customers' data pipelines, ensuring load data reliability to cloud warehouses.The Hevo platform core services will provide no-code-low pipeline building capabilities, making a significant impact on the company's business.About the...

  • Senior Data Solutions Consultant

    1 month ago

    Bengaluru, Karnataka, India HEVO DATA Full time

    Overview: We're seeking a skilled Senior Data Solutions Consultant to join our Sales team. This role involves working closely with the sales team to understand customers' business and technical needs and demonstrating how Hevo's platform can address their real-time data replication requirements.

  • Senior Data Engineer

    2 months ago

    Bengaluru, India Rakuten Symphony Full time

    Job Title: Senior Data EngineerLocation: Bangalore, IndiaCompany: Rakuten Symphony, IndiaOverviewRakuten Symphony is seeking an experienced and strategic Head/Senior Data Engineer to lead our data engineering initiatives and transform mobile connectivity through innovative data solutions. This senior leadership role requires deep expertise in large-scale...

  • Senior data engineer

    2 months ago

    Bengaluru, India Rakuten Symphony Full time

    Job Title: Senior Data EngineerLocation: Bangalore, IndiaCompany: Rakuten Symphony, IndiaOverviewRakuten Symphony is seeking an experienced and strategic Head/Senior Data Engineer to lead our data engineering initiatives and transform mobile connectivity through innovative data solutions. This senior leadership role requires deep expertise in large-scale...

  • Senior Data Engineer

    2 months ago

    Bengaluru, India Rakuten Symphony Full time

    Job Title: Senior Data EngineerLocation: Bangalore, IndiaCompany: Rakuten Symphony, IndiaOverviewRakuten Symphony is seeking an experienced and strategic Head/Senior Data Engineer to lead our data engineering initiatives and transform mobile connectivity through innovative data solutions. This senior leadership role requires deep expertise in large-scale...

  • Senior Data Engineer

    2 months ago

    Bengaluru, India Rakuten Symphony Full time

    Job Title: Senior Data Engineer Location: Bangalore, India Company: Rakuten Symphony, India Overview Rakuten Symphony is seeking an experienced and strategic Head/Senior Data Engineer to lead our data engineering initiatives and transform mobile connectivity through innovative data solutions. This senior leadership role requires deep expertise in...

  • Senior Data Engineer

    2 months ago

    Bengaluru, India Rakuten Symphony Full time

    Job Title: Senior Data EngineerLocation: Bangalore, IndiaCompany: Rakuten Symphony, IndiaOverviewRakuten Symphony is seeking an experienced and strategic Head/Senior Data Engineer to lead our data engineering initiatives and transform mobile connectivity through innovative data solutions. This senior leadership role requires deep expertise in large-scale...

  • Senior Data Engineer

    4 weeks ago

    Bengaluru, Karnataka, India Prodcon Tech Services Pvt Ltd Full time

    Job Title: Senior Data EngineerOverview of the Role: We are seeking a highly skilled Senior Data Engineer to join our team at Prodcon Tech Services Pvt Ltd. This individual will be responsible for designing, developing, and maintaining data pipelines and data warehousing solutions.The ideal candidate will have strong expertise in Amazon Data Warehouse (ADB)...

  • Senior Data Engineer

    4 weeks ago

    Bengaluru, Karnataka, India TMRW Full time

    About TMRWTMRW is a company that values innovation and excellence in data engineering. We are looking for a talented Senior Data Engineer to join our team.

  • Senior Data Engineer

    2 months ago

    Bengaluru, India ThoughtFocus Full time

    Job Description: Senior Data EngineerPosition: Senior Data Engineer Experience: 8-13 Years Location: Bangalore Employment Type: Full-TimeMust have experience in Capital Market/Investment banking/BFSI domain.ONLY IMMEDIATE JOINERS (0-15 DAYS) Summary:We are looking for a highly skilled Senior Data Engineer with 8-13 years of experience in SQL, Python, Pyspark...

  • Data Engineer

    2 months ago

    Bengaluru, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data EngineerLocation: Yemalur, BangaloreYour Key Responsibilities:Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices.Deliver assigned projects within budgeted time, cost and agreed...

  • Data engineer

    2 months ago

    Bengaluru, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data EngineerLocation: Yemalur, BangaloreYour Key Responsibilities:Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices.Deliver assigned projects within budgeted time, cost and agreed...

  • Data Engineering Leader

    2 weeks ago

    Bengaluru, Karnataka, India Data Inc Full time

    Job SummaryWe are seeking a highly skilled and experienced Data Architect to lead our team at Data Inc. This is a key role in shaping the future of our data architecture and driving business growth.About the RoleThe successful candidate will be responsible for overseeing the development and implementation of our data architecture strategy, ensuring alignment...

  • Senior data engineer

    4 days ago

    Bengaluru, India NeoScript Full time

    Role DescriptionLooking for a Senior Data Engineer.The Senior Data Engineer will be responsible for data engineering, modeling, ETL processes, data warehousing, and data analytics on a day-to-day basis.Location: BangaloreQualificationsExperience required: 5+ years5 years of experience developing strategy and building data and analytics solutions5+ years of...

  • Data Engineering Manager Lead

    4 weeks ago

    Bengaluru, Karnataka, India HEVO DATA Full time

    Scale Data Pipeline Services with HevoWe are seeking an experienced Data Engineering Manager to lead the development of scalable platform services supporting thousands of customers' data pipelines.This role offers a unique opportunity to build and own critical components of our cloud warehouse solution, ensuring reliable data loading and streamlining...

  • Senior Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Coffeee Full time

    We are seeking a skilled Data Engineer to join our innovative team at Coffeee.About the RoleWe are looking for a highly motivated and experienced professional to lead our Hadoop development efforts. As a Senior Data Engineer, you will be responsible for designing, implementing, and supporting large-scale data processing systems using the Hadoop...