Data Engineering Technical Specialist

18 hours ago

INDIA NOIDA BIRLASOFT OFFICE IN Birlasoft Limited Full time ₹ 5,00,000 - ₹ 12,00,000 per year

Technical Specialist-Data Engg

Area(s) of responsibility

ROLE SUMMARY

We are seeking a highly skilled PySpark Developer with hands-on experience in Databricks to join IT Systems Development unit in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on the Databricks Unified Analytics Platform. The ideal candidate will have expertise in big data frameworks, distributed computing, and cloud platforms, with a deep understanding of Databricks architecture. This is an excellent opportunity to work with cutting-edge technologies in a dynamic, fast-paced environment.

ROLE RESPONSIBILITIES

Data Engineering and Processing:

  • Develop and manage data pipelines using PySpark on Databricks.
  • Implement ETL/ELT processes to process structured and unstructured data at scale.
  • Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.

Databricks Platform Expertise:

  • Experience in Perform Design, Development & Deployment using Azure Services (Data Factory, Databricks, PySpark, SQL)
  • Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.
  • Leverage the Databricks Lakehouse architecture for advanced analytics and machine learning workflows.
  • Manage Delta Lake for ACID transactions and data versioning.
  • Develop notebooks and workflows for end-to-end data solutions.

Cloud Platforms and Deployment:

  • Deploy and manage Databricks on Azure (e.g., Azure Databricks).
  • Use Databricks Jobs, Clusters, and Workflows to orchestrate data pipelines.
  • Optimize resource utilization and troubleshoot performance issues on the Databricks platform.

CI/CD and Testing:

  • Build and maintain CI/CD pipelines for Databricks workflows using tools like Azure DevOps, GitHub Actions, or Jenkins.
  • Write unit and integration tests for PySpark code using frameworks like Pytest or unittest.

Collaboration and Documentation:

  • Work closely with data scientists, data analysts, and IT teams to deliver robust data solutions.
  • Document Databricks workflows, configurations, and best practices for internal use.
Experience LevelMid Level

  • Technical Specialist-Data Engineering

    21 hours ago

    INDIA - HYDERABAD - BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Technical Specialist-Data Engg Area(s) of responsibility Skill Sets: Experience in EDQ, Integration development preferred. Oracle SQL DB, PL/SQL a must. Knowledge on JDE, PDH, CDH, SQM are preferred. Job Description: Understand data feed from feeder systems and design Interfaces for importing the data Good understanding of the API or...

  • Technical Specialist-BI

    3 days ago

    INDIA - NOIDA- BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Technical Specialist-BI Area(s) of responsibility · 6+ years of IT experience · 5 yrs of SQL Server experience · Experience in writing Tsql'S, Stored Procedure and Complex Queries USING Microsoft SQL Server. · Troubleshoot and resolve production issues. · Prior experience working in Insurance domain (Lloyds/London market/Part7). ...

  • Data Engineer

    5 days ago

    INDIA - BENGALURU - BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Technical Specialist-Data Engg Area(s) of responsibility Job Overview We are seeking a highly skilled and motivated Data Engineer to join our team. The successful candidate will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross-functional teams. The ideal...

  • Technical Lead-Data Engineering

    3 days ago

    INDIA - NOIDA- BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Technical Lead-Data Engg Area(s) of responsibility Functional Job Description Job Title: Business Analyst – Finance & Data Management Job Summary: We are seeking a Business Analyst with a strong finance background and expertise in SQL and ETL/DW concepts. The ideal candidate will have experience in data analysis, financial reporting, and...

  • Technical Specialist-Cloud

    19 hours ago

    INDIA - NOIDA- BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Backup - Technical Specialist-Cloud & Infra Engg Area(s) of responsibility Backup L2 Exp 5 Years Activities- • Plan and schedule Changes, Coordinating with different stakeholders. Perform RCA for Major Incidents. • Follow quality / security process defined for the engagement. Perform Trend analysis, identify top few incidents and work...

  • Data Science Specialist

    3 days ago

    INDIA - NOIDA- BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Data Science Specialist Area(s) of responsibility About Birlasoft: Birlasoft, a powerhouse where domain expertise, enterprise solutions, and digital technologies converge to redefine business processes. We take pride in our consultative and design thinking approach, driving societal progress by enabling our customers to run businesses with unmatched...

  • Technical Specialist

    3 days ago

    INDIA - HYDERABAD - BIRLASOFT OFFICE Birlasoft Limited Full time ₹ 40,00,000 - ₹ 60,00,000 per year

    Technical Specialist-Enterprise Apps Area(s) of responsibility Skills with M/O flag are part of Specialization Knowledge Management -PL2 (Functional)Programming/Software Development -PL3 (Functional)Win the Customer -PL2 (Behavioural)One Birlasoft -PL2 (Behavioural)Results Matter -PL2 (Behavioural)Release And Deployment -PL3 (Functional)Software...

  • Technical Specialist

    1 week ago

    INDIA - HYDERABAD - BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 4,00,000 - ₹ 12,00,000 per year

    Technical Specialist-Wiremock Area(s) of responsibility Primary: - 3-5 years in wiremock framework and stub design. - Thorough understanding of Service Mocking concepts. - Create and configure API mocks to mimic real-world scenarios. - Develop dynamic responses for complex requests using Wiremock. - Set up wiremock server to support development...

  • Senior Technical Lead Data Engineering

    3 days ago

    INDIA - BENGALURU - BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Matallion -Sr Technical Lead-Data Engg Area(s) of responsibility JD - Matillion LeadExperience in end-to-end data pipeline development/troubleshooting using Snowflake and Matillion Cloud. 10 years of experience in DWH, 2-4 years of experience in implementing DWH on Snowflake using Matillion. Design, develop, and maintain ETL processes using...

  • Senior Technical Lead Data Engineering

    4 days ago

    INDIA - BENGALURU - BIRLASOFT OFFICE, IN Birlasoft Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Sr Technical Lead-Data Engg Area(s) of responsibility JD - Matillion LeadExperience in end-to-end data pipeline development/troubleshooting using Snowflake and Matillion Cloud. 10 years of experience in DWH, 2-4 years of experience in implementing DWH on Snowflake using Matillion. Design, develop, and maintain ETL processes using Matillion to...