Data Engineer

4 weeks ago


Pune, Maharashtra, India EDGESOFT Full time

Data Engineer - DBX


We are seeking a skilled Data Engineer with 4 to 5 years of experience specializing in DataBricks. The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.

Responsibilities :

- Design, develop, and maintain data pipelines using DataBricks and PySpark to process and manipulate large scale datasets.

- Proven experience in optimizing Apache Spark batch processing workflows.

- Extensive experience in building and maintaining streaming data pipelines.

- Optimize and finetune existing DataBricks jobs and PySpark scripts for enhanced performance and reliability.

- Troubleshoot issues related to data pipelines, identify bottlenecks, and implement effective solutions.

- Implement best practices for data governance, security, and compliance within DataBricks environments.

- Work closely with Data Scientists and Analysts to support their data requirements and enable efficient access to relevant datasets.

- Stay updated with industry trends and advancements in DataBricks and PySpark technologies to propose and implement innovative solutions.

- Demonstrated expertise in optimizing systems for low-latency and high-throughput performance.

- Proficiency in using Spark SQL and DataFrame API for dynamic data transformations.

- Experience with using programming languages such as Python or Scala to implement advanced filtering logic in Databricks notebooks or scripts.

- Familiarity with the principles of distributed systems and their application in message broking.

- Collaborate with cross functional teams to gather requirements, understand data needs, and implement scalable solutions.

Requirements :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 5 to 10 years of proven experience as a Data Engineer with a strong emphasis on DataBricks.

- Proficiency in PySpark and extensive hands on experience in building and optimizing data pipelines using DataBricks.

- Solid understanding of different components within DataBricks such as clusters, notebooks, jobs, and libraries.

- Strong knowledge of SQL, data modeling, and ETL processes.

- Ability to analyzxcellent communication skills with the ability to collaborate with cross functional teams.

- Experience working in e complex problems, propose solutions, and implement them effectively.

- Ea cloud environment (AWS, Azure, or GCP) is a plus.

Preferred Skills :

- Certifications or training in DataBricks or PySpark.

- Familiarity with other big data technologies (Hadoop, Spark, Kafka, etc.).

- Experience with version control systems (Git, SVN, etc.) and CI/CD pipelines.

- Knowledge of at least one repository like GIT and software tool like JIRA

(ref:hirist.tech)
  • Senior Data Engineer

    2 weeks ago


    Pune, Maharashtra, India timesjobs Full time

    Senior Data EngineerPosition: Senior Data EngineerExperience: 8 yearsJob Location: PuneSkills Required:Developing, maintaining, and documenting reliable and scalable data infrastructure and data products in complex environmentsDevelopment experience in one or more object-oriented programming languages (e.g., Py-spark, Python, Java, C )Experience of working...


  • Pune, Maharashtra, India TeacherOn Full time

    I want to learn from someone who can train me well in Azure Data Engineering topics so that i can take a job as a Data Engineer and be confident while doing it in the real time. I have 6 years of experience in market Reasearch company and know the basics of SQL, Power Bi and Alteryx. If you think you can help me in my career transition, i would be happy to...

  • AWS Data Engineer

    4 days ago


    Pune, Maharashtra, India HNM Solutions Full time

    Note : Only Immediate JoinerRole : AWS Data EngineerLocation : Pune, IndiaExperience : 8 YearsJD: Roles and responsibilities :- With over 8 yrs. of hands-on experience with Data Technologies - Requirement analysis and assess the technical feasibility of proposed solutions. - Act as the technical specialist in designing and recommending architecture for...


  • Pune, Maharashtra, India EDGESOFT Full time

    Project Role : Data Platform EngineerProject Role Description : Assists with the data platform blueprint and design, encompassing the relevant data platform components. Collaborates with the Integration Architects and Data Architects to ensure cohesive integration between systems and data models.Must have Skills : Google BigQueryGood to Have Skills : No...

  • Azure Data Engineer

    4 weeks ago


    Pune, Maharashtra, India ACZ Global Pvt Ltd Full time

    Azure Data EngineerMandatory Skills : Pyspark + Databricks + Adf + Synapse is mandatoryJob Description:We are looking for a skilled and motivated Azure Data Engineer to join our team. As an Azure Data Engineer, you will play a crucial role in designing, implementing, and maintaining data solutions on the Microsoft Azure platform. The ideal candidate will...

  • Senior Data Engineer

    1 month ago


    Pune, Maharashtra, India Procore Full time

    Job DescriptionWe're looking for a Data Engineer to join Procore's data platform and who wants to make an impact. This is a great opportunity to work with an amazing team and help them achieve their mission to deliver actionable insights from data to the business. As a member of our Data team, you'll help define the future of our next-generation global data...


  • Pune, Maharashtra, India NTT Data Vertex Software Inc. Full time

    Apply for Digital Engineering Staff Engineer, Career Progress Consultants in Pune for Year of Experience on

  • Senior Data Engineer

    4 weeks ago


    Pune, Maharashtra, India Tekfortune IT India Pvt Ltd Full time

    Role/Title: Senior Data EngineerExperience: 6-10 Years' experienceLocation: PuneNotice period: 0-30days only.Key Skills: Oracle HCM Cloud, Fast formulas, HDL, OTBI, BI PublisherJob Summary:As a Senior Data Engineer, you will support the European World Area using the Windows & Azure suite of Analytics & Data platforms. The focus of the role is on the...

  • Big Data Engineer

    4 weeks ago


    Pune, Maharashtra, India Www.Huquo.com Full time

    Big Data Engineer( Lead assistant manager-Manager) Job Description : Must Have : Tech savvy engineer - willing and able to learn new skills, track industry trend.- 5 - 7 years of total experience of solid data engineering experience, especially in Open Source, data-intensive, distributed environments with minimum 5 years of experience in big data related...

  • Data Engineer

    1 day ago


    Pune, Maharashtra, India HNM Solutions Full time

    Note : immediate joinerRole : Data EngineerLocation : Onsite (Pune)Language : EnglishRoles and responsibilities :With over 3+ yrs. of hands-on experience with Data TechnologiesRequirement analysis and assess the technical feasibility of proposed solutions.Act as the technical specialist in designing and recommending architecture for application...


  • Pune, Maharashtra, India Antal International Full time

    Role :As a Senior Software Engineer, you will collaborate closely with the data engineering team and work on developing and maintaining data pipelines, data processing applications, and data integration solutions. You will get a chance to work with brilliant minds in the industry, work on complex use cases enabling steep learning curve, and build products...

  • AWS Data Engineer

    2 days ago


    Pune, Maharashtra, India AAR Consultancy Full time

    Role : AWS Data EngineerJob Description :Your role includes :- Collect, manage and convert raw data into usable information for business analysts- Improve data reliability and quality by combining raw information from different sources to create consistent and machine-readable formats- Able to integrate various data types and sources (stream, API, various...

  • Data Engineer

    4 weeks ago


    Pune, Maharashtra, India SGS Technical Services Pvt. Ltd Full time

    Experience : Total 4+ Years of experience Notice Period : Immediate to 15 daysLocation : PuneJob Title : Data Engineer (Snowflake and Matillion) Job Description : Overview : - Design, implement, and optimize data pipelines and workflows using Snowflake and Matillion to support our data integration, transformation, and analytics initiatives. - Collaborate...

  • Data Engineer

    4 weeks ago


    Pune, Maharashtra, India ADVANSOFT Full time

    Position: Data EngineerNotice Period: Immediately -20 Days (Early Joiners / Candidates Serving Notice Period will be Preferred)Location: PuneExperience: 5+ yearsMust Skills : Python, Hive, SQL, Airflow, Pyspark, Hadoop, Data Structure, GitHub, Backend Technologies Duration: Full TimeJob Description :Technical & Other Skills :- 5+ years of working experience...


  • Pune, Maharashtra, India Intellify Full time

    Job Description :Job Title : Data Engineer and Power BI Data Visualization EngineerWork location : Pune (Kothrud)Job Type : PERMANENT (Full-time and on company payroll)No. of Working days (Per Week) : 5-days week (MON to FRI)Mode of Work : Work From Office (WFO) for 5-days is required (initially from Kothrud office and thereafter from Baner as they are...

  • Data Engineer

    1 day ago


    Pune, Maharashtra, India EDGESOFT Full time

    Job Description :- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.- 5 to 10 years of proven experience as a Data Engineer with a strong emphasis on DataBricks.- Proficiency in PySpark and extensive hands on experience in building and optimizing data pipelines using DataBricks.- Solid understanding of different...

  • Data Engineer

    5 days ago


    Pune, Maharashtra, India GetHyr Full time

    Responsibilities : - Create prototypes and proof-of-concepts for iterative development.- Experience with NoSQL databases, such as DynamoDB, MongoDB, etc.- Create and maintain optimal data pipeline architecture.- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure...

  • Data Engineer

    1 week ago


    Pune, Maharashtra, India timesjobs Full time

    Skills Required:Experience developing, maintaining, and documenting reliable and scalable data infrastructure and data products in complex environmentsDevelopment experience in one or more object-oriented programming languages (e.g., Py-spark, Python, Java, C )Experience working in a cloud environment AWS / AzureExcellent data modeling and dimensional...


  • Pune, Maharashtra, India QAAgility Technologies Full time

    Job Title : Senior Data and Analytics EngineerWork Experience : 3-5 yearsJob Location : PunePosition Overview : We are looking for an experienced Senior Data and Analytics Engineer who will be responsible for designing, developing, and maintaining our data infrastructure and analytics solutions. The ideal candidate will have strong expertise in Hadoop,...

  • Data Engineer

    1 week ago


    Pune, Maharashtra, India Nurturing Interns Private Limited Full time

    Role Summery:You will be responsible for Analysing archiving and reporting requirements, source and destination systems and their data models and designing data transformation, archiving, reporting and analytics solutions that meet the functional and non-functional requirements.- You will work closely with Project Managers to deliver the program.Dimension of...