Data Engineer

4 weeks ago


HyderabadPuneAhmedabad, IN Talentoj Full time

Azure Data Engineer :

We are seeking a highly skilled Azure Data Engineer to join the data engineering team and play a crucial role in an optimization initiative. The current problem is that read speed is compromised and faster responses from delta are required.

The ideal candidate should have extensive experience in Python programming and be proficient in PySpark and Databricks. The selected candidate will be responsible for designing and implementing efficient data pipelines and solutions for data-driven projects in Azure Data Factory and Azure Databricks

Responsibilities :

- Design, build, and maintain scalable data pipelines using PySpark and Databricks

- Optimize data processing and storage for maximum performance and efficiency

- Troubleshoot and debug data-related issues, and implement solutions to prevent reoccurrence

- Collaborate with data scientists, software engineers, and other stakeholders to ensure that data solutions are aligned with business goals

Requirements :

- Strong experience in Python programming and PySpark, and SparkSQL

- Clear understanding of Spark Data structures, RDD, Dataframe, dataset

- Expertise in Databricks and ADLS

- Expertise handling data type, from dictionaries, lists, tuples, sets, arrays, pandas dataframes, and spark dataframes

- Expertise working with complex data types such as, structs, and JSON strings.

- Clear understanding of Spark Broadcast, Repartition, Bloom index filters

- Experience with ADLS optimization, partitioning, shuffling and shrinking

- Ideal experience with disk caching

- Ideal Experience with cost based optimizer

- Experience with data modeling, data warehousing, data-lake, delta-lake and ETL/ELT processes in ADF

- Strong analytical and problem-solving skills

- Excellent documentation, communication and collaboration skills

- Experience of Leading team of engineers and have mentored them

(ref:hirist.tech)

  • Hyderabad/Pune, IN AMAZATIC SOLUTIONS LLP Full time

    Senior AZURE DE Our Values : Passion, Continuous Learning, Adaptability, Teamwork, Customer Centricity, ReliabilityPosition : Data Engineer / Senior Data EngineerExperience : 8+ YearsKey Skills :- Minimum 5Y+ on Azure cloud services.- Proficiency in using Azure Synapse, Azure Storage, and Azure Data Factory for building data solutions.- Strong understanding...

  • Data Engineer

    2 weeks ago


    Pune/Hyderabad, IN Promaynaov Advisory Services Pvt Ltd Full time

    Role : Data EngineerLocation : Pune/HyderabadIn this role, you will : Principal Responsibilities :As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of our collaborative Agile development process - Software design, Scala & Spark development,...

  • Data Engineer

    24 hours ago


    Hyderabad/Chennai/Pune, IN Freelancer HR Full time

    Description :Job Description :- Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Data bricks and/or HDInsights- Experience with other Open Source big data products Hadoop (incl. Hive, Pig, Impala)- Experience with Open Source non-relational / NoSQL data repositories (incl. MongoDB,...

  • Data Engineer

    4 weeks ago


    Pune/Hyderabad, IN Huquo Full time

    Must-Have :- 4+ years of experience in Data Engineering experience on Azure and Databricks- Strong knowledge of Databricks, Azure Data Factory, Azure Blob, and Azure SQL is required- Experience Developing and tuning Spark Application and SparkSQL queries-- Expertise to Implementing Data warehousing Solutions - Good understanding of Azure Databricks platform...

  • Data Engineer

    4 weeks ago


    Pune/Hyderabad/Remote, IN HARP Technologies and Services Full time

    Job Location : Pune & Hyderabad (Initial 1 -2 months hybrid and then complete remote job. Client will offer accommodation(stay), food & travel exp. )Exp range : 6+ years Shift timings : General 9am - 6pm IST or 10am - 7pm IST Mandatory skills : Data engineering (4+years), Databricks (3.5+ years), Pyspark (3+ years), Python (preferred) or bash, Data pipeline,...

  • Azure Data Engineer

    2 weeks ago


    Bangalore/Hyderabad/Pune, IN Kezan Consulting Full time

    Work Timings : 10 AM - 7 PMNumber of Openings : 5Joining Date : Within a weekJob Description :We are seeking experienced Azure Data Engineers to join our team on a contractual basis. As an Azure Data Engineer, you will be responsible for designing, implementing, and maintaining data solutions on the Azure platform. The ideal candidate will have a strong...

  • Python Data Engineer

    4 weeks ago


    Bangalore/Hyderabad/Pune/Mumbai/Chennai, IN Swift Strategic Staff Solutions INC Full time

    Job Description :We are seeking a talented and experienced Python Data Engineer to join our data team and play a crucial role in building, maintaining, and optimizing data pipelines for our cloud-based data warehouse.Responsibilities :- Design, develop, and implement data pipelines for ingesting, transforming, and loading data into Snowflake using Python...


  • Pune/Hyderabad, IN Idyllic Services Pvt Ltd Full time

    About the job :Job Description :Job Title : Fullstack Data EngineerExperience : 7+ Yrs,Location : Pune, Hyderabad.Full Stack Data Engineer role is expected to own and manage end to end activities within the team. Person would be entrusted to developed solutions/design ideas related to Data Engineering, service upkeep, developing new capabilities and identify...

  • AWS Data Engineer

    4 weeks ago


    Pune/Hyderabad, IN Huquo Full time

    Role : Data EngineerMust-Have : - 4 to 6 years of Relevant experience in Data Engineering and delivery.- 3 to 4 years of relevant work experience in large-scale Data applications- AWS Glue is mandatory.- Strong 4-6 years of working experience in SQL Spark Python, Snowflake, and AWS cloud knowledge.- Good aptitude, strong problem-solving abilities, and...

  • Kanerika Software

    3 weeks ago


    Hyderabad/Ahmedabad, IN KANERIKA SOFTWARE PRIVATE LIMITED Full time

    Role :As a Data Engineer Lead / Power BI Lead, your role will involve designing, developing, and implementing interactive dashboards and reports using Power BI. You will work closely with stakeholders to gather requirements and translate them into effective data visualizations that provide valuable insights. Additionally, you will be responsible for...

  • Data Engineer

    4 weeks ago


    Pune/Mumbai/Hyderabad/Chennai/Bangalore, IN RapidBraiins Full time

    Position Overview :As a Talend - Data Engineering Specialist, you will play a key role in designing, developing, and maintaining data pipelines using Talend, AWS, and SQL. You will work closely with cross-functional teams to gather requirements, design data solutions, and implement scalable and efficient ETL processes. Additionally, you will leverage your...


  • Bangalore/Hyderabad/Chennai/Pune, IN CGI Information Systems and Management Consultants Full time

    Required Skills :- 5+ years of experience in database administration duties- Bachelor's degree (or equivalent) in computer science, information technology, engineering, or related discipline- Experience in building or maintaining ETL processes.- Experienced in working with big data analytics tools.- Need to be proficient in various database types and...

  • Lead Data Engineer

    4 weeks ago


    Bangalore/Hyderabad/Pune, IN iMind Your Business Solutions Private Limited Full time

    Job Description :We are hiring Lead Data Engineers for the leaders In Analytics & Cloud Applications, headquartered in San Francisco and have offices in India (Bangalore, Hyderabad and Pune)Our client provides strategic guidance and technology systems for clients wishing to solve their most complex and interesting business challenges involving cloud...

  • Senior Data Engineer

    4 weeks ago


    Hyderabad/Pune/Bangalore, IN iMind Your Business Solutions Private Limited Full time

    Job Description :We are hiring Data Engineers for the leaders In Analytics & Cloud Applications, headquartered in San Francisco and have offices in India (Bangalore, Hyderabad and Pune).Our client provides strategic guidance and technology systems for clients wishing to solve their most complex and interesting business challenges involving cloud...

  • AWS Data Engineer

    2 weeks ago


    Bangalore/Pune/Hyderabad, IN Kezan Consulting Full time

    Company Overview : Kezan Consulting is a leading provider of Human Resources Services in the industry. We specialize in providing comprehensive HR solutions to our clients, helping them achieve their goals and drive business success. With a team of highly skilled professionals and a commitment to delivering excellence, we have established ourselves as a...

  • Lead Data Engineer

    3 weeks ago


    Pune/Hyderabad/Gurgaon/Gurugram, IN LnS Infotech Full time

    Position : Lead Data EngineerLocation : : 8+ YearsRoles & Responsibilities : This role is a hybrid across Data Engineering skill sets and DBA. Ideally someone with a background in Oracle DBA roles who has branched into data engineering.- 8+ years of hands-on experience in SQL, Py Spark & Python, should be experienced in shell scriptingJob Description :-...

  • Data Engineer

    1 week ago


    Bangalore/Pune/Kolkata/Gurgaon/Gurugram/Mumbai/Ahmedabad, IN Sterlite Technologies Limited Full time

    Role as a Data engineer :- Develop and build AI/ML model ready data ponds using Azure Databricks Delta lake- Improve upon the data ingestion models, ETL jobs, and alarm to maintain data integrity and data availability.- Partner with AI/ML product teams proactively identify the data needs for rapid experimentation, stable production deployment and ensure the...


  • Pune/Mumbai/Hyderabad, IN Zallery Full time

    Role : Principal Salesforce Engineer (Data Migration)Location : Pune, Hyderabad, or Mumbai - IndiaRole Overview :We are seeking a Principal Salesforce Engineer with a deep specialization in data migration to architect and lead our Salesforce data migration initiatives. This strategic role focuses on ensuring that our data migration processes are seamlessly...

  • AI/ML Engineer

    4 weeks ago


    Bangalore/Pune/Hyderabad/Chennai, IN Nilasu consulting Full time

    Summary :We are seeking a talented and passionate AI/Machine Learning Engineer to join our team. You will play a key role in the entire lifecycle of machine learning projects, from data engineering and classical machine learning to potentially exploring deep learning and natural language processing techniques. Your strong foundation in Python coding,...

  • Senior Data Engineer

    24 hours ago


    Pune/Mysore, IN IT Full time

    We are seeking a talented Senior Data Engineer with a strong background in developing ETLs and data pipelines using Apache Spark with the PySpark library. The ideal candidate will have proven experience in data modeling for data warehousing use cases, designing patterns, and building highly scalable and secure solutions. Additionally, experience with Azure...