Data Engineer

3 weeks ago

PuneHyderabad, IN EDGESOFT Full time

Job Description :


The ideal candidate should have a robust understanding and hands-on expertise in PySpark and various components within DataBricks. As a crucial member of our data team, you will play a pivotal role in developing, optimizing, and maintaining our data infrastructure, ensuring seamless and efficient data processing.

Responsibilities :

- Design, develop, and maintain data pipelines using DataBricks and PySpark to process and manipulate large scale datasets.

- Proven experience in optimizing Apache Spark batch processing workflows.

- Extensive experience in building and maintaining streaming data pipelines.

- Optimize and finetune existing DataBricks jobs and PySpark scripts for enhanced performance and reliability.

- Troubleshoot issues related to data pipelines, identify bottlenecks, and implement effective solutions.

- Implement best practices for data governance, security, and compliance within DataBricks environments.

- Work closely with Data Scientists and Analysts to support their data requirements and enable efficient access to relevant datasets.

- Stay updated with industry trends and advancements in DataBricks and PySpark technologies to propose and implement innovative solutions.

- Demonstrated expertise in optimizing systems for low-latency and high-throughput performance.

- Proficiency in using Spark SQL and DataFrame API for dynamic data transformations.

- Experience with using programming languages such as Python or Scala to implement advanced filtering logic in Databricks notebooks or scripts.

- Familiarity with the principles of distributed systems and their application in message broking.

- Collaborate with cross functional teams to gather requirements, understand data needs, and implement scalable solutions.

Requirements :

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- 5 to 10 years of proven experience as a Data Engineer with a strong emphasis on DataBricks.

- Proficiency in PySpark and extensive hands on experience in building and optimizing data pipelines using DataBricks.

- Solid understanding of different components within DataBricks such as clusters, notebooks, jobs, and libraries.

- Strong knowledge of SQL, data modeling, and ETL processes.

- Ability to analyzxcellent communication skills with the ability to collaborate with cross functional teams.

- Experience working in e complex problems, propose solutions, and implement them effectively.

- Ea cloud environment (AWS, Azure, or GCP) is a plus.

Preferred Skills :

- Certifications or training in DataBricks or PySpark.

- Familiarity with other big data technologies (Hadoop, Spark, Kafka, etc.).

- Experience with version control systems (Git, SVN, etc.) and CI/CD pipelines.

- Knowledge of at least one repository like GIT and software tool like JIRA

(ref:hirist.tech)

  • Data Engineer

    2 months ago

    Pune/Hyderabad, IN Promaynaov Advisory Services Pvt Ltd Full time

    Role : Data EngineerLocation : Pune/HyderabadIn this role, you will : Principal Responsibilities :As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of our collaborative Agile development process - Software design, Scala & Spark development,...

  • Data Engineer

    3 weeks ago

    Pune/Hyderabad, IN Promaynaov Advisory Services Pvt Ltd Full time

    Role : Data EngineerLocation : Pune/HyderabadIn this role, you will : Principal Responsibilities :As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of our collaborative Agile development process - Software design, Scala & Spark development,...

  • Data Engineer

    1 month ago

    Hyderabad/Chennai/Pune, IN Freelancer HR Full time

    Description :Job Description :- Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Data bricks and/or HDInsights- Experience with other Open Source big data products Hadoop (incl. Hive, Pig, Impala)- Experience with Open Source non-relational / NoSQL data repositories (incl. MongoDB,...

  • Data Engineer

    3 weeks ago

    Hyderabad/Chennai/Pune, IN Freelancer HR Full time

    Description :Job Description :- Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure Data bricks and/or HDInsights- Experience with other Open Source big data products Hadoop (incl. Hive, Pig, Impala)- Experience with Open Source non-relational / NoSQL data repositories (incl. MongoDB,...

  • Aptus Data Labs

    4 days ago

    Mumbai/Bangalore/Hyderabad/Gurgaon/Gurugram, IN Aptus Data Labs Full time

    Role - Data Scientist (Azure)Location - Bangalore/Hyderabad/Mumbai/Gurgaon (Onsite)Apply if you can join us within 30 days.The ideal candidate's favorite words are learning, data, scale, and agility. You will leverage your strong collaboration skills and ability to extract valuable insights from highly complex data sets to ask the right questions and...

  • Aptus Data Labs

    2 weeks ago

    Bangalore/Hyderabad/Gurgaon/Gurugram/Mumbai, IN Aptus Data Labs Full time

    Description :We are seeking an experienced and talented individual for the position of Data Governance with Azure experience. In this role, you will be responsible for managing the data governance framework using Azure tools and technologies, and collaborating closely with cross-functional teams to implement and maintain best practices.Location :...

  • Data Engineer

    3 weeks ago

    Pune/Hyderabad/Remote, IN HARP Technologies and Services Full time

    Job Location : Pune & Hyderabad (Initial 1 -2 months hybrid and then complete remote job. Client will offer accommodation(stay), food & travel exp. )Exp range : 6+ years Shift timings : General 9am - 6pm IST or 10am - 7pm IST Mandatory skills : Data engineering (4+years), Databricks (3.5+ years), Pyspark (3+ years), Python (preferred) or bash, Data pipeline,...

  • Data Engineer

    2 months ago

    Pune/Hyderabad/Remote, IN HARP Technologies and Services Full time

    Job Location : Pune & Hyderabad (Initial 1 -2 months hybrid and then complete remote job. Client will offer accommodation(stay), food & travel exp. )Exp range : 6+ years Shift timings : General 9am - 6pm IST or 10am - 7pm IST Mandatory skills : Data engineering (4+years), Databricks (3.5+ years), Pyspark (3+ years), Python (preferred) or bash, Data pipeline,...

  • Azure Data Engineer

    2 months ago

    Bangalore/Hyderabad/Pune, IN Kezan Consulting Full time

    Work Timings : 10 AM - 7 PMNumber of Openings : 5Joining Date : Within a weekJob Description :We are seeking experienced Azure Data Engineers to join our team on a contractual basis. As an Azure Data Engineer, you will be responsible for designing, implementing, and maintaining data solutions on the Azure platform. The ideal candidate will have a strong...

  • Azure Data Engineer

    3 weeks ago

    Bangalore/Hyderabad/Pune, IN Kezan Consulting Full time

    Work Timings : 10 AM - 7 PMNumber of Openings : 5Joining Date : Within a weekJob Description :We are seeking experienced Azure Data Engineers to join our team on a contractual basis. As an Azure Data Engineer, you will be responsible for designing, implementing, and maintaining data solutions on the Azure platform. The ideal candidate will have a strong...

  • AWS Data Engineer

    2 months ago

    Pune/Hyderabad, IN Huquo Full time

    Role : Data EngineerMust-Have : - 4 to 6 years of Relevant experience in Data Engineering and delivery.- 3 to 4 years of relevant work experience in large-scale Data applications- AWS Glue is mandatory.- Strong 4-6 years of working experience in SQL Spark Python, Snowflake, and AWS cloud knowledge.- Good aptitude, strong problem-solving abilities, and...

  • AWS Data Engineer

    3 weeks ago

    Pune/Hyderabad, IN Huquo Full time

    Role : Data EngineerMust-Have : - 4 to 6 years of Relevant experience in Data Engineering and delivery.- 3 to 4 years of relevant work experience in large-scale Data applications- AWS Glue is mandatory.- Strong 4-6 years of working experience in SQL Spark Python, Snowflake, and AWS cloud knowledge.- Good aptitude, strong problem-solving abilities, and...

  • Aptus Data Labs

    4 days ago

    Hyderabad/Bangalore/Mumbai/Gurgaon/Gurugram, IN Aptus Data Labs Full time

    Description :We are seeking a highly skilled and experienced Vertica developer to join our team. As a Vertica developer, you will be responsible for designing, developing, and maintaining our Vertica database. You will collaborate with cross-functional teams to understand business requirements and implement solutions, optimizing performance and ensuring data...

  • Senior Data Quality Engineer

    3 weeks ago

    Pune/Hyderabad, IN HuQuo Consulting Pvt. Ltd. Full time

    Job Description - Senior Data Quality Engineer (Azure Specialist, Automation) Experience : 6-8+ yearsWork Location - Pune, Hyderabad(Hybrid)The Senior Data Quality Engineer is a collaborative role responsible for designing sound, scalable, and performant data quality metrics that meet the requirements in a large data volume enterprise.Technical SkillsMust...

  • Data Engineer

    3 weeks ago

    Pune/Mumbai/Hyderabad/Chennai/Bangalore, IN RapidBraiins Full time

    Position Overview :As a Talend - Data Engineering Specialist, you will play a key role in designing, developing, and maintaining data pipelines using Talend, AWS, and SQL. You will work closely with cross-functional teams to gather requirements, design data solutions, and implement scalable and efficient ETL processes. Additionally, you will leverage your...

  • Data Engineer

    2 months ago

    Pune/Mumbai/Hyderabad/Chennai/Bangalore, IN RapidBraiins Full time

    Position Overview :As a Talend - Data Engineering Specialist, you will play a key role in designing, developing, and maintaining data pipelines using Talend, AWS, and SQL. You will work closely with cross-functional teams to gather requirements, design data solutions, and implement scalable and efficient ETL processes. Additionally, you will leverage your...

  • Lead Data Engineer

    3 weeks ago

    Hyderabad/Bangalore/Pune, IN SKANDHA IT SERVICES PRIVATE LIMITED Full time

    Lead Data Engineer Locations : Hyderabad, Bengaluru, Pune. Skills :- Programming : Python , Pyspark/Scala, Flask.- API Microservices , ECS Fargate.- Bigdata AWS : EMR, Glue, Athena, Redshift, Lambda.- Analytics : AWS Quicksight.- Database : Postgres RDS,Mongo (SQL/NoSQL).- Orchestration : preferably Airflow.About the Role : We are searching for a talented...

  • Data Engineer

    3 weeks ago

    Mumbai/Bangalore/Hyderabad/Chennai/Pune/Delhi, IN VARITE Inc. Full time

    Job Title: Data EngineerLocation City : Hyderabad, bangalore, chennai, mumbai, new delhi, pune.Experience Required : 4+Shift: Day ShiftWork Mode: OnsitePosition Type: ContractOpenings: 3Company Name : VARITE INDIA PRIVATE LIMITEDAbout The Client: A multinational technology and e-commerce company based in the United States is widely recognized as one of the...

  • Data Engineer

    3 weeks ago

    Mumbai/Bangalore/Hyderabad/Chennai/Pune/Delhi, IN VARITE Inc. Full time

    Job Title: Data EngineerLocation City : Hyderabad, bangalore, chennai, mumbai, new delhi, pune.Experience Required : 4+Shift: Day ShiftWork Mode: OnsitePosition Type: ContractOpenings: 3Company Name : VARITE INDIA PRIVATE LIMITEDAbout The Client: A multinational technology and e-commerce company based in the United States is widely recognized as one of the...

  • Senior Data Engineer

    2 months ago

    Hyderabad/Pune/Bangalore, IN iMind Your Business Solutions Private Limited Full time

    Job Description :We are hiring Data Engineers for the leaders In Analytics & Cloud Applications, headquartered in San Francisco and have offices in India (Bangalore, Hyderabad and Pune).Our client provides strategic guidance and technology systems for clients wishing to solve their most complex and interesting business challenges involving cloud...