Senior Big Data Engineer

3 days ago

MumbaiBangalorePune, IN Techno Wise Full time

Job Role : Big Data Engineer

Position : Big Data Engineer

Relevant Experience : 5+ years

Location : Navi Mumbai /Bengaluru/Pune

Notice Period : Immediate or serving Notice Period

Primary Skills : Big Data, PySpark, Cloud- Azure or AWS, Databricks, SQL, Python

Overview :

We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a pivotal role in designing, implementing, and maintaining scalable data solutions leveraging the Big Data technologies like PySpark. This role requires a strong understanding of big data technologies, data engineering best practices, and the ability to collaborate effectively with cross-functional teams.

Responsibilities :

1. Design, develop, and implement end-to-end data pipelines using PySpark to process, cleanse, and transform large-scale datasets.

2. Collaborate with data scientists, analysts, and stakeholders to understand data requirements and translate them into technical solutions.

3. Optimize and tune Spark jobs for performance and scalability, ensuring efficient resource utilization and processing speed.

4. Develop and maintain data ingestion processes from various sources such as databases, streams, and file systems into the Databricks environment.

5. Implement data quality checks and validation processes to ensure the accuracy, completeness, and reliability of the data.

6. Have worked on Data Orchestration Tools like Airflow.

7. Working on story\task using Agile Methodology.

8. Must have hand-on on GitHub and GitLab

9. Work closely with DevOps teams to automate deployment processes and ensure the reliability and availability of data pipelines.

10. Monitor, troubleshoot, and resolve issues related to data pipeline failures, performance degradation, and system errors.

11. Stay updated on emerging technologies and industry trends in big data, distributed computing, and cloud platforms to drive continuous improvement and innovation.

12. Implementing Data Solutions by using Azure Services like Azure Data Factory.

13. Creating Data Pipelines using Notebooks using different Clusters in Databricks.

Qualifications :

1. Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

2. Proven experience as a Big Data Engineer with expertise in building and optimizing data pipelines using PySpark.

3. Strong programming skills in Python and proficiency with SQL for data manipulation and analysis.

4. Familiarity with distributed computing frameworks like Hadoop, Scala.

5. Hands-on experience with cloud platforms such as Azure or AWS.

6. Working experience on Databricks

7. Excellent problem-solving skills and the ability to work independently as well as collaboratively in a fast-paced environment.

8. Effective communication skills with the ability to explain complex technical concepts to non-technical stakeholders.

Benefits You'll Get :

- Unlimited opportunities to learn on our multiple Training Platforms

- Certifications Reimbursement

- Flexibility

- Opportunity to work on multiple technologies

- Medical Coverage &Life Insurance

- Company Events and Outings

- Tech Thursdays and Fun Fridays

- 5 days working

- Work-Fun Environment

(ref:hirist.tech)

  • Senior Big Data Engineer

    4 weeks ago

    Bangalore/Pune, IN Recrud Full time

    Senior Big Data Engineer Position. Experience Required: 6 -10 Years. Employment Type: Full-Time. About Our Client:. Our client is a distinguished product engineering consulting firm based in Bangalore, dedicated to providing innovative offshore development solutions. With over half a century of combined expertise in product engineering and consulting...

  • Big Data Engineer

    2 weeks ago

    Mumbai/Bangalore/Pune, IN Techno Wise Full time

    Job Role : Big Data EngineerRelevant Experience : 3+ yearsLocation : Navi Mumbai /Bengaluru/PuneNotice Period : Immediate or serving Notice PeriodPrimary Skills : Big Data, PySpark, Cloud- Azure or AWS, Databricks, SQL, PythonOverview : We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a...

  • Lead Big Data/Data Engineer

    5 days ago

    Remote/Bangalore/Pune, IN TechTiv Software Solutions Full time

    The Opportunity :Digit88 development team is establishing a new offshore product development team for its partner , that is building next-generation Big Data, Cloud-Based Business Operation Support technology for utilities, retail energy suppliers and Community Choice Aggregators (CCA). The candidate would be joining an existing team of outstanding data...

  • Big Data Engineer

    2 weeks ago

    Mumbai/Pune/Bangalore, IN Techno Wise Full time

    Job Description : We are seeking a talented Big Data Engineer proficient in PySpark to join our dynamic team. The ideal candidate will play a pivotal role in designing, implementing, and maintaining scalable data solutions leveraging the Big Data technologies like PySpark. This role requires a strong understanding of big data technologies, data engineering...

  • Big Data Developer

    4 weeks ago

    Bangalore/Pune, IN Xcelyst Full time

    Big Data + Scala developer Experience : 4-8 Yrs. experience overallMandatory Skills :Programming Languages : Scala (SCALA IS MAND)Big Data Frameworks : Spark, Hadoop/HDFS/Hive/Sqoop/Oozie, Kafka, HBase or any other NoSQL StoreGood to Have : Streaming with Spark/Kafka, Cloud knowledge (AWS/Azure)Scheduling and creatinLocation : Bangalore / PuneNotice Period :...

  • Big Data Engineer

    4 weeks ago

    Bangalore/Chennai/Pune, IN ADVANSOFT Full time

    We are hiring for Data engineer with top most MNC client;Exp : 5+yearsNp : immediate to 15daysLocation : & Responsibilities : - 5+ years of working experience in ETL scalable data pipeline usin61Scala, Python, Pyspark, Hadoop, Apache Spark, Spark SQL, Kafka, Nill, and incremental Data Load with Big data technologies.- Experience working with Databases like...

  • Publicis Sapient

    4 weeks ago

    Bangalore/Gurgaon/Gurugram, IN Publicis Sapient Full time

    Job Description :As a Senior Data Engineer, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to...

  • Data Architect

    4 weeks ago

    Bangalore/Hyderabad/Pune, IN Blanket Technologies Full time

    Responsibilities :- Lead a team of data engineers in designing and implementing scalable data solutions. - Collaborate with data scientists to understand data requirements and ensure data integrity. - Develop and maintain data pipelines for efficient data processing. - Data Quality and Governance - Handling customer sensitive ( PII ,GDPR,HIPAA ) with...

  • Spark Engineer

    4 weeks ago

    Bangalore/Onsite In Bangalore, IN Cyber Sphere LLC Full time

    Job Description :As a Spark Engineer you will help customers to be successful with the Data Intelligence platform by resolving important technical customer escalations and the support team. You will be the technical bridge between support and engineering and the first line of defense for engineering. You will ensure that all issues are vetted by you before...

  • Valiance Solutions

    4 weeks ago

    Noida/Bangalore/Mumbai, IN Valiance Analytics Pvt. Ltd. Full time

    About Us : Valiance is a global AI & Data analytics firm helping clients build cutting-edge technology solutions for digital transformation. We work with some of the marquee brands across India, US and APAC to build transformative solutions for Credit Risk, Fraud, Predictive Maintenance, Quality Inspection, Data lake, IOT analytics etc. Our team comprises...

  • AWS Data Engineer

    4 weeks ago

    Bangalore/Pune, IN Technomech Placements Full time

    This is a full-time hybrid role for a Senior AWS Data Engineer. The Senior AWS Data Engineer will be responsible for day-to-day tasks such as data engineering, data modeling, ETL (Extract Transform Load), data warehousing, and data analytics.Qualifications :- Data Engineering, Data Modeling, and ETL (Extract Transform Load) skills- Data Warehousing and Data...

  • Publicis Sapient

    4 weeks ago

    Bangalore/Gurgaon/Gurugram, IN Publicis Sapient Full time

    Job Description :As Manager, Data Engineering, you will be responsible for translating client requirements into design, architecting, and implementing Azure Cloud based big data solutions for clients. Your role will be focused on delivering high quality solutions by independently driving design discussions related to Data Ingestion, Transformation &...

  • Big Data/Databricks Developer

    4 weeks ago

    Bangalore/Pune, IN Xcelyst Full time

    Big Data + Data Bricks DeveloperExperience : 5-8Years- 3+ Years in Big Data Engineer/Min 1 yr in DatabricksNotice Period : Immediate - 30 Days Location : Bangalore / PuneSkills : - Experience in Big Data/Hadoop Ecosystem. Hands on experience in Spark and Hive. Mandatory in DatabricksData Pipeline Development : Design, develop, and maintain scalable and...

  • Big Data Quality Assurance Engineer

    4 weeks ago

    Bangalore/Anywhere in India/Multiple Locations, IN Leadway Resource Full time

    We are seeking a highly skilled and experienced Big Data QA Engineer to join our team. You will play a critical role in ensuring the quality of our big data platform and data solutions. You will leverage your expertise in Big Data testing methodologies, automation frameworks, and scripting languages to design, develop, and execute comprehensive test...

  • Big Data Engineer

    4 weeks ago

    Any Location/Bangalore, IN Careerfit.ai Full time

    Responsibilities :Solution Design :- Design, develop, and maintain scalable big data solutions and data pipelines for efficient data ingestion, processing, storage, and analysis.- Collaborate with data scientists, analysts, and business stakeholders to identify data requirements, use cases, and business objectives.- Translate business needs into technical...

  • Big Data Solutions Engineer

    4 weeks ago

    Bangalore/Any Location, IN Gethyr Full time

    Our company is seeking a talented Big Data Solutions Engineer to join our team.- The ideal candidate will be responsible for designing, developing, and implementing our proprietary Bigdata-powered Fin Crime solution for our clients.- The candidate will work closely with the software development team, data scientists, and product managers to understand...

  • Senior Data Engineer

    4 weeks ago

    Bangalore/Any Location, IN Randstad Full time

    Job Description:We are looking for a highly skilled Senior Data Engineer with over 5 years of experience in the data engineering field. The ideal candidate will have strong expertise in PySpark, SQL databases, and Big Data programming for data transformation and aggregations. This role offers an exciting opportunity to work on cutting-edge projects,...

  • Big Data Engineer

    4 weeks ago

    Bangalore/Anywhere in India/Multiple Locations, IN Connectio IT Pvt Ltd Full time

    Key Skills : Java, Apache Spark, Apache Kafka, Hadoop, AWSJob Description :- Designing, installing, testing, and maintaining scalable data management systems- Ensuring systems meet business requirements and industry standards- Creating custom software components and analytics applications- Research data acquisition opportunities and new uses for existing...

  • Big Data Engineer

    2 weeks ago

    Bangalore/Pune, IN Confidentail Full time

    Experience : 4+ YearsLocations : Pune & BangaloreWorking Mode : Hybrid Job Responsibilities :- Design, develop, and maintain highly scalable data processing applications- Write efficient, reusable and well-documented code- Deliver big data projects using Spark, Scala, Python, SQL, HQL, Hive- Maintain and tune existing Hadoop/Hbase applications- Work closely...

  • Data Engineer

    4 weeks ago

    Pune/Hyderabad, IN Promaynaov Advisory Services Pvt Ltd Full time

    Role : Data EngineerLocation : Pune/HyderabadIn this role, you will : Principal Responsibilities :As a key member of the technical team alongside Engineers, Data Scientists and Data Users, you will be expected to define and contribute at a high-level to many aspects of our collaborative Agile development process - Software design, Scala & Spark development,...