Big Data Engineer

4 weeks ago


Solapur, India Pert Telecom Solutions Full time
Job Description: We are seeking a skilled Big Data Engineer with deep expertise in PySpark, Python, MySQL, Hadoop and Linux. The ideal candidate should have extensive experience deploying and managing these tools within on-premises environments (no public cloud). Proficiency in cluster management, performance optimization, and monitoring is essential for this role.

Key Responsibilities:

- Develop and implement data ingestion pipelines and ETL processes using PySpark and Python capable of managing and processing large volumes of data, ranging in the millions of records.- Design and develop APIs using Python to facilitate data integration and processing.- Deploy, configure, and maintain Hadoop ecosystem components including HDFS, YARN, Hive, and HBase in on-premises environments.- Optimize the performance of Big Data applications and queries, ensuring efficient resource utilization.- Manage and monitor cluster health, including tuning and troubleshooting performance issues.- Ensure data security, integrity, and availability within the Big Data infrastructure.- Collaborate with cross-functional teams to integrate data analytics solutions and support business requirements.

Required Skills and Qualifications:

- Bachelor’s degree in computer science, Engineering, or a related field;- Proven experience 3+ years working with Big Data technologies such as Hadoop, PySpark, and MySQL in on-premises environments and handling large-scale data volumes, including datasets with millions of records, in on-premises environments.- Strong programming skills in Python, with experience in developing APIs.- Hands-on experience in Linux.- Expertise in deploying, managing, and optimizing Big Data clusters in a local data center.- Proficiency in monitoring and performance tuning of Hadoop and PySpark clusters.- Ability to troubleshoot and resolve complex issues related to data processing and infrastructure.- Excellent problem-solving skills and the ability to work independently as well as part of a team.- Strong communication skills with the ability to convey technical concepts to non-technical stakeholders.- Experience in dockers

Preferred Skills:

- Experience with Apache Airflow, Kafka, Apache Flink, and Apache NiFi for data orchestration and streaming.- Certification in Hadoop or related Big Data technologies.

Application Process: Interested candidates should submit their resume and a cover letter outlining their qualifications and experience in Big Data engineering and on-premises cluster management and include details of at least one relevant project involving the handling of large data volumes (in millions of records), you have worked on, including a brief description of the project, your role and the technologies used. This is mandatory requirement for our screening process.

  • Solapur, Maharashtra, India Cardinal Health Full time

    About UsAt Cardinal Health, we are a leading distributor of pharmaceuticals and a global manufacturer and distributor of medical and laboratory products. We strive to be the most trusted partner in healthcare by delivering innovative digital solutions and end-to-end services that advance healthcare and improve lives.We have a long history of success, with...


  • Solapur, India Gravity Infosolutions Full time

    Role: Sr. Data Architect-ContractExperience: 14-20 yearsShift: Regular IST shiftType: ContractDuration: 6 monthsLocation: Mumbai-Hybrid(2 Days Work from Office in a week)Job Description:We are seeking an accomplished Senior Data Architect to join our team and lead our data architecture efforts. As a Senior Data Architect, you will be responsible for...

  • Data Engineer

    4 weeks ago


    Solapur, India Truelancer.com Full time

    Job Title: Data Engineer - DBT (Data Build Tool)Experience: 5-8 yrsMust-Have Skills: Airflow, S3, Snowflake, SQL/No SQL, Talend, AWSEssential Skills, Education and Experience:Should have a B.E. / B.Tech. / MCA or equivalent degree along with 4-7 years of experience in Data EngineeringStrong experience in DBT concepts like Model building and configurations,...

  • Data Engineer

    4 weeks ago


    Solapur, India Bonzer Business Solutions Pvt. Ltd. Full time

    As part of the team, you will be responsible for building and running the data pipelines and services that are required to support business functions/reports/dashboard.. We are heavily dependent on BigQuery/Snowflake, Airflow, Stitch/Fivetran, dbt , Tableau/Looker for our business intelligence and embrace AWS with some GCP.As a Data Engineer you’ll...


  • Solapur, Maharashtra, India Azilen Technologies Full time

    Company OverviewAzilen Technologies is a leading technology firm that specializes in data engineering and analytics solutions.Job SummaryWe are seeking an experienced Data Engineering Lead to join our team. As a key member of our data engineering group, you will be responsible for designing and developing scalable data pipelines, ensuring high-quality data...

  • Senior Data Engineer

    4 weeks ago


    Solapur, India GS Lab Full time

    Experience Range : 7- 12 Years Responsibilities:Design, implement and maintain scalable ETL jobs and data pipelines for processing and analyzing large data setsBuild warehouses, datalakes and integrations with CRMs and analytics tools.Build and maintain data pipelines using Python for data extraction, transformation and loading (ETL) workflows.Ensure high...


  • Solapur, India MPC Cloud Consulting Pvt Ltd Full time

    We are looking for Data Analytics Engineers Key Details:Role: Data Analytics Engineer (5-8 years of experience, ability to work independently) Skillset: Azure Data Factory, Azure Synapse Analytics, and other Azure services. SQL Server, Azure SQL Python software development stacks, ecosystems, frameworks, and tools such as NumPy, SciPy, Pandas, sci-kit-learn...


  • Solapur, India RecourseTech Full time

    Recourse Tech is a dynamic and fast-growing software and technology agency that builds cutting-edge solutions for businesses across industries. We specialize in delivering high-impact, data-driven solutions, leveraging advanced technologies, and empowering organizations with actionable insights.We are looking for a Data Science Engineer with 1-3 years of...


  • Solapur, India v4c.ai Full time

    Overview: As the apex of the data science hierarchy, the Principal Data Scientist will set the strategic direction for data-driven initiatives, ensuring alignment with organizational goals, and lead in the adoption of advanced technologies including Machine Learning, Deep Learning, and Generative AI.Job Description:Strategically lead the planning and...


  • Solapur, Maharashtra, India Hexaware Technologies Full time

    Hexaware Technologies is a leading IT consulting and business solutions company that offers end-to-end services to clients worldwide. Our team of experts provides innovative solutions to businesses, helping them achieve their goals and stay ahead in the market.We are seeking an experienced Snowflake Data Engineer who will play a key role in designing,...


  • solapur, India Coinbase Full time

    At Coinbase, our mission is to increase economic freedom around the world, and we couldn’t do this without hiring the best people. We’re a group of hard-working overachievers who are deeply focused on building the future of finance and Web3 for our users across the globe, whether they’re trading, storing, staking or using crypto. Know those people who...

  • GCP Data Engineer

    4 weeks ago


    Solapur, India True Tech Professionals Full time

    Notice Period: Only 15 days or less will be consideredLocation: Remote (But will have to visit Gurgaon or Bangalore office – as and when required)Experience: 5 Plus YearsRoles/Responsibilities (Candidate’s responsibilities):Experience in GCP, Google BigQuery (Including BQ query standards, partitioning/clustering, data-model)Python Programming ExpertSQL...


  • solapur, India Maltem Australia Full time

    About Maltem AustraliaMaltem Australia is a global boutique data and analytics consulting firm specialising in supporting organisations in designing and delivering enterprise data platforms and advanced analytics solutions. With strong expertise in building robust data architectures and leveraging cutting-edge technologies, we help businesses harness the...

  • Data Analyst

    1 week ago


    Solapur, India snooplay.in Full time

    Location: Noida (Full time)Education: B.Tech (preferred)About Snooplay:Snooplay is India’s leading marketplace for toys and games, where data drives every decision. With multiple data nodes fueling our platform, we rely on meaningful insights to improve experiences for parents and children. If you’re passionate about digging into data, deriving patterns,...

  • SAS Developer

    1 week ago


    Solapur, Maharashtra, India Cardinal Health Full time

    We are seeking a highly skilled SAS Revenue Management professional to join our Advanced Analytics and Automation Team at Cardinal Health. This is an exciting opportunity to leverage your expertise in data analytics and engineering to drive business innovation and success.About the RoleThis Data Engineer position is critical for the success of supporting our...


  • Solapur, Maharashtra, India v4c Full time

    Overview:v4c is a leading innovator in the data science industry, shaping the future of technology and business.Salary: $250,000 - $350,000 per yearJob Description:Data Science ProjectsWe are seeking an exceptional Principal Data Scientist to lead strategic planning and execution of innovative data science projects, focusing on Machine Learning, Deep...

  • Lead DevOps Engineer

    4 weeks ago


    Solapur, India DDN Full time

    This is an incredible opportunity to be part of a company that has been at the forefront of AI and high-performance data storage innovation for over two decades. DataDirect Networks (DDN) is a global market leader renowned for powering many of the world's most demanding AI data centers, in industries ranging from life sciences and healthcare to financial...


  • solapur, India Cognitio Analytics Inc Full time

    Experience:1 to 8 years - DA/ Sr. DA/ AM /ManagerMust Haves:Python/PySpark SQL Advanced ExcelComm Skills Data Analysis and interpretation Project Management Story Telling Interest and ability to get to grips with the underlying business problem and value drivers of a projectGood to have:FlaskTeam ManagementAny kind of health care background eg, -...

  • Site civil engineer

    2 months ago


    Solapur, India MIT Vishwaprayag University Full time

    ABOUT US MIT Vishwaprayag University MIT-VPU is built on the motto of seamlessly preparing the next generation for sustainable living with a holistic approach by integrating scientific, aesthetic, and ethical aspects into education across disciplines. “That is Education which gives Lasting Delight ”. ABOUT JOB Designation: Site...


  • Solapur, India Aezion, Inc Full time

    Sr. Salesforce Developer/LeadLocation: Bengaluru/Coimbatore/MysuruFull timeNote: Immediate Joiners PreferredResponsibilities:• Develop and implement Salesforce declarative solutions, customizations, applications, extensions and integrations to deliver solutions that impact multiple platforms and products• Perform code reviews, ad-hoc pair programming;...