Senior Big Data Engineer

3 days ago


Remote, IN MindBrain Full time

Job Title - Big Data Engineer.

Location - Remote.

Contractual - Duration 1 yr.

Responsibilities :

- Design, build, and maintain large-scale data processing systems (batch and streaming) using technologies such as Hadoop, Spark, Kafka, etc.

- Develop and optimize data pipelines, data ingestion processes, and data integration workflows.

- Implement data governance and security best practices to ensure data quality and integrity.

- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.

- Perform data profiling, analysis, and troubleshooting to ensure optimal performance and reliability of data systems.

- Monitor and maintain the health, performance, and reliability of data pipelines and infrastructure components.

- Implement best practices for data storage, data lake architecture, and data warehousing solutions.

- Work closely with DevOps and IT teams to deploy and manage big data solutions in cloud environments (AWS, Azure, GCP) or on-premises.

- Stay updated with emerging technologies and trends in big data and data engineering to drive continuous improvement.

Requirements :

- Proven experience as a Big Data Engineer or in a similar role designing and implementing large-scale data systems.

- Strong proficiency in programming languages such as Python, Java, Scala, etc.

- Hands-on experience with big data technologies such as Hadoop ecosystem (HDFS, Hive, HBase), Spark, Kafka, etc.

- Experience with data pipeline orchestration tools like Airflow, Luigi, or similar.

- Proficiency in SQL and NoSQL databases (e.g, MySQL, PostgreSQL, MongoDB).

- Familiarity with cloud platforms and services (AWS, Azure, GCP) and containerization technologies (Docker, Kubernetes).

- Solid understanding of data modeling, ETL processes, and data warehousing concepts.

- Excellent problem-solving skills and ability to work independently as well as in a team environment.

- Bachelor's degree in Computer Science, Engineering, or a related field; Master's degree preferred.

Exp 6 to 8 years. Location- Remote

(ref:hirist.tech)

  • Remote/Bangalore/Pune, IN TechTiv Software Solutions Full time

    The Opportunity :Digit88 development team is establishing a new offshore product development team for its partner , that is building next-generation Big Data, Cloud-Based Business Operation Support technology for utilities, retail energy suppliers and Community Choice Aggregators (CCA). The candidate would be joining an existing team of outstanding data...


  • Remote, IN Gama Advisory Services LLP Full time

    Overview :We are seeking a proficient Big Data/ETL Test Engineer with 4-6 years of experience in testing Big Data platforms and ETL processes, specifically using Hadoop and Hive technologies. As a Test Engineer, you will play a critical role in ensuring the quality and reliability of our Big Data solutions by designing and executing comprehensive test...

  • Java Developer

    5 days ago


    Remote, IN Gama Advisory Services LLP Full time

    Overview :We are looking for a talented Java Developer with 2-3 years of experience in Big Data technologies to join our team. As a Java Developer specializing in Big Data, you will be responsible for designing, developing, and maintaining scalable applications and data pipelines using Java and Big Data frameworks. Your expertise in Java programming and Big...


  • Remote, IN Suitable.AI Full time

    Role Objective :Data Engineer will be responsible for expanding and optimizing our data and database architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building. The Data Engineer will support our software...


  • Remote, IN MindBrain Full time

    Roles & Responsibilities :- Sound knowledge in Spark architecture and distributed computing- Proficient in Spark - including RDD and Dataframes core functions, troubleshooting and performance tuning.- Good understanding in object-oriented concepts and hands on experience on Scala or on java with excellent programming logic and technique- Good in functional...

  • Data Scientist

    5 days ago


    Remote, IN Georgia IT, Inc. Full time

    Job Description :- We are seeking a skilled Data Scientist / Data Analyst with a solid background in Python programming, SQL, and data visualization. - The ideal candidate will have at least 3 years of relevant experience in data science or data analysis, with specific expertise in NLP techniques and LLM. - This role requires a creative problem-solver who...

  • Aptus Data Labs

    5 days ago


    Remote, IN Aptus Data Labs Full time

    Description :We are seeking a highly skilled and experienced individual to join our team as a Data Governance expert with AWS knowledge. In this role, you will play a critical role in ensuring data integrity, quality, and security across our organization's data assets. As a data governance expert, your main responsibility will be to develop and implement...

  • Data Engineer

    2 weeks ago


    Remote, IN Suitable.AI Full time

    Role Objective :Data Engineer will be responsible for expanding and optimizing our data and database architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building. The Data Engineer will support our software...

  • Senior Data Engineer

    4 weeks ago


    Remote, IN Nthinsight Full time

    Job Description :We are seeking a skilled and experienced Data Engineer with expertise in Python, data warehousing, ETL (Extract, Transform, Load), and SQL. As a Data Engineer, you will play a crucial role in designing, developing, and maintaining our data infrastructure to support the organization's data needs. You will collaborate with cross-functional...

  • Neurones IT

    3 days ago


    Remote, IN IT AISA Full time

    Job Description :- As a Data engineer, you will be a part of a highly qualified team, leading the delivery of modern data platforms using Azure Services and Databricks.- Designing and delivering software using an agile and iterative approach based on Scrum or Kanban.- Following and contributing to improvement of client's software engineering practices...


  • Remote, IN Avalara Technologies Pvt ltd Full time

    We are seeking a top-performing, versatile, and passionate Senior Software Development Engineer in Test (SDET) with over 10 years of extensive experience in software testing for an open position in our Content Engineering department, where we curate all the tax content needs for various Avalara product suites. The ideal candidate will have solid expertise in...


  • Remote, IN Diatoz Solutions Full time

    A Senior Data Engineer Specializing In Snowflake And ETL (Extract, Transform, Load) Processes Is Responsible For Designing, Implementing, And Maintaining Data Infrastructure And Pipelines To Ensure Data Is Efficiently Extracted From Various Sources, Transformed Into Usable Formats, And Loaded Into Data Warehouses Or Data Lakes.Responsibilities :- Developing...


  • Remote, IN Avalara Technologies Pvt ltd Full time

    - ONLY ENGINEERING GRADUATES ( COMPUTER SCIENCE AND RELATED FIELDS O NLY )- Candidates should be from product firms only ( No services firms )- Candidates from top tier institutes ( IIT / REC NIT ) & BITSWe are seeking a top-performing, versatile, and passionate Senior Software Development Engineer in Test (SDET) with over 10 years of extensive experience in...

  • Senior Data Engineer

    14 hours ago


    Remote, IN Hirextra Full time

    Responsibilities :- Design, develop, and implement cloud-based data pipelines for efficient data movement and transformation.- Migrate data workloads from on-premise to cloud storage solutions (e.g, AWS S3).- Utilize cloud-based SQL and NoSQL databases (e.g, AWS Redshift, DynamoDB) for data storage and retrieval.- Develop and maintain data pipelines using...


  • Remote, IN Prime Infosoft Full time

    What Does a Data Scientist Do?Data scientist roles and responsibilities include: - Data mining or extracting usable data from valuable data sources - Using machine learning tools to select features, create and optimize classifiers - Carrying out preprocessing of structured and unstructured data - Enhancing data collection procedures to include all relevant...


  • Remote, IN RiskInsight Consulting Pvt Ltd Full time

    About the Role : We are seeking a highly skilled and experienced Senior Machine Learning Engineer to join our innovative team and play a pivotal role in developing and deploying impactful machine learning solutions. In this role, you will leverage your expertise in machine learning algorithms, data analysis, and software engineering to tackle complex...


  • Remote, IN Avalara Technologies Pvt ltd Full time

    We are seeking a highly skilled and experienced Senior AI & Machine Learning Technical Manager to lead our dynamic team in developing cutting-edge AI & ML solutions. This role is perfect for someone passionate about leveraging Artificial Intelligence and Machine Learning to solve complex problems, drive innovation, and create impactful products. You will be...


  • Remote, IN Blue Silicon Infotech Private Limited Full time

    Senior Software Engineer PythonA Senior Software Engineer Python is a professional with extensive experience and expertise in Python programming language. They are responsible for designing, developing, and maintaining high-quality software applications using Python and related frameworks.Role and Responsibilities :- R&D on new technologies to solve problems...


  • Remote, IN FENL IT SERVICES FINANCIAL ENGINES LLP Full time

    Senior Staff Engineer - Salesforce Development At Edelman Financial Engines (EFE), we believe everyone deserves to move their financial life forward! We know that assets have the power to fund goals. Those numbers represent individual lifetimes filled with hard work and dreams for our clients and generations to follow. Our clients trust us to guide them...

  • Snowflake Developer

    2 weeks ago


    Remote, IN Flexiventures Pvt. Ltd Full time

    We are seeking a skilled Data Engineer with expertise in Snowflake to join our dynamic team. The ideal candidate will have a strong background in data engineering and a deep understanding of Snowflake architecture and development. You will be responsible for designing, building, and optimizing data pipelines and warehouse solutions within the Snowflake...