Big Data Engineer

1 month ago

Any LocationBangalore, IN Careerfit.ai Full time

Responsibilities :

Solution Design :

- Design, develop, and maintain scalable big data solutions and data pipelines for efficient data ingestion, processing, storage, and analysis.

- Collaborate with data scientists, analysts, and business stakeholders to identify data requirements, use cases, and business objectives.

- Translate business needs into technical designs and select appropriate big data technologies for optimal performance.

Stakeholder Collaboration :

- Partner with data scientists, analysts, and business stakeholders to understand data needs, answer technical questions, and provide data access solutions.

- Communicate complex technical concepts clearly and concisely to non-technical audiences.

- Foster strong working relationships across departments to ensure successful data-driven projects.

Data Processing :

- Implement and optimize data ingestion pipelines using tools like Apache Kafka to acquire data from various sources in real-time or batch mode.

- Utilize big data technologies such as Apache Hadoop, Spark, and HBase to efficiently process and store large datasets.

- Develop and maintain data quality checks and data cleansing routines to ensure data integrity and consistency.

Data Transformation :

- Design and maintain data models and schemas for structured and unstructured data.

- Develop and implement ETL (Extract, Transform, Load) processes to transform raw data into usable formats for analysis.

- Perform data cleansing, normalization, and deduplication tasks to improve data quality for downstream applications.

Performance Optimization :

- Monitor the performance, scalability, and reliability of big data solutions.

- Identify and troubleshoot performance bottlenecks to ensure efficient data processing and access.

- Continuously optimize data pipelines and infrastructure to meet evolving data volumes and processing demands.

Data Governance :

- Champion data governance best practices throughout the data lifecycle.

- Implement robust data security measures to safeguard sensitive information.

- Ensure adherence to data privacy regulations and compliance requirements.

Continuous Learning :

- Stay up-to-date with emerging big data technologies, trends, and best practices through ongoing learning initiatives.

- Participate in professional development activities, such as conferences and workshops, to enhance your technical skillset.

Requirements :

- Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.

- Proven experience in big data engineering or related roles (minimum X years).

- Proficiency in programming languages like Java, Python, or Scala.

- Solid understanding of distributed computing frameworks (Hadoop, Spark).

- Experience with data modeling, schema design, and ETL processes.

- Excellent problem-solving, analytical, and communication skills.

- Ability to work effectively in a team environment.

Skills :

- Big Data Engineering : Proficient in designing, developing, and maintaining big data solutions and pipelines.

- Data Processing : Skilled in implementing and optimizing data ingestion, processing, and storage using big data technologies.

- Collaboration : Ability to collaborate effectively with cross-functional teams.

- Problem-Solving : Strong problem-solving skills to identify and resolve technical challenges.

Preferred Qualifications :

- Certification in big data technologies (Cloudera Certified Professional, Hortonworks Certified Developer).

- Experience with cloud platforms (AWS, Azure, GCP) for big data deployments.

- Experience with data visualization tools (Tableau, Power BI).

- Experience with scripting languages (Bash, Shell).

(ref:hirist.tech)

  • Big Data Solutions Engineer

    3 days ago

    Bangalore/Any Location, IN Gethyr Full time

    Our company is seeking a talented Big Data Solutions Engineer to join our team.- The ideal candidate will be responsible for designing, developing, and implementing our proprietary Bigdata-powered Fin Crime solution for our clients.- The candidate will work closely with the software development team, data scientists, and product managers to understand...

  • Senior Machine Learning Engineer

    3 days ago

    Any Location, IN Tekvaly Full time

    Job Description :As a Senior Machine Learning Engineer, you will be responsible for leading the development and deployment of machine learning models. You will work closely with data scientists, data engineers, and other stakeholders to understand business requirements and translate them into scalable ML solutions.Key Responsibilities :Model Development and...

  • Data Engineer

    1 month ago

    Bangalore/Any Location, IN INSIGHT GLOBAL SOLUTIONS Full time

    Data Engineer Must Haves :- Bachelors Degree in science, technology, or another related field.- 3+ years of data engineering experience.- 3+ years of experience with Cloud platforms like GCP, Azure, Airflow.- 4+ years of data warehouse, relational database, and SQL experience.- 4+ years of supporting ETL process and job scheduling experience.- 3+ years of...

  • Senior Data Engineer

    1 month ago

    Bangalore/Any Location, IN Randstad Full time

    Job Description:We are looking for a highly skilled Senior Data Engineer with over 5 years of experience in the data engineering field. The ideal candidate will have strong expertise in PySpark, SQL databases, and Big Data programming for data transformation and aggregations. This role offers an exciting opportunity to work on cutting-edge projects,...

  • Senior Machine Learning Engineer

    1 month ago

    Any Location, IN K2 Data & Analytics Full time

    Job Description : We are on the lookout for an enthusiastic ML Engineer with over five years of experience to be part of our team, which is focused on developing cutting-edge Video Ad Data technologies. The perfect fit for this role would be someone with a solid grasp of machine learning algorithms and a background in designing and executing large-scale...

  • Senior Data Engineer

    3 days ago

    Any Location, IN Coders Brain Technology Pvt. Ltd. Full time

    Position Name : Sr Data EngineerExperience Required : 8+yearsSalary : As per the Market Standard or Else whatever budget you haveNotice period : Immediate joiner Job Type : Full timeLocation : RemoteJob Description :Solution Architect (Databricks) :Candidate having 8+ years of total experience in Data engineering (development/design), in which at least 5+...

  • Big Data DevOps Engineer

    1 month ago

    Any Location, IN Freelancer Recruiter Full time

    Job Requirements :- 5+ years of DevOps experience managing the Big Data application stack including HDFS, YARN, Spark, Hive and Hbase- A deeper understanding of all the configurations required for installing and maintaining the infrastructure in the long run- Experience setting up high availability, configuring resource allocation, setting up capacity...

  • Data Engineer

    3 weeks ago

    Any Location/Bangalore, IN RS Consultants Full time

    About the job :Responsibilities :- Build, document, test and launch new data extract, transform and load (ETL) processes for one or more core data sets.- Collaborate with Business Intelligence team and other stakeholders for data requirements- Collaborate with the Data Management team to provide quality data to stakeholders.- Build data expertise and learn...

  • Azure Data Engineer

    1 month ago

    Any Location, IN Visionyle Full time

    Position Name: Azure Data EngineerPosition Mode: PermanentExperience: 6 YearsJoining Location: RemoteNotice: ImmediateMode of Interview : 2-3 roundsSUMMARY:Azure Data Engineer should have experience on Azure data lake, Azure SQL database, Databricks, Azure Data Factory, and Azure SQL data warehouse and controlling and granting database access and migrating o...

  • Senior Big Data Engineer

    1 month ago

    Bangalore/Pune, IN Recrud Full time

    Senior Big Data Engineer Position. Experience Required: 6 -10 Years. Employment Type: Full-Time. About Our Client:. Our client is a distinguished product engineering consulting firm based in Bangalore, dedicated to providing innovative offshore development solutions. With over half a century of combined expertise in product engineering and consulting...

  • Victrix Systems and Labs

    1 month ago

    Any Location, IN Victrix Systems & Labs Full time

    Job Title : GCP Cloud Data EngineerLocation : RemoteExperience : 2 to 7 yearsJob Description :We are seeking a talented Cloud Data Engineer to join our team on a remote basis.The ideal candidate will have a strong background in cloud migration, building ETL pipelines, data integrations, and the development of Operational Data Stores (ODS) and Data Warehouses...

  • Big Data Quality Assurance Engineer

    1 month ago

    Bangalore/Anywhere in India/Multiple Locations, IN Leadway Resource Full time

    We are seeking a highly skilled and experienced Big Data QA Engineer to join our team. You will play a critical role in ensuring the quality of our big data platform and data solutions. You will leverage your expertise in Big Data testing methodologies, automation frameworks, and scripting languages to design, develop, and execute comprehensive test...

  • Big Data Developer

    1 month ago

    Gurgaon/Gurugram/Bangalore, IN Workassist.in Full time

    Our hiring partner is looking for :- This is a full-time on-site role for a Big Data Developer at Big Four Consulting Firm Gurugram.- The Big Data Developer will be responsible for tasks related to Data Engineering, Extract Transform Load (ETL), Big Data, Data Warehousing, and Software Data Engineering, Big Data, Extract Transform Load (ETL), and Data...

  • Big Data Engineering Manager

    1 month ago

    Any Location/Bangalore, IN Pylon Management Consulting Full time

    Responsibilities :- Have an agile mindset, easy to adapt to changes, and comfortable working in a dynamic environment.- Prioritize, delegate, and supervise all engineers' activities and deliverables following on the Adform best practices and standards.- Manage, support, and evaluate engineers' performance.- Provide input in strategic technical...

  • Big Data Engineer

    1 month ago

    Bangalore/Chennai/Pune, IN ADVANSOFT Full time

    We are hiring for Data engineer with top most MNC client;Exp : 5+yearsNp : immediate to 15daysLocation : & Responsibilities : - 5+ years of working experience in ETL scalable data pipeline usin61Scala, Python, Pyspark, Hadoop, Apache Spark, Spark SQL, Kafka, Nill, and incremental Data Load with Big data technologies.- Experience working with Databases like...

  • MLOps Developer

    3 weeks ago

    Any Location, IN GOOGNU DATA SERVICES PRIVATE LIMITED Full time

    Job Description :- Design and Implement MLOps Infrastructure: Build and maintain robust MLOps workflows using Azure Machine Learning and other Azure services to automate machine learning lifecycle from data preparation to model deployment and monitoring.- Collaborate with Cross-Functional Teams: Work closely with data scientists, software developers, and IT...

  • Big Data Engineer

    1 month ago

    Bangalore/Anywhere in India/Multiple Locations, IN Connectio IT Pvt Ltd Full time

    Key Skills : Java, Apache Spark, Apache Kafka, Hadoop, AWSJob Description :- Designing, installing, testing, and maintaining scalable data management systems- Ensuring systems meet business requirements and industry standards- Creating custom software components and analytics applications- Research data acquisition opportunities and new uses for existing...

  • Tableau Developer/Lead

    1 month ago

    Bangalore/Any Location, IN Rrootshell Technologiiss Pvt Ltd Full time

    Job Description :- 6 to 8 Years of IT experience with last 5 years in Tableau Development.- Experience with Structured Query Language (SQL), data analysis, data profiling, data validation, PLSQL, Stored Procedures, Exasol Lua script.- Experience in designing Tableau models with Live & Extract data source.- Good to have experience in Tableau Prep, data...

  • Valiance Solutions

    1 month ago

    Noida/Bangalore/Mumbai, IN Valiance Analytics Pvt. Ltd. Full time

    About Us : Valiance is a global AI & Data analytics firm helping clients build cutting-edge technology solutions for digital transformation. We work with some of the marquee brands across India, US and APAC to build transformative solutions for Credit Risk, Fraud, Predictive Maintenance, Quality Inspection, Data lake, IOT analytics etc. Our team comprises...

  • Data Engineer

    4 weeks ago

    Any Location, IN Reliable Group Full time

    POSITION DESCRIPTION :WHAT YOU'LL DO :- Architect, Design, document & implement the data pipelines to feed data models for subsequent consumption in Snowflake using dbt, airflow.- Ensure correctness and completeness of the data being transformed via engineering pipelines for end consumption in Analytical Dashboards.- Actively monitor and triage technical...