Big Data Engineering Solution Specialist

4 days ago

Shimoga, Karnataka, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 25,00,000
Job Title: Solution Architect for Big Data Engineering

We are seeking an experienced Solution Architect to lead the design and implementation of large-scale data engineering solutions. The ideal candidate will have a strong background in Azure Data Engineering, with expertise in Data Factory, Synapse, Fabric, and Databricks. Additionally, they should have hands-on experience with PySpark for big data processing and proficiency in SQL, T-SQL, and Python for data processing and transformation.

About the Role:
As a Solution Architect, you will be responsible for designing and implementing scalable data architectures following Medallion Architecture and best practices. This includes developing and optimizing ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Synapse Analytics. You will also perform data ingestion, transformation, and orchestration across structured and unstructured datasets. Furthermore, you will develop and maintain data lake solutions in Azure Data Lake Storage (ADLS) Gen2.

Key Responsibilities:
  • Design and implement scalable data architectures.
  • Develop and optimize ETL/ELT pipelines.
  • Perform data ingestion, transformation, and orchestration.
  • Develop and maintain data lake solutions.
  • Implement CI/CD pipelines for data workflows.
  • Collaborate with stakeholders to understand data requirements.
  • Ensure data governance, security, and compliance.
  • Monitor and troubleshoot data pipelines.

Requirements:
The ideal candidate will possess:
  • A bachelor's degree in Computer Science or related field.
  • Strong experience in Azure Data Engineering with Data Factory, Synapse, Fabric, Databricks.
  • Expertise in SQL, T-SQL, and Python for data processing and transformation.
  • Hands-on experience with PySpark for big data processing.
  • Experience in designing ETL/ELT pipelines and working with large datasets.
  • Proficiency with Azure Data Lake (ADLS) Gen2 for data storage and processing.
  • CI/CD and DevOps knowledge, including Azure DevOps, Git, and automated deployments.
  • Understanding of data modeling, data warehousing, and schema design principles.

Bonus Points:
  • Experience in New Age Technologies like Low Code, No Code, DevOps.
  • Familiarity with Data Governance and Security best practices in cloud-based data solutions.
  • Experience with real-time data processing using Kafka, Event Hub, or Stream Analytics.
  • Knowledge of DataOps methodologies and best practices.
  • Experience with APIs, REST services, and integrating third-party data sources.
  • Familiarity with Power BI or other visualization tools.

About Us:

We are a global technology company with more than two decades of core IT experience. Our differentiators are technological prowess with unwavering back-end support on a wide range of technologies and industry-leading platforms. We take pride in building innovative, efficient, and robust IT solutions for our clients. We match the client's business goals with our technology expertise and immaculate execution capabilities to solve issues that matter to the end-user. At our company, we believe in empowering our people to perform and grow by offering opportunities, learning, and inspiration—to 'act and accomplish'. If you are a self-starter looking for an open and collaborative work culture to excel in your career, we invite you to join us.

  • Chief Architect: Big Data Solutions

    3 days ago

    Shimoga, Karnataka, India beBeeEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Big Data LeaderLead a team of data engineers, working closely with clients and team members to understand their requirements and develop architectures that meet their needs.Key responsibilities include:Building pipelines to bring in various data from multiple sources within the organization and external sources.Collaborating with cross-functional teams to...

  • Senior Big Data Specialist

    6 days ago

    Shimoga, Karnataka, India beBeeBigDataSpecialist Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Senior Big Data SpecialistSkill Set:AWS - Amazon Web ServicesPythonPysparkAbout the Role:We are seeking a skilled big data specialist to join our team. The ideal candidate will have hands-on experience with AWS, Python, and Pyspark.Responsibilities:Built pySpark applications using Spark Dataframes in PythonOptimized spark jobs that process huge...

  • Big Data Architectural Specialist

    6 days ago

    Shimoga, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Transform Your Career as a Senior Data EngineerWe seek a highly skilled data professional to join our team of consultants.This role involves designing, constructing, and optimizing large-scale data processing systems that enable advanced analytics and business intelligence.About the Role:You will collaborate closely with data scientists, analysts, and other...

  • Big Data Expert

    4 days ago

    Shimoga, Karnataka, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    **Unlocking Big Data Opportunities:**We are seeking a highly skilled data expert to design and develop innovative solutions that drive business growth.Essential Skills:PySparkHiveDesirable Skills:SparkHBaseData Quality ToolsAgile Methodologies ExperienceExposure in Data Ingestion from Various Sources onto Big Data PlatformsRequirements:A minimum of 6-12...

  • Cloud Data Solutions Engineer

    6 days ago

    Shimoga, Karnataka, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Data Engineering LeaderAs a seasoned data engineering leader, you will spearhead the design, development, and maintenance of scalable, secure, and high-performance data solutions on Databricks. Your expertise will be pivotal in delivering actionable insights that drive business growth.Key Responsibilities:Develop and implement efficient data...

  • Master Data Engineer

    3 days ago

    Shimoga, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Opportunity:We are seeking a seasoned data engineer with expertise in cloud-based services and big data technologies to craft scalable data solutions.Key Responsibilities Include:Design, develop, and maintain large-scale data pipelines using cloud-based tools like AWS and Databricks.Build and optimize ETL/ELT processes for complex structured and...

  • Data Development Specialist

    4 days ago

    Shimoga, Karnataka, India beBeeDataDevelopment Full time ₹ 15,00,000 - ₹ 20,00,000

    Senior Data Development SpecialistJob Description:We are seeking an accomplished data development specialist to take on the key challenge of developing and maintaining high-performance data pipelines, collaborating with analytics and business teams, and implementing best practices to monitor data quality.Develops and maintains scalable data pipelines to...

  • Cloud Data Specialist

    3 days ago

    Shimoga, Karnataka, India beBeeCloudData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Cloud Data SpecialistAbout the Role:We are seeking a highly skilled Cloud Data Specialist to collaborate with data science teams and implement complex algorithms and analytical solutions.Collaborate with cross-functional teams to design and develop scalable data pipelines.Develop predictive models to drive actionable insights for business...

  • Data Solutions Architect

    5 days ago

    Shimoga, Karnataka, India beBeeData Full time ₹ 1,56,56,568 - ₹ 2,61,94,866

    Job DescriptionAs a seasoned data professional, you will play a pivotal role in leading multiple data engagements on cloud-based platforms for data lakes, engineering, migration, warehouse, and business intelligence.Design and develop distributed systems for high-volume multi-thread batch, real-time, and event processing.Implement robust processes to...

  • Data Solutions Architect

    4 days ago

    Shimoga, Karnataka, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Data Solutions ArchitectAbout the RoleWe are seeking an experienced data solutions architect with expertise in large-scale data processing, distributed systems, and cloud-based architectures to join our team.Key Responsibilities:Design and develop scalable data pipelines and distributed data processing systems.Work extensively with big data...