Delta Lake Architect

3 weeks ago

bangalore, India Thoucentric Full time



As a Big Data Engineer, you will be responsible for designing, developing, and maintaining our big data infrastructure. You will work with large datasets, perform data processing, and support various business functions by creating data pipelines, data processing jobs, and data integration solutions. You will be working in a dynamic and collaborative environment, leveraging your expertise in Hive, Hadoop, and PySpark to unlock valuable insights from our data.

 

Key Responsibilities:

Data Ingestion and Integration:

·         Develop and maintain data ingestion processes to collect data from various sources.

·         Integrate data from different platforms and databases into a unified data lake.

 

Data Processing:

·         Create data processing jobs using Hive and PySpark for large-scale data transformation.

·         Optimize data processing workflows to ensure efficiency and performance.

 

Data Pipeline Development:

·         Design and implement ETL pipelines to move data from raw to processed formats.

·         Monitor and troubleshoot data pipelines, ensuring data quality and reliability.

 

Data Modeling and Optimization:

·         Develop data models for efficient querying and reporting using Hive.

·         Implement performance tuning and optimization strategies for Hadoop and Spark.

 

Data Governance:

·         Implement data security and access controls to protect sensitive information.

·         Ensure compliance with data governance policies and best practices.

 

Collaboration:

·         Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide data support.

 


Requirements

Qualifications:

    • Bachelor’s degree in computer science, Information Technology, or a related field.
    • 8+ years of experience in big data engineering and data processing.
    • Proficiency in Hive, Hadoop, Airflow and PySpark.4
    • Strong SQL and NoSQL database experience.
    • Experience with data warehousing and data modeling.
    • Knowledge of data integration, ETL processes, and data quality.
    • Strong problem-solving and troubleshooting skills.

     

    Preferred Qualifications:

    • Experience with cloud-based big data technologies (e.g., AWS, Azure and GCP).
    • Certification in Hadoop, Hive, or PySpark.

This 
Benefits What a Consulting role at Thoucentric will offer you?
  • Opportunity to define your career path and not as enforced by a manager.
  • A great consulting environment with a chance to work with Fortune 500 companies and startups alike.
  • A dynamic but relaxed and supportive working environment that encourages personal development.
  • Be part of One Extended Family. We bond beyond work - sports, get-togethers, common interests etc.
  • Work in a very enriching environment with Open Culture, Flat Organization and Excellent Peer Group.
  • Be part of the exciting Growth Story of Thoucentric

  • Technical Architect

    6 days ago

    bangalore, India Impetus Full time

    Job DescriptionOverall 8-14 yrs. of Data Engineering experience with 4+ years of hands on experience in Databricks.Proven experience as a Databricks Architect or similar role with a deep understanding of the Databricks platform and its capabilities.Analyze business requirements and translate them into technical specifications for data pipelines, data lakes,...

  • Databricks Architect

    7 days ago

    bangalore, India Impetus Full time

    Joh Description Overall 8-14 yrs. of Data Engineering experience with 4+ years of hands on experience in Databricks. Proven experience as a Databricks Architect or similar role with a deep understanding of the Databricks platform and its capabilities. Analyze business requirements and translate them into technical specifications for data pipelines, data...

  • Azure data engineer

    3 weeks ago

    bangalore, India LTIMindtree Full time

    Role : ADF+ADBExperience : 3 to 12 yearsLocation : Pan indiaNotice : Immediate to 30 days maxResponsibilities:1. Collaborate with technical architect and cloud solutions teams to architect and design data pipelines, data marts, and reporting solutions.2. Code, test, and optimize Databricks Jobs for efficient data processing, ETL tasks, and report...

  • Data Architect

    1 month ago

    bangalore, India Bristlecone Full time

    Specific Requirements at a Glance15 years of experience having architected multiple data solutions on Google Cloud primarily, like Data Warehouse, Data Lake, Data Lakehouse, etc. (experience in designing Data Mesh would be a huge plus)Good data modeling skills, having modelled Common Data Platforms for enterprisesUnderstanding of data integration / ETL...

  • Data Architect

    3 weeks ago

    bangalore, India Bristlecone Full time

    Specific Requirements at a Glance15 years of experience having architected multiple data solutions on Google Cloud primarily, like Data Warehouse, Data Lake, Data Lakehouse, etc. (experience in designing Data Mesh would be a huge plus)Good data modeling skills, having modelled Common Data Platforms for enterprisesUnderstanding of data integration / ETL...

  • Data Architect

    4 days ago

    bangalore, India Synechron Full time

    Good-day,We have immediate opportunity for Big Data Architect– 15 to 20 years.Job Role: Big data architectJob Location: BangaloreExperience- 12+Notice Period: ImmediateAbout Company:At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver...

  • Data Architect

    3 weeks ago

    bangalore, India Greenway Health Full time

    You Belong at Greenway Bring your best and truest self. We celebrate what makes us different and what brings us all together. At Greenway Health, we are committed to an inclusive environment and a culture of belonging as we pursue our purpose of healthier communities, successful providers, and empowered patients. We are united in our goal to build the...

  • Data Architect

    1 month ago

    bangalore, India Greenway Health Full time

    You Belong at Greenway Bring your best and truest self. We celebrate what makes us different and what brings us all together. At Greenway Health, we are committed to an inclusive environment and a culture of belonging as we pursue our purpose of healthier communities, successful providers, and empowered patients. We are united in our goal to build the...

  • Databricks Architect

    2 months ago

    Bangalore, Karnataka, India TecHHire Global Full time

    Job Description :Function : Technical Management - Engineering Management, Software Engineering - Backend Development, Big Data / DWH / ETLResponsibilities :- Design and implement large-scale data processing systems using the Databricks platform.- Work closely with clients to understand their requirements and design solutions that meet their needs.-...

  • Databricks Architect

    3 weeks ago

    Bangalore, India TecHHire Global Full time

    Job Description :Function : Technical Management - Engineering Management, Software Engineering - Backend Development, Big Data / DWH / ETLResponsibilities :- Design and implement large-scale data processing systems using the Databricks platform.- Work closely with clients to understand their requirements and design solutions that meet their needs.-...

  • Databricks Architect

    2 months ago

    bangalore, India TecHHire Global Full time

    Job Description :Function : Technical Management - Engineering Management, Software Engineering - Backend Development, Big Data / DWH / ETLResponsibilities :- Design and implement large-scale data processing systems using the Databricks platform.- Work closely with clients to understand their requirements and design solutions that meet their needs.-...

  • Databricks Architect

    5 days ago

    bangalore, India TecHHire Global Full time

    Job Description :Function : Technical Management - Engineering Management, Software Engineering - Backend Development, Big Data / DWH / ETLResponsibilities :- Design and implement large-scale data processing systems using the Databricks platform.- Work closely with clients to understand their requirements and design solutions that meet their needs.-...

  • Databricks Architect

    2 months ago

    Bangalore, India TecHHire Global Full time

    Job Description :Function : Technical Management - Engineering Management, Software Engineering - Backend Development, Big Data / DWH / ETLResponsibilities :- Design and implement large-scale data processing systems using the Databricks platform.- Work closely with clients to understand their requirements and design solutions that meet their needs.-...

  • Databricks Architect

    3 weeks ago

    Bangalore, Karnataka, India TecHHire Global Full time

    Job Description :Function : Technical Management - Engineering Management, Software Engineering - Backend Development, Big Data / DWH / ETLResponsibilities :- Design and implement large-scale data processing systems using the Databricks platform.- Work closely with clients to understand their requirements and design solutions that meet their needs.-...

  • Azure Cloud Data Solution Architect

    2 months ago

    Bangalore/Noida/Hyderabad/Pune, IN Nilasu consulting Full time

    Position Overview:We are seeking a highly skilled and experienced Azure-Cloud Data Solution Architect to join our team. The Solution Architect will be responsible for designing and implementing end-to-end data solutions, including Data Warehouse, ETL, Data Lake, and Cloud Migration projects. The ideal candidate will have extensive experience in data...

  • Azure Cloud Data Solution Architect

    2 months ago

    Bangalore/Noida/Hyderabad/Pune, India Nilasu consulting Full time

    Position Overview:We are seeking a highly skilled and experienced Azure-Cloud Data Solution Architect to join our team. The Solution Architect will be responsible for designing and implementing end-to-end data solutions, including Data Warehouse, ETL, Data Lake, and Cloud Migration projects. The ideal candidate will have extensive experience in data...

  • Azure Cloud Data Solution Architect

    3 weeks ago

    Bangalore/Noida/Hyderabad/Pune, India Nilasu consulting Full time

    Position Overview:We are seeking a highly skilled and experienced Azure-Cloud Data Solution Architect to join our team. The Solution Architect will be responsible for designing and implementing end-to-end data solutions, including Data Warehouse, ETL, Data Lake, and Cloud Migration projects. The ideal candidate will have extensive experience in data...

  • Azure Cloud Data Solution Architect

    3 weeks ago

    Bangalore/Noida/Hyderabad/Pune, IN Nilasu consulting Full time

    Position Overview:We are seeking a highly skilled and experienced Azure-Cloud Data Solution Architect to join our team. The Solution Architect will be responsible for designing and implementing end-to-end data solutions, including Data Warehouse, ETL, Data Lake, and Cloud Migration projects. The ideal candidate will have extensive experience in data...

  • Azure Databricks Engineer

    2 months ago

    Bangalore, India ResourceTree Global Services Full time

    Job Title : Senior Databricks Data EngineerSkillset : Databricks, Datalake Architecture, AzureExperience : 6 - 8 yearsLocation : ChennaiJob Summary : We are seeking an experienced Senior Databricks Data Engineer with a very good understanding of Databricks, Lakehouse concepts, with architecture experience, and a sound knowledge of Azure and its services....

  • Data Architect

    7 days ago

    bangalore, India Synechron Full time

    Good-day, We have immediate opportunity for Big Data Architect– 15 to 20 years. Job Role: Big data architect Job Location: Bangalore Experience- 12+ Notice Period: Immediate About Company: At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology...