Delta Lake Architect

2 weeks ago


Delhi, Delhi, India Thoucentric Full time
Job Description

As a Big Data Engineer, you will be responsible for designing, developing, and maintaining our big data infrastructure.

You will work with large datasets, perform data processing, and support various business functions by creating data pipelines, data processing jobs, and data integration solutions.

You will be working in a dynamic and collaborative environment, leveraging your expertise in Hive, Hadoop, and PySpark to unlock valuable insights from our data.


Key Responsibilities:

Data Ingestion and Integration:

  • Develop and maintain data ingestion processes to collect data from various sources.
  • Integrate data from different platforms and databases into a unified data lake.

Data Processing:

  • Create data processing jobs using Hive and PySpark for large-scale data transformation.
  • Optimize data processing workflows to ensure efficiency and performance.

Data Pipeline Development:

  • Design and implement ETL pipelines to move data from raw to processed formats.
  • Monitor and troubleshoot data pipelines, ensuring data quality and reliability.

Data Modeling and Optimization:

  • Develop data models for efficient querying and reporting using Hive.
  • Implement performance tuning and optimization strategies for Hadoop and Spark.

Data Governance:

  • Implement data security and access controls to protect sensitive information.
  • Ensure compliance with data governance policies and best practices.

Collaboration:

  • Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide data support.
Requirements


Qualifications:

Bachelor's degree in computer science, Information Technology, or a related field.

8+ years of experience in big data engineering and data processing.

Proficiency in Hive, Hadoop, Airflow and PySpark.4

Strong SQL and NoSQL database experience.

Experience with data warehousing and data modeling.

Knowledge of data integration, ETL processes, and data quality.

Strong problem-solving and troubleshooting skills.

Preferred

Qualifications:

Experience with cloud-based big data technologies (e.g., AWS, Azure and GCP).

Certification in Hadoop, Hive, or PySpark.

ThisBenefits What a Consulting role at Thoucentric will offer you?

Opportunity to define your career path and not as enforced by a manager.

A great consulting environment with a chance to work with Fortune 500 companies and startups alike.

A dynamic but relaxed and supportive working environment that encourages personal development.

Be part of One Extended Family. We bond beyond work - sports, get-togethers, common interests etc.

Work in a very enriching environment with Open Culture, Flat Organization and Excellent Peer Group.

Be part of the exciting Growth Story of Thoucentric

RequirementsHive, Hadoop, Pyspark, delta lake
  • Solutions Architect

    2 weeks ago


    Delhi, Delhi, India Koantek Full time

    The AWS/Azure/GCP Architect at Koantek builds secure, highly scalable big data solutions to achieve tangible, data-driven outcomes all the while keeping simplicity and operational effectiveness in mind.This role collaborates with teammates, product teams, and cross-functional project teams to lead the adoption and integration of the Databricks Lakehouse...

  • Senior Data Engineer

    2 weeks ago


    Delhi, Delhi, India Coders Brain Technology Pvt. Ltd. Full time

    Position: Senior Data EngineerExperience Required: 8+ yearsSalary: Market Standard or as per budgetNotice Period: ImmediateJob Type: Full-timeLocation: RemoteJob Description:Candidate should have 8+ years of experience in Data engineering, with at least 5+ years as a Big Data architect/solution architect.Proven experience as an Architect/solution architect...

  • Data Architect

    2 weeks ago


    Delhi, Delhi, India FutureBridge Full time

    Job Responsibilities:Interact closely with Business Leaders and Product Team to design, develop and implement an overall organizational data strategy that is in line with business processes.These data strategies include data model designs (Conceptual, Logical and Physical), implementation and management of Data Warehouses / Data Lakes / Data Lakehouse and...


  • Delhi, Delhi, India Marlabs LLC Full time

    :What you'll do Spearhead the development of conceptual, logical, and physical data models, while implementing data marts/data warehouse/data lake/data lakehouse on Spark-based platforms Analyse existing models to identify variances/gaps and devise strategies to achieve alignment. Establish and enforce best practices and standards for data models. Generate...


  • Delhi, Delhi, India Exusia Full time

    About The PositionDepartment:Sales and Delivery Team - EmpowerIndustry:Information Technology & Services, Computer Software, Management ConsultingLocation:BangaloreExperience Range: 5+ yearsBasic Qualification:Master/Bachelor of Engineering or EquivalentTravel Requirements:Not requiredWebsite:Exusia, a cutting-edge digital transformation consultancy, seeks...

  • Data Engineer

    2 weeks ago


    Delhi, Delhi, India Neurones IT Asia Full time

    As a Data engineer , you will be a part of a highly qualified team, leading the delivery of modern data platforms using Azure Services and Databricks.Designing and delivering software using an agile and iterative approach based on Scrum or KanbanFollowing and contributing to improvement of client's software engineering practices e.g., source code management,...

  • Data Architect

    2 weeks ago


    Delhi, Delhi, India Ascendion Full time

    Job Title:Data Architect (with Team Management experience)Location:Bangalore (Remote)Position Overview:We are in search of a seasoned Senior Data Architect with 12-18 years of experience, particularly someone with a solid background in SQL Server-based platforms or application architectures and a profound expertise in Azure data pipelines, Data Lake,...

  • Data Architect

    2 weeks ago


    Delhi, Delhi, India Ascendion Full time

    Role: Data Architect/AI LeaderLocation: Bangalore (Remote)We are in search of a seasoned Senior Data Architect with10-15 years of experience, particularly someone with a solid background inSQL Server-basedplatforms or application architectures and a profound expertise in Azuredata pipelines, Data Lake, Databricks, and Power BI. The role is pivotal for...

  • Data Architect

    2 weeks ago


    Delhi, Delhi, India Valiance Solutions Full time

    About Us:Valiance is a global AI & Data analytics firm helping clients build cutting-edge technology solutions for digital transformation. We work with some of the marquee brands across India, US and APAC to build transformative solutions for Credit Risk, Fraud, Predictive Maintenance, Quality Inspection, Data lake, IOT analytics etc. Our team comprises 120+...


  • Delhi, Delhi, India Visvero, Inc. Full time

    Job DescriptionThis is a remote position.Architect and develop robust frameworks for efficiently onboarding new users, user groups, and security configuration onto the Databricks Platform and Unity Catalog.Design and implement security control processes tailored for integrating new data and users, ensuring compliance.Build scalable and maintainable...

  • Senior Developer

    2 weeks ago


    Delhi, Delhi, India Delta Air Lines Full time

    Responsibilities Include, But Are Not Limited To :Implement & support code for break-fix, maintenance, and enhancements, and integrating container platforms with automated CI/CD pipelinesDesign and develop cloud-native applications, features, and enhancements with an emphasis on clean, highly efficient, supportable, and well documented code for the full...

  • Consultant

    2 weeks ago


    Delhi, Delhi, India Tredence Full time

    About TredenceTredence is a global analytics services and solutions company. We are one of the fastest growing private companies in the country for three straight years according to the Inc. 5000 and we continue to set ourselves apart from our competitors by attracting the greatest talent in the data analytics and data science space. Our capabilities range...


  • Delhi, Delhi, India Marlabs LLC Full time

    :Job Title: Data Architect - Azure DatabricksLocation: Bangalore, IndiaPosition Overview:We are looking for a Data Architect with 10 to 12 years of experience specializing in Azure Databricks and Azure native services. The ideal candidate will have a proven track record of designing and implementing scalable, high-performance data solutions in cloud...


  • Delhi, Delhi, India LTIMindtree Full time

    LTIMindtree is proud to be an equal opportunity employer. We are committed to equal employment opportunity regardless of race, ethnicity, nationality, gender, gender-identity, gender expression, language, age, sexual orientation, religion, marital status, veteran status, socio-economic status, disability or any other characteristic protected by applicable...

  • Data Architect

    2 weeks ago


    Delhi, Delhi, India Indium Software Full time

    Title:Data ArchitectExperience:10-13 YearsWork Location:Chennai/BangaloreWork Mode:HybridSkills:Data Modelling, Database Concepts, OLAP, OLTPRoles and Responsibilities:Experience with Big Data Technologies like Apache Spark, Kafka, Cloud Native, Data Lakes in a customer-oriented post-sales, technical architecture or consulting position.6+ years working on...

  • Data Architect

    2 weeks ago


    Delhi, Delhi, India Photon Full time

    We are seeking a Data Architect to lead a Generative AI initiative. The ideal candidate will have a deep understanding of Azure services and capabilities, and experience with architecting and implementing solutions using Azure. They must also possess knowledge of Python, APIs and machine learning frameworks.ResponsibilitiesDesign and architect solutions that...

  • Architect T500-12019

    2 weeks ago


    Delhi, Delhi, India Delta Air Lines Full time

    Responsibilities include, but are not limited to :Owns technical decisions and direction of the Crew Portfolio.Design applications required to automate business processes as an integrated solution.Plays an active role in the overall technical architecture and owns the technical responsibility of the success of cloud/non-cloud solutions.Overseeing each phase...

  • Data Architect

    2 weeks ago


    Delhi, Delhi, India Trigyn Technologies Full time

    Job Description:Required Profile:10+ years of professional experience in Data Management, Information Systems and other BI-related fields is required.+2 years of data architect experience.+1 year of public cloud developmentsExperience with Incorta and Oracle CloudExperience with Data Modeler tool.Experience working as a member of distributed data...

  • Data Architect

    2 weeks ago


    Delhi, Delhi, India Trigyn Technologies Full time

    Job Description: Required Profile:•10+ years of professional experience in Data Management, Information Systems and other BI-related fields is required.•+2 years of data architect experience.•+1 year of public cloud developments•Experience with Incorta and Oracle Cloud•Experience with Data Modeler tool.•Experience working as a member of...

  • Azure Data Architect

    2 weeks ago


    Delhi, Delhi, India Ray Business Technologies (A CMMI Level 3 Company) Full time

    Azure Data ArchitectResponsibilities:Design and implement scalable, reliable, and secure data solutions on AzureCollaborate with data engineers and developers to integrate data solutions into existing systemsDevelop and maintain data models, data flows, and data dictionariesOptimize data storage, processing, and retrieval for performance and...