Databricks with AWS

1 day ago

Hyderabad, Telangana, India Cognizant Technology Solutions Full time ₹ 35,00,000 - ₹ 1,20,00,000 per year

AIA - Hyderabed Drive

Job Summary

We are seeking a Sr. Developer with 7 to 12 years of experience to join our team. The ideal candidate will have expertise in Spark in Scala Delta Sharing and Databricks Unity Catalog Admin. This role involves working with cutting-edge technologies like Databricks CLI Delta Live Pipelines and Structured Streaming. The candidate will contribute to risk management and data solutions using Apache Airflow Amazon S3 and Amazon Redshift. Proficiency in Python Databricks SQL and PySpark is esse

Responsibilities

  • Develop and maintain scalable data pipelines using Spark in Scala to process large datasets efficiently.
  • Implement Delta Sharing and Databricks Unity Catalog Admin to ensure secure and seamless data sharing across platforms.
  • Utilize Databricks CLI and Delta Live Pipelines to automate data workflows and enhance operational efficiency.
  • Design and execute Structured Streaming applications to process real-time data streams with precision.
  • Collaborate with cross-functional teams to integrate risk management solutions into data processing frameworks.
  • Leverage Apache Airflow to orchestrate complex data workflows and ensure timely data delivery.
  • Manage data storage and retrieval using Amazon S3 and Amazon Redshift to optimize performance and cost.
  • Apply Python programming skills to develop robust data processing scripts and applications.
  • Utilize Databricks SQL to perform advanced data analysis and generate actionable insights.
  • Implement Databricks Delta Lake to ensure data reliability and consistency across the data lifecycle.
  • Configure and manage Databricks Workflows to streamline data operations and improve productivity.
  • Develop PySpark applications to process and analyze large-scale datasets with high efficiency.
  • Contribute to the companys data strategy by delivering innovative solutions that drive business growth and societal impact.

Qualifications

  • Possess a strong understanding of Spark in Scala and its application in data processing.
  • Demonstrate expertise in Delta Sharing and Databricks Unity Catalog Admin for secure data management.
  • Have experience with Databricks CLI and Delta Live Pipelines for workflow automation.
  • Be proficient in Structured Streaming for real-time data processing.
  • Show capability in risk management integration within data frameworks.
  • Have hands-on experience with Apache Airflow for workflow orchestration.
  • Be skilled in using Amazon S3 and Amazon Redshift for data storage solutions.
  • Exhibit proficiency in Python for data processing and scripting.
  • Be adept at using Databricks SQL for data analysis.
  • Understand the principles of Databricks Delta Lake for data consistency.
  • Have experience managing Databricks Workflows for operational efficiency.
  • Be skilled in developing PySpark applications for large-scale data processing.

  • AWS-Databricks Engineer

    2 weeks ago

    Hyderabad, Telangana, India Coforge Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    We are seeking an experiencedAWS Databricks Developerto design, develop, and maintain scalable data processing and analytics solutions on theAWS and Databricksplatform. The ideal candidate will have strong expertise in building data pipelines, optimizing performance, and integrating multiple AWS services to deliver high-quality data solutions.Key...

  • aws databricks with devops

    5 days ago

    Hyderabad, Telangana, India Primus Global Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Type of RoleClient-facing roleMust to Have SkillSenior: 6+ YOE overall in DevOps2+ hands on databricksDevOps LeadDriving automationCI/CD pipelines optimising deploymentsInfra + code deploymentsCollab with data engineers across teamsAutomation process and DevOps solutionsGitHub and Databricks DevOps, AWSAzure DevOps/Integrated with GitHubControl

  • Sr. Data Engineer – AWS Databricks

    2 weeks ago

    Hyderabad, Telangana, India Gravity Infosolutions, Inc. Full time ₹ 8,00,000 - ₹ 12,00,000 per year

    Sr. Data Engineer – AWS Databricks / Data Tech LeadType: ContractLocation : Chennai / HydExperience: 5+ yearsDuration: 3 Months+ ExtendableKey ResponsibilitiesData Pipeline Development: Design and implement robust ETL/ELT pipelines using Databricks, PySpark, and Delta Lake to process structured and unstructured data efficiently.Manage job orchestration,...

  • AWS, Databricks, Python, Pyspark

    1 day ago

    Hyderabad, Telangana, India Infosys Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Job Description:Lead Data Engineer AWS Databricks Python PysparkKey Responsibilities:A day in the life of an InfoscionAs part of the Infosys consulting team your primary role would be to lead the engagement effort of providing high quality and value adding consulting solutions to customers at different stages from problem definition to diagnosis to solution...

  • Senior Databricks Engineer

    1 week ago

    Hyderabad, Telangana, India SID Information Technologies Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Role: Senior Data Engineer (Python, Spark/Databricks, SQL, AWS)Experience:6–12 yearsLocation:HyderabadWork Mode:Hybrid (3 days/week in-office)Join Time:ImmediateMust-Have Technical Skills:Strong programming skills in Python, DatabricksHands-on experience with Apache Spark & Databricks for big data processing on AWS cloudProficiency with AWS services ...

  • Databricks Engineer

    2 weeks ago

    Hyderabad, Telangana, India Anblicks Solutions Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Role & responsibilities5+ years of experience in data engineering roles.• Proficiency in programming languages such as Scala or Python, or Java.• Expertise in distributed data processing frameworks like Apache Spark or Flink.• Strong knowledge of cloud platforms (e.g., AWS) and cloud-native data platforms like Databricks or Redshift.• Solid...

  • Databricks Pre-Sales and Solutions Architect

    1 week ago

    Hyderabad, Telangana, India Tata Consultancy Services Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    I hope you're doing well I came across your profile and was impressed by your extensive experience as an/a (mention role/skill). We have a similar opportunity to join our dynamic team at TCS.Role: Databricks Pre-Sales and Solutions ArchitectLocation: PAN IndiaMode: Full TimeExperience: 10-16 yearsPre-Sales Support and Technical Leadership:a. Engage with...

  • Senior Databricks Administrator

    1 week ago

    Hyderabad, Telangana, India Sonatype Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    DescriptionKey Responsibilities :User Provisioning And ManagementOnboard and offboard users, groups, and service principals within Databricks, including integration with identity providers (IdPs) like Azure Active Directory or Okta via SCIM.Manage user roles and entitlements at both the account and workspace levels (Account Admins, Workspace Admins,...

  • Senior Databricks Engineer

    1 week ago

    Hyderabad, Telangana, India Xenon7 Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About us:Where elite tech talent meets world-class opportunitiesAt Xenon7, we work with leading enterprises and innovative startups on exciting, cutting-edge projects that leverage the latest technologies across various domains of IT including Data, Web, Infrastructure, AI, and many others. Our expertise in IT solutions development and on-demand resources...

  • Senior Databricks Administrator

    1 day ago

    Hyderabad, Telangana, India Sonatype Full time ₹ 15,00,000 - ₹ 30,00,000 per year

    Sonatype is the software supply chain security company. We provide the world's best end-to-end software supply chain security solution, combining the only proactive protection against malicious open source, the only enterprise grade SBOM management and the leading open source dependency management platform. This empowers enterprises to create and maintain...