Specialist Data/AI Engineering

Job Description:

Roles & Responsibilities

• Minimum 6+ years of experience in the fundamental technology work of architecting/designing/building/managing/Optimizing the core frameworks for our cloud-based system platforms and end-to-end solutions.

• Responsible for optimization of cloud infrastructure, including Databricks, ADLS, Blob, VMs etc.,

• Conduct Proof-of-Concept activities in support of advanced use cases.

• Must have architectural focus for ensuring availability, reliability and scalability are necessary

• Responsible for documenting the technical solutions

• Administer and manage Azure Databricks workspaces, clusters, and jobs with a focus on maximizing efficiency and performance.

• Implement and manage Unity Catalog to provide centralized governance and fine-grained access control across data assets.

• Design and execute cost optimization strategies, including monitoring cluster usage, autoscaling, and resource allocation to minimize cloud expenses.

• Collaborate with data engineers and security teams to enforce data governance, compliance, and security policies using Unity Catalog.

• Automate deployment, scaling, and monitoring tasks using scripting and cloud-native tools.

• Troubleshoot and resolve issues related to Databricks clusters, jobs, and Unity Catalog configurations.

• Responsible for working with international teams spread across multiple time zones.

Must Have:

• 6 to 8 years of hands-on experience administering Azure Databricks in cloud environments.

• In-depth knowledge and experience with Unity Catalog for data governance and security.

• Proven expertise in cost optimization for Databricks clusters and cloud resources.

• Strong understanding of Databricks architecture, cluster management, and job orchestration.

• experience in RDBMS, ANSI SQL/PL SQL, Hadoop, Python and/or Java,spark/Pyspark.

• Proficiency in Spark architecture, including cluster management, job scheduling, and performance tuning.

• Experience in  Azure DevOps, Git/Github.

• Azure Event Hub/ Azure Event Grid / Kafka

• Hands-on experience with scripting languages such as Python, Scala, or SQL.

• Experience with infrastructure-as-code tools like Terraform or ARM templates.

• Solid understanding of data security, governance, and compliance standards.

• Excellent problem-solving skills and ability to work in a collaborative environment.

• Strong communication skills and ability to explain technical concepts to non-technical stakeholders.

Additional experience:

• Hadoop Administration and/or Azure cloud Administration 

• Working with Object storage ( ADLS/Blob or S3), Kubernetes (preferably AKS)

• Azure AD/ OpenID /OAuth/ SAML

• NSG /Firewall deployment and usage

• DNS configuration and management

• Automation through Ansible or Teraform or Azure DevOps

• Basic Linux administration and Shell scripting (Bash)

• Good documentation skills

• Ability to troubleshoot to isolate and diagnose common technical problems

• Monitoring and use of monitoring tools for managing critical infrastructure

Preferred Certifications:

• Microsoft Certified: Azure Administrator Associate/ Microsoft Certified: Azure Solutions Architect Expert

• Databricks Accredited Platform Administrator Accreditation

#DataEngineering

Weekly Hours:

Time Type:

Location:

It is the policy of AT&T to provide equal employment opportunity (EEO) to all persons regardless of age, color, national origin, citizenship status, physical or mental disability, race, religion, creed, gender, sex, sexual orientation, gender identity and/or expression, genetic information, marital status, status with regard to public assistance, veteran status, or any other characteristic protected by federal, state or local law. In addition, AT&T will provide reasonable accommodations for qualified individuals with disabilities. AT&T is a fair chance employer and does not initiate a background check until an offer is made.