Splunk Engineer

Job Description

Role Profile: Splunk Engineer
Shift: 7 pm IST to 4 am IST

Responsibilities

- Architect, engineer, implement, and administer Splunk solutions in highly available, redundant, distributed computing environments.
- Lead design and deployment of new Splunk environments, including clustered, multi-site, and large-scale configurations.
- Perform Splunk forwarder deployment, configuration, and troubleshooting across diverse platforms.
- Integrate, curate, and normalize diverse log sources into Splunk, ensuring CIM compliance and high data fidelity.
- Configure and maintain Splunk dashboards, searches, and alerts to meet PCI DSS logging requirements, and deliver evidentiary reports to auditors to support compliance verification
- Develop advanced content for SIEM correlation, including custom correlation searches, dashboards, and alerts.
- Administer, maintain, and tune Splunk components (Indexers, Search Heads, Forwarders, Cluster Masters, Deployer, Deployment Server, and License Master).
- Proactively monitor platform health using internal logs, KPIs, and custom monitoring solutions to identify and address performance bottlenecks.
- Lead capacity planning, storage forecasting, and continuity of operations for large Splunk deployments.
- Optimize Splunk performance through configuration tuning, search optimization, and data model acceleration strategies.
- Troubleshoot complex ingestion, performance, and search-related issues, identifying root causes and implementing sustainable fixes or workarounds.
- Reproduce customer or internal issues, document findings, and work with Splunk Support or vendor engineers for resolution.
- Create, maintain, and enforce Splunk engineering documentation, including SOPs, design diagrams, architecture runbooks, and KB articles.
- Develop custom scripts and automation tools (e.g., Python, Bash, PowerShell) to improve Splunk administration, onboarding, and operational workflows.
- Utilize Splunk APIs for integration with enterprise tools and automation frameworks.
- Serve as a technical escalation point for Splunk Engineer I/II and Splunk Admin roles.
- Administer, tune, and troubleshoot Splunk Enterprise Security, maintaining data models, correlation searches, and notable events pipeline.
- Configure and manage HEC (HTTP Event Collector) connections and onboard new data sources.
- Manage Splunk RBAC (Role-Based Access Control) including SAML and AD group integrations for search heads and API endpoints.
- Collaborate with security, infrastructure, application, and DevOps teams to ensure Splunk aligns with enterprise monitoring, compliance, and operational goals.
- Design and implement Splunk solutions supporting compliance frameworks (e.g., PCI DSS, HIPAA, SOX), including dashboard/report development and audit evidence.
- Research, evaluate, and implement new Splunk apps, add-ons, and integrations to enhance platform capabilities.
- Mentor junior Splunk engineers and guide cross-functional teams on Splunk best practices, search optimization, and data onboarding.

Requirements

- 8+ years of IT experience in technical engineering, security operations, or infrastructure roles.
- 5+ years of direct, hands-on Splunk engineering and administration experience in large-scale, distributed environments.
- Expert-level knowledge of Splunk Enterprise and Splunk Enterprise Security, including architecture, clustering, and scaling strategies.
- Proficiency in Linux/Unix administration and shell scripting.
- Strong knowledge of Splunk APIs, including use for automation and tool integrations.
- Expertise in regex, field extractions, and key-value parsing.
- Strong programming/scriptingskills in one or more languages (Python, Bash, PowerShell, Perl, JavaScript).
- Experience with storage systems (DAS, SAN, object storage) and understanding of their performance implications for Splunk indexing.
- Solid understanding of networking (switches, routers, firewalls, load balancers, DNS, SSL/TLS) and how it impacts Splunk architecture.
- Familiarity with Enterprise Management and automation tools.
- Experience with Splunk ITSI (preferred) and other premium Splunk apps.
- Strong knowledge of data formats including JSON, XML, and CSV.
- Demonstrated experience delivering Splunk-based compliance reporting and audit support.
- Strong communication skills for interacting with technical and non-technical stakeholders.
- Proven ability to lead projects, mentor team members, and provide architectural guidance.

Education & Certifications

- Bachelor's degree in Computer Science, Information Systems, or related technical field (or equivalent experience).
- Splunk Certified Architect and/or Splunk Certified Consultant preferred.
- Additional certifications in security, cloud, or automation tools are a plus.