Support in SIEM (Splunk) software administration and log collection

EXPERIENCE AND EDUCATION:

Essential Qualifications/Experience:

·         Splunk Automated Management: Challenges of managing 350+ nodes via Git/Ansible while ensuring zero manual intervention and configuration consistency

·         Architectural Understanding: Deep dive into the T2/T3 logic and data flow across a multi-site, distributed environment where all data are stored at T2

·         Role Delineation: Clear distinction between the Splunk Administrator’s technical maintenance duties and the Security Analyst’s investigative functions

·         Technical Integration & Co-management: Challenges of administering Splunk services and log collection on Linux servers where the OS is managed by a separate entity (e.g., privileged access, SELinux, Syslog)

·         Stakeholder Coordination: A structured approach to the end-to-end onboarding process, including technical support for external log source owners

·         Previous similar experience

·         Expert Splunk Administrator (Minimum 2 years of experience working on complex & distributed environments)

ü  Distributed Architecture: Proven experience managing and scaling complex Splunk environments, including Indexer Clustering, Search Head Clustering, and multi-site deployments

ü  Advanced Management: Deep knowledge of Splunk configuration files, data lifecycle management, and managing large-scale deployments via Deployment Servers

ü  Critical Distinction: Proposed team members have to specifically be Splunk Administrators. Proposals with team members whose experience is primarily as a Security Analysts (end-user of the UI/investigating alerts) will not be considered

·         2+ years of hands-on experience in a Linux environment, with a proven track record in:

ü  CLI & System Navigation: Advanced command-line operations, file system management, and permissions (UID/GID, ACLs).

ü  Service Management: Demonstrated ability to independently install, configure, and troubleshoot application services (specifically Splunk) on Linux-based servers.

ü  Technical Scope: Focus is on application-level deployment; hardware configuration and kernel-level patching are excluded from the minimum requirements

·         Practical Networking & IT Security Knowledge

ü  Networking Fundamentals: Solid understanding of core protocols, specifically: DNS, HTTP(S), SSH, syslog, TCP/IP, and TLS/SSL.

ü  Security Mindset: Strong grasp of IT security principles, including:

ü  Log integrity

ü  Encryption in transit

ü  Role-Based Access Control (RBAC)

·         Practical Automation & Programming Knowledge

ü  Infrastructure Automation: Demonstrated ability to create and execute Ansible playbooks for automated infrastructure and configuration management

ü  Scripting & Development: Proven ability to write and maintain functional scripts in Python and Bash for data processing or task automation

ü  Version Control: Proficiency in using GitHub (or similar Git-based tools) for configuration management, including branching, committing, and merging code

DUTIES/ROLE: 

·         SIEM Infrastructure and Software Management

ü  Management of Splunk components deployed within 50+ T3 enclaves across high-side and low-side networks

ü  Operation and maintenance of a T2 SIEM environment composed of 80+ Linux servers (virtual and physical)

ü  Administration of the full Splunk software stack, including:

o   Splunk Enterprise

o   Splunk Enterprise Security

o   Splunk SOAR

o   Splunk UBA

ü  Management of Splunk deployments across more than 350 servers spanning T2 and T3 environments

ü  Implementation and operation of fully automated deployment and configuration mechanisms based on Ansible and Git

·         Log Collection and Data Management

ü  Collection of logs from more than 20,000 endpoints, appliances, and cloud-based solutions.

ü  Ensuring end-to-end log lifecycle management, including:

o   Data collection and ingestion

o   Parsing and normalization

o   Storage and retention

o   Categorization and enrichment

o   Monitoring of data flows and data quality

ü  Support and integration of new data sources into the T2 Splunk environment, including:

o   Project-driven onboarding

o   Continuous log collection improvements

o   Customer-driven requests

ü  Coordination with customers for the deployment and configuration of devices hosting log sources, including:

o   Acting as the technical point of contact for log collection setup

o   Supporting customers during the configuration of endpoints, appliances, and other log sources

o   Ensuring proper follow-up with customers until log sources are correctly configured and successfully integrated into the Splunk platform

o   Clarifying that endpoints, appliances, and other log sources are configured by the customer, with technical guidance and support provided by the SIEM engineer

·         Platform Configuration and System-Level Support

ü  Configuration and management of Splunk components hosted on Linux servers within T2 and T3 environments

ü  Execution of system-level activities requiring privileged access, including but not limited to:

o   Syslog server configuration

o   SELinux configuration

o   Other OS-level configurations necessary for proper Splunk operation

ü  Coordination with the entity responsible for Linux operating system management where responsibilities overlap

·         SIEM Reliability and Operational Quality

ü  Ensuring that Splunk Enterprise Security is properly configured, operational, and functioning as intended

ü  Verification that correlation rules are correctly deployed and operate reliably

ü  Ensuring the overall quality, stability, and reliability of SIEM services delivered to security analysts

ü  Continuous monitoring of platform health and service performance