Principal Engineer - Agentic Security Operations (remote)

They are passionate about what they do, and we are equally passionate about fostering an environment where all feel welcome, supported, and empowered to be innovative and reach their full potential. We have also earned awards as a best place to work for women, diversity and LGBTQ+ employees, and have been included on more than 50 regional best places to work lists. Our Security Operations Center (SOC) protects the enterprise by detecting, investigating, and responding to cybersecurity threats.
As a member of our Threat Detection Engineering team operating within our SOC, you will build governed agent capabilities to support data onboarding, detection engineering, threat hunting, investigation, response, automation, and platform operations. The Principal Engineer provides hands-on technical leadership to turn this strategy into secure, reliable, and supportable production capabilities across the SOC.

*We are open to remote candidates for this role

What You’ll Do

Lead the architecture and hands-on implementation of Agentic Security Operations capabilities across the SOC.
Define and evolve the reference architecture for governed data services, agent capabilities, workload routing, execution environments, and deterministic orchestration.
Establish engineering standards for creating, testing, registering, approving, releasing, monitoring, and retiring agents and their capabilities.
Advance detection-as-code practices, including versioning, simulation, deployment, tuning, migration, and rollback.
Build governed knowledge and context services that provide authoritative documentation, policies, operational history, citations, freshness information, and permission-aware retrieval.
Establish agent evaluation and observability through historical replay, synthetic scenarios, expected-versus-actual comparisons, policy tests, traces, outcome verification, and regression testing.
Measure quality, security, reliability, latency, cost, analyst acceptance, and operational impact.
Engineer reliable integrations across search, case management, workflow, communications, and response systems, including retries, idempotency, partial failures, and reconciliation.
Design production reliability, continuity, recovery, and support models for agent capabilities and security data services.
Collaborate with enterprise architecture, infrastructure, data, risk, privacy, legal, and compliance teams.
Mentor engineers, establish technical standards, review designs, and influence Threat Detection Engineering strategy.
Present complex technical decisions, risks, tradeoffs, and investment recommendations clearly to engineering and executive stakeholders.
Partner across Threat Detection and Response groups, including engineering, threat hunting, security analysis, investigation, and incident response stakeholders on initiatives and to translate operational feedback into improvements within our agentic harness, evaluations, safeguards, workflows, and other platforms.
Extensive experience designing and operating enterprise security, software, data, or distributed systems at scale.
Demonstrated success as a principal-level technical leader while remaining actively involved in architecture, prototyping, coding, testing, and production delivery.
Hands-on experience delivering production AI-assisted or agentic systems that use tools, memory, retrieval, planning, and structured workflows.
Strong understanding of security operations, including telemetry management, detection engineering, threat hunting, triage, investigation, response, and automation.
Experience with high-volume streaming and batch data pipelines, object-based storage, elastic analytics, distributed query, schema management, and data-quality controls.
Experience designing modular agents, reusable capabilities, workload routing, local and managed execution, and human-in-the-loop workflows.
Experience evaluating nondeterministic systems using versioned test sets, historical replay, synthetic scenarios, regression testing, and measurable outcome criteria.
Strong software-engineering practices, including source control, automated testing, continuous delivery, infrastructure automation, peer review, and rollback.
Knowledge of AI security risks such as prompt injection, data poisoning, excessive agency, sensitive-data exposure, insecure tool use, and model or workflow drift.
Strong production-engineering experience covering observability, service objectives, capacity, failure isolation, continuity, recovery, and operational support.
Ability to evaluate emerging capabilities critically, distinguish demonstrated functionality from product claims, and validate architectural assumptions through controlled testing.
Excellent written and verbal communication skills, including the ability to document architectures, operating models, engineering standards, and executive recommendations.
Hands-on experience with cloud-scale data and analytics platforms supporting security operations, including distributed query, high-volume data processing, object storage, schema management, data pipelines, and the operational patterns required to run security analytics at scale (e.g., Databricks, Snowflake, etc.)
Experience operating and migrating security analytics and detection workloads across modern platforms, along with strong working knowledge of major public cloud environments (AWS, Azure, GCP) and the identity, networking, logging, data protection, and workload-security controls required to secure them.
A minimum of 10 years’ experience in cybersecurity or relevant fields.
Bachelor’s degree in computer science, Information Technology, Cybersecurity, a related field, or equivalent work experience.
Experience modernizing a centralized security analytics environment into a decoupled architecture with customer-controlled data and replaceable compute or control layers.
Experience migrating security content while validating semantic equivalence, historical fidelity, and operational parity.
Background in detection engineering, incident response, threat hunting, adversary emulation, or security automation.
Experience defining governance and autonomy frameworks for AI-enabled production systems.
Experience building internal platforms or shared engineering services used by multiple security functions.
A leadership-approved Agentic Security Operations reference architecture and governance model.
Reusable capabilities spanning data, detection, automation, and other SOC functions.
Measurable reductions in engineering effort and operational cycle time without sacrificing security or human accountability.
A production platform with clear ownership, support procedures, recovery controls, and an intentional path away from vendor or architecture lock-in.
First American will consider for employment all qualified applicants, including those with arrest or conviction records, in a manner consistent with the requirements of applicable state and local laws (e.g., First American reasonably believes that a criminal history may have a direct, adverse and negative relationship with the following material job duties for this position potentially resulting in the withdrawal of the conditional offer of employment: handling of confidential, proprietary or trade secret information belonging to First American or its customers, administrating or facilitating financial transactions, and the ability to meet customer-imposed criminal history requirements.
By choice, we don’t simply accept individuality – we embrace it, we support it, and we thrive on it! Based on eligibility, First American offers a comprehensive benefits package including medical, dental, vision, 401k, PTO/paid sick leave and other great benefits like an employee stock purchase plan.