(USA) Principal, Software Engineer

About the role

Position Summary...

What you'll do... Role summary:

The Principal Software Engineer on Walmart's Private Cloud Storage Engineering team architects and operates some of the largest Ceph storage clusters in the industry, supporting hundreds of petabytes of mission-critical data. This hands-on technical leader drives the design, scalability, performance optimization, and automation of distributed storage platforms across multiple global production environments. Partnering closely with computer, networking, security, and cloud engineering teams, they deliver highly available storage solutions that meet demanding SLAs for latency, throughput, resiliency, and scalability. The role requires deep expertise in Ceph, Linux internals, OpenStack, and large-scale infrastructure, along with a proven ability to mentor engineers, influence technical strategy, and contribute to open-source storage communities.

About the team:

Walmart’s Private Cloud Storage Engineering team designs and operates some of the largest Ceph storage clusters worldwide, managing extensive data across multiple sites. The team supports critical workloads for business operations, customer platforms, and innovation projects. Working at the convergence of distributed storage, Linux internals, networking, and cloud infrastructure, the team addresses complex challenges in scalability, performance, and resilience. Emphasizing technical expertise, automation, and observability, the team collaborates with engineers across computer, networking, security, and cloud domains to advance enterprise storage solutions at Walmart’s scale.

What you'll do:

- Architect, deploy, and manage large-scale Ceph storage clusters across global production environments, ensuring high availability, durability, and scalability through advanced CRUSH map design, erasure coding, and replication strategies.

- Design, develop, and maintain Ceph CSI and Cinder CSI drivers to provide persistent, resilient, and high-performance storage services for containerized workloads on Walmart's Cloud Native Platform.

- Troubleshoot and optimize performance across the storage, Linux, and networking stacks, leveraging tools such as perf, blktrace, bpftrace, and tcpdump to identify and resolve complex bottlenecks.

- Build and enhance automation for cluster provisioning, lifecycle management, capacity expansion, monitoring, and operations using Ansible, Terraform, and Python.

- Lead root cause analysis for critical production incidents, mentor engineers on storage architecture and best practices, and contribute to the Ceph and Kubernetes CSI open-source communities.

What you'll bring:

- 15+ years of experience in distributed storage systems and Linux infrastructure engineering, including 10+ years of hands-on expertise designing, deploying, and operating large-scale Ceph environments at petabyte scale.

- Extensive experience developing, enhancing, and supporting Ceph CSI and Cinder CSI drivers to provide persistent storage for cloud-native, containerized platforms.