Visible to the public Hardware Remediation at Scale

TitleHardware Remediation at Scale
Publication TypeConference Paper
Year of Publication2018
AuthorsLin, F., Beadon, M., Dixit, H. D., Vunnam, G., Desai, A., Sankar, S.
Conference Name2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W)
Date Publishedjun
Keywordsanomaly detection, Automated Response Actions, composability, datacenter, fault tolerant computing, Film bulk acoustic resonators, Hardware, hardware availability, hardware failure modes, hardware issues, hardware remediation, learning (artificial intelligence), machine learning, maintenance engineering, Monitoring, natural language processing, pubcrawl, remediation efficiency, remediation flow, remediation system, Resiliency, Servers, Transient analysis, transient errors
AbstractLarge scale services have automated hardware remediation to maintain the infrastructure availability at a healthy level. In this paper, we share the current remediation flow at Facebook, and how it is being monitored. We discuss a class of hardware issues that are transient and typically have higher rates during heavy load. We describe how our remediation system was enhanced to be efficient in detecting this class of issues. As hardware and systems change in response to the advancement in technology and scale, we have also utilized machine learning frameworks for hardware remediation to handle the introduction of new hardware failure modes. We present an ML methodology that uses a set of predictive thresholds to monitor remediation efficiency over time. We also deploy a recommendation system based on natural language processing, which is used to recommend repair actions for efficient diagnosis and repair. We also describe current areas of research that will enable us to improve hardware availability further.
Citation Keylin_hardware_2018