Reverse debugging at scale

Say you receive an email notification that a service is crashing just after your last code change deploys. The crash happens in only 0.1 percent of the servers where it runs. But you’re at a large-scale company, so 0.1 percent equals thousands of servers — and this issue is going to be hard to reproduce. Several [...] Read More... The post Reverse debugging at scale appeared first on Facebook Engineering.
http://dlvr.it/RyYLx0

Komentar

Posting Komentar

Postingan populer dari blog ini

How Meta enforces purpose limitation via Privacy Aware Infrastructure at scale

Gambar
At Meta, we’ve been diligently working to incorporate privacy into different systems of our software stack over the past few years. Today, we’re excited to share some cutting-edge technologies that are part of our Privacy Aware Infrastructure (PAI) initiative. These innovations mark a major milestone in our ongoing commitment to honoring user privacy.  PAI offers [...] Read More... The post How Meta enforces purpose limitation via Privacy Aware Infrastructure at scale appeared first on Engineering at Meta. http://dlvr.it/TCRbJ9
Baca selengkapnya

Fully Sharded Data Parallel: faster AI training with fewer GPUs

Gambar
Training AI models at a large scale isn’t easy. Aside from the need for large amounts of computing power and resources, there is also considerable engineering complexity behind training very large models. At Facebook AI Research (FAIR) Engineering, we have been working on building tools and infrastructure to make training large AI models easier. Our [...] Read More... The post Fully Sharded Data Parallel: faster AI training with fewer GPUs appeared first on Facebook Engineering. http://dlvr.it/S3ndNR
Baca selengkapnya

Risk-driven backbone management during COVID-19 and beyond

Gambar
What the research is:  A first-of-its-kind study detailing our backbone management strategy to ensure high service performance throughout the COVID-19 pandemic. The pandemic moved most social interactions online and caused an unprecedented stress test on our global network infrastructure with tens of data center regions. At this scale, failures such as fiber cuts, router misconfigurations, [...] Read More... The post Risk-driven backbone management during COVID-19 and beyond appeared first on Facebook Engineering. http://dlvr.it/S5Kkd7
Baca selengkapnya