Desa aikmual

-Transparent memory offloading (TMO) is Meta’s data center solution for offering more memory at a fraction of the cost and power of existing technologies -In production since 2021, TMO saves 20 percent to 32 percent of memory per server across millions of servers in our data center fleet We are witnessing massive growth in the [...] Read More... The post Transparent memory offloading: more memory at a fraction of the cost and power appeared first on Engineering at Meta. http://dlvr.it/SSX5qT

What the research is: Federated learning with differential privacy (FL-DP) is one of the latest privacy-enhancing technologies being evaluated at Meta as we constantly work to enhance user privacy and further safeguard users’ data in the products we design, build, and maintain. FL-DP enhances privacy in two important ways: It allows machine learning (ML) models [...] Read More... The post Applying federated learning to protect data on mobile devices appeared first on Engineering at Meta. http://dlvr.it/SSBDht

The promise of cloud gaming is a promise to democratize gaming. Anyone who loves games should be able to enjoy them and share the experience with their friends, no matter where they’re located, and even if they don’t have the latest, most expensive gaming hardware. Facebook launched its cloud gaming platform in 2020 to give [...] Read More... The post Under the hood: Meta’s cloud gaming infrastructure appeared first on Engineering at Meta. http://dlvr.it/SRvvyd

Within large-scale services, durable storage, distributed leases, and coordination primitives such as distributed locks, semaphores, and events should be strongly consistent. At Meta, we have historically used Apache ZooKeeper as a centralized service for these primitives. However, as Meta’s workload has scaled, we’ve found ourselves pushing the limits of ZooKeeper’s capabilities. Modifying and tuning ZooKeeper [...] Read More... The post Introducing Zelos: A ZooKeeper API leveraging Delos appeared first on Engineering at Meta. http://dlvr.it/SRrb2j

Caches help reduce latency, scale read-heavy workloads, and save cost. They are literally everywhere. Caches run on your phone and in your browser. For example, CDNs and DNS are essentially geo-replicated caches. It’s thanks to many caches working behind the scenes that you can read this blog post right now. Phil Karlton famously said, “There [...] Read More... The post Cache made consistent: Meta’s cache invalidation solution appeared first on Engineering at Meta. http://dlvr.it/SRrb12

Meta Open Source is officially transferring Jest, its open source JavaScript testing framework, to the OpenJS Foundation.  With over 17 million weekly downloads and over 38,000 GitHub stars, Jest is the most used testing framework in the JavaScript ecosystem and is used by companies of all sizes, including Amazon, Google, Microsoft, and Stripe. We believe [...] Read More... The post Meta Open Source is transferring Jest to the OpenJS Foundation appeared first on Engineering at Meta. http://dlvr.it/SQ9kkK

More than 3 billion people around the world rely on our services each month. On mobile, around 57 percent of people on Facebook for Android and 49 percent of those on Facebook for iOS use the app in a language other than English. Delivering the best experience for these people, in their own language, is [...] Read More... The post Language packs: Meta’s mobile localization solution appeared first on Engineering at Meta. http://dlvr.it/SQ2X6Z

Building infrastructure that can easily recover from outages, particularly outages involving adjacent infrastructure, too often becomes a murky exploration of nuanced fate-sharing between systems. Untangling dependencies and uncovering side effects of unavailability has historically been time-consuming work. A lack of great tooling built for this, and the rarity of infrastructure outages, makes reasoning about them [...] Read More... The post BellJar: A new framework for testing system recoverability at scale appeared first on Engineering at Meta. http://dlvr.it/SPqvD7

Over the years, Meta has invested in a number of storage service offerings that cater to different use cases and workload characteristics. Along the way, we’ve aimed to reduce and converge the systems in the storage space. At the same time, having a dedicated solution for critical package workload makes everyone happier. Having this in [...] Read More... The post Delta: A highly available, strongly consistent storage service using chain replication appeared first on Engineering at Meta. http://dlvr.it/SPm5Sk

Since Instagram runs one of the world’s largest deployments of the Django web framework, we have natural interest in finding ways to optimize Python so we can speed up our production application. As part of this effort, we’ve recently open-sourced Cinder, our Python runtime that is a fork of CPython. Cinder includes optimizations like immortal [...] Read More... The post How the Cinder JIT’s function inliner helps us optimize Instagram appeared first on Engineering at Meta. http://dlvr.it/SPdWLk

Delivering on our vision for the metaverse will require us to reimagine a network infrastructure capable of supporting the computing platforms of the future. Although the metaverse is still a ways off, elements of it are already underway and we are already collaborating with telecommunications companies around the world to develop shared, open access optical [...] Read More... The post A new way to survey potential fiber routes — without access to paved roads appeared first on Engineering at Meta. http://dlvr.it/SPc61L

At Meta, our internal data tools are the main channel from our data scientists to our production engineers. As such, it’s important for us to empower our scientists and engineers not only to use data to make decisions, but also to do so in a secure and compliant way. We’ve developed SQL Notebooks, a new [...] Read More... The post SQL Notebooks: Combining the power of Jupyter and SQL editors for data analytics appeared first on Engineering at Meta. http://dlvr.it/SPHhb7

Data minimization — collecting the minimum amount of data required to support our services — is one of our core principles at Meta as we continue developing new privacy-enhancing technologies (PETs). We are constantly seeking ways to improve privacy and protect user data on our family of products. Previously, we’ve approached data minimization by exploring [...] Read More... The post How Meta enables de-identified authentication at scale appeared first on Engineering at Meta. http://dlvr.it/SMfzs6

Silent data corruptions (SDCs), data errors that go undetected by the larger system, are a widespread problem for large-scale infrastructure systems. Left undetected, these types of corruptions can cause data loss and propagate across the stack and manifest as application-level problems. Silent data corruptions (SDC) in hardware impact computational integrity for large-scale applications. Sources of [...] Read More... The post Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scale appeared first on Engineering at Meta. http://dlvr.it/SLtFLm

What the research is: Recent research has demonstrated that binary optimization is important for achieving peak performance for various applications. For instance, the state-of-the-art BOLT binary optimizer developed at Meta, which is part of the LLVM Compiler Project, significantly improves the performance of highly optimized binaries produced using compilers’ most aggressive optimizations, such as profile-guided [...] Read More... The post VESPA: Static profiling for binary optimization appeared first on Engineering at Meta. http://dlvr.it/SLlBfr

Since WhatsApp introduced multi-device capability last year, we’ve seen an increase in people accessing WhatsApp directly through their web browser via WhatsApp Web. With this shift in mind, we’ve been looking at ways to add additional layers of security to the WhatsApp Web experience. Starting today, you can now use Code Verify, an open source [...] Read More... The post Code Verify: An open source browser extension for verifying code authenticity on the web appeared first on Engineering at Meta. http://dlvr.it/SLSSyb

What the research is: We’ve developed a new static analyzer that catches deadlocks in Java code for Android without ever running the code. What distinguishes our analyzer from past research is its ability to analyze revisions in codebases with hundreds of millions of lines of code. We have deployed our analyzer in Meta’s continuous integration [...] Read More... The post An open source compositional deadlock detector for Android Java appeared first on Engineering at Meta. http://dlvr.it/SLKR5y

We’ve improved read availability in LogDevice, Meta’s scalable distributed log storage system, by removing a fundamental trade-off in Flexible Paxos, the algorithm used to gain consensus among our distributed systems. At Meta’s scale, systems need to be reliable, even in the face of organic failures like power loss events, or when systems are undergoing hardware [...] Read More... The post Augmenting Flexible Paxos in LogDevice to improve read availability appeared first on Engineering at Meta. http://dlvr.it/SLFrWV

Facebook Ordered Queueing Service (FOQS) is a fully managed, distributed priority queueing service used for reliable message delivery among many services. FOQS has evolved from a regional deployment into a geo-distributed, global deployment to help ensure that data stored within logical queues is highly available, even through large-scale disaster scenarios. Migrating to a global architecture [...] Read More... The post FOQS: Making a distributed priority queue disaster-ready appeared first on Engineering at Meta. http://dlvr.it/SHMRDZ

There are thousands of distributed services running on millions of servers in Meta’s data centers. Part of ensuring the reliability of those services means making them resilient to power loss events as our data center fleet grows. To help increase resiliency, we built the Power Loss Siren (PLS) — a rack level, low latency, distributed [...] Read More... The post Power Loss Siren: Making Meta resilient to power loss events appeared first on Engineering at Meta. http://dlvr.it/SFVvTJ

We’re tackling the industry-wide issue of scraping by expanding our bug bounty program to reward valid reports of scraping bugs and unprotected data sets. To the best of our knowledge, this is an industry first.  Looking toward the future, we’re also launching new educational opportunities for researchers and hosting our first BountyConEDU — a three-day [...] Read More... The post Charting the future of our bug bounty program appeared first on Engineering at Meta. http://dlvr.it/SFRHvS

To support the people and communities who use our apps and products, we need to stay in constant contact with them. We want to provide the experiences we offer reliably. We also need to establish trust with the larger community we support. This can be especially challenging in a large-scale, quickly evolving environment like Meta, [...] Read More... The post SLICK: Adopting SLOs for improved reliability appeared first on Engineering at Meta. http://dlvr.it/SFHsfm

2,500 km cable will expand access to high-speed internet [...] Read More... The post Malbec subsea cable connects Argentina and Brazil with the rest of the world appeared first on Facebook Engineering. http://dlvr.it/SCKygZ

Open infrastructure technologies and networking hardware will play an important role as we build new technologies for the metaverse, where billions of people will someday come together in virtual spaces. As we head toward the next major computing platform with a continued spirit of embracing openness and disaggregation, we’re announcing two new milestones for our [...] Read More... The post OCP Summit 2021: Open networking hardware lays the groundwork for the metaverse appeared first on Facebook Engineering. http://dlvr.it/SCBrwD

What the research is:  Kangaroo is a new flash cache that enables more efficient caching of tiny objects (objects that are ~100 bytes or less) and overcomes the challenges presented by existing flash cache designs. Since Kangaroo is implemented within CacheLib, Facebook’s open source caching engine, developers can use Kangaroo through CacheLib’s API to build [...] Read More... The post Kangaroo: A new flash cache optimized for tiny objects appeared first on Facebook Engineering. http://dlvr.it/SBLmkL

Enabling developers to prototype, test, and iterate on new features quickly is important to Facebook’s success. To do this effectively, it’s key to have a stable infrastructure that doesn’t introduce unnecessary friction. This gets significantly more challenging when the infrastructure in question must also scale to support more than 3 billion people around the world, [...] Read More... The post Autonomous testing of services at scale appeared first on Facebook Engineering. http://dlvr.it/S9yYyZ

Until recently, static analysis tools weren’t seen by our industry as a reliable element of securing code at scale. After nearly a decade of investing in refining these systems, I’m so proud to celebrate our engineering teams today for being awarded the IEEE Computer Society’s Cybersecurity Award for Practice for development and deployment of static [...] Read More... The post Facebook engineers receive 2021 IEEE Computer Society Cybersecurity Award for static analysis tools appeared first on Facebook Engineering. http://dlvr.it/S9yYvY

What it is: Real Time Messaging Protocol (RTMP) is a popular media streaming protocol that uses Transmission Control Protocol (TCP) persistent connections. When a connection between a live-streaming client and the platform is interrupted, data from the live event is lost until the client can reconnect to a new server. RTMP Go Away is a [...] Read More... The post RTMP Go Away: Lossless reconnections for live streaming appeared first on Facebook Engineering. http://dlvr.it/S9tgNf

[...] Read More... The post Inside the Lab: Expanding connectivity by sea, land, and air appeared first on Facebook Engineering. http://dlvr.it/S96XgG

Now that our platforms are up and running as usual after yesterday’s outage, I thought it would be worth sharing a little more detail on what happened and why — and most importantly, how we’re learning from it.  This outage was triggered by the system that manages our global backbone network capacity. The backbone is [...] Read More... The post More details about the October 4 outage appeared first on Facebook Engineering. http://dlvr.it/S8zNSJ

To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms. We’ve been working as hard as we can to restore access, and our systems are now back up and running. The underlying cause of this outage also impacted many [...] Read More... The post Update about the October 4th outage appeared first on Facebook Engineering. http://dlvr.it/S8wnRz

We’re sharing details about Mariana Trench (MT), a tool we use to spot and prevent security and privacy bugs in Android and Java applications. As part of our effort to help scale security through building automation, we recently open-sourced MT to support security engineers at Facebook and across the industry.  This post is the third [...] Read More... The post Open-sourcing Mariana Trench: Analyzing Android and Java app security in depth appeared first on Facebook Engineering. http://dlvr.it/S8bx96

Facebook invests in subsea cables as part of our continued efforts to build the infrastructure that carries internet traffic and helps bring more people online to a faster internet. Today, we are thrilled to announce, along with regional and global partners, a new segment of subsea cable called 2Africa Pearls, which connects three continents — [...] Read More... The post 2Africa Pearls subsea cable connects Africa, Europe, and Asia to bring affordable, high-speed internet to 3 billion people appeared first on Facebook Engineering. http://dlvr.it/S8TtnD

Managing app size at Facebook is a unique challenge: Every day, developers check in large volumes of code, and each line of code translates into additional bits in the apps that people ultimately download onto their phones. Left unchecked, this added code would make the app bigger and bigger until eventually the time it takes [...] Read More... The post Superpack: Pushing the limits of compression in Facebook’s mobile apps appeared first on Facebook Engineering. http://dlvr.it/S7TLKk

For years, in order to safeguard the privacy of people’s messages, WhatsApp has provided end-to-end encryption by default ​​so messages can be seen only by the sender and recipient, and no one in between. Now, we’re planning to give people the option to protect their WhatsApp backups using end-to-end encryption as well. People can already [...] Read More... The post How WhatsApp is enabling end-to-end encrypted backups appeared first on Facebook Engineering. http://dlvr.it/S7KQpL

Caching plays an important role in helping people access their information efficiently. For example, when an email app loads, it temporarily caches some messages, so the user can refresh the page without the app retrieving the same messages. However, large-scale caching has long been a complex engineering challenge. Companies must balance the fast experience people [...] Read More... The post CacheLib, Facebook’s open source caching engine for web-scale services appeared first on Facebook Engineering. http://dlvr.it/S6qMPL

What the research is:  RAMP-TAO is a new protocol that improves the developer experience on TAO, Facebook’s online social graph store, by providing stronger transactional guarantees. It is the first protocol to provide transactional semantics over an eventually consistent massive-scale data store while still preserving the system’s overall reliability and performance. RAMP-TAO enables an intuitive [...] Read More... The post RAMP-TAO: Layering atomic transactions on Facebook’s online graph store appeared first on Facebook Engineering. http://dlvr.it/S5ts30

We are excited to announce our participation in the Apricot subsea cable system, together with leading regional and global partners. When completed, the project (which is still subject to regulatory approvals) will deliver much-needed internet capacity, redundancy, and reliability to expand connections in the Asia-Pacific region. The 12,000-kilometer-long cable will connect Japan, Taiwan, Guam, the [...] Read More... The post Apricot subsea cable will boost internet capacity, speeds in the Asia-Pacific region appeared first on Facebook Engineering. http://dlvr.it/S5jbVp

Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure. To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance. With the help of the OCP community, we [...] Read More... The post Open-sourcing a more precise time appliance appeared first on Facebook Engineering. http://dlvr.it/S5Rd13

What the research is:  A first-of-its-kind study detailing our backbone management strategy to ensure high service performance throughout the COVID-19 pandemic. The pandemic moved most social interactions online and caused an unprecedented stress test on our global network infrastructure with tens of data center regions. At this scale, failures such as fiber cuts, router misconfigurations, [...] Read More... The post Risk-driven backbone management during COVID-19 and beyond appeared first on Facebook Engineering. http://dlvr.it/S5Kkd7

ZippyDB is the largest strongly consistent, geographically distributed key-value store at Facebook. Since we first deployed ZippyDB in 2012, this key-value store has expanded rapidly, and today, ZippyDB serves a number of use cases, ranging from metadata for a distributed filesystem, counting events for both internal and external purposes, to product data that’s used for [...] Read More... The post How we built a general purpose key value store for Facebook with ZippyDB appeared first on Facebook Engineering. http://dlvr.it/S59Rv2

We are releasing Winterfell, our implementation of a STARK prover/verifier to Crates.io  Winterfell is an easy to use open source implementation of STARKs for security and privacy applications. One potential application for Winterfell’s zero-knowledge proofs is blockchain privacy and scalability.  “Any sufficiently advanced technology is indistinguishable from magic.” —Clarke’s Third Law What if the average [...] Read More... The post Open sourcing Winterfell: A STARK prover and verifier appeared first on Facebook Engineering. http://dlvr.it/S52Rfx

Whether it’s iterating on Facebook’s News Feed ranking algorithm or delivering the most relevant ads to users, we are constantly exploring new features to help improve our machine learning (ML) models. Every time we add new features, we create a challenging data engineering problem that requires us to think strategically about the choices we make. [...] Read More... The post A linear programming approach for optimizing features in ML models appeared first on Facebook Engineering. http://dlvr.it/S4gMgq

MySQL, an open source database developed by Oracle, powers some of Facebook’s most important workloads. We actively develop new features in MySQL to support our evolving requirements. These features change many different areas of MySQL, including client connectors, storage engine, optimizer, and replication. Each new major version of MySQL requires significant time and effort to [...] Read More... The post Migrating Facebook to MySQL 8.0 appeared first on Facebook Engineering. http://dlvr.it/S4DVdZ

Training AI models at a large scale isn’t easy. Aside from the need for large amounts of computing power and resources, there is also considerable engineering complexity behind training very large models. At Facebook AI Research (FAIR) Engineering, we have been working on building tools and infrastructure to make training large AI models easier. Our [...] Read More... The post Fully Sharded Data Parallel: faster AI training with fewer GPUs appeared first on Facebook Engineering. http://dlvr.it/S3ndNR

For years, people have been asking us to create a true multi-device experience that allows people to use WhatsApp on other devices without requiring a smartphone connection. Today, we’re announcing the rollout of a limited public beta test for WhatsApp’s updated multi-device capability.  With this new capability, you can now use WhatsApp on your phone [...] Read More... The post How WhatsApp enables multi-device capability appeared first on Facebook Engineering. http://dlvr.it/S3kR3H

Our infrastructure supports thousands of services that handle billions of requests per second. We’ve previously discussed how we built our service encryption infrastructure to keep these globally distributed services operating securely and performantly. This post discusses the system we designed to enforce encryption policies within our network and shares some of the lessons we learned [...] Read More... The post Enforcing encryption at scale appeared first on Facebook Engineering. http://dlvr.it/S3ZzZn

What the research is: The Ribbon filter is a new data structure that is more space-efficient than the popular Bloom filters that are widely used for optimizing data retrieval. One of the ways that Bloom, and now Ribbon, filters solve real engineering problems is by providing smooth configurability unmatched by other filters. Bloom filters work [...] Read More... The post Ribbon filter: Practically smaller than Bloom and Xor appeared first on Facebook Engineering. http://dlvr.it/S3Pw13

We will be hosting a talk about our work on, “A Platform Agnostic Observability System for AI Accelerators” during our virtual Systems @Scale event at 10:20 a.m. PT on Wednesday, June 30, followed by a live Q&A session. Please submit any questions to systemsatscale@fb.com before the event. Accelerators are special-purpose hardware devices optimized for specific [...] Read More... The post Asicmon: A platform agnostic observability system for AI accelerators appeared first on Facebook Engineering. http://dlvr.it/S2dhyb

Connectivity is an integral part of Facebook’s mission to bring people closer together, and the COVID-19 pandemic has only heightened the demand for critical internet access. According to the latest edition of our Inclusive Internet Index, nearly 70 percent of people around the world believe that increased internet usage in all aspects of their lives [...] Read More... The post Driving towards an open internet ecosystem to help tackle the digital divide appeared first on Facebook Engineering. http://dlvr.it/S2cXBx