Desa aikmual

We’ve made architecture changes to Meta’s event driven asynchronous computing platform that have  enabled easy integration with multiple event-sources.  We’re sharing our learnings from handling various workloads and how to tackle trade offs made with certain design choices in building the platform. Asynchronous computing is a paradigm where the user does not expect a workload [...] Read More... The post Asynchronous computing at Meta: Overview and learnings appeared first on Engineering at Meta. http://dlvr.it/Shk6rc

Managing network solutions amidst a growing scale inherently brings challenges around performance, deployment, and operational complexities.  At Meta, we’ve found that these challenges broadly fall into three themes: 1.)   Data center networking: Over the past decade, on the physical front, we have seen a rise in vendor-specific hardware that comes with heterogeneous feature and [...] Read More... The post Watch Meta’s engineers discuss optimizing large-scale networks appeared first on Engineering at Meta. http://dlvr.it/ShXR9F

The technical journey discusses the motivations, challenges, and technical solutions employed for warehouse schematization, especially a change to the wire serialization format employed in Meta’s data platform for data interchange related to Warehouse Analytics Logging. Here, we discuss the engineering, scaling, and nontechnical challenges of modernizing  Meta’s exabyte-scale data platform by migrating to the new [...] Read More... The post Tulip: Modernizing Meta’s data platform appeared first on Engineering at Meta. http://dlvr.it/ShTYlR

Meta has open-sourced Anonymous Credential Service (ACS), a highly available multitenant service that allows clients to authenticate in a de-identified manner. ACS enhances privacy and security while also being compute-conscious. By open-sourcing and fostering a community for ACS, we believe we can accelerate the pace of innovation in de-identified authentication. Data minimization — collecting the [...] Read More... The post Open-sourcing Anonymous Credential Service appeared first on Engineering at Meta. http://dlvr.it/SfDm7Q

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis [...] Read More... The post Enabling static analysis of SQL queries at Meta appeared first on Engineering at Meta. http://dlvr.it/Sdd0LJ

We developed a new static analysis tool called Nullsafe that is used at Meta to detect NullPointerException (NPE) errors in Java code. Interoperability with legacy code and gradual deployment model were key to Nullsafe’s wide adoption and allowed us to recover some null-safety properties in the context of an otherwise null-unsafe language in a multimillion-line [...] Read More... The post Retrofitting null-safety onto Java at Meta appeared first on Engineering at Meta. http://dlvr.it/SdBLRw

Implementing Precision Time Protocol (PTP) at Meta allows us to synchronize the systems that drive our products and services down to nanosecond precision. PTP’s predecessor, Network Time Protocol (NTP), provided us with millisecond precision, but as we scale to more advanced systems on our way to building the next computing platform, the metaverse and AI, [...] Read More... The post How Precision Time Protocol is being deployed at Meta appeared first on Engineering at Meta. http://dlvr.it/Sd7KVb

Code reviews are one of the most important parts of the software development process At Meta we’ve recognized the need to make code reviews as fast as possible without sacrificing quality We’re sharing several tools and steps we’ve taken at Meta to reduce the time waiting for code reviews When done well, code reviews can [...] Read More... The post Move faster, wait less: Improving code review time at Meta appeared first on Engineering at Meta. http://dlvr.it/SctM6V

Sapling is a new Git-compatible source control client. Sapling emphasizes usability while also scaling to the largest repositories in the world. ReviewStack is a demonstration code review UI for GitHub pull requests that integrates with Sapling to make reviewing stacks of commits easy. You can get started using Sapling today.  Source control is one of [...] Read More... The post Sapling: Source control that’s user-friendly and scalable appeared first on Engineering at Meta. http://dlvr.it/ScqDLF

We’re sharing Tulip, a binary serialization protocol supporting schema evolution.  Tulip assists with data schematization by addressing protocol reliability and other issues simultaneously.  It replaces multiple legacy formats used in Meta’s data platform and has achieved significant performance and efficiency gains. There are numerous heterogeneous services, such as warehouse data storage and various real-time systems, [...] Read More... The post Tulip: Schematizing Meta’s data platform appeared first on Engineering at Meta. http://dlvr.it/ScVlWp

In our constant quest to prioritize efficiency, Instagram’s engineers have developed a way to process new videos that reduces the cost to produce basic video encodings by 94 percent. With this method in place, Meta’s video infrastructure can continue to scale without needing to add more machines. This frees up resources so more people can [...] Read More... The post Reducing Instagram’s basic video compute time by 94 percent appeared first on Engineering at Meta. http://dlvr.it/ScDbnK

We’re sharing how Meta is applying statistics and machine learning (ML) to improve notification personalization and management on Instagram – particularly on daily digest push notifications. By using causal inference and ML to identify highly active users who are likely to see more content organically, we have been able to reduce the number of notifications [...] Read More... The post Improving Instagram notification management with machine learning and causal inference appeared first on Engineering at Meta. http://dlvr.it/Sc12XM

We’re sharing lessons learned from shifting our Android development from Java to Kotlin. Kotlin is a popular language for Android development and offers some key advantages over Java.  As of today, our Android codebase contains over 10 million lines of Kotlin code. We’re open sourcing various examples and utilities we used to manipulate Kotlin code [...] Read More... The post From zero to 10 million lines of Kotlin appeared first on Engineering at Meta. http://dlvr.it/SbdkwR

At OCP Summit 2022, we’re announcing Grand Teton, our next-generation platform for AI at scale that we’ll contribute to the OCP community. We’re also sharing new innovations designed to support data centers as they advance to support new AI technologies: A new, more efficient version of Open Rack. Our Air-Assisted Liquid Cooling (AALC) – design. [...] Read More... The post OCP Summit 2022: Open hardware for AI infrastructure appeared first on Engineering at Meta. http://dlvr.it/SbJSr2

Our in-app browser for Facebook on Android has historically relied on an Android System WebView based on Chromium, the open source project that powers many browsers on Android and other operating systems. On other mobile operating systems, the System WebView component cannot be updated without updating the entire operating system. On Android, this works differently, [...] Read More... The post Launching a new Chromium-based WebView for Android appeared first on Engineering at Meta. http://dlvr.it/SZHjM3

Data center optimization has always played an important role at Meta. By optimizing our data centers’ environmental controls, we can reduce our environmental impact  while ensuring that people can always depend on our products. With most other complex systems, optimization of energy consumption is a trial-and-error process. But experimenting on any component of a live [...] Read More... The post How thermal simulation helps optimize Meta’s data centers appeared first on Engineering at Meta. http://dlvr.it/SYKcXy

We’ve open-sourced MemLab, a JavaScript memory testing framework that automates memory leak detection. Finding and addressing the root cause of memory leaks is important for delivering a quality user experience on web applications. MemLab has helped engineers and developers at Meta improve user experience and make significant improvements in memory optimization. We hope it will [...] Read More... The post MemLab: An open source framework for finding JavaScript memory leaks appeared first on Engineering at Meta. http://dlvr.it/SYBy38

Meta’s overall network usage and traffic volume has increased as we’ve continued to add new services. Due to the scarcity of fiber resources, we’re developing an explicit resource reservation framework to effectively plan, manage, and operate the shared consumption of network bandwidth, which will help us keep up with demand and limit network disruptions during [...] Read More... The post Network Entitlement: A contract-based network sharing solution appeared first on Engineering at Meta. http://dlvr.it/SXwyjm

Meta currently operates 14 data centers around the world. This rapidly expanding global data center footprint poses new challenges for service owners and for our infrastructure management systems. Systems like Twine, which we use to scale cluster management, and RAS, which handles perpetual region-wide resource allocation, have provided the abstractions and automation necessary for service [...] Read More... The post Viewing the world as a computer: Global capacity management appeared first on Engineering at Meta. http://dlvr.it/SXslZX

Meta is introducing Velox, an open source unified execution engine aimed at accelerating data management systems and streamlining their development. Velox is under active development. Experimental results from our paper published at the International Conference on Very Large Data Bases (VLDB) 2022 show how Velox improves efficiency and consistency in data management systems. Velox helps [...] Read More... The post Introducing Velox: An open source unified execution engine appeared first on Engineering at Meta. http://dlvr.it/SXXv99

When we focus on minimizing errors and downtime here at Meta, we place a lot of attention on service-level indicators (SLIs) and service-level objectives (SLOs). Consider Instagram, for example. There, SLIs represent metrics from different product surfaces, like the volume of error response codes to certain endpoints, or the number of successful media uploads. Based [...] Read More... The post Improving Meta’s SLO workflows with data annotations appeared first on Engineering at Meta. http://dlvr.it/SXR48r

A touring alien from a galaxy far, far away is an avid Instagram user. Her Instagram Feed is dominated by: Friends and family posts Some space travel magazines A few general news accounts Lots of science fiction blogs She logs in, scrolls through her feed gently — catching up with friends and family, keeping pace [...] Read More... The post How Instagram suggests new content appeared first on Engineering at Meta. http://dlvr.it/SWWyVv

Many of Meta’s products, such as search and language translations, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastructure increases, we need to scale our data ingestion infrastructure accordingly to handle workloads more efficiently. GPUs, which are used for training infrastructure, tend to double in [...] Read More... The post Scaling data ingestion for machine learning training at Meta appeared first on Engineering at Meta. http://dlvr.it/SWPMWQ

At Meta, our messaging apps help billions of people around the world stay connected to those who matter most to them. This scale brings potential threats from criminals and hackers, so we have a responsibility to keep people and their data safe. We’re sharing a set of principles to ensure that security is central to [...] Read More... The post Five security principles for billions of messages across Meta’s apps appeared first on Engineering at Meta. http://dlvr.it/SVfzks

– Supporting a programming language at Meta is a very careful and deliberate decision. – We’re sharing our internal programming language guidance that helps our engineers and developers choose the best language for their projects. – Rust is the latest addition to Meta’s list of supported server-side languages. At Meta, we use many different programming [...] Read More... The post Programming languages endorsed for server-side use at Meta appeared first on Engineering at Meta. http://dlvr.it/SVbgc3

In 2020 we launched Instagram Messaging (referred to in this post simply as “Messaging”) for personal desktop computers. We believe that this feature will improve everyday experiences and enable new use cases for all of our desktop web users. In this post, we go through some of our overall learnings from our desktop users, and [...] Read More... The post Launching Instagram Messaging on desktop appeared first on Engineering at Meta. http://dlvr.it/SVXTRM

The leap second concept was first introduced in 1972 by the International Earth Rotation and Reference Systems Service (IERS) in an attempt to periodically update Coordinated Universal Time (UTC) due to imprecise observed solar time (UT1) and the long-term slowdown in the Earth’s rotation. This periodic adjustment mainly benefits scientists and astronomers as it allows [...] Read More... The post It’s time to leave the leap second in the past appeared first on Engineering at Meta. http://dlvr.it/SVTQVw

With more than 75 percent of our internet traffic set to use QUIC and HTTP/3 together, QUIC is slowly moving to become the de facto protocol used for internet communication at Meta. For Meta’s data center network, TCP remains the primary network transport protocol that supports thousands of services on top of it. As our [...] Read More... The post Watch Meta’s engineers discuss QUIC and TCP innovations for our network appeared first on Engineering at Meta. http://dlvr.it/STSR78

-Transparent memory offloading (TMO) is Meta’s data center solution for offering more memory at a fraction of the cost and power of existing technologies -In production since 2021, TMO saves 20 percent to 32 percent of memory per server across millions of servers in our data center fleet We are witnessing massive growth in the [...] Read More... The post Transparent memory offloading: more memory at a fraction of the cost and power appeared first on Engineering at Meta. http://dlvr.it/SSX5qT

What the research is: Federated learning with differential privacy (FL-DP) is one of the latest privacy-enhancing technologies being evaluated at Meta as we constantly work to enhance user privacy and further safeguard users’ data in the products we design, build, and maintain. FL-DP enhances privacy in two important ways: It allows machine learning (ML) models [...] Read More... The post Applying federated learning to protect data on mobile devices appeared first on Engineering at Meta. http://dlvr.it/SSBDht

The promise of cloud gaming is a promise to democratize gaming. Anyone who loves games should be able to enjoy them and share the experience with their friends, no matter where they’re located, and even if they don’t have the latest, most expensive gaming hardware. Facebook launched its cloud gaming platform in 2020 to give [...] Read More... The post Under the hood: Meta’s cloud gaming infrastructure appeared first on Engineering at Meta. http://dlvr.it/SRvvyd

Within large-scale services, durable storage, distributed leases, and coordination primitives such as distributed locks, semaphores, and events should be strongly consistent. At Meta, we have historically used Apache ZooKeeper as a centralized service for these primitives. However, as Meta’s workload has scaled, we’ve found ourselves pushing the limits of ZooKeeper’s capabilities. Modifying and tuning ZooKeeper [...] Read More... The post Introducing Zelos: A ZooKeeper API leveraging Delos appeared first on Engineering at Meta. http://dlvr.it/SRrb2j

Caches help reduce latency, scale read-heavy workloads, and save cost. They are literally everywhere. Caches run on your phone and in your browser. For example, CDNs and DNS are essentially geo-replicated caches. It’s thanks to many caches working behind the scenes that you can read this blog post right now. Phil Karlton famously said, “There [...] Read More... The post Cache made consistent: Meta’s cache invalidation solution appeared first on Engineering at Meta. http://dlvr.it/SRrb12

Meta Open Source is officially transferring Jest, its open source JavaScript testing framework, to the OpenJS Foundation.  With over 17 million weekly downloads and over 38,000 GitHub stars, Jest is the most used testing framework in the JavaScript ecosystem and is used by companies of all sizes, including Amazon, Google, Microsoft, and Stripe. We believe [...] Read More... The post Meta Open Source is transferring Jest to the OpenJS Foundation appeared first on Engineering at Meta. http://dlvr.it/SQ9kkK

More than 3 billion people around the world rely on our services each month. On mobile, around 57 percent of people on Facebook for Android and 49 percent of those on Facebook for iOS use the app in a language other than English. Delivering the best experience for these people, in their own language, is [...] Read More... The post Language packs: Meta’s mobile localization solution appeared first on Engineering at Meta. http://dlvr.it/SQ2X6Z

Building infrastructure that can easily recover from outages, particularly outages involving adjacent infrastructure, too often becomes a murky exploration of nuanced fate-sharing between systems. Untangling dependencies and uncovering side effects of unavailability has historically been time-consuming work. A lack of great tooling built for this, and the rarity of infrastructure outages, makes reasoning about them [...] Read More... The post BellJar: A new framework for testing system recoverability at scale appeared first on Engineering at Meta. http://dlvr.it/SPqvD7

Over the years, Meta has invested in a number of storage service offerings that cater to different use cases and workload characteristics. Along the way, we’ve aimed to reduce and converge the systems in the storage space. At the same time, having a dedicated solution for critical package workload makes everyone happier. Having this in [...] Read More... The post Delta: A highly available, strongly consistent storage service using chain replication appeared first on Engineering at Meta. http://dlvr.it/SPm5Sk

Since Instagram runs one of the world’s largest deployments of the Django web framework, we have natural interest in finding ways to optimize Python so we can speed up our production application. As part of this effort, we’ve recently open-sourced Cinder, our Python runtime that is a fork of CPython. Cinder includes optimizations like immortal [...] Read More... The post How the Cinder JIT’s function inliner helps us optimize Instagram appeared first on Engineering at Meta. http://dlvr.it/SPdWLk

Delivering on our vision for the metaverse will require us to reimagine a network infrastructure capable of supporting the computing platforms of the future. Although the metaverse is still a ways off, elements of it are already underway and we are already collaborating with telecommunications companies around the world to develop shared, open access optical [...] Read More... The post A new way to survey potential fiber routes — without access to paved roads appeared first on Engineering at Meta. http://dlvr.it/SPc61L

At Meta, our internal data tools are the main channel from our data scientists to our production engineers. As such, it’s important for us to empower our scientists and engineers not only to use data to make decisions, but also to do so in a secure and compliant way. We’ve developed SQL Notebooks, a new [...] Read More... The post SQL Notebooks: Combining the power of Jupyter and SQL editors for data analytics appeared first on Engineering at Meta. http://dlvr.it/SPHhb7

Data minimization — collecting the minimum amount of data required to support our services — is one of our core principles at Meta as we continue developing new privacy-enhancing technologies (PETs). We are constantly seeking ways to improve privacy and protect user data on our family of products. Previously, we’ve approached data minimization by exploring [...] Read More... The post How Meta enables de-identified authentication at scale appeared first on Engineering at Meta. http://dlvr.it/SMfzs6

Silent data corruptions (SDCs), data errors that go undetected by the larger system, are a widespread problem for large-scale infrastructure systems. Left undetected, these types of corruptions can cause data loss and propagate across the stack and manifest as application-level problems. Silent data corruptions (SDC) in hardware impact computational integrity for large-scale applications. Sources of [...] Read More... The post Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scale appeared first on Engineering at Meta. http://dlvr.it/SLtFLm

What the research is: Recent research has demonstrated that binary optimization is important for achieving peak performance for various applications. For instance, the state-of-the-art BOLT binary optimizer developed at Meta, which is part of the LLVM Compiler Project, significantly improves the performance of highly optimized binaries produced using compilers’ most aggressive optimizations, such as profile-guided [...] Read More... The post VESPA: Static profiling for binary optimization appeared first on Engineering at Meta. http://dlvr.it/SLlBfr

Since WhatsApp introduced multi-device capability last year, we’ve seen an increase in people accessing WhatsApp directly through their web browser via WhatsApp Web. With this shift in mind, we’ve been looking at ways to add additional layers of security to the WhatsApp Web experience. Starting today, you can now use Code Verify, an open source [...] Read More... The post Code Verify: An open source browser extension for verifying code authenticity on the web appeared first on Engineering at Meta. http://dlvr.it/SLSSyb

What the research is: We’ve developed a new static analyzer that catches deadlocks in Java code for Android without ever running the code. What distinguishes our analyzer from past research is its ability to analyze revisions in codebases with hundreds of millions of lines of code. We have deployed our analyzer in Meta’s continuous integration [...] Read More... The post An open source compositional deadlock detector for Android Java appeared first on Engineering at Meta. http://dlvr.it/SLKR5y

We’ve improved read availability in LogDevice, Meta’s scalable distributed log storage system, by removing a fundamental trade-off in Flexible Paxos, the algorithm used to gain consensus among our distributed systems. At Meta’s scale, systems need to be reliable, even in the face of organic failures like power loss events, or when systems are undergoing hardware [...] Read More... The post Augmenting Flexible Paxos in LogDevice to improve read availability appeared first on Engineering at Meta. http://dlvr.it/SLFrWV

Facebook Ordered Queueing Service (FOQS) is a fully managed, distributed priority queueing service used for reliable message delivery among many services. FOQS has evolved from a regional deployment into a geo-distributed, global deployment to help ensure that data stored within logical queues is highly available, even through large-scale disaster scenarios. Migrating to a global architecture [...] Read More... The post FOQS: Making a distributed priority queue disaster-ready appeared first on Engineering at Meta. http://dlvr.it/SHMRDZ

There are thousands of distributed services running on millions of servers in Meta’s data centers. Part of ensuring the reliability of those services means making them resilient to power loss events as our data center fleet grows. To help increase resiliency, we built the Power Loss Siren (PLS) — a rack level, low latency, distributed [...] Read More... The post Power Loss Siren: Making Meta resilient to power loss events appeared first on Engineering at Meta. http://dlvr.it/SFVvTJ

We’re tackling the industry-wide issue of scraping by expanding our bug bounty program to reward valid reports of scraping bugs and unprotected data sets. To the best of our knowledge, this is an industry first.  Looking toward the future, we’re also launching new educational opportunities for researchers and hosting our first BountyConEDU — a three-day [...] Read More... The post Charting the future of our bug bounty program appeared first on Engineering at Meta. http://dlvr.it/SFRHvS

To support the people and communities who use our apps and products, we need to stay in constant contact with them. We want to provide the experiences we offer reliably. We also need to establish trust with the larger community we support. This can be especially challenging in a large-scale, quickly evolving environment like Meta, [...] Read More... The post SLICK: Adopting SLOs for improved reliability appeared first on Engineering at Meta. http://dlvr.it/SFHsfm