Desa aikmual

Meta is introducing Velox, an open source unified execution engine aimed at accelerating data management systems and streamlining their development. Velox is under active development. Experimental results from our paper published at the International Conference on Very Large Data Bases (VLDB) 2022 show how Velox improves efficiency and consistency in data management systems. Velox helps [...] Read More... The post Introducing Velox: An open source unified execution engine appeared first on Engineering at Meta. http://dlvr.it/SXXv99

When we focus on minimizing errors and downtime here at Meta, we place a lot of attention on service-level indicators (SLIs) and service-level objectives (SLOs). Consider Instagram, for example. There, SLIs represent metrics from different product surfaces, like the volume of error response codes to certain endpoints, or the number of successful media uploads. Based [...] Read More... The post Improving Meta’s SLO workflows with data annotations appeared first on Engineering at Meta. http://dlvr.it/SXR48r

A touring alien from a galaxy far, far away is an avid Instagram user. Her Instagram Feed is dominated by: Friends and family posts Some space travel magazines A few general news accounts Lots of science fiction blogs She logs in, scrolls through her feed gently — catching up with friends and family, keeping pace [...] Read More... The post How Instagram suggests new content appeared first on Engineering at Meta. http://dlvr.it/SWWyVv

Many of Meta’s products, such as search and language translations, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastructure increases, we need to scale our data ingestion infrastructure accordingly to handle workloads more efficiently. GPUs, which are used for training infrastructure, tend to double in [...] Read More... The post Scaling data ingestion for machine learning training at Meta appeared first on Engineering at Meta. http://dlvr.it/SWPMWQ

At Meta, our messaging apps help billions of people around the world stay connected to those who matter most to them. This scale brings potential threats from criminals and hackers, so we have a responsibility to keep people and their data safe. We’re sharing a set of principles to ensure that security is central to [...] Read More... The post Five security principles for billions of messages across Meta’s apps appeared first on Engineering at Meta. http://dlvr.it/SVfzks

– Supporting a programming language at Meta is a very careful and deliberate decision. – We’re sharing our internal programming language guidance that helps our engineers and developers choose the best language for their projects. – Rust is the latest addition to Meta’s list of supported server-side languages. At Meta, we use many different programming [...] Read More... The post Programming languages endorsed for server-side use at Meta appeared first on Engineering at Meta. http://dlvr.it/SVbgc3

In 2020 we launched Instagram Messaging (referred to in this post simply as “Messaging”) for personal desktop computers. We believe that this feature will improve everyday experiences and enable new use cases for all of our desktop web users. In this post, we go through some of our overall learnings from our desktop users, and [...] Read More... The post Launching Instagram Messaging on desktop appeared first on Engineering at Meta. http://dlvr.it/SVXTRM

The leap second concept was first introduced in 1972 by the International Earth Rotation and Reference Systems Service (IERS) in an attempt to periodically update Coordinated Universal Time (UTC) due to imprecise observed solar time (UT1) and the long-term slowdown in the Earth’s rotation. This periodic adjustment mainly benefits scientists and astronomers as it allows [...] Read More... The post It’s time to leave the leap second in the past appeared first on Engineering at Meta. http://dlvr.it/SVTQVw

With more than 75 percent of our internet traffic set to use QUIC and HTTP/3 together, QUIC is slowly moving to become the de facto protocol used for internet communication at Meta. For Meta’s data center network, TCP remains the primary network transport protocol that supports thousands of services on top of it. As our [...] Read More... The post Watch Meta’s engineers discuss QUIC and TCP innovations for our network appeared first on Engineering at Meta. http://dlvr.it/STSR78

-Transparent memory offloading (TMO) is Meta’s data center solution for offering more memory at a fraction of the cost and power of existing technologies -In production since 2021, TMO saves 20 percent to 32 percent of memory per server across millions of servers in our data center fleet We are witnessing massive growth in the [...] Read More... The post Transparent memory offloading: more memory at a fraction of the cost and power appeared first on Engineering at Meta. http://dlvr.it/SSX5qT

What the research is: Federated learning with differential privacy (FL-DP) is one of the latest privacy-enhancing technologies being evaluated at Meta as we constantly work to enhance user privacy and further safeguard users’ data in the products we design, build, and maintain. FL-DP enhances privacy in two important ways: It allows machine learning (ML) models [...] Read More... The post Applying federated learning to protect data on mobile devices appeared first on Engineering at Meta. http://dlvr.it/SSBDht

The promise of cloud gaming is a promise to democratize gaming. Anyone who loves games should be able to enjoy them and share the experience with their friends, no matter where they’re located, and even if they don’t have the latest, most expensive gaming hardware. Facebook launched its cloud gaming platform in 2020 to give [...] Read More... The post Under the hood: Meta’s cloud gaming infrastructure appeared first on Engineering at Meta. http://dlvr.it/SRvvyd

Within large-scale services, durable storage, distributed leases, and coordination primitives such as distributed locks, semaphores, and events should be strongly consistent. At Meta, we have historically used Apache ZooKeeper as a centralized service for these primitives. However, as Meta’s workload has scaled, we’ve found ourselves pushing the limits of ZooKeeper’s capabilities. Modifying and tuning ZooKeeper [...] Read More... The post Introducing Zelos: A ZooKeeper API leveraging Delos appeared first on Engineering at Meta. http://dlvr.it/SRrb2j

Caches help reduce latency, scale read-heavy workloads, and save cost. They are literally everywhere. Caches run on your phone and in your browser. For example, CDNs and DNS are essentially geo-replicated caches. It’s thanks to many caches working behind the scenes that you can read this blog post right now. Phil Karlton famously said, “There [...] Read More... The post Cache made consistent: Meta’s cache invalidation solution appeared first on Engineering at Meta. http://dlvr.it/SRrb12

Meta Open Source is officially transferring Jest, its open source JavaScript testing framework, to the OpenJS Foundation.  With over 17 million weekly downloads and over 38,000 GitHub stars, Jest is the most used testing framework in the JavaScript ecosystem and is used by companies of all sizes, including Amazon, Google, Microsoft, and Stripe. We believe [...] Read More... The post Meta Open Source is transferring Jest to the OpenJS Foundation appeared first on Engineering at Meta. http://dlvr.it/SQ9kkK

More than 3 billion people around the world rely on our services each month. On mobile, around 57 percent of people on Facebook for Android and 49 percent of those on Facebook for iOS use the app in a language other than English. Delivering the best experience for these people, in their own language, is [...] Read More... The post Language packs: Meta’s mobile localization solution appeared first on Engineering at Meta. http://dlvr.it/SQ2X6Z

Building infrastructure that can easily recover from outages, particularly outages involving adjacent infrastructure, too often becomes a murky exploration of nuanced fate-sharing between systems. Untangling dependencies and uncovering side effects of unavailability has historically been time-consuming work. A lack of great tooling built for this, and the rarity of infrastructure outages, makes reasoning about them [...] Read More... The post BellJar: A new framework for testing system recoverability at scale appeared first on Engineering at Meta. http://dlvr.it/SPqvD7

Over the years, Meta has invested in a number of storage service offerings that cater to different use cases and workload characteristics. Along the way, we’ve aimed to reduce and converge the systems in the storage space. At the same time, having a dedicated solution for critical package workload makes everyone happier. Having this in [...] Read More... The post Delta: A highly available, strongly consistent storage service using chain replication appeared first on Engineering at Meta. http://dlvr.it/SPm5Sk

Since Instagram runs one of the world’s largest deployments of the Django web framework, we have natural interest in finding ways to optimize Python so we can speed up our production application. As part of this effort, we’ve recently open-sourced Cinder, our Python runtime that is a fork of CPython. Cinder includes optimizations like immortal [...] Read More... The post How the Cinder JIT’s function inliner helps us optimize Instagram appeared first on Engineering at Meta. http://dlvr.it/SPdWLk

Delivering on our vision for the metaverse will require us to reimagine a network infrastructure capable of supporting the computing platforms of the future. Although the metaverse is still a ways off, elements of it are already underway and we are already collaborating with telecommunications companies around the world to develop shared, open access optical [...] Read More... The post A new way to survey potential fiber routes — without access to paved roads appeared first on Engineering at Meta. http://dlvr.it/SPc61L

At Meta, our internal data tools are the main channel from our data scientists to our production engineers. As such, it’s important for us to empower our scientists and engineers not only to use data to make decisions, but also to do so in a secure and compliant way. We’ve developed SQL Notebooks, a new [...] Read More... The post SQL Notebooks: Combining the power of Jupyter and SQL editors for data analytics appeared first on Engineering at Meta. http://dlvr.it/SPHhb7

Data minimization — collecting the minimum amount of data required to support our services — is one of our core principles at Meta as we continue developing new privacy-enhancing technologies (PETs). We are constantly seeking ways to improve privacy and protect user data on our family of products. Previously, we’ve approached data minimization by exploring [...] Read More... The post How Meta enables de-identified authentication at scale appeared first on Engineering at Meta. http://dlvr.it/SMfzs6

Silent data corruptions (SDCs), data errors that go undetected by the larger system, are a widespread problem for large-scale infrastructure systems. Left undetected, these types of corruptions can cause data loss and propagate across the stack and manifest as application-level problems. Silent data corruptions (SDC) in hardware impact computational integrity for large-scale applications. Sources of [...] Read More... The post Detecting silent errors in the wild: Combining two novel approaches to quickly detect silent data corruptions at scale appeared first on Engineering at Meta. http://dlvr.it/SLtFLm

What the research is: Recent research has demonstrated that binary optimization is important for achieving peak performance for various applications. For instance, the state-of-the-art BOLT binary optimizer developed at Meta, which is part of the LLVM Compiler Project, significantly improves the performance of highly optimized binaries produced using compilers’ most aggressive optimizations, such as profile-guided [...] Read More... The post VESPA: Static profiling for binary optimization appeared first on Engineering at Meta. http://dlvr.it/SLlBfr

Since WhatsApp introduced multi-device capability last year, we’ve seen an increase in people accessing WhatsApp directly through their web browser via WhatsApp Web. With this shift in mind, we’ve been looking at ways to add additional layers of security to the WhatsApp Web experience. Starting today, you can now use Code Verify, an open source [...] Read More... The post Code Verify: An open source browser extension for verifying code authenticity on the web appeared first on Engineering at Meta. http://dlvr.it/SLSSyb

What the research is: We’ve developed a new static analyzer that catches deadlocks in Java code for Android without ever running the code. What distinguishes our analyzer from past research is its ability to analyze revisions in codebases with hundreds of millions of lines of code. We have deployed our analyzer in Meta’s continuous integration [...] Read More... The post An open source compositional deadlock detector for Android Java appeared first on Engineering at Meta. http://dlvr.it/SLKR5y

We’ve improved read availability in LogDevice, Meta’s scalable distributed log storage system, by removing a fundamental trade-off in Flexible Paxos, the algorithm used to gain consensus among our distributed systems. At Meta’s scale, systems need to be reliable, even in the face of organic failures like power loss events, or when systems are undergoing hardware [...] Read More... The post Augmenting Flexible Paxos in LogDevice to improve read availability appeared first on Engineering at Meta. http://dlvr.it/SLFrWV

Facebook Ordered Queueing Service (FOQS) is a fully managed, distributed priority queueing service used for reliable message delivery among many services. FOQS has evolved from a regional deployment into a geo-distributed, global deployment to help ensure that data stored within logical queues is highly available, even through large-scale disaster scenarios. Migrating to a global architecture [...] Read More... The post FOQS: Making a distributed priority queue disaster-ready appeared first on Engineering at Meta. http://dlvr.it/SHMRDZ

There are thousands of distributed services running on millions of servers in Meta’s data centers. Part of ensuring the reliability of those services means making them resilient to power loss events as our data center fleet grows. To help increase resiliency, we built the Power Loss Siren (PLS) — a rack level, low latency, distributed [...] Read More... The post Power Loss Siren: Making Meta resilient to power loss events appeared first on Engineering at Meta. http://dlvr.it/SFVvTJ

We’re tackling the industry-wide issue of scraping by expanding our bug bounty program to reward valid reports of scraping bugs and unprotected data sets. To the best of our knowledge, this is an industry first.  Looking toward the future, we’re also launching new educational opportunities for researchers and hosting our first BountyConEDU — a three-day [...] Read More... The post Charting the future of our bug bounty program appeared first on Engineering at Meta. http://dlvr.it/SFRHvS

To support the people and communities who use our apps and products, we need to stay in constant contact with them. We want to provide the experiences we offer reliably. We also need to establish trust with the larger community we support. This can be especially challenging in a large-scale, quickly evolving environment like Meta, [...] Read More... The post SLICK: Adopting SLOs for improved reliability appeared first on Engineering at Meta. http://dlvr.it/SFHsfm

2,500 km cable will expand access to high-speed internet [...] Read More... The post Malbec subsea cable connects Argentina and Brazil with the rest of the world appeared first on Facebook Engineering. http://dlvr.it/SCKygZ

Open infrastructure technologies and networking hardware will play an important role as we build new technologies for the metaverse, where billions of people will someday come together in virtual spaces. As we head toward the next major computing platform with a continued spirit of embracing openness and disaggregation, we’re announcing two new milestones for our [...] Read More... The post OCP Summit 2021: Open networking hardware lays the groundwork for the metaverse appeared first on Facebook Engineering. http://dlvr.it/SCBrwD

What the research is:  Kangaroo is a new flash cache that enables more efficient caching of tiny objects (objects that are ~100 bytes or less) and overcomes the challenges presented by existing flash cache designs. Since Kangaroo is implemented within CacheLib, Facebook’s open source caching engine, developers can use Kangaroo through CacheLib’s API to build [...] Read More... The post Kangaroo: A new flash cache optimized for tiny objects appeared first on Facebook Engineering. http://dlvr.it/SBLmkL

Enabling developers to prototype, test, and iterate on new features quickly is important to Facebook’s success. To do this effectively, it’s key to have a stable infrastructure that doesn’t introduce unnecessary friction. This gets significantly more challenging when the infrastructure in question must also scale to support more than 3 billion people around the world, [...] Read More... The post Autonomous testing of services at scale appeared first on Facebook Engineering. http://dlvr.it/S9yYyZ

Until recently, static analysis tools weren’t seen by our industry as a reliable element of securing code at scale. After nearly a decade of investing in refining these systems, I’m so proud to celebrate our engineering teams today for being awarded the IEEE Computer Society’s Cybersecurity Award for Practice for development and deployment of static [...] Read More... The post Facebook engineers receive 2021 IEEE Computer Society Cybersecurity Award for static analysis tools appeared first on Facebook Engineering. http://dlvr.it/S9yYvY

What it is: Real Time Messaging Protocol (RTMP) is a popular media streaming protocol that uses Transmission Control Protocol (TCP) persistent connections. When a connection between a live-streaming client and the platform is interrupted, data from the live event is lost until the client can reconnect to a new server. RTMP Go Away is a [...] Read More... The post RTMP Go Away: Lossless reconnections for live streaming appeared first on Facebook Engineering. http://dlvr.it/S9tgNf

[...] Read More... The post Inside the Lab: Expanding connectivity by sea, land, and air appeared first on Facebook Engineering. http://dlvr.it/S96XgG

Now that our platforms are up and running as usual after yesterday’s outage, I thought it would be worth sharing a little more detail on what happened and why — and most importantly, how we’re learning from it.  This outage was triggered by the system that manages our global backbone network capacity. The backbone is [...] Read More... The post More details about the October 4 outage appeared first on Facebook Engineering. http://dlvr.it/S8zNSJ

To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms. We’ve been working as hard as we can to restore access, and our systems are now back up and running. The underlying cause of this outage also impacted many [...] Read More... The post Update about the October 4th outage appeared first on Facebook Engineering. http://dlvr.it/S8wnRz

We’re sharing details about Mariana Trench (MT), a tool we use to spot and prevent security and privacy bugs in Android and Java applications. As part of our effort to help scale security through building automation, we recently open-sourced MT to support security engineers at Facebook and across the industry.  This post is the third [...] Read More... The post Open-sourcing Mariana Trench: Analyzing Android and Java app security in depth appeared first on Facebook Engineering. http://dlvr.it/S8bx96

Facebook invests in subsea cables as part of our continued efforts to build the infrastructure that carries internet traffic and helps bring more people online to a faster internet. Today, we are thrilled to announce, along with regional and global partners, a new segment of subsea cable called 2Africa Pearls, which connects three continents — [...] Read More... The post 2Africa Pearls subsea cable connects Africa, Europe, and Asia to bring affordable, high-speed internet to 3 billion people appeared first on Facebook Engineering. http://dlvr.it/S8TtnD

Managing app size at Facebook is a unique challenge: Every day, developers check in large volumes of code, and each line of code translates into additional bits in the apps that people ultimately download onto their phones. Left unchecked, this added code would make the app bigger and bigger until eventually the time it takes [...] Read More... The post Superpack: Pushing the limits of compression in Facebook’s mobile apps appeared first on Facebook Engineering. http://dlvr.it/S7TLKk

For years, in order to safeguard the privacy of people’s messages, WhatsApp has provided end-to-end encryption by default ​​so messages can be seen only by the sender and recipient, and no one in between. Now, we’re planning to give people the option to protect their WhatsApp backups using end-to-end encryption as well. People can already [...] Read More... The post How WhatsApp is enabling end-to-end encrypted backups appeared first on Facebook Engineering. http://dlvr.it/S7KQpL

Caching plays an important role in helping people access their information efficiently. For example, when an email app loads, it temporarily caches some messages, so the user can refresh the page without the app retrieving the same messages. However, large-scale caching has long been a complex engineering challenge. Companies must balance the fast experience people [...] Read More... The post CacheLib, Facebook’s open source caching engine for web-scale services appeared first on Facebook Engineering. http://dlvr.it/S6qMPL

What the research is:  RAMP-TAO is a new protocol that improves the developer experience on TAO, Facebook’s online social graph store, by providing stronger transactional guarantees. It is the first protocol to provide transactional semantics over an eventually consistent massive-scale data store while still preserving the system’s overall reliability and performance. RAMP-TAO enables an intuitive [...] Read More... The post RAMP-TAO: Layering atomic transactions on Facebook’s online graph store appeared first on Facebook Engineering. http://dlvr.it/S5ts30

We are excited to announce our participation in the Apricot subsea cable system, together with leading regional and global partners. When completed, the project (which is still subject to regulatory approvals) will deliver much-needed internet capacity, redundancy, and reliability to expand connections in the Asia-Pacific region. The 12,000-kilometer-long cable will connect Japan, Taiwan, Guam, the [...] Read More... The post Apricot subsea cable will boost internet capacity, speeds in the Asia-Pacific region appeared first on Facebook Engineering. http://dlvr.it/S5jbVp

Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure. To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance. With the help of the OCP community, we [...] Read More... The post Open-sourcing a more precise time appliance appeared first on Facebook Engineering. http://dlvr.it/S5Rd13

What the research is:  A first-of-its-kind study detailing our backbone management strategy to ensure high service performance throughout the COVID-19 pandemic. The pandemic moved most social interactions online and caused an unprecedented stress test on our global network infrastructure with tens of data center regions. At this scale, failures such as fiber cuts, router misconfigurations, [...] Read More... The post Risk-driven backbone management during COVID-19 and beyond appeared first on Facebook Engineering. http://dlvr.it/S5Kkd7

ZippyDB is the largest strongly consistent, geographically distributed key-value store at Facebook. Since we first deployed ZippyDB in 2012, this key-value store has expanded rapidly, and today, ZippyDB serves a number of use cases, ranging from metadata for a distributed filesystem, counting events for both internal and external purposes, to product data that’s used for [...] Read More... The post How we built a general purpose key value store for Facebook with ZippyDB appeared first on Facebook Engineering. http://dlvr.it/S59Rv2