Desa aikmual

At Meta, Bento is our internal Jupyter notebooks platform that is leveraged by many internal users. Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL) that need to be repeated at certain intervals. Users with such notebooks would have to remember to manually run their notebooks at the [...] Read More... The post Scheduling Jupyter Notebooks at Meta appeared first on Engineering at Meta. http://dlvr.it/SvLZcq

[...] Read More... The post Code Llama: Meta’s state-of-the-art LLM for coding appeared first on Engineering at Meta. http://dlvr.it/Sv71Yx

Instagram has introduced Immortal Objects – PEP-683 – to Python. Now, objects can bypass reference count checks and live throughout the entire execution of the runtime, unlocking exciting avenues for true parallelism. At Meta, we use Python (Django) for our frontend server within Instagram. To handle parallelism, we rely on a multi-process architecture along with [...] Read More... The post Introducing Immortal Objects for Python appeared first on Engineering at Meta. http://dlvr.it/SthLjW

[...] Read More... The post Meta Connect 2023: September 27 – 28 appeared first on Engineering at Meta. http://dlvr.it/StdxNz

Explore is one of the largest recommendation systems on Instagram. We leverage machine learning to make sure people are always seeing content that is the most interesting and relevant to them. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and [...] Read More... The post Scaling the Instagram Explore recommendations system appeared first on Engineering at Meta. http://dlvr.it/StQZhS

Meta is developing new privacy-enhancing technologies (PETs) to innovate and solve problems with less data. These technologies enable teams to build and launch privacy-enhanced products in a way that’s verifiable and safeguards user data. Using state-of-the-art cryptographic techniques, we have developed Private Data Lookup (PDL) that allows users to privately query a server-side data set. [...] Read More... The post How Meta is improving password security and preserving privacy appeared first on Engineering at Meta. http://dlvr.it/StN2n1

Fixit is dead! Long live Fixit 2 – the latest version of our open-source auto-fixing linter. Fixit 2 allows developers to efficiently build custom lint rules and perform auto-fixes for their codebases. Fixit 2 is available today on PyPI. Python is one of the most popular languages in use at Meta. Meta’s production engineers (PEs) [...] Read More... The post Fixit 2: Meta’s next-generation auto-fixing linter appeared first on Engineering at Meta. http://dlvr.it/StKV0Y

Short-lived certificates (SLCs) are part of our latest efforts to further secure our Transport Layer Security (TLS) private keys on our edge networks. SLCs have a very short exposure compared to traditional certificates and lower the chances of a compromised private key being abused. Implementing SLCs has required us to address tradeoffs between operability and [...] Read More... The post Using short-lived certificates to protect TLS secrets appeared first on Engineering at Meta. http://dlvr.it/StK5X7

Meta has made it possible for people to upload high dynamic range (HDR) videos from their phone’s camera roll to Reels on Facebook and Instagram. To show standard dynamic range (SDR) UI elements and overlays legibly on top of HDR video, we render them at a brightness level comparable to the video itself. We solved [...] Read More... The post Bringing HDR video to Reels appeared first on Engineering at Meta. http://dlvr.it/SsK75f

Meta is transferring its IP for Evenstar, a program to accelerate the adoption of open RAN technologies, to the Open Compute Project (OCP). Meta will contribute Evenstar’s radio unit design to OCP, giving the telecom industry its first open, white box radio unit solution. The TIP Open RAN community will leverage the Evenstar radio unit [...] Read More... The post Meta’s Evenstar is transitioning to OCP to accelerate open RAN adoption appeared first on Engineering at Meta. http://dlvr.it/SrQx05

Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we [...] Read More... The post Meta developer tools: Working at scale appeared first on Engineering at Meta. http://dlvr.it/SrKRff

When we first conceived of our aerial fiber deployment solution, Bombyx (the Latin name for a silk moth), we imagined a robot weaving strands of fiber-optic cables over powerlines, helping human workers quickly connect communities even in very rural or remote locations. Now, after years of successful research, Bombyx is taking the next steps in [...] Read More... The post Bombyx is being licensed for product development appeared first on Engineering at Meta. http://dlvr.it/SpRWGm

[...] Read More... The post Meta introduces its first-generation AI inference accelerator appeared first on Engineering at Meta. http://dlvr.it/SpFgWT

  We’re rolling out MySQL Raft with the aim to eventually replace our current MySQL semisynchronous databases.  The biggest win of MySQL Raft was simplification of the operation and making MySQL servers take care of promotions and membership. This gave the provable safety of Raft and reduced significant operational pain. Making MySQL server a true [...] Read More... The post Building and deploying MySQL Raft at Meta appeared first on Engineering at Meta. http://dlvr.it/Sp7QX5

We’re sharing our latest threat research and technical analysis into persistent malware campaigns targeting businesses across the internet, including threat indicators to help raise our industry’s collective defenses across the internet. These malware families – including Ducktail, NodeStealer and newer malware posing as ChatGPT and other similar tools– targeted people through malicious browser extensions, ads, [...] Read More... The post The malware threat landscape: NodeStealer, DuckTail, and more appeared first on Engineering at Meta. http://dlvr.it/SnSr9H

What the research is:  Millisampler is one of Meta’s latest characterization tools and allows us to observe, characterize, and debug network performance at high-granularity timescales efficiently. This lightweight network traffic characterization tool for continual monitoring operates at fine, configurable timescales. It collects time series of ingress and egress traffic volumes, number of active flows, incoming [...] Read More... The post A fine-grained network traffic analysis with Millisampler appeared first on Engineering at Meta. http://dlvr.it/SmdBPl

WhatsApp has launched a new security feature that further helps prevent attackers from using vectors like on-device malware. This security feature, called Device Verification, requires no action or additional steps from users and helps protect your account. This feature is part of our broader work to increase security for our users from the growing threat [...] Read More... The post How Device Verification protects your WhatsApp account appeared first on Engineering at Meta. http://dlvr.it/SmQlTn

We’re sharing how Meta delivers high-quality audio at scale with the xHE-AAC audio codec. xHE-AAC has already been deployed on Facebook and Instagram to provide enhanced audio for features like Reels and Stories.  At Meta, we serve every media use case imaginable for billions of people across the world — from short-form, user-generated content, such [...] Read More... The post Why xHE-AAC is being embraced at Meta appeared first on Engineering at Meta. http://dlvr.it/SmKJCw

Buck2, our new open source, large-scale build system, is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient.  In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. Buck2, Meta’s open [...] Read More... The post Build faster with Buck2: Our open source build system appeared first on Engineering at Meta. http://dlvr.it/Sm5M7x

Meta is introducing Velox, an open source unified execution engine aimed at accelerating data management systems and streamlining their development. Velox is under active development. Experimental results from our paper published at the International Conference on Very Large Data Bases (VLDB) 2022 show how Velox improves efficiency and consistency in data management systems. Velox helps [...] Read More... The post Introducing Velox: An open source unified execution engine appeared first on Engineering at Meta. http://dlvr.it/Skchbp

While VR headsets have been with us for at least a decade, AR hardware barely exists today; indeed, the very components that will comprise the hardware scarcely exist, making it a truly zero-to-one innovation challenge. Meta’s Head of AR Glasses Hardware, Caitlin Kalinowski is helping to lead that charge. Kalinowski hails from Portsmouth, NH and [...] Read More... The post Meta’s head of AR glasses on the future of AR hardware appeared first on Engineering at Meta. http://dlvr.it/SjxfMR

We’re sharing how we’re enabling production and delivery of AV1 for Facebook Reels and Instagram Reels. We believe AV1 is the most viable codec for Meta for the coming years. It offers higher quality at a much lower bit rate compared with previous generations of video codecs. Meta has worked closely with the open source [...] Read More... The post How Meta brought AV1 to Reels appeared first on Engineering at Meta. http://dlvr.it/Sjlwk7

What’s new: Meta is sharing the inside story of how it developed the Ray-Ban Stories smart glasses. Why it matters: Creating Ray-Ban Stories meant Meta’s engineers had to take on new challenges to build smart glasses that married complex engineering dynamics. How do you make something that features cameras, microphones, audio, and touch controls, all [...] Read More... The post Inside Meta’s first smart glasses appeared first on Engineering at Meta. http://dlvr.it/SjWknx

Meta’s augmented reality (AR) platform is one of the largest in the world, helping the billions of people on Meta’s apps experience AR every day and giving hundreds of thousands of creators a means to express themselves Meta’s AR tools are unique because they can be used on a wide variety of devices — from [...] Read More... The post Building a cross-platform runtime for AR appeared first on Engineering at Meta. http://dlvr.it/SjLgYb

A lot has changed since the initial launch of our basemap in late 2020. We’re Meta now, but our mission remains the same: Giving people the power to build community and bring the world closer together. Across Meta, our family of applications (Facebook, Instagram, WhatsApp, among others) are using our basemap to connect people through [...] Read More... The post Improving Meta’s global maps appeared first on Engineering at Meta. http://dlvr.it/Sj3vdp

Facebook for iOS (FBiOS) is the oldest mobile codebase at Meta. Since the app was rewritten in 2012, it has been worked on by thousands of engineers and shipped to billions of users, and it can support hundreds of engineers iterating on it at a time. After years of iteration, the Facebook codebase does not [...] Read More... The post The evolution of Facebook’s iOS app architecture appeared first on Engineering at Meta. http://dlvr.it/Sj1Bps

We’ve made architecture changes to Meta’s event driven asynchronous computing platform that have  enabled easy integration with multiple event-sources.  We’re sharing our learnings from handling various workloads and how to tackle trade offs made with certain design choices in building the platform. Asynchronous computing is a paradigm where the user does not expect a workload [...] Read More... The post Asynchronous computing at Meta: Overview and learnings appeared first on Engineering at Meta. http://dlvr.it/Shk6rc

Managing network solutions amidst a growing scale inherently brings challenges around performance, deployment, and operational complexities.  At Meta, we’ve found that these challenges broadly fall into three themes: 1.)   Data center networking: Over the past decade, on the physical front, we have seen a rise in vendor-specific hardware that comes with heterogeneous feature and [...] Read More... The post Watch Meta’s engineers discuss optimizing large-scale networks appeared first on Engineering at Meta. http://dlvr.it/ShXR9F

The technical journey discusses the motivations, challenges, and technical solutions employed for warehouse schematization, especially a change to the wire serialization format employed in Meta’s data platform for data interchange related to Warehouse Analytics Logging. Here, we discuss the engineering, scaling, and nontechnical challenges of modernizing  Meta’s exabyte-scale data platform by migrating to the new [...] Read More... The post Tulip: Modernizing Meta’s data platform appeared first on Engineering at Meta. http://dlvr.it/ShTYlR

Meta has open-sourced Anonymous Credential Service (ACS), a highly available multitenant service that allows clients to authenticate in a de-identified manner. ACS enhances privacy and security while also being compute-conscious. By open-sourcing and fostering a community for ACS, we believe we can accelerate the pace of innovation in de-identified authentication. Data minimization — collecting the [...] Read More... The post Open-sourcing Anonymous Credential Service appeared first on Engineering at Meta. http://dlvr.it/SfDm7Q

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis [...] Read More... The post Enabling static analysis of SQL queries at Meta appeared first on Engineering at Meta. http://dlvr.it/Sdd0LJ

We developed a new static analysis tool called Nullsafe that is used at Meta to detect NullPointerException (NPE) errors in Java code. Interoperability with legacy code and gradual deployment model were key to Nullsafe’s wide adoption and allowed us to recover some null-safety properties in the context of an otherwise null-unsafe language in a multimillion-line [...] Read More... The post Retrofitting null-safety onto Java at Meta appeared first on Engineering at Meta. http://dlvr.it/SdBLRw

Implementing Precision Time Protocol (PTP) at Meta allows us to synchronize the systems that drive our products and services down to nanosecond precision. PTP’s predecessor, Network Time Protocol (NTP), provided us with millisecond precision, but as we scale to more advanced systems on our way to building the next computing platform, the metaverse and AI, [...] Read More... The post How Precision Time Protocol is being deployed at Meta appeared first on Engineering at Meta. http://dlvr.it/Sd7KVb

Code reviews are one of the most important parts of the software development process At Meta we’ve recognized the need to make code reviews as fast as possible without sacrificing quality We’re sharing several tools and steps we’ve taken at Meta to reduce the time waiting for code reviews When done well, code reviews can [...] Read More... The post Move faster, wait less: Improving code review time at Meta appeared first on Engineering at Meta. http://dlvr.it/SctM6V

Sapling is a new Git-compatible source control client. Sapling emphasizes usability while also scaling to the largest repositories in the world. ReviewStack is a demonstration code review UI for GitHub pull requests that integrates with Sapling to make reviewing stacks of commits easy. You can get started using Sapling today.  Source control is one of [...] Read More... The post Sapling: Source control that’s user-friendly and scalable appeared first on Engineering at Meta. http://dlvr.it/ScqDLF

We’re sharing Tulip, a binary serialization protocol supporting schema evolution.  Tulip assists with data schematization by addressing protocol reliability and other issues simultaneously.  It replaces multiple legacy formats used in Meta’s data platform and has achieved significant performance and efficiency gains. There are numerous heterogeneous services, such as warehouse data storage and various real-time systems, [...] Read More... The post Tulip: Schematizing Meta’s data platform appeared first on Engineering at Meta. http://dlvr.it/ScVlWp

In our constant quest to prioritize efficiency, Instagram’s engineers have developed a way to process new videos that reduces the cost to produce basic video encodings by 94 percent. With this method in place, Meta’s video infrastructure can continue to scale without needing to add more machines. This frees up resources so more people can [...] Read More... The post Reducing Instagram’s basic video compute time by 94 percent appeared first on Engineering at Meta. http://dlvr.it/ScDbnK

We’re sharing how Meta is applying statistics and machine learning (ML) to improve notification personalization and management on Instagram – particularly on daily digest push notifications. By using causal inference and ML to identify highly active users who are likely to see more content organically, we have been able to reduce the number of notifications [...] Read More... The post Improving Instagram notification management with machine learning and causal inference appeared first on Engineering at Meta. http://dlvr.it/Sc12XM

We’re sharing lessons learned from shifting our Android development from Java to Kotlin. Kotlin is a popular language for Android development and offers some key advantages over Java.  As of today, our Android codebase contains over 10 million lines of Kotlin code. We’re open sourcing various examples and utilities we used to manipulate Kotlin code [...] Read More... The post From zero to 10 million lines of Kotlin appeared first on Engineering at Meta. http://dlvr.it/SbdkwR

At OCP Summit 2022, we’re announcing Grand Teton, our next-generation platform for AI at scale that we’ll contribute to the OCP community. We’re also sharing new innovations designed to support data centers as they advance to support new AI technologies: A new, more efficient version of Open Rack. Our Air-Assisted Liquid Cooling (AALC) – design. [...] Read More... The post OCP Summit 2022: Open hardware for AI infrastructure appeared first on Engineering at Meta. http://dlvr.it/SbJSr2

Our in-app browser for Facebook on Android has historically relied on an Android System WebView based on Chromium, the open source project that powers many browsers on Android and other operating systems. On other mobile operating systems, the System WebView component cannot be updated without updating the entire operating system. On Android, this works differently, [...] Read More... The post Launching a new Chromium-based WebView for Android appeared first on Engineering at Meta. http://dlvr.it/SZHjM3

Data center optimization has always played an important role at Meta. By optimizing our data centers’ environmental controls, we can reduce our environmental impact  while ensuring that people can always depend on our products. With most other complex systems, optimization of energy consumption is a trial-and-error process. But experimenting on any component of a live [...] Read More... The post How thermal simulation helps optimize Meta’s data centers appeared first on Engineering at Meta. http://dlvr.it/SYKcXy

We’ve open-sourced MemLab, a JavaScript memory testing framework that automates memory leak detection. Finding and addressing the root cause of memory leaks is important for delivering a quality user experience on web applications. MemLab has helped engineers and developers at Meta improve user experience and make significant improvements in memory optimization. We hope it will [...] Read More... The post MemLab: An open source framework for finding JavaScript memory leaks appeared first on Engineering at Meta. http://dlvr.it/SYBy38

Meta’s overall network usage and traffic volume has increased as we’ve continued to add new services. Due to the scarcity of fiber resources, we’re developing an explicit resource reservation framework to effectively plan, manage, and operate the shared consumption of network bandwidth, which will help us keep up with demand and limit network disruptions during [...] Read More... The post Network Entitlement: A contract-based network sharing solution appeared first on Engineering at Meta. http://dlvr.it/SXwyjm

Meta currently operates 14 data centers around the world. This rapidly expanding global data center footprint poses new challenges for service owners and for our infrastructure management systems. Systems like Twine, which we use to scale cluster management, and RAS, which handles perpetual region-wide resource allocation, have provided the abstractions and automation necessary for service [...] Read More... The post Viewing the world as a computer: Global capacity management appeared first on Engineering at Meta. http://dlvr.it/SXslZX

Meta is introducing Velox, an open source unified execution engine aimed at accelerating data management systems and streamlining their development. Velox is under active development. Experimental results from our paper published at the International Conference on Very Large Data Bases (VLDB) 2022 show how Velox improves efficiency and consistency in data management systems. Velox helps [...] Read More... The post Introducing Velox: An open source unified execution engine appeared first on Engineering at Meta. http://dlvr.it/SXXv99

When we focus on minimizing errors and downtime here at Meta, we place a lot of attention on service-level indicators (SLIs) and service-level objectives (SLOs). Consider Instagram, for example. There, SLIs represent metrics from different product surfaces, like the volume of error response codes to certain endpoints, or the number of successful media uploads. Based [...] Read More... The post Improving Meta’s SLO workflows with data annotations appeared first on Engineering at Meta. http://dlvr.it/SXR48r

A touring alien from a galaxy far, far away is an avid Instagram user. Her Instagram Feed is dominated by: Friends and family posts Some space travel magazines A few general news accounts Lots of science fiction blogs She logs in, scrolls through her feed gently — catching up with friends and family, keeping pace [...] Read More... The post How Instagram suggests new content appeared first on Engineering at Meta. http://dlvr.it/SWWyVv

Many of Meta’s products, such as search and language translations, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastructure increases, we need to scale our data ingestion infrastructure accordingly to handle workloads more efficiently. GPUs, which are used for training infrastructure, tend to double in [...] Read More... The post Scaling data ingestion for machine learning training at Meta appeared first on Engineering at Meta. http://dlvr.it/SWPMWQ

At Meta, our messaging apps help billions of people around the world stay connected to those who matter most to them. This scale brings potential threats from criminals and hackers, so we have a responsibility to keep people and their data safe. We’re sharing a set of principles to ensure that security is central to [...] Read More... The post Five security principles for billions of messages across Meta’s apps appeared first on Engineering at Meta. http://dlvr.it/SVfzks