Desa aikmual

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency.  We’re sharing how we’ve achieved this, in part, by leveraging Velox, Meta’s open source execution engine, as well as work ahead as we continue to rethink our data management systems.  Data is at [...] Read More... The post Composable data management at Meta appeared first on Engineering at Meta. http://dlvr.it/T7Fm52

Today, the internet (like most digital infrastructure in general) relies heavily on the security offered by public-key cryptosystems such as RSA, Diffie-Hellman (DH), and elliptic curve cryptography (ECC). But the advent of quantum computers has raised real questions about the long-term privacy of data exchanged over the internet. In the future, significant advances in quantum [...] Read More... The post Post-quantum readiness for TLS at Meta appeared first on Engineering at Meta. http://dlvr.it/T7Flq2

When Threads first launched one of the top feature requests was for a web client. In this episode of the Meta Tech Podcast, Pascal Hartig (@passy) sits down with Ally C. and Kevin C., two engineers on the Threads Web Team that delivered the basic version of Threads for web in just under three months. [...] Read More... The post Behind the scenes of Threads for web appeared first on Engineering at Meta. http://dlvr.it/T6t653

[...] Read More... The post Building an infrastructure for AI’s future appeared first on Engineering at Meta. http://dlvr.it/T5NLK3

[...] Read More... The post Introducing the next-gen Meta Training and Inference Accelerator appeared first on Engineering at Meta. http://dlvr.it/T5KvGc

Meta’s family of apps serves trillions of image download requests every day. And if you’re into high-quality images, you’ve probably noticed that Instagram and Threads have added support for high dynamic range (HDR) photos. Now people on Threads and Instagram can upload and share images that are more true-to-life, with the full color and range [...] Read More... The post Bringing HDR photo support to Instagram and Threads appeared first on Engineering at Meta. http://dlvr.it/T4fLq4

Threads has entered the fediverse! As part of our beta experience, now available in a few countries, Threads users aged 18+ with public profiles can now choose to share their Threads posts to other ActivityPub-compliant servers. People on those servers can now follow federated Threads profiles and see, like, reply to, and repost posts from [...] Read More... The post Threads has entered the fediverse appeared first on Engineering at Meta. http://dlvr.it/T4QNyG

Bandwidth estimation (BWE) and congestion control play an important role in delivering high-quality real-time communication (RTC) across Meta’s family of apps. We’ve adopted a machine learning (ML)-based approach that allows us to solve networking problems holistically across cross-layers such as BWE, network resiliency, and transport. We’re sharing our experiment results from this approach, some of [...] Read More... The post Optimizing RTC bandwidth estimation with machine learning appeared first on Engineering at Meta. http://dlvr.it/T4Mvz3

At Meta, we support real-time communication (RTC) for billions of people through our apps, including Messenger, Instagram, and WhatsApp. We’ve seen significant benefits by adopting the AV1 codec for RTC. Here’s how we are improving the RTC video quality for our apps with tools like the AV1 codec, the challenges we face, and how we [...] Read More... The post Better video for mobile RTC with AV1 and HD appeared first on Engineering at Meta. http://dlvr.it/T4MvlT

Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. Logarithm is a hosted, serverless, multitenant service, used only internally at Meta, that consumes and indexes these logs and provides an interactive query interface to retrieve and view logs. In this post, we present the design behind Logarithm, and [...] Read More... The post Logarithm: A logging engine for AI training workflows and services appeared first on Engineering at Meta. http://dlvr.it/T4Fp7Z

Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We are strongly committed to open [...] Read More... The post Building Meta’s GenAI Infrastructure appeared first on Engineering at Meta. http://dlvr.it/T3z2gC

To comply with a new EU law, the Digital Markets Act (DMA), which comes into force on March 7th, we’ve made major changes to WhatsApp and Messenger to enable interoperability with third-party messaging services.  We’re sharing how we enabled third-party interoperability (interop) while maintaining end-to-end encryption (E2EE) and other privacy guarantees in our services as [...] Read More... The post Making messaging interoperability with third parties safe for users in Europe appeared first on Engineering at Meta. http://dlvr.it/T3h4SQ

Andres Suarez and Michael Bolin, two software engineers at Meta, join Pascal Hartig (@passy) on the Meta Tech Podcast to discuss the ins and outs of DotSlash, a new open source tool from Meta. DotSlash takes the pain out of distributing binaries and toolchains to developers. Instead of committing large, platform-specific executables to a repository, [...] Read More... The post How DotSlash makes executable deployment simpler appeared first on Engineering at Meta. http://dlvr.it/T3HXYH

We’ve partnered with Voltron Data and the Arrow community to align and converge Apache Arrow with Velox, Meta’s open source execution engine. Apache Arrow 15 includes three new format layouts developed through this partnership: StringView, ListView, and Run-End-Encoding (REE). This new convergence helps Meta and the larger community build data management systems that are unified, [...] Read More... The post Aligning Velox and Apache Arrow: Towards composable data management appeared first on Engineering at Meta. http://dlvr.it/T31740

By now you’re already aware that Python 3.12 has been released. But did you know that several of its new features were developed by Meta? Meta engineer Pascal Hartig (@passy) is joined on the Meta Tech Podcast by Itamar Oren and Carl Meyer, two software engineers at Meta, to discuss their teams’ contributions to the [...] Read More... The post Meta loves Python appeared first on Engineering at Meta. http://dlvr.it/T2dx3M

While deploying Precision Time Protocol (PTP) at Meta, we’ve developed a simplified version of the protocol (Simple Precision Time Protocol – SPTP), that can offer the same level of clock synchronization as unicast PTPv2 more reliably and with fewer resources. In our own tests, SPTP boasts comparable performance to PTP, but with significant improvements in [...] Read More... The post Simple Precision Time Protocol at Meta appeared first on Engineering at Meta. http://dlvr.it/T2Rjvv

We’ve open sourced DotSlash, a tool that makes large executables available in source control with a negligible impact on repository size, thus avoiding I/O-heavy clone operations. With DotSlash, a set of platform-specific executables is replaced with a single script containing descriptors for the supported platforms. DotSlash handles transparently fetching, decompressing, and verifying the appropriate remote [...] Read More... The post DotSlash: Simplified executable deployment appeared first on Engineering at Meta. http://dlvr.it/T2NRkB

Slow build times and inefficiencies in packaging and distributing execution files were costing our ML/AI engineers a significant amount of time while working on our training stack. By addressing these issues head-on, we were able to reduce this overhead by double-digit percentages.  In the fast-paced world of AI/ML development, it’s crucial to ensure that our [...] Read More... The post Improving machine learning iteration speed with faster application build and packaging appeared first on Engineering at Meta. http://dlvr.it/T22Qcp

At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime. The outcome? Up to 40 percent time to first batch (TTFB) improvements, along with a 20 percent reduction in Jupyter kernel startup times. This advancement facilitates swifter experimentation capabilities and elevates the [...] Read More... The post Lazy is the new fast: How Lazy Imports and Cinder accelerate machine learning at Meta appeared first on Engineering at Meta. http://dlvr.it/T1Z9nc

What’s going on with generative AI (GenAI) at Meta? And what does the future have in store? In this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) speaks with Devi Parikh, an AI research director at Meta. They cover a wide range of topics, including the history and future of GenAI and the most [...] Read More... The post How Meta is advancing GenAI appeared first on Engineering at Meta. http://dlvr.it/T1G1WR

On July 5, 2023, Meta launched Threads, the newest product in our family of apps, to an unprecedented success that saw it garner over 100 million sign ups in its first five days. A small, nimble team of engineers built Threads over the course of only five months of technical work. While the app’s production [...] Read More... The post How Meta built the infrastructure for Threads appeared first on Engineering at Meta. http://dlvr.it/T0LrDX

HawkEye is the powerful toolkit used internally at Meta for monitoring, observability, and debuggability of the end-to-end machine learning (ML) workflow that powers ML-based products. HawkEye supports recommendation and ranking models across several products at Meta. Over the past two years, it has facilitated order of magnitude improvements in the time spent debugging production issues. [...] Read More... The post AI debugging at Meta with HawkEye appeared first on Engineering at Meta. http://dlvr.it/T0Lqqs

We are beginning to upgrade people’s personal conversations on Messenger to use end-to-end encryption (E2EE) by default Meta is publishing two technical white papers on end-to-end encryption: Our Messenger end-to-end encryption whitepaper describes the core cryptographic protocol for transmitting messages between clients. The Labyrinth encrypted storage protocol whitepaper explains our protocol for end-to-end encrypting stored [...] Read More... The post Building end-to-end security for Messenger appeared first on Engineering at Meta. http://dlvr.it/Szpb5c

Python plays a big part at Meta. It powers Instagram’s backend and plays an important role in our configuration systems, as well as much of our AI work. Meta even made contributions to Python 3.12, the latest version of Python. On this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) is joined by Amethyst [...] Read More... The post Writing and linting Python at scale appeared first on Engineering at Meta. http://dlvr.it/Sz7cLc

Meta is building for the future of AI at every level – from hardware like MTIA v1, Meta’s first-generation AI inference accelerator to publicly released models like Llama 2, Meta’s next-generation large language model, as well as new generative AI (GenAI) tools like Code Llama. Delivering next-generation AI products and services at Meta’s scale also [...] Read More... The post Watch: Meta’s engineers on building network infrastructure for AI appeared first on Engineering at Meta. http://dlvr.it/Syt3dm

New optional features in WhatsApp have helped make calling on WhatsApp more secure. “Silence Unknown Callers” is a new setting on WhatsApp that not only quiets annoying calls but also blocks sophisticated cyber attacks. “Protect IP Address in Calls” is a new setting on WhatsApp that helps hide your location from other parties on the [...] Read More... The post Enhancing the security of WhatsApp calls appeared first on Engineering at Meta. http://dlvr.it/SyYh1w

In about five short months, a small team of engineers at Meta took Threads, the new text-based conversations app, from from an idea to the most successful app launch of all time, pulling in over 100M users in its first five days. But this achievement wouldn’t have been possible without Meta’s existing systems and infrastructure. [...] Read More... The post How Meta built Threads in 5 months appeared first on Engineering at Meta. http://dlvr.it/SySxL4

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. SCARF avoids tedious manual work and ensures that product data is correctly removed when a product is shut down. This [...] Read More... The post Automating data removal appeared first on Engineering at Meta. http://dlvr.it/SyCHSl

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis of programs to detect dead code from both a business and programming language perspective. SCARF automatically creates change requests that delete the dead code identified from the program analysis, minimizing developer costs. [...] Read More... The post Automating dead code cleanup appeared first on Engineering at Meta. http://dlvr.it/SxtrD1

Meta has a very large monorepo, with many  different programming languages. To optimize build and performance, we developed our own build system called Buck, which was first open-sourced in 2013.  Buck2 is the recently open-sourced successor. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Below [...] Read More... The post 5 Things you didn’t know about Buck2 appeared first on Engineering at Meta. http://dlvr.it/SxrGC5

Olivia Wu, Meta’s Technical Lead for Infra Silicon, discusses the design and development of Meta’s first-generation AI inference accelerator. [...] Read More... The post How Meta is creating custom silicon for AI appeared first on Engineering at Meta. http://dlvr.it/SxcsFy

Systematic Code and Asset Removal Framework (SCARF) is Meta’s unused code and data deletion framework. SCARF guides engineers through deprecating a product safely and efficiently via an internal tool. SCARF combines this tooling with automation to reduce load on engineers. At Meta, we are constantly innovating and experimenting by building and shipping many different products, [...] Read More... The post Automating product deprecation appeared first on Engineering at Meta. http://dlvr.it/SxZJwV

Python 3.12 is out! It includes new features and performance improvements – some contributed by Meta – that we believe will benefit all Python users. We’re sharing details about these new features that we worked closely with the Python community to develop. This week’s release of Python 3.12 marks a milestone in our efforts to [...] Read More... The post Meta contributes new features to Python 3.12 appeared first on Engineering at Meta. http://dlvr.it/Sx308L

Meta’s Native Assurance team regularly performs manual code reviews as part of our ongoing commitment to improve the security posture of Meta’s products.  In 2021, we discovered a vulnerability in the Meta Quest 2’s Android-based OS that never made it to production but helped us find new ways to improve the security of Meta Quest [...] Read More... The post Meta Quest 2: Defense through offense appeared first on Engineering at Meta. http://dlvr.it/Sw0XM0

Earlier this year, a small team of engineers at Meta started working on an idea for a new app. It would have all the features people expect from a text-based conversations app, but with one very key, distinctive goal – being an app that would allow people to share their content across multiple platforms. We [...] Read More... The post Threads: The inside story of Meta’s newest social app appeared first on Engineering at Meta. http://dlvr.it/Svn4ZX

Ever wonder what it’s like to write code at Meta’s scale? On the latest episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) sits down with Dustin Shahidehpour and Katherine Zak,  two software engineers at Meta, about their careers and what it’s really like to ship code at Meta. Why does Meta have a monorepo? [...] Read More... The post What’s it like to write code at Meta? appeared first on Engineering at Meta. http://dlvr.it/SvgNp5

At Meta, Bento is our internal Jupyter notebooks platform that is leveraged by many internal users. Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL) that need to be repeated at certain intervals. Users with such notebooks would have to remember to manually run their notebooks at the [...] Read More... The post Scheduling Jupyter Notebooks at Meta appeared first on Engineering at Meta. http://dlvr.it/SvLZcq

[...] Read More... The post Code Llama: Meta’s state-of-the-art LLM for coding appeared first on Engineering at Meta. http://dlvr.it/Sv71Yx

Instagram has introduced Immortal Objects – PEP-683 – to Python. Now, objects can bypass reference count checks and live throughout the entire execution of the runtime, unlocking exciting avenues for true parallelism. At Meta, we use Python (Django) for our frontend server within Instagram. To handle parallelism, we rely on a multi-process architecture along with [...] Read More... The post Introducing Immortal Objects for Python appeared first on Engineering at Meta. http://dlvr.it/SthLjW

[...] Read More... The post Meta Connect 2023: September 27 – 28 appeared first on Engineering at Meta. http://dlvr.it/StdxNz

Explore is one of the largest recommendation systems on Instagram. We leverage machine learning to make sure people are always seeing content that is the most interesting and relevant to them. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and [...] Read More... The post Scaling the Instagram Explore recommendations system appeared first on Engineering at Meta. http://dlvr.it/StQZhS

Meta is developing new privacy-enhancing technologies (PETs) to innovate and solve problems with less data. These technologies enable teams to build and launch privacy-enhanced products in a way that’s verifiable and safeguards user data. Using state-of-the-art cryptographic techniques, we have developed Private Data Lookup (PDL) that allows users to privately query a server-side data set. [...] Read More... The post How Meta is improving password security and preserving privacy appeared first on Engineering at Meta. http://dlvr.it/StN2n1

Fixit is dead! Long live Fixit 2 – the latest version of our open-source auto-fixing linter. Fixit 2 allows developers to efficiently build custom lint rules and perform auto-fixes for their codebases. Fixit 2 is available today on PyPI. Python is one of the most popular languages in use at Meta. Meta’s production engineers (PEs) [...] Read More... The post Fixit 2: Meta’s next-generation auto-fixing linter appeared first on Engineering at Meta. http://dlvr.it/StKV0Y

Short-lived certificates (SLCs) are part of our latest efforts to further secure our Transport Layer Security (TLS) private keys on our edge networks. SLCs have a very short exposure compared to traditional certificates and lower the chances of a compromised private key being abused. Implementing SLCs has required us to address tradeoffs between operability and [...] Read More... The post Using short-lived certificates to protect TLS secrets appeared first on Engineering at Meta. http://dlvr.it/StK5X7

Meta has made it possible for people to upload high dynamic range (HDR) videos from their phone’s camera roll to Reels on Facebook and Instagram. To show standard dynamic range (SDR) UI elements and overlays legibly on top of HDR video, we render them at a brightness level comparable to the video itself. We solved [...] Read More... The post Bringing HDR video to Reels appeared first on Engineering at Meta. http://dlvr.it/SsK75f

Meta is transferring its IP for Evenstar, a program to accelerate the adoption of open RAN technologies, to the Open Compute Project (OCP). Meta will contribute Evenstar’s radio unit design to OCP, giving the telecom industry its first open, white box radio unit solution. The TIP Open RAN community will leverage the Evenstar radio unit [...] Read More... The post Meta’s Evenstar is transitioning to OCP to accelerate open RAN adoption appeared first on Engineering at Meta. http://dlvr.it/SrQx05

Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we [...] Read More... The post Meta developer tools: Working at scale appeared first on Engineering at Meta. http://dlvr.it/SrKRff

When we first conceived of our aerial fiber deployment solution, Bombyx (the Latin name for a silk moth), we imagined a robot weaving strands of fiber-optic cables over powerlines, helping human workers quickly connect communities even in very rural or remote locations. Now, after years of successful research, Bombyx is taking the next steps in [...] Read More... The post Bombyx is being licensed for product development appeared first on Engineering at Meta. http://dlvr.it/SpRWGm

[...] Read More... The post Meta introduces its first-generation AI inference accelerator appeared first on Engineering at Meta. http://dlvr.it/SpFgWT

  We’re rolling out MySQL Raft with the aim to eventually replace our current MySQL semisynchronous databases.  The biggest win of MySQL Raft was simplification of the operation and making MySQL servers take care of promotions and membership. This gave the provable safety of Raft and reduced significant operational pain. Making MySQL server a true [...] Read More... The post Building and deploying MySQL Raft at Meta appeared first on Engineering at Meta. http://dlvr.it/Sp7QX5

We’re sharing our latest threat research and technical analysis into persistent malware campaigns targeting businesses across the internet, including threat indicators to help raise our industry’s collective defenses across the internet. These malware families – including Ducktail, NodeStealer and newer malware posing as ChatGPT and other similar tools– targeted people through malicious browser extensions, ads, [...] Read More... The post The malware threat landscape: NodeStealer, DuckTail, and more appeared first on Engineering at Meta. http://dlvr.it/SnSr9H