Postingan

Composable data management at Meta

Gambar
In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency.  We’re sharing how we’ve achieved this, in part, by leveraging Velox, Meta’s open source execution engine, as well as work ahead as we continue to rethink our data management systems.  Data is at [...] Read More... The post Composable data management at Meta appeared first on Engineering at Meta. http://dlvr.it/T7Fm52

Post-quantum readiness for TLS at Meta

Gambar
Today, the internet (like most digital infrastructure in general) relies heavily on the security offered by public-key cryptosystems such as RSA, Diffie-Hellman (DH), and elliptic curve cryptography (ECC). But the advent of quantum computers has raised real questions about the long-term privacy of data exchanged over the internet. In the future, significant advances in quantum [...] Read More... The post Post-quantum readiness for TLS at Meta appeared first on Engineering at Meta. http://dlvr.it/T7Flq2

Behind the scenes of Threads for web

Gambar
When Threads first launched one of the top feature requests was for a web client. In this episode of the Meta Tech Podcast, Pascal Hartig (@passy) sits down with Ally C. and Kevin C., two engineers on the Threads Web Team that delivered the basic version of Threads for web in just under three months. [...] Read More... The post Behind the scenes of Threads for web appeared first on Engineering at Meta. http://dlvr.it/T6t653

Building an infrastructure for AI’s future

Gambar
[...] Read More... The post Building an infrastructure for AI’s future appeared first on Engineering at Meta. http://dlvr.it/T5NLK3

Introducing the next-gen Meta Training and Inference Accelerator

Gambar
[...] Read More... The post Introducing the next-gen Meta Training and Inference Accelerator appeared first on Engineering at Meta. http://dlvr.it/T5KvGc

Bringing HDR photo support to Instagram and Threads

Gambar
Meta’s family of apps serves trillions of image download requests every day. And if you’re into high-quality images, you’ve probably noticed that Instagram and Threads have added support for high dynamic range (HDR) photos. Now people on Threads and Instagram can upload and share images that are more true-to-life, with the full color and range [...] Read More... The post Bringing HDR photo support to Instagram and Threads appeared first on Engineering at Meta. http://dlvr.it/T4fLq4

Threads has entered the fediverse

Gambar
Threads has entered the fediverse! As part of our beta experience, now available in a few countries, Threads users aged 18+ with public profiles can now choose to share their Threads posts to other ActivityPub-compliant servers. People on those servers can now follow federated Threads profiles and see, like, reply to, and repost posts from [...] Read More... The post Threads has entered the fediverse appeared first on Engineering at Meta. http://dlvr.it/T4QNyG

Optimizing RTC bandwidth estimation with machine learning

Gambar
Bandwidth estimation (BWE) and congestion control play an important role in delivering high-quality real-time communication (RTC) across Meta’s family of apps. We’ve adopted a machine learning (ML)-based approach that allows us to solve networking problems holistically across cross-layers such as BWE, network resiliency, and transport. We’re sharing our experiment results from this approach, some of [...] Read More... The post Optimizing RTC bandwidth estimation with machine learning appeared first on Engineering at Meta. http://dlvr.it/T4Mvz3

Better video for mobile RTC with AV1 and HD

Gambar
At Meta, we support real-time communication (RTC) for billions of people through our apps, including Messenger, Instagram, and WhatsApp. We’ve seen significant benefits by adopting the AV1 codec for RTC. Here’s how we are improving the RTC video quality for our apps with tools like the AV1 codec, the challenges we face, and how we [...] Read More... The post Better video for mobile RTC with AV1 and HD appeared first on Engineering at Meta. http://dlvr.it/T4MvlT

Logarithm: A logging engine for AI training workflows and services

Gambar
Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. Logarithm is a hosted, serverless, multitenant service, used only internally at Meta, that consumes and indexes these logs and provides an interactive query interface to retrieve and view logs. In this post, we present the design behind Logarithm, and [...] Read More... The post Logarithm: A logging engine for AI training workflows and services appeared first on Engineering at Meta. http://dlvr.it/T4Fp7Z

Building Meta’s GenAI Infrastructure

Gambar
Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We are strongly committed to open [...] Read More... The post Building Meta’s GenAI Infrastructure appeared first on Engineering at Meta. http://dlvr.it/T3z2gC

Making messaging interoperability with third parties safe for users in Europe

Gambar
To comply with a new EU law, the Digital Markets Act (DMA), which comes into force on March 7th, we’ve made major changes to WhatsApp and Messenger to enable interoperability with third-party messaging services.  We’re sharing how we enabled third-party interoperability (interop) while maintaining end-to-end encryption (E2EE) and other privacy guarantees in our services as [...] Read More... The post Making messaging interoperability with third parties safe for users in Europe appeared first on Engineering at Meta. http://dlvr.it/T3h4SQ

How DotSlash makes executable deployment simpler

Gambar
Andres Suarez and Michael Bolin, two software engineers at Meta, join Pascal Hartig (@passy) on the Meta Tech Podcast to discuss the ins and outs of DotSlash, a new open source tool from Meta. DotSlash takes the pain out of distributing binaries and toolchains to developers. Instead of committing large, platform-specific executables to a repository, [...] Read More... The post How DotSlash makes executable deployment simpler appeared first on Engineering at Meta. http://dlvr.it/T3HXYH

Aligning Velox and Apache Arrow: Towards composable data management

Gambar
We’ve partnered with Voltron Data and the Arrow community to align and converge Apache Arrow with Velox, Meta’s open source execution engine. Apache Arrow 15 includes three new format layouts developed through this partnership: StringView, ListView, and Run-End-Encoding (REE). This new convergence helps Meta and the larger community build data management systems that are unified, [...] Read More... The post Aligning Velox and Apache Arrow: Towards composable data management appeared first on Engineering at Meta. http://dlvr.it/T31740

Meta loves Python

Gambar
By now you’re already aware that Python 3.12 has been released. But did you know that several of its new features were developed by Meta? Meta engineer Pascal Hartig (@passy) is joined on the Meta Tech Podcast by Itamar Oren and Carl Meyer, two software engineers at Meta, to discuss their teams’ contributions to the [...] Read More... The post Meta loves Python appeared first on Engineering at Meta. http://dlvr.it/T2dx3M

Simple Precision Time Protocol at Meta

Gambar
While deploying Precision Time Protocol (PTP) at Meta, we’ve developed a simplified version of the protocol (Simple Precision Time Protocol – SPTP), that can offer the same level of clock synchronization as unicast PTPv2 more reliably and with fewer resources. In our own tests, SPTP boasts comparable performance to PTP, but with significant improvements in [...] Read More... The post Simple Precision Time Protocol at Meta appeared first on Engineering at Meta. http://dlvr.it/T2Rjvv

DotSlash: Simplified executable deployment

Gambar
We’ve open sourced DotSlash, a tool that makes large executables available in source control with a negligible impact on repository size, thus avoiding I/O-heavy clone operations. With DotSlash, a set of platform-specific executables is replaced with a single script containing descriptors for the supported platforms. DotSlash handles transparently fetching, decompressing, and verifying the appropriate remote [...] Read More... The post DotSlash: Simplified executable deployment appeared first on Engineering at Meta. http://dlvr.it/T2NRkB

Improving machine learning iteration speed with faster application build and packaging

Gambar
Slow build times and inefficiencies in packaging and distributing execution files were costing our ML/AI engineers a significant amount of time while working on our training stack. By addressing these issues head-on, we were able to reduce this overhead by double-digit percentages.  In the fast-paced world of AI/ML development, it’s crucial to ensure that our [...] Read More... The post Improving machine learning iteration speed with faster application build and packaging appeared first on Engineering at Meta. http://dlvr.it/T22Qcp

Lazy is the new fast: How Lazy Imports and Cinder accelerate machine learning at Meta

Gambar
At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime. The outcome? Up to 40 percent time to first batch (TTFB) improvements, along with a 20 percent reduction in Jupyter kernel startup times. This advancement facilitates swifter experimentation capabilities and elevates the [...] Read More... The post Lazy is the new fast: How Lazy Imports and Cinder accelerate machine learning at Meta appeared first on Engineering at Meta. http://dlvr.it/T1Z9nc

How Meta is advancing GenAI

Gambar
What’s going on with generative AI (GenAI) at Meta? And what does the future have in store? In this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) speaks with Devi Parikh, an AI research director at Meta. They cover a wide range of topics, including the history and future of GenAI and the most [...] Read More... The post How Meta is advancing GenAI appeared first on Engineering at Meta. http://dlvr.it/T1G1WR

How Meta built the infrastructure for Threads

Gambar
On July 5, 2023, Meta launched Threads, the newest product in our family of apps, to an unprecedented success that saw it garner over 100 million sign ups in its first five days. A small, nimble team of engineers built Threads over the course of only five months of technical work. While the app’s production [...] Read More... The post How Meta built the infrastructure for Threads appeared first on Engineering at Meta. http://dlvr.it/T0LrDX

AI debugging at Meta with HawkEye

Gambar
HawkEye is the powerful toolkit used internally at Meta for monitoring, observability, and debuggability of the end-to-end machine learning (ML) workflow that powers ML-based products. HawkEye supports recommendation and ranking models across several products at Meta. Over the past two years, it has facilitated order of magnitude improvements in the time spent debugging production issues. [...] Read More... The post AI debugging at Meta with HawkEye appeared first on Engineering at Meta. http://dlvr.it/T0Lqqs

Building end-to-end security for Messenger

Gambar
We are beginning to upgrade people’s personal conversations on Messenger to use end-to-end encryption (E2EE) by default Meta is publishing two technical white papers on end-to-end encryption: Our Messenger end-to-end encryption whitepaper describes the core cryptographic protocol for transmitting messages between clients. The Labyrinth encrypted storage protocol whitepaper explains our protocol for end-to-end encrypting stored [...] Read More... The post Building end-to-end security for Messenger appeared first on Engineering at Meta. http://dlvr.it/Szpb5c

Writing and linting Python at scale

Gambar
Python plays a big part at Meta. It powers Instagram’s backend and plays an important role in our configuration systems, as well as much of our AI work. Meta even made contributions to Python 3.12, the latest version of Python. On this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) is joined by Amethyst [...] Read More... The post Writing and linting Python at scale appeared first on Engineering at Meta. http://dlvr.it/Sz7cLc

Watch: Meta’s engineers on building network infrastructure for AI

Gambar
Meta is building for the future of AI at every level – from hardware like MTIA v1, Meta’s first-generation AI inference accelerator to publicly released models like Llama 2, Meta’s next-generation large language model, as well as new generative AI (GenAI) tools like Code Llama. Delivering next-generation AI products and services at Meta’s scale also [...] Read More... The post Watch: Meta’s engineers on building network infrastructure for AI appeared first on Engineering at Meta. http://dlvr.it/Syt3dm

Enhancing the security of WhatsApp calls

Gambar
New optional features in WhatsApp have helped make calling on WhatsApp more secure. “Silence Unknown Callers” is a new setting on WhatsApp that not only quiets annoying calls but also blocks sophisticated cyber attacks. “Protect IP Address in Calls” is a new setting on WhatsApp that helps hide your location from other parties on the [...] Read More... The post Enhancing the security of WhatsApp calls appeared first on Engineering at Meta. http://dlvr.it/SyYh1w

How Meta built Threads in 5 months

Gambar
In about five short months, a small team of engineers at Meta took Threads, the new text-based conversations app, from from an idea to the most successful app launch of all time, pulling in over 100M users in its first five days. But this achievement wouldn’t have been possible without Meta’s existing systems and infrastructure. [...] Read More... The post How Meta built Threads in 5 months appeared first on Engineering at Meta. http://dlvr.it/SySxL4

Automating data removal

Gambar
Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. SCARF avoids tedious manual work and ensures that product data is correctly removed when a product is shut down. This [...] Read More... The post Automating data removal appeared first on Engineering at Meta. http://dlvr.it/SyCHSl

Automating dead code cleanup

Gambar
Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis of programs to detect dead code from both a business and programming language perspective. SCARF automatically creates change requests that delete the dead code identified from the program analysis, minimizing developer costs. [...] Read More... The post Automating dead code cleanup appeared first on Engineering at Meta. http://dlvr.it/SxtrD1

5 Things you didn’t know about Buck2

Gambar
Meta has a very large monorepo, with many  different programming languages. To optimize build and performance, we developed our own build system called Buck, which was first open-sourced in 2013.  Buck2 is the recently open-sourced successor. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Below [...] Read More... The post 5 Things you didn’t know about Buck2 appeared first on Engineering at Meta. http://dlvr.it/SxrGC5

How Meta is creating custom silicon for AI

Olivia Wu, Meta’s Technical Lead for Infra Silicon, discusses the design and development of Meta’s first-generation AI inference accelerator. [...] Read More... The post How Meta is creating custom silicon for AI appeared first on Engineering at Meta. http://dlvr.it/SxcsFy

Automating product deprecation

Gambar
Systematic Code and Asset Removal Framework (SCARF) is Meta’s unused code and data deletion framework. SCARF guides engineers through deprecating a product safely and efficiently via an internal tool. SCARF combines this tooling with automation to reduce load on engineers. At Meta, we are constantly innovating and experimenting by building and shipping many different products, [...] Read More... The post Automating product deprecation appeared first on Engineering at Meta. http://dlvr.it/SxZJwV

Meta contributes new features to Python 3.12

Gambar
Python 3.12 is out! It includes new features and performance improvements – some contributed by Meta – that we believe will benefit all Python users. We’re sharing details about these new features that we worked closely with the Python community to develop. This week’s release of Python 3.12 marks a milestone in our efforts to [...] Read More... The post Meta contributes new features to Python 3.12 appeared first on Engineering at Meta. http://dlvr.it/Sx308L

Meta Quest 2: Defense through offense

Gambar
Meta’s Native Assurance team regularly performs manual code reviews as part of our ongoing commitment to improve the security posture of Meta’s products.  In 2021, we discovered a vulnerability in the Meta Quest 2’s Android-based OS that never made it to production but helped us find new ways to improve the security of Meta Quest [...] Read More... The post Meta Quest 2: Defense through offense appeared first on Engineering at Meta. http://dlvr.it/Sw0XM0

Threads: The inside story of Meta’s newest social app

Gambar
Earlier this year, a small team of engineers at Meta started working on an idea for a new app. It would have all the features people expect from a text-based conversations app, but with one very key, distinctive goal – being an app that would allow people to share their content across multiple platforms. We [...] Read More... The post Threads: The inside story of Meta’s newest social app appeared first on Engineering at Meta. http://dlvr.it/Svn4ZX

What’s it like to write code at Meta?

Gambar
Ever wonder what it’s like to write code at Meta’s scale? On the latest episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) sits down with Dustin Shahidehpour and Katherine Zak,  two software engineers at Meta, about their careers and what it’s really like to ship code at Meta. Why does Meta have a monorepo? [...] Read More... The post What’s it like to write code at Meta? appeared first on Engineering at Meta. http://dlvr.it/SvgNp5

Scheduling Jupyter Notebooks at Meta

Gambar
At Meta, Bento is our internal Jupyter notebooks platform that is leveraged by many internal users. Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL) that need to be repeated at certain intervals. Users with such notebooks would have to remember to manually run their notebooks at the [...] Read More... The post Scheduling Jupyter Notebooks at Meta appeared first on Engineering at Meta. http://dlvr.it/SvLZcq

Code Llama: Meta’s state-of-the-art LLM for coding

Gambar
[...] Read More... The post Code Llama: Meta’s state-of-the-art LLM for coding appeared first on Engineering at Meta. http://dlvr.it/Sv71Yx

Introducing Immortal Objects for Python

Gambar
Instagram has introduced Immortal Objects – PEP-683 – to Python. Now, objects can bypass reference count checks and live throughout the entire execution of the runtime, unlocking exciting avenues for true parallelism. At Meta, we use Python (Django) for our frontend server within Instagram. To handle parallelism, we rely on a multi-process architecture along with [...] Read More... The post Introducing Immortal Objects for Python appeared first on Engineering at Meta. http://dlvr.it/SthLjW

Meta Connect 2023: September 27 – 28

Gambar
[...] Read More... The post Meta Connect 2023: September 27 – 28 appeared first on Engineering at Meta. http://dlvr.it/StdxNz

Scaling the Instagram Explore recommendations system

Gambar
Explore is one of the largest recommendation systems on Instagram. We leverage machine learning to make sure people are always seeing content that is the most interesting and relevant to them. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and [...] Read More... The post Scaling the Instagram Explore recommendations system appeared first on Engineering at Meta. http://dlvr.it/StQZhS

How Meta is improving password security and preserving privacy

Gambar
Meta is developing new privacy-enhancing technologies (PETs) to innovate and solve problems with less data. These technologies enable teams to build and launch privacy-enhanced products in a way that’s verifiable and safeguards user data. Using state-of-the-art cryptographic techniques, we have developed Private Data Lookup (PDL) that allows users to privately query a server-side data set. [...] Read More... The post How Meta is improving password security and preserving privacy appeared first on Engineering at Meta. http://dlvr.it/StN2n1

Fixit 2: Meta’s next-generation auto-fixing linter

Gambar
Fixit is dead! Long live Fixit 2 – the latest version of our open-source auto-fixing linter. Fixit 2 allows developers to efficiently build custom lint rules and perform auto-fixes for their codebases. Fixit 2 is available today on PyPI. Python is one of the most popular languages in use at Meta. Meta’s production engineers (PEs) [...] Read More... The post Fixit 2: Meta’s next-generation auto-fixing linter appeared first on Engineering at Meta. http://dlvr.it/StKV0Y

Using short-lived certificates to protect TLS secrets

Gambar
Short-lived certificates (SLCs) are part of our latest efforts to further secure our Transport Layer Security (TLS) private keys on our edge networks. SLCs have a very short exposure compared to traditional certificates and lower the chances of a compromised private key being abused. Implementing SLCs has required us to address tradeoffs between operability and [...] Read More... The post Using short-lived certificates to protect TLS secrets appeared first on Engineering at Meta. http://dlvr.it/StK5X7

Bringing HDR video to Reels

Gambar
Meta has made it possible for people to upload high dynamic range (HDR) videos from their phone’s camera roll to Reels on Facebook and Instagram. To show standard dynamic range (SDR) UI elements and overlays legibly on top of HDR video, we render them at a brightness level comparable to the video itself. We solved [...] Read More... The post Bringing HDR video to Reels appeared first on Engineering at Meta. http://dlvr.it/SsK75f

Meta’s Evenstar is transitioning to OCP to accelerate open RAN adoption

Gambar
Meta is transferring its IP for Evenstar, a program to accelerate the adoption of open RAN technologies, to the Open Compute Project (OCP). Meta will contribute Evenstar’s radio unit design to OCP, giving the telecom industry its first open, white box radio unit solution. The TIP Open RAN community will leverage the Evenstar radio unit [...] Read More... The post Meta’s Evenstar is transitioning to OCP to accelerate open RAN adoption appeared first on Engineering at Meta. http://dlvr.it/SrQx05

Meta developer tools: Working at scale

Gambar
Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we [...] Read More... The post Meta developer tools: Working at scale appeared first on Engineering at Meta. http://dlvr.it/SrKRff

Bombyx is being licensed for product development

Gambar
When we first conceived of our aerial fiber deployment solution, Bombyx (the Latin name for a silk moth), we imagined a robot weaving strands of fiber-optic cables over powerlines, helping human workers quickly connect communities even in very rural or remote locations. Now, after years of successful research, Bombyx is taking the next steps in [...] Read More... The post Bombyx is being licensed for product development appeared first on Engineering at Meta. http://dlvr.it/SpRWGm

Meta introduces its first-generation AI inference accelerator

Gambar
[...] Read More... The post Meta introduces its first-generation AI inference accelerator appeared first on Engineering at Meta. http://dlvr.it/SpFgWT

Building and deploying MySQL Raft at Meta

Gambar
  We’re rolling out MySQL Raft with the aim to eventually replace our current MySQL semisynchronous databases.  The biggest win of MySQL Raft was simplification of the operation and making MySQL servers take care of promotions and membership. This gave the provable safety of Raft and reduced significant operational pain. Making MySQL server a true [...] Read More... The post Building and deploying MySQL Raft at Meta appeared first on Engineering at Meta. http://dlvr.it/Sp7QX5

The malware threat landscape: NodeStealer, DuckTail, and more

Gambar
We’re sharing our latest threat research and technical analysis into persistent malware campaigns targeting businesses across the internet, including threat indicators to help raise our industry’s collective defenses across the internet. These malware families – including Ducktail, NodeStealer and newer malware posing as ChatGPT and other similar tools– targeted people through malicious browser extensions, ads, [...] Read More... The post The malware threat landscape: NodeStealer, DuckTail, and more appeared first on Engineering at Meta. http://dlvr.it/SnSr9H