This talk presents our ongoing research on improving communication efficiency in distributed ML systems. I will share techniques that leverage the inherent sparsity found in ML models to significantly reduce communication time, and present initial set of results and findings.

Microsoft, with their Catapult research, and continuing with their research to support AI and smart networks, are one of the few hyper scalars that have publicly deployed FPGAs broadly. In addition, Amazon, and more recently, Microsoft and others, have deployed FPGAs as compute that anyone can rent for an hourly fee. Users can now deploy these accelerators quickly, and accessibility has never been better. But, given massive deployments of GPUs, it is unknown what the future will hold for FPGAs in the datacenter. Microsoft, for example, has now developed a new AI accelerator ASIC — but how does this compare to their BrainWave FPGA solution and why are they moving to ASICs now? In this talk, I will provide a background into our recent spatial accelerator research, and aim to present the FPGA, as well as other spatial accelerators as a strong contender as a path forward. I posit that with the right interfaces and programming models, spatial accelerators might move closer to becoming a more general-purpose acceleration solution.

This talk presents PulseNet, a system that eliminates this trade-off by melding a conventional control plane with an expedited, parallel track. PulseNet intelligently distinguishes between predictable, sustainable traffic—handled by standard, full-featured, regular function instances for maximum compatibility—and sudden traffic bursts that trigger function cold starts. These bursts are routed to a fast path that uses node-local agents to rapidly spawn lightweight, emergency instances, bypassing the central manager’s overhead. We prototype PulseNet in vHive, showing how this dual-track approach allows to achieve 1.5-3.5x faster performance and reduce the operational cost by up to 70%, compared to the state-of-the-art systems — all while remaining fully integrated with the conventional Kubernetes manager for over 98% of invocation traffic.

This talk presents a trajectory from energy-aware compression using reinforcement learning, to on-device quantized training, and finally to emerging quantum approaches that frame compression as an optimization problem solvable by quantum annealing. We highlight the efficiency gains, limitations of classical methods, and the potential of quantum computing to accelerate model compression in the future.

As data volume and complexity grew, we introduced BG3 (ByteGraph 3.0)—a cost-effective and scalable system featuring: (1) a query-efficient storage engine using BW-tree indices and cloud storage, (2) workload-aware space reclamation for better storage utilization, and (3) a lightweight synchronization mechanism for real-time consistency. BG3 significantly reduces operational costs while supporting fast, large-scale graph processing across ByteDance’s global platforms.

In this paper, we introduce AxCore, a quantization-aware, approximate GEMM unit that combines weight-only quantization with floating-point multiplication approximation (FPMA) to deliver highly efficient and accurate LLM inference. Unlike traditional GEMM units, AxCore eliminates multipliers entirely, replacing them with low-bit integer additions in a novel systolic array. AxCore features several key innovations: (1) a mixed-precision FPMAbased processing element that supports direct computation on compressed weights and high-precision activations; (2) a lightweight co-designed accuracy preservation strategy, including subnormal number handling, error compensation and format-aware quantization; and (3) a set of systolic array optimizations, including shared correction and normalization logic, as well as FPMA-based dequantization, to enhance hardware efficiency. Evaluations on open-source LLMs show that AxCore achieves up to 6.3×-12.5× higher compute density than conventional FP GEMM units. Compared to stateof-the-art INT4-based accelerators, FIGLUT and FIGNA, AxCore improves compute density by 53% and 70%, respectively, while also delivering lower perplexity.

Even though these heterogeneous systems achieve better performance than CPU-only homogeneous system due to the dying of Moore’s law, we still identify that simple heterogeneous systems cannot harvest the potential of each heterogeneous device. For example, GPU-only LLMs suffer from severe IO issues. To this end, we argue for hyper-heterogeneous LLM systems. The key idea is to offload communiation and storage from GPUs to the nework, e.g., SmartNIC and SmartSwitch, while only keeping computation in GPUs. As such, the GPU utilization is maximized.

The system incorporates two core innovations: Layer Sliding Mechanism and Asynchronous Offloading Technology. Our system supports dual offloading to both CPU memory and NVMe storage, enabling fine-tuning of the latest 123B+ parameter models on a single RTX 4090 GPU. When offloading to CPU, it maintains >95% peak performance on both NVIDIA and AMD GPUs. Experiments show that compared to baseline methods, SlideFormer achieves a 1.40–6.27× throughput improvement while reducing CPU and GPU memory usage by ~50%. In this talk, we will also report some recent results on serving large LLMs on a single RTX 4090 GPU.

These workloads bring new challenges in responsiveness, state management, and resource efficiency. In this talk, we introduce SAGE, a dataflow-native inference framework designed to support always-on, high-performance, and evolvable generative AI applications. SAGE integrates dataflow orchestration, programmable memory, resource-aware scheduling, and full-stack observability into a unified execution engine. Using an AIOps intelligent diagnosis assistant as a motivating example, we explain why such scenarios are better served by globally optimized dataflow systems rather than traditional message-driven multi-agent frameworks. We will outline the core design principles of SAGE, highlight representative use cases, and briefly discuss ongoing research directions.

Currently, RISC-V already provides diversity of extensions supporting various computing tasks including SIMD, vector and matrix which benefits the AI/ML scenario. At the meanwhile, it is very easy to develop the customized instruction set to facilitate the user-defined acceleration. The open-source community support and the customizable feature greatly reduces the development cost of the accelerator IP as well as the supporting software and middleware which leads to the bloom of the ECO system of heterogeneous computing. In this talk, we present the benefits, technical trend and recent updates from industry to apply RISC-V in different and versatile of computing systems.