Backend
IA-MoE: An Intensity-Aware Mixture of Experts for DNN Inference
arXiv cs.DC0 views
arXiv:2611.19669v2 Announce Type: replace Abstract: In edge-cloud collaborative inference, the layers of a deep neural network (DNN) are placed on edge and cloud virtual machines (VMs) for every request, while the edge load and the cross-cloud bandwidth change over time. We propose IA-MoE, an intensity-aware Mixture of Experts (MoE) framework that scores each layer-VM pair by end-to-end delay and energy and places the layers one at a time on the best-scoring VM. Such greedy placement is myopic when trained on per-layer labels: offloading to the cloud slows the first offloaded layer while the gain appears only in later layers, so the policy keeps too many requests at the edge. Training on the remaining request time after each decision removes this myopia and reduces the mean delay by 12-56% on congested workloads. In CloudSimSDN simulations with three DNNs and two public request traces, IA-MoE has the lowest mean delay of all compared methods on congested VGG16 workloads, across cross-cloud bandwidths of 150-600 Mbps, fluctuating bandwidth and saturated edge servers. A random forest trained on the same label is 5-27% slower, and only a heuristic that sends whole requests to the cloud uses less energy, at up to 20 times the delay. When the edge is lightly loaded, offloading does not pay and simple heuristics perform as well. IA-MoE also beats a Double Deep Q-Network (DQN) agent with the same features in all 19 tested settings, with a mean delay lower by a factor of 1.3 to 10.5, and trains in minutes instead of hours.
Source:https://arxiv.org/abs/2611.19669PreviousWhy Kubernetes Cannot Contain a Full Lazy-Pulling Cache: Two Impossibility Results and a Bounded, Honest Design for Model ServingNextFlux: Optimal Scheduling of Optical Circuit Switches for LLM Training
Related posts
Autoscaling Adds Waiters, Not Ovens. Here's What Actually Breaks at 10x Load
0dev.to
[백엔드 모의면접] 중복 결제는 어떻게 막을까?
0dev.to
Server-Side Pagination Looked Easy. It Took Me Longer Than Any React Concept Yet.
0dev.to
Before You Create an AWS Account, Spend 45 Minutes Learning These Security Tools
0dev.to
Claude “Invalid Signature” in a Thinking Block: How to Debug It Safely
0dev.to
How to catch an expiring or broken SSL certificate before your visitors do
0dev.to