Latest Research Papers
2024-03-04
arXiv
Wukong: Towards a Scaling Law for Large-Scale Recommendation
This paper introduces Wukong, a network architecture based on stacked factorization machines and an upscaling strategy, to establish a scaling law for recommendation models. Wukong effectively captures diverse interactions and outperforms state-of-the-art models in quality and scalability. The results show that Wukong maintains its superiority across a wide range of model complexities.
Scaling laws play an instrumental role in the sustainable improvement in
model quality. Unfortunately, recommendation models to date do not exhibit such
laws similar to those observed in the domain of large language models, due to
the inefficiencies of their upscaling mechanisms. This limitation poses
significant challenges in adapting these models to increasingly more complex
real-world datasets. In this paper, we propose an effective network
architecture based purely on stacked factorization machines, and a synergistic
upscaling strategy, collectively dubbed Wukong, to establish a scaling law in
the domain of recommendation. Wukong's unique design makes it possible to
capture diverse, any-order of interactions simply through taller and wider
layers. We conducted extensive evaluations on six public datasets, and our
results demonstrate that Wukong consistently outperforms state-of-the-art
models quality-wise. Further, we assessed Wukong's scalability on an internal,
large-scale dataset. The results show that Wukong retains its superiority in
quality over state-of-the-art models, while holding the scaling law across two
orders of magnitude in model complexity, extending beyond 100 GFLOP/example,
where prior arts fall short.