Learning for Low-Latency Inference in Deep Neural Networks

Home

Discover

Bookmarks

Keep Up With Top Trending Papers In CS / AI / ML ...

Learning for Low-Latency Inference in Deep Neural Networks

Accelerating Deep Learning Inference via Learned Caches

We present the design of gati, an end-to-end prediction serving system that incorporates learned caches for low-latency deep neural network inference.

We observe that caching hidden layer outputs of the deep neural network can introduce a form of late-binding where inference requests only consume the amount of computation needed.

This enables a mechanism for achieving low latencies, coupled with anability to exploit temporal locality in prediction serving workloads.

Results show that gati can reduce inference latency by up to 7.69x on realistic workloads.

Authors

Arjun Balasubramanian, Adarsh Kumar, Yuhan Liu, Han Cao, Shivaram Venkataraman, Aditya Akella

◐ Latest Activity

◐ Recommended Members