ML Research Labs, Canberra
Research
Our ML research team works with the Australian National University to make large models faster and speech AI more accurate, and publishes what it finds.
- 3.16×faster AI text generation on average, with no loss of accuracy
- +44%more speedup than EAGLE-2, the previous best method
- ANUour research partner, the Australian National University
[ 01 · Papers ]
Papers
Dynamic Depth Decoding: Faster Speculative Decoding for LLMs(opens in a new tab)
3.16× faster
Large language models write text a word at a time, which is slow. Speculative decoding speeds this up by drafting several words ahead and checking them all at once. Dynamic Depth Decoding decides how far ahead to draft as it goes, making generation 3.16 times faster on average, with no loss of accuracy.
Read the full paper (opens in a new tab)Read the abstract
The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by 44%, giving DDD an average speedup of 3.16x.
Using Fine-tuning and Min Lookahead Beam Search to Improve Whisper(opens in a new tab)
Better low-resource speech
Whisper is a widely used speech-to-text model, but it struggles with languages that have little training data. We trained it on more data and improved the way it chooses the most likely words, to make its transcripts in those languages more accurate.
Read the full paper (opens in a new tab)Read the abstract
The performance of Whisper in low-resource languages is still far from perfect. In addition to a lack of training data on low-resource languages, we identify some limitations in the beam search algorithm used in Whisper. To address these issues, we fine-tune Whisper on additional data and propose an improved decoding algorithm.
[ 02 · Research areas ]
What we work on
Faster inference
Speculative decoding that makes large language models generate text faster, with no loss of accuracy.
Read the paper : Dynamic Depth Decoding: Faster Speculative Decoding for LLMs (opens in a new tab)Speech AI
Better transcription for low-resource languages and difficult audio, through fine-tuning and improved decoding.
Read the paper : Using Fine-tuning and Min Lookahead Beam Search to Improve Whisper (opens in a new tab)Explainable AI
AI whose answers trace back to their sources, for organisations making high-stakes decisions.
See it in our platform