Despite the popularity of large language models (LLMs) used to quickly summarise large documents, a recent study by the Australian Securities and Investments Commission (ASIC) has revealed significant shortcomings of such systems. The study found that the summaries produced by the Llama2-70B model were much worse than those produced by humans.

As part of this research, ASIC, together with Amazon Web Services, analysed the ability of the LLM to summarise public submissions to Parliament. The assessment was based on five criteria, including consistency, focus on key aspects and references to ASIC. Human summarisers received a score of 12.2, while AI scored only 7.

The main problems with AI were its inability to understand complex contexts, analyse nuances, and avoid factual errors. Such results can even increase the amount of work due to the need for fact-checking.

However, the study has limitations - only a week was allocated to optimise the models. ASIC notes that LLM performance may improve in the future as the technology continues to evolve.