A Supervised Framework for Document Processing at Scale with Large Language Models in Credit-Risk Research

Abstract

In a time of exponentially growing unstructured digital content, automatic information extraction from huge corpora is an indispensable ability for the modern economy and research. Credit risk research is based on a wide scientific literature content, hard to be synthesized and compared because of its multiple layers of models and results. Recent breakthroughs in large language models mitigate this limitation for partially automated extraction, but the majority of tools available today suffer from low reproducibility, accuracy, and faithfulness of extracted information. This paper presents an approach of a supervised large language model to generate semi-structured data used for performing automatic document processing on credit risk research papers. This work challenges how well a supervised approach can extract structured information, such as model names, performance measures, validation types and reference lists from diverse sources and digital formats. The method uses schema based guidance with automatic text retrieval. Documents are processed independently and transformed into a structured schema that is compared with a ground truth to evaluate accuracy, completeness, and faithfulness. The key evaluation parameters are title, authors, year, DOI, journal, model identification, metrics and references. Results demonstrate that the proposed supervised strategy is capable of producing higher precision, completeness and reproducibility than using general large language model extractions. It demonstrates much lower hallucination and omission error and increased content coverage. The framework is able to parse a variety of academic papers, and provides robust results across required metadata. Outside of the scientific literature, the architecture is applicable to other types of unstructured documents to credit risk research, such as annual financial statements, audit reports or legal verdicts. These results validate that a supervised and tractable pipeline forges the basis of large scale evidence mapping and analytic automation, which enable knowledge integration and future predictive modeling in risk analytics and beyond.