Published on: 2026-10-20
Source: Novosibirsk State University –
An important disclaimer is at the bottom of this article.
The new development was presented by scientists from the Artificial Intelligence Center of Novosibirsk State University at the 8th International Forum on Artificial Intelligence Technologies and Data Analytics “Open Data,” which took place in the Altai Republic on October 5-6. The event brought together more than 300 participants from CIS and BRICS countries — representatives from over 80 universities, 30 companies, and associations. The forum organizers were the University Consortium of Big Data Researchers and Tomsk State University.
In the open track, where universities and industry partners present new research, pilot developments, and educational initiatives, the head of the Laboratory of Stream Data Analytics and Machine Learning at the Faculty of Mechanics and Mathematics, senior researcher at the NSU Center for Artificial Intelligence, gave a report titled “Kappa Framework: Dataset Management and Benchmarking AI Models for Trustworthy Artificial Intelligence Systems.” Evgeny Pavlovsky Following the presentation, representatives from NSU proposed to other universities to use “Kappa” for free — through the version hosted on NSU’s servers. There is also the possibility to install the framework within the university’s own network.
The “Kappa” framework (kappa.nsu.ru) — developed at the AI Center of NSU. The system manages the full lifecycle of training and control datasets for AI models: annotation, verification, versioning, publication, model benchmarking, and telemetry of their operation.
— The quality of an artificial intelligence model directly depends on the data it was trained on. Due to errors in the data and the lack of transparency about their origin, many models never reach industrial deployment: the customer cannot verify what the system was trained on and does not trust it. “Kappa” solves both problems: the framework supports the full lifecycle of datasets — creation, annotation, verification, versioning, and publication (in a closed loop, on request, or open), as well as the lifecycle of the AI models themselves, — explained Evgeniy Pavlovsky.
The system guarantees data quality, experiment reproducibility, and full traceability: it always records who made which annotation and when. Any discovered error can be corrected and fixed in a new version of the dataset, rather than repeating it in each subsequent training. A natural continuation of working with data is benchmarking: prepared datasets can be used to measure the quality of any models, both proprietary and public.
Existing data repositories — Figshare, Zenodo, Mendeley Data, Dataverse, and the Russian ERIS system — are primarily designed for storing and citing files. “Kappa,” however, is entirely tailored for working with machine learning datasets: annotation, verification, versioning, and benchmarking. Moreover, the framework is not only software but also a conceptual foundation: it supports a project prepared by a team for a national standard of humanitarian expertise on AI implementation in city management practices and implements this expertise as a separate stage of work.
“The quality of an artificial intelligence model is determined by the data we input. It is precisely due to errors in the data that many models do not reach industrial deployment, and the lack of transparency in data origin breeds mistrust. The situation becomes ‘Garbage in, garbage out.’ We guarantee the quality of datasets, reproducibility of experiments, and complete traceability — you can see the authorship of everyone involved in data preparation,” noted Evgeny Pavlovsky.
The trust mechanism in “Kappa” covers the entire model lifecycle — from data to deployment. Before training, dataset verification is performed, bias metric checks and cross-validation of annotations by multiple annotators; during training, the data version and parameters are fixed; after training, evaluation on benchmark datasets is carried out; and during deployment, a telemetry module collects usage statistics and monitors errors, signaling when deviations occur that the model needs retraining.
The new software platform has already been tested in real projects. The largest project on the framework is a collaboration with the State Public Scientific and Technical Library of the Siberian Branch of the Russian Academy of Sciences. The library provided about 400,000 records — digitized cards and their catalog descriptions, with about 100 employees participating in the markup. “Kappa” kept statistics for each annotator. The result was a trained model and a control benchmark on which other algorithms can be tested.
Among other applications are narration of scientific books, facade documentation, work with archival documents, and school datasets. The very first project on the framework was datasets that schoolchildren collected in the forest by photographing and describing mushrooms and birds. Currently, the team of the NSU Artificial Intelligence Center is working on a dataset for assessing students’ academic performance.
In the master’s program at the Faculty of Mechanics and Mathematics of NSU, there is already a mandatory practice of preserving student datasets: along with the code of term papers and theses, the data sets are also archived. According to Evgeny Pavlovsky, such a practice is hardly common in universities, although it has a direct effect: the scientific advisor accepts the result based on clear criteria, and the next student continues the work — first understanding the existing one, then developing it further. This system has already demonstrated its effectiveness.
NSU offered universities two options for using the framework. The first involves using the free version of “Kappa,” hosted on NSU servers. For this, it is enough to register at kappa.nsu.ru. If necessary, a cooperation agreement can be concluded. At the same time, NSU provides technical support and supplies methodological materials. The second option is to purchase a license to install the framework within the university’s own network: the data does not leave the university infrastructure. This solution is suitable for sensitive data — for example, medical data, access to which is regulated by the decision of the ethics committee. The license is provided on a paid basis. Its cost is 100 thousand rubles per year. In addition, the NSU Artificial Intelligence Center team is ready to discuss a pilot project with a specific university within 30 days: specialists will assess how well its datasets are prepared for reuse and help identify potential biases in the data.
— Frameworks are rarely monetized — no one pays for PyTorch, for example. We see the value of “Kappa” primarily in education and science, so universities can use our version for free. And for those who need to keep data within their own perimeter, we have made local installation available, — emphasized Evgeny Pavlovsky.
Please be advised; This information is raw content obtained directly from the source. It represents an accurate account of what the source asserts and does not necessarily reflect the position of MIL-OSI or its clients.