A multi-layer big data platform for intelligent monitoring and diagnostics of agrocoenosis parameters based on hybrid machine learning methods

A multi-tier big data platform for intelligent monitoring and diagnostics of agrocenoses parameters has been developed using hybrid machine learning methods. The relevance of this platform is determined by the gap between the growing volumes of sensor, satellite, and aerial imagery data and the performance of single-machine instruments. It is based on a fourtier architecture (device – edge node – distributed core – application circuit) based on Apache Kafka 3.7, Apache Spark 3.5, and TimescaleDB 2.14. The database was collected over 182 days on 2,850 hectares in the Republic of Tatarstan (1,240 sensor nodes, 28 edge gateways); the volume of raw data is 4.2 TB, 12.8 million records. Six models for yield forecasting, anomaly detection, and crop diagnostics were compared. The best result was achieved by CatBoost with five-fold cross-validation: R² = 0.89, RMSE = 2.4 c/ha, MAE = 1.7 c/ha for spring wheat; in the diagnostic task of 14 parameters, Accuracy = 0.898 and AUC-ROC = 0.943 were achieved at a latency of 75 s. Scaling the Spark cluster from 1 to 8 nodes reduced the processing time from 47.3 to 7.1 min (speedup 6.67; efficiency 83.4%). The author’s integral index of the IDC takes into account the accuracy, completeness, and timeliness of the response; its final value was 0.82, which is 0.04–0.17 higher than the values of the review papers.

1. Kharchenko K.V. State support for digitalization of the agricultural sector: current situation and outlook for the future. Food Policy and Security. 2024; 11(3): 541–552 (in Russian). https://doi.org/10.18334/ppib.11.3.121575

2. Vostrukhin A.V., Mastepanenko M.A., Vakhtina E.A. Power-saving asynchronous interface for wireless sensors. Tomsk State University Journal of Control and Computer Science. 2023; 63: 92–102 (in Russian). EDN MCGNMX

3. Monteiro A., Santos S., Gonçalves P. Precision Agriculture for Crop and Livestock Farming—Brief Review. Animals. 2021; 11(8): 2345. https://doi.org/10.3390/ani11082345

4. Morchid A., El Alami R., Raezah A.A., Sabbar Y. Applications of Internet of Things (IoT) and sensors technology to increase food security and agricultural Sustainability: Benefits and challenges. Ain Shams Engineering Journal. 2024; 15(3): 102509. https://doi.org/10.1016/j.asej.2023.102509

5. Vostrukhin A.V., Mastepanenko M.A., Vorotnikov I.N., Vakhtina E.A. Algorithm of software development for a wireless system monitoring the temperature of agricultural premises. Agricultural Engineering (Moscow). 2025; 27(1): 75–81 (in Russian). https://doi.org/10.26897/2687-1149-2025-1-75-81

6. Debauche O., Mahmoudi S., Manneback P., Lebeau F. Cloud and distributed architectures for data management in agriculture 4.0: Review and future trends. Journal of King Saud University - Computer and Information Sciences. 2022; 34(9): 7494–7514. https://doi.org/10.1016/j.jksuci.2021.09.015

7. Li M., Yan Y. Comparative Analysis of Machine-Learning Models for Soil Moisture Estimation Using High-Resolution Remote-Sensing Data. Land. 2024; 13(8): 1331. https://doi.org/10.3390/land13081331

8. Dhillon M.S. et al. Integrating random forest and crop modeling improves the crop yield prediction of winter wheat and oil seed rape. Frontiers in Remote Sensing. 2022; 3: 1010978. https://doi.org/10.3389/frsen.2022.1010978

9. Araújo S.O., Peres R.S., Barata J., Lidon F., Ramalho J.C. Characterising the Agriculture 4.0 Landscape—Emerging Trends, Challenges and Opportunities. Agronomy. 2021; 11(4): 667. https://doi.org/10.3390/agronomy11040667

10. Galkin A.I. Application of big data and neural network technologies for optimization of agricultural production. Agrarian science. 2025; (2): 38–40 (in Russian). EDN UEWPMT

11. Dimitrov V.P., Borisova L.V., Nurutdinova I.N. Method for Determining the Initial Values of the Adjustable Parameters of the Combine Harvester Cutting Unit. Engineering Technologies and Systems. 2022; 32(4): 552–566 (in Russian). https://doi.org/10.15507/2658-4123.032.202204.552-566

12. Kostomakhin M.N., Pestryakov E.V. Software Package for Remote Diagnostics of Agricultural Machinery Condition. Agricultural Machinery and Technologies. 2022; 16(4): 19–25 (in Russian). https://doi.org/10.22314/2073-7599-2022-16-4-19-25

13. Kuznet sov P.N., Kotelnikov D.Yu., Voronin D.Yu. Technology of automated monitoring of the vineyard condition. Agrarian science. 2023; (3): 109–116 (in Russian). https://doi.org/10.32634/0869-8155-2023-368-3-109-116

14. Rogachev A.F., Melikhova E.V. Multi-class recognition of aerial images of agricultural fields. Proceedings of Nizhnevolzskiy Agrouniversity Complex: Science and Higher Vocational Education. 2020; (3): 142–152 (in Russian). https://doi.org/10.32786/2071-9485-2020-03-14

15. Pestryakov E.V. Software for diagnosing and predicting the technical condition of agricultural machines. Machinery and Equipment for Rural Area. 2021; (12): 37–41 (in Russian). EDN NUTMPU

16. Pavlova A.I. Agro-Ecological Classification of Agricultural Land using Machine Learning, GIS and Remote Sensing Data. Siberian Journal of Life Sciences and Agriculture. 2023; 15(6): 72–88 (in Russian). https://doi.org/10.12731/2658-6649-2023-15-6-960

17. Vasilyev S.A., Limonov S.Ye., Mishin S.A. Intelligent Field Sensor Station for Monitoring Agrophysical Parameters and Phenotyping in Precision Agriculture System. Agricultural Machinery and Technologies. 2024; 18(4): 79–85 (in Russian). https://doi.org/10.22314/2073-7599-2024-18-4-79-85

18. Stepnykh N.V., Zargaryan A.M. Improving the competitiveness of crop production due to the differentiated application of and control of the consumption of resources. Agrarian Bulletin of the Urals. 2014; (7): 27–29 (in Russian). EDN SWEGE