Collaborative Architecture Design and Performance Optimization for Cloud-Native Big Data Platforms

Abstract

This research addresses the critical challenges of resource inefficiency and architectural rigidity in traditional big data environments by proposing a novel collaborative architecture design tailored for cloud-native ecosystems. As organizations transition from monolithic on-premise clusters to dynamic cloud environments, the decoupling of compute and storage becomes a fundamental requirement for achieving elasticity and cost-efficiency. This study introduces a multi-layered collaborative framework that integrates container orchestration with distributed data processing engines, facilitating seamless resource sharing and automated scaling. The methodology focuses on the development of a latency-aware scheduling algorithm and a dynamic resource allocation policy that optimizes the interaction between microservices and data nodes. Through extensive empirical testing and performance modeling, the research evaluates the architecture's impact on system throughput, query latency, and resource utilization rates. The findings indicate that the proposed collaborative design significantly reduces operational overhead and mitigates the performance bottlenecks typically associated with data shuffling in cloud environments. Specifically, the implementation of intelligent caching layers and adaptive concurrency controls results in a measurable improvement in processing speeds for complex analytical workloads. This paper provides a comprehensive theoretical and practical foundation for building next-generation big data platforms that are inherently scalable, resilient, and optimized for high-concurrency cloud-native scenarios. The discussion further explores the trade-offs between consistency and availability within this collaborative framework, offering insights into the long-term sustainability of cloud-native data infrastructures.