Data-Intensive Distributed Systems LaboratoryProject -- Optimizing Resource Management and Dynamic Adaptation in Data Science Ecosystems (André Bauer)
As scientific computing and data-driven applications continue to grow in scale in complexity, managing computational resources efficiently while ensuring performance, scalability, and sustainability has become a major challenge. Cloud-based and containerized environments must dynamically adapt to changing workloads and optimize resource usage in real time. Additionally, AI models used in these systems must continuously adapt to new data to remain effective in evolving conditions. This project aims to develop new approaches that enhance both the efficiency and adaptability of data science ecosystems, ensuring that applications can scale seamlessly and operate sustainably in dynamic environments.
Students participating in this project will gain hands-on experience in designing and implementing optimization strategies for cloud-based and containerized environments. They will investigate challenges in resource management and workload distribution, learning how performance bottlenecks and inefficiencies arise in large-scale data science applications. Through experiments on real-world and simulated datasets, students will develop and evaluate strategies for dynamic scaling, intelligent scheduling, and adaptive system behavior. Beyond resource optimization, students will explore methods for the continuous adaptation of AI models to new data. They will design and implement techniques that allow models to update dynamically in response to changing workloads and evolving input distributions, ensuring that predictions and decisions remain accurate over time. This will involve investigating online learning techniques, reinforcement learning, and self-adaptive optimization frameworks.
By working with cloud-based infrastructures and container orchestration frameworks, students will understand how modern computing platforms manage resources and will explore advanced techniques for optimizing their operation. Additionally, they will apply machine learning and optimization methods to improve decision-making in both resource allocation and AI model adaptation. The project will also introduce students to performance benchmarking, allowing them to assess the impact of different optimization techniques on system efficiency and scalability. In a nutshell, students will develop a deep understanding of cloud computing architectures, performance engineering, and intelligent resource management while building practical skills in system analysis, AI-driven adaptation, optimization, and experimental evaluation.