Leveraging run time knowledge about event rates to improve memory utilization in wide area data stream filtering
Proceedings 11th IEEE International Symposium on High Performance Distributed Computing - Trang 171-178
Tóm tắt
The dQUOB system conceptualization of data streams as database and its SQL interface to data streams is an intuitive way for users to think about their data needs in a large scale application containing hundreds if not thousands of data streams. Experience with dQUOB has shown the need for more aggressive memory management to achieve the scalability we desire. This paper addresses the problem with a two-fold solution. The first one is replacement of the existing first-come first-served scheduling algorithm with an earliest job first algorithm which we demonstrate to yield better average service time. The second one is an introspection algorithm that sets and adapts the sizes of join windows in response to the knowledge acquired at runtime about event rates. In addition to the potential for significant improvements in memory utilization, the algorithm presented here also provides a means by which the user can reason about join window sizes. Wide area measurements demonstrate the adaptive capability required by the introspection technique.
Từ khóa
#Scalability #Scheduling algorithm #Relational databases #Filtering #Computer science #Large-scale systems #Application software #Memory management #Runtime #Wide area measurementsTài liệu tham khảo
10.1145/378993.379239
liu, 1999, Continual queries for internet scale event-driven information delivery, IEEE Transactions on Knowledge and Data Engineering
10.1109/ICDE.2002.994774
10.1109/71.238299
plale, 1999, Realizing distributed computational laboratories, Int J Parallel Distrib Syst Networks, 2, 180
plale, 2000, dQUOB: Managing large data flows using dynamic embedded queries, Proc 9th IEEE Intl High Performance Distributed Computing (HPDC)
ribler, 1999, Autopilot: Adaptive control of distributed applications, High Performance Distributed Computing (HPDC)
smallen, 2001, Applying scheduling and tuning to online parallel tomography, Proc ACM/IEEE Supercomputing 2001, 10.1145/582034.582046
10.1145/42186.42323
terry, 1992, Continuous queries over appendonly databases, International Conference on Management of Data (SIGMOD), 10.1145/141484.130333
10.1145/603867.603884
10.1145/342009.335420
10.1109/HPDC.2000.868641
10.1006/jnca.2000.0110
10.1109/2.809249
10.1142/S0129626499000190
avery, 2001, GriPhyn Grid Physics Network
10.1109/SIMSYM.1995.393571
kilpatrick, 1990, Using languages for describing capture, analysis, and display of performance information for parallel and distributed applications, IEEE International Conference on Computer Languages