Directory::Queue(3pm)

1Directory::Queue(3)   User Contributed Perl Documentation  Directory::Queue(3)
2
3
4

NAME

6       Directory::Queue - object oriented interface to a directory based queue
7

SYNOPSIS

9         use Directory::Queue;
10
11         #
12         # sample producer
13         #
14
15         $dirq = Directory::Queue->new(path => "/tmp/test");
16         foreach $count (1 .. 100) {
17             $name = $dirq->add(... some data ...);
18             printf("# added element %d as %s\n", $count, $name);
19         }
20
21         #
22         # sample consumer (one pass only)
23         #
24
25         $dirq = Directory::Queue->new(path => "/tmp/test");
26         for ($name = $dirq->first(); $name; $name = $dirq->next()) {
27             next unless $dirq->lock($name);
28             printf("# reading element %s\n", $name);
29             $data = $dirq->get($name);
30             # one could use $dirq->unlock($name) to only browse the queue...
31             $dirq->remove($name);
32         }
33

DESCRIPTION

35       The goal of this module is to offer a queue system using the underlying
36       filesystem for storage, security and to prevent race conditions via
37       atomic operations. It focuses on simplicity, robustness and
38       scalability.
39
40       This module allows multiple concurrent readers and writers to interact
41       with the same queue. A Python implementation of the same algorithm is
42       available at <https://github.com/cern-mig/python-dirq>, a Java
43       implementation at <https://github.com/cern-mig/java-dirq> and a C
44       implementation at <https://github.com/cern-mig/c-dirq> so readers and
45       writers can be written in different programming languages.
46
47       There is no knowledge of priority within a queue. If multiple
48       priorities are needed, multiple queues should be used.
49

TERMINOLOGY

51       An element is something that contains one or more pieces of data. With
52       Directory::Queue::Simple queues, an element can only contain one binary
53       string. With Directory::Queue::Normal queues, more complex data schemas
54       can be used.
55
56       A queue is a "best effort" FIFO (First In - First Out) collection of
57       elements.
58
59       It is very hard to guarantee pure FIFO behavior with multiple writers
60       using the same queue. Consider for instance:
61
62       •   Writer1: calls the add() method
63
64       •   Writer2: calls the add() method
65
66       •   Writer2: the add() method returns
67
68       •   Writer1: the add() method returns
69
70       Who should be first in the queue, Writer1 or Writer2?
71
72       For simplicity, this implementation provides only "best effort" FIFO,
73       i.e. there is a very high probability that elements are processed in
74       FIFO order but this is not guaranteed. This is achieved by using a
75       high-resolution timer and having elements sorted by the time their
76       final directory gets created.
77

QUEUE TYPES

79       Different queue types are supported. More detailed information can be
80       found in the modules implementing these types:
81
82       •   Directory::Queue::Normal
83
84       •   Directory::Queue::Simple
85
86       •   Directory::Queue::Null
87
88       Compared to Directory::Queue::Normal, Directory::Queue::Simple:
89
90       •   is simpler
91
92       •   is faster
93
94       •   uses less space on disk
95
96       •   can be given existing files to store
97
98       •   does not support schemas
99
100       •   can only store and retrieve binary strings
101
102       •   is not compatible (at filesystem level) with
103           Directory::Queue::Normal
104
105       Directory::Queue::Null is special: it is a kind of black hole with the
106       same API as the other directory queues.
107

LOCKING

109       Adding an element is not a problem because the add() method is atomic.
110
111       In order to support multiple reader processes interacting with the same
112       queue, advisory locking is used. Processes should first lock an element
113       before working with it. In fact, the get() and remove() methods report
114       a fatal error if they are called on unlocked elements.
115
116       If the process that created the lock dies without unlocking the
117       element, we end up with a staled lock. The purge() method can be used
118       to remove these staled locks.
119
120       An element can basically be in only one of two states: locked or
121       unlocked.
122
123       A newly created element is unlocked as a writer usually does not need
124       to do anything more with it.
125
126       Iterators return all the elements, regardless of their states.
127
128       There is no method to get an element state as this information is
129       usually useless since it may change at any time. Instead, programs
130       should directly try to lock elements to make sure they are indeed
131       locked.
132

CONSTRUCTOR

134       The new() method of this module can be used to create a
135       Directory::Queue object that will later be used to interact with the
136       queue. It can have a "type" attribute specifying the queue type to use.
137       If not specified, the type defaults to "Simple".
138
139       This method is however only a wrapper around the constructor of the
140       underlying module implementing the functionality. So:
141
142         $dirq = Directory::Queue->new(type => Foo, ... options ...);
143
144       is identical to:
145
146         $dirq = Directory::Queue::Foo->new(... options ...);
147

INHERITANCE

149       Regardless of how the directory queue object is created, it inherits
150       from the "Directory::Queue" class. You can therefore test if an object
151       is a directory queue (of any kind) by using:
152
153         if ($object->isa("Directory::Queue")) ...
154

BASE METHODS

156       Here are the methods available in the base class and inherited by all
157       directory queue implementations:
158
159       new(PATH)
160           return a new object (class method)
161
162       copy()
163           return a copy of the object
164
165       path()
166           return the queue toplevel path
167
168       id()
169           return a unique identifier for the queue
170
171       first()
172           return the first element in the queue, resetting the iterator;
173           return an empty string if the queue is empty
174
175       next()
176           return the next element in the queue, incrementing the iterator;
177           return an empty string if there is no next element
178

SECURITY

180       There are no specific security mechanisms in this module.
181
182       The elements are stored as plain files and directories. The filesystem
183       security features (owner, group, permissions, ACLs...) should be used
184       to adequately protect the data.
185
186       By default, the process' umask is respected. See the class constructor
187       documentation if you want an other behavior.
188
189       If multiple readers and writers with different uids are expected, the
190       easiest solution is to have all the files and directories inside the
191       toplevel directory world-writable (i.e. umask=0). Then, the permissions
192       of the toplevel directory itself (e.g. group-writable) are enough to
193       control who can access the queue.
194

AUTHOR

200       Lionel Cons <http://cern.ch/lionel.cons>
201
202       Copyright (C) CERN 2010-2022
203
204
205
206perl v5.36.0                      2023-01-20               Directory::Queue(3)