Directory::Queue::Simple(3pm)

1Directory::Queue::SimplUes(e3r)Contributed Perl DocumentDaitrieocntory::Queue::Simple(3)
2
3
4

NAME

6       Directory::Queue::Simple - object oriented interface to a simple
7       directory based queue
8

SYNOPSIS

10         use Directory::Queue::Simple;
11
12         #
13         # sample producer
14         #
15
16         $dirq = Directory::Queue::Simple->new(path => "/tmp/test");
17         foreach $count (1 .. 100) {
18             $name = $dirq->add("element $count\n");
19             printf("# added element %d as %s\n", $count, $name);
20         }
21
22         #
23         # sample consumer (one pass only)
24         #
25
26         $dirq = Directory::Queue::Simple->new(path => "/tmp/test");
27         for ($name = $dirq->first(); $name; $name = $dirq->next()) {
28             next unless $dirq->lock($name);
29             printf("# reading element %s\n", $name);
30             $data = $dirq->get($name);
31             # one could use $dirq->unlock($name) to only browse the queue...
32             $dirq->remove($name);
33         }
34

DESCRIPTION

36       The goal of this module is to offer a "simple" (as opposed to "normal")
37       queue system using the underlying filesystem for storage, security and
38       to prevent race conditions via atomic operations.
39
40       It only allows binary strings to be stored but it is fast and small.
41
42       Please refer to Directory::Queue for general information about
43       directory queues.
44

CONSTRUCTOR

46       The new() method can be used to create a Directory::Queue::Simple
47       object that will later be used to interact with the queue. The
48       following attributes are supported:
49
50       path
51           the queue toplevel directory (mandatory)
52
53       rndhex
54           the "random" hexadecimal digit to use in element names (aka R) as a
55           number between 0 and 15 (default: randomly generated)
56
57       umask
58           the umask to use when creating files and directories (default: use
59           the running process' umask)
60
61       maxlock
62           default maximum time for a locked element (in seconds, default 600)
63           as used by the purge() method
64
65       maxtemp
66           default maximum time for a temporary element (in seconds, default
67           300) as used by the purge() method
68
69       granularity
70           the time granularity for intermediate directories, see "DIRECTORY
71           STRUCTURE" (default: 60)
72

METHODS

74       The following methods are available:
75
76       new()
77           return a new Directory::Queue::Simple object (class method)
78
79       copy()
80           return a copy of the object; this can be useful to have independent
81           iterators on the same queue
82
83       path()
84           return the queue toplevel path
85
86       id()
87           return a unique identifier for the queue
88
89       count()
90           return the number of elements in the queue
91
92       first()
93           return the first element in the queue, resetting the iterator;
94           return an empty string if the queue is empty
95
96       next()
97           return the next element in the queue, incrementing the iterator;
98           return an empty string if there is no next element
99
100       add(DATA)
101           add the given data (a binary string) to the queue and return the
102           corresponding element name
103
104       add_ref(REF)
105           add the given data (a reference to a binary string) to the queue
106           and return the corresponding element name, this can avoid string
107           copies with large strings
108
109       add_path(PATH)
110           add the given file (identified by its path) to the queue and return
111           the corresponding element name, the file must be on the same
112           filesystem and will be moved to the queue
113
114       lock(ELEMENT[, PERMISSIVE])
115           attempt to lock the given element and return true on success; if
116           the PERMISSIVE option is true (which is the default), it is not a
117           fatal error if the element cannot be locked and false is returned
118
119       unlock(ELEMENT[, PERMISSIVE])
120           attempt to unlock the given element and return true on success; if
121           the PERMISSIVE option is true (which is not the default), it is not
122           a fatal error if the element cannot be unlocked and false is
123           returned
124
125       touch(ELEMENT)
126           update the access and modification times on the element's file to
127           indicate that it is still being used; this is useful for elements
128           that are locked for long periods of time (see the purge() method)
129
130       remove(ELEMENT)
131           remove the given element (which must be locked) from the queue
132
133       get(ELEMENT)
134           get the data from the given element (which must be locked) and
135           return a binary string
136
137       get_ref(ELEMENT)
138           get the data from the given element (which must be locked) and
139           return a reference to a binary string, this can avoid string copies
140           with large strings
141
142       get_path(ELEMENT)
143           get the file path of the given element (which must be locked), this
144           file can be read but not removed, you must use the remove() method
145           for this
146
147       purge([OPTIONS])
148           purge the queue by removing unused intermediate directories,
149           removing too old temporary elements and unlocking too old locked
150           elements (aka staled locks); note: this can take a long time on
151           queues with many elements; OPTIONS can be:
152
153           maxtemp
154               maximum time for a temporary element (in seconds); if set to 0,
155               temporary elements will not be removed
156
157           maxlock
158               maximum time for a locked element (in seconds); if set to 0,
159               locked elements will not be unlocked
160

DIRECTORY STRUCTURE

162       The toplevel directory contains intermediate directories that contain
163       the stored elements, each of them in a file.
164
165       The names of the intermediate directories are time based: the element
166       insertion time is used to create a 8-digits long hexadecimal number.
167       The granularity (see the new() method) is used to limit the number of
168       new directories. For instance, with a granularity of 60 (the default),
169       new directories will be created at most once per minute.
170
171       Since there is usually a filesystem limit in the number of directories
172       a directory can hold, there is a trade-off to be made. If you want to
173       support many added elements per second, you should use a low
174       granularity to keep small directories. However, in this case, you will
175       create many directories and this will limit the total number of
176       elements you can store.
177
178       The elements themselves are stored in files (one per element) with a
179       14-digits long hexadecimal name SSSSSSSSMMMMMR where:
180
181       SSSSSSSS
182           represents the number of seconds since the Epoch
183
184       MMMMM
185           represents the microsecond part of the time since the Epoch
186
187       R   is a random hexadecimal digit used to reduce name collisions
188
189       A temporary element (being added to the queue) will have a ".tmp"
190       suffix.
191
192       A locked element will have a hard link with the same name and the
193       ".lck" suffix.
194

AUTHOR

199       Lionel Cons <http://cern.ch/lionel.cons>
200
201       Copyright (C) CERN 2010-2022
202
203
204
205perl v5.36.0                      2022-07-22       Directory::Queue::Simple(3)