1Data::Stream::Bulk(3) User Contributed Perl DocumentationData::Stream::Bulk(3)
2
3
4

NAME

6       Data::Stream::Bulk - N at a time iteration API
7

SYNOPSIS

9           # get a bulk stream from somewere
10           my $s = Data::Stream::Bulk::Foo->new( ... );
11
12           # can be used like this:
13           until ( $s->is_done ) {
14               foreach my $item ( $s->items ) {
15                   process($item);
16               }
17           }
18
19           # or like this:
20           while( my $block = $s->next ) {
21               foreach my $item ( @$block ) {
22                   process($item);
23               }
24           }
25

DESCRIPTION

27       This module tries to find middle ground between one at a time and all
28       at once processing of data sets.
29
30       The purpose of this module is to avoid the overhead of implementing an
31       iterative api when this isn't necessary, without breaking forward
32       compatibility in case that becomes necessary later on.
33
34       The API optimizes for when a data set typically fits in memory and is
35       returned as an array, but the consumer cannot assume that the data set
36       is bounded.
37
38       The API is destructive in order to minimize the chance that resultsets
39       are leaked due to improper usage.
40

API

42   Required Methods
43       The API requires two methods to be implemented:
44
45       is_done
46           Should return true if the stream is exhausted.
47
48           As long as this method returns a false value (not done) "next"
49           could potentially return another block.
50
51       next
52           Returns the next block.
53
54           Note that "next" is not guaranteed to return an array reference,
55           even if "is_done" returned false prior to calling it.
56
57   Convenience Methods
58       items
59           This method calls "next" and dereferences the result if there are
60           pending items.
61
62       all Force evaluation of the entire resultset.
63
64           Note that for large data sets this might cause swap thrashing of
65           various other undesired effects. Use with caution.
66
67       cat @streams
68           Concatenates this stream with @streams, returning a single stream.
69
70       list_cat @tail
71           Returns a possibly cleaned up list of streams.
72
73           Used by "cat".
74
75           Overridden by Data::Stream::Bulk::Array, Data::Stream::Bulk::Cat
76           and Data::Stream::Bulk::Nil to implement some simple short
77           circuiting.
78
79       filter $filter
80           Applies a per-block block filter to the stream.
81
82           Returns a possibly new stream with the filtering layered.
83
84           $filter is invoked once per block and should return an array
85           reference to the filtered block.
86
87       loaded
88           Should be overridden to return true if all the items are already
89           realized (e.g.  in the case of Data::Stream::Bulk::Array).
90
91           Returns false by default.
92
93           When true calling "all" is supposed to be safe (memory usage should
94           be in the same order of magnitude as stream's own usage).
95
96           This is typically useful when tranforming an array is easier than
97           transorming a stream (e.g. optional duplicate filtering).
98

CLASSES

100       Data::Stream::Bulk::Array
101           This class is not a stream at all, but just one block. When the
102           data set easily fits in memory this class can be used, while
103           retaining forward compatibility with larger data sets.
104
105       Data::Stream::Bulk::Callback
106           Callback driven iteration.
107
108       Data::Stream::Bulk::DBI
109           Bulk fetching of data from DBI statement handles.
110
111       Data::Stream::Bulk::DBIC
112           DBIx::Class::ResultSet iteration.
113
114       Data::Stream::Bulk::Nil
115           An empty result set.
116
117       Data::Stream::Bulk::Cat
118           A concatenation of several streams.
119
120       Data::Stream::Bulk::Filter
121           A filter wrapping a stream.
122

SEE ALSO

124       HOP::Stream, Iterator, Class::Iterator etc for one by one iteration
125
126       DBI, DBIx::Class::ResultSet
127
128       POE::Filter
129
130       Data::Page
131
132       Parallel::Iterator
133
134       <http://en.wikipedia.org/wiki/MapReduce>, LISP, and all that other kool
135       aid
136

TODO

138       Sorted streams
139           Add a hint for sorted streams (like "loaded" but as an attribute in
140           the base role).
141
142           Introduce a "merge" operation for merging of sorted streams.
143
144           Optimize "unique" to make use of sorting hints for constant space
145           uniquing.
146
147       More utility functions
148           To assist in proccessing and creating streams.
149
150       Coercion tables
151           Moose::Util::TypeConstraints
152

AUTHOR

154       Yuval Kogman <nothingmuch@woobling.org>
155

157       This software is copyright (c) 2010 by Yuval Kogman.
158
159       This is free software; you can redistribute it and/or modify it under
160       the same terms as the Perl 5 programming language system itself.
161
162
163
164perl v5.12.1                      2010-08-24             Data::Stream::Bulk(3)
Impressum