1MCE::Map(3) User Contributed Perl Documentation MCE::Map(3)
2
6 MCE::Map - Parallel map model similar to the native map function
7
9 This document describes MCE::Map version 1.838
10
12 ## Exports mce_map, mce_map_f, and mce_map_s
13 use MCE::Map;
14 ## Array or array_ref
16 my @a = mce_map { $_ * $_ } 1..10000;
17 my @b = mce_map { $_ * $_ } [ 1..10000 ];
18 ## File_path, glob_ref, or scalar_ref
20 my @c = mce_map_f { chomp; $_ } "/path/to/file";
21 my @d = mce_map_f { chomp; $_ } $file_handle;
22 my @e = mce_map_f { chomp; $_ } \$scalar;
23 ## Sequence of numbers (begin, end [, step, format])
25 my @f = mce_map_s { $_ * $_ } 1, 10000, 5;
26 my @g = mce_map_s { $_ * $_ } [ 1, 10000, 5 ];
27 my @h = mce_map_s { $_ * $_ } {
29 begin => 1, end => 10000, step => 5, format => undef
30 };
31
33 This module provides a parallel map implementation via Many-Core
34 Engine. MCE incurs a small overhead due to passing of data. A fast
35 code block will run faster natively. However, the overhead will likely
36 diminish as the complexity increases for the code.
37 my @m1 = map { $_ * $_ } 1..1000000; ## 0.127 secs
39 my @m2 = mce_map { $_ * $_ } 1..1000000; ## 0.304 secs
40 Chunking, enabled by default, greatly reduces the overhead behind the
42 scene. The time for mce_map below also includes the time for data
43 exchanges between the manager and worker processes. More
44 parallelization will be seen when the code incurs additional CPU time.
45 sub calc {
47 sqrt $_ * sqrt $_ / 1.3 * 1.5 / 3.2 * 1.07
48 }
49 my @m1 = map { calc } 1..1000000; ## 0.367 secs
51 my @m2 = mce_map { calc } 1..1000000; ## 0.365 secs
52 Even faster is mce_map_s; useful when input data is a range of numbers.
54 Workers generate sequences mathematically among themselves without any
55 interaction from the manager process. Two arguments are required for
56 mce_map_s (begin, end). Step defaults to 1 if begin is smaller than
57 end, otherwise -1.
58 my @m3 = mce_map_s { calc } 1, 1000000; ## 0.270 secs
60 Although this document is about MCE::Map, the MCE::Stream module can
62 write results immediately without waiting for all chunks to complete.
63 This is made possible by passing the reference to an array (in this
64 case @m4 and @m5).
65 use MCE::Stream;
67 sub calc {
69 sqrt $_ * sqrt $_ / 1.3 * 1.5 / 3.2 * 1.07
70 }
71 my @m4; mce_stream \@m4, sub { calc }, 1..1000000;
73 ## Completes in 0.272 secs. This is amazing considering the
75 ## overhead for passing data between the manager and workers.
76 my @m5; mce_stream_s \@m5, sub { calc }, 1, 1000000;
78 ## Completed in 0.176 secs. Like with mce_map_s, specifying a
80 ## sequence specification turns out to be faster due to lesser
81 ## overhead for the manager process.
82
84 The following list options which may be overridden when loading the
85 module.
86 use Sereal qw( encode_sereal decode_sereal );
88 use CBOR::XS qw( encode_cbor decode_cbor );
89 use JSON::XS qw( encode_json decode_json );
90 use MCE::Map
92 max_workers => 4, # Default 'auto'
93 chunk_size => 100, # Default 'auto'
94 tmp_dir => "/path/to/app/tmp", # $MCE::Signal::tmp_dir
95 freeze => \&encode_sereal, # \&Storable::freeze
96 thaw => \&decode_sereal # \&Storable::thaw
97 ;
98 From MCE 1.8 onwards, Sereal 3.015+ is loaded automatically if
100 available. Specify "Sereal =" 0> to use Storable instead.
101 use MCE::Map Sereal => 0;
103
105 MCE::Map->init ( options )
106 MCE::Map::init { options }
107 The init function accepts a hash of MCE options. The gather option,
108 if specified, is ignored due to being used internally by the module.
109 use MCE::Map;
111 MCE::Map::init {
113 chunk_size => 1, max_workers => 4,
114 user_begin => sub {
116 print "## ", MCE->wid, " started\n";
117 },
118 user_end => sub {
120 print "## ", MCE->wid, " completed\n";
121 }
122 };
123 my @a = mce_map { $_ * $_ } 1..100;
125 print "\n", "@a", "\n";
127 -- Output
129 ## 2 started
131 ## 1 started
132 ## 3 started
133 ## 4 started
134 ## 1 completed
135 ## 4 completed
136 ## 2 completed
137 ## 3 completed
138 1 4 9 16 25 36 49 64 81 100 121 144 169 196 225 256 289 324 361
140 400 441 484 529 576 625 676 729 784 841 900 961 1024 1089 1156
141 1225 1296 1369 1444 1521 1600 1681 1764 1849 1936 2025 2116 2209
142 2304 2401 2500 2601 2704 2809 2916 3025 3136 3249 3364 3481 3600
143 3721 3844 3969 4096 4225 4356 4489 4624 4761 4900 5041 5184 5329
144 5476 5625 5776 5929 6084 6241 6400 6561 6724 6889 7056 7225 7396
145 7569 7744 7921 8100 8281 8464 8649 8836 9025 9216 9409 9604 9801
146 10000
147
149 MCE::Map->run ( sub { code }, list )
150 mce_map { code } list
151 Input data may be defined using a list or an array reference. Unlike
152 MCE::Loop, Flow, and Step, specifying a hash reference as input data
153 isn't allowed.
154 my @a = mce_map { $_ * 2 } 1..1000;
156 my @b = mce_map { $_ * 2 } \@list;
157 my @z = mce_map { $_ * 2 } \%hash; # not supported
159 MCE::Map->run_file ( sub { code }, file )
161 mce_map_f { code } file
162 The fastest of these is the /path/to/file. Workers communicate the
163 next offset position among themselves with zero interaction by the
164 manager process.
165 my @c = mce_map_f { chomp; $_ . "\r\n" } "/path/to/file"; # faster
167 my @d = mce_map_f { chomp; $_ . "\r\n" } $file_handle;
168 my @e = mce_map_f { chomp; $_ . "\r\n" } \$scalar;
169 MCE::Map->run_seq ( sub { code }, $beg, $end [, $step, $fmt ] )
171 mce_map_s { code } $beg, $end [, $step, $fmt ]
172 Sequence may be defined as a list, an array reference, or a hash
173 reference. The functions require both begin and end values to run.
174 Step and format are optional. The format is passed to sprintf (% may
175 be omitted below).
176 my ($beg, $end, $step, $fmt) = (10, 20, 0.1, "%4.1f");
178 my @f = mce_map_s { $_ } $beg, $end, $step, $fmt;
180 my @g = mce_map_s { $_ } [ $beg, $end, $step, $fmt ];
181 my @h = mce_map_s { $_ } {
183 begin => $beg, end => $end,
184 step => $step, format => $fmt
185 };
186 MCE::Map->run ( sub { code }, iterator )
188 mce_map { code } iterator
189 An iterator reference may be specified for input_data. Iterators are
190 described under section "SYNTAX for INPUT_DATA" at MCE::Core.
191 my @a = mce_map { $_ * 2 } make_iterator(10, 30, 2);
193
195 MCE::Map->finish
196 MCE::Map::finish
197 Workers remain persistent as much as possible after running.
198 Shutdown occurs automatically when the script terminates. Call
199 finish when workers are no longer needed.
200 use MCE::Map;
202 MCE::Map::init {
204 chunk_size => 20, max_workers => 'auto'
205 };
206 my @a = mce_map { ... } 1..100;
208 MCE::Map::finish;
210
212 MCE, MCE::Core
213
215 Mario E. Roy, <marioeroy AT gmail DOT com>
216perl v5.28.1 2019-01-23 MCE::Map(3)