MCE(3pm) - f35

1MCE(3)                User Contributed Perl Documentation               MCE(3)
2
3
4

NAME

6       MCE - Many-Core Engine for Perl providing parallel processing
7       capabilities
8

VERSION

10       This document describes MCE version 1.874
11
12       Many-Core Engine (MCE) for Perl helps enable a new level of performance
13       by maximizing all available cores.
14

DESCRIPTION

16       MCE spawns a pool of workers and therefore does not fork a new process
17       per each element of data. Instead, MCE follows a bank queuing model.
18       Imagine the line being the data and bank-tellers the parallel workers.
19       MCE enhances that model by adding the ability to chunk the next n
20       elements from the input stream to the next available worker.
21

SYNOPSIS

23       This is a simplistic use case of MCE running with 5 workers.
24
25        # Construction using the Core API
26
27        use MCE;
28
29        my $mce = MCE->new(
30           max_workers => 5,
31           user_func => sub {
32              my ($mce) = @_;
33              $mce->say("Hello from " . $mce->wid);
34           }
35        );
36
37        $mce->run;
38
39        # Construction using a MCE model
40
41        use MCE::Flow max_workers => 5;
42
43        mce_flow sub {
44           my ($mce) = @_;
45           MCE->say("Hello from " . MCE->wid);
46        };
47
48       The following is a demonstration for parsing a huge log file in
49       parallel.
50
51        use MCE::Loop;
52
53        MCE::Loop->init( max_workers => 8, use_slurpio => 1 );
54
55        my $pattern  = 'something';
56        my $hugefile = 'very_huge.file';
57
58        my @result = mce_loop_f {
59           my ($mce, $slurp_ref, $chunk_id) = @_;
60
61           # Quickly determine if a match is found.
62           # Process the slurped chunk only if true.
63
64           if ($$slurp_ref =~ /$pattern/m) {
65              my @matches;
66
67              # The following is fast on Unix, but performance degrades
68              # drastically on Windows beyond 4 workers.
69
70              open my $MEM_FH, '<', $slurp_ref;
71              binmode $MEM_FH, ':raw';
72              while (<$MEM_FH>) { push @matches, $_ if (/$pattern/); }
73              close   $MEM_FH;
74
75              # Therefore, use the following construction on Windows.
76
77              while ( $$slurp_ref =~ /([^\n]+\n)/mg ) {
78                 my $line = $1; # save $1 to not lose the value
79                 push @matches, $line if ($line =~ /$pattern/);
80              }
81
82              # Gather matched lines.
83
84              MCE->gather(@matches);
85           }
86
87        } $hugefile;
88
89        print join('', @result);
90
91       The next demonstration loops through a sequence of numbers with
92       MCE::Flow.
93
94        use MCE::Flow;
95
96        my $N = shift || 4_000_000;
97
98        sub compute_pi {
99           my ( $beg_seq, $end_seq ) = @_;
100           my ( $pi, $t ) = ( 0.0 );
101
102           foreach my $i ( $beg_seq .. $end_seq ) {
103              $t = ( $i + 0.5 ) / $N;
104              $pi += 4.0 / ( 1.0 + $t * $t );
105           }
106
107           MCE->gather( $pi );
108        }
109
110        # Compute bounds only, workers receive [ begin, end ] values
111
112        MCE::Flow->init(
113           chunk_size  => 200_000,
114           max_workers => 8,
115           bounds_only => 1
116        );
117
118        my @ret = mce_flow_s sub {
119           compute_pi( $_->[0], $_->[1] );
120        }, 0, $N - 1;
121
122        my $pi = 0.0;  $pi += $_ for @ret;
123
124        printf "pi = %0.13f\n", $pi / $N;  # 3.1415926535898
125

CORE MODULES

127       Four modules make up the core engine for MCE.
128
129       MCE::Core
130          This is the POD documentation describing the core Many-Core Engine
131          (MCE) API.  Go here for help with the various MCE options. See also,
132          MCE::Examples for additional demonstrations.
133
134       MCE::Mutex
135          Provides a simple semaphore implementation supporting threads and
136          processes.  Two implementations are provided; one via pipes or
137          socket depending on the platform and the other using Fcntl.
138
139       MCE::Signal
140          Provides signal handling, temporary directory creation, and cleanup
141          for MCE.
142
143       MCE::Util
144          Provides utility functions for MCE.
145

MCE EXTRAS

147       There are 5 add-on modules for use with MCE.
148
149       MCE::Candy
150          Provides a collection of sugar methods and output iterators for
151          preserving output order.
152
153       MCE::Channel
154          Introduced in MCE 1.839, provides queue-like and two-way
155          communication capability. Three implementations "Simple", "Mutex",
156          and "Threads" are provided. "Simple" does not involve locking
157          whereas "Mutex" and "Threads" do locking transparently using
158          "MCE::Mutex" and "threads" respectively.
159
160       MCE::Child
161          Also introduced in MCE 1.839, provides a threads-like
162          parallelization module that is compatible with Perl 5.8. It is a
163          fork of MCE::Hobo. The difference is using a common "MCE::Channel"
164          object when yielding and joining.
165
166       MCE::Queue
167          Provides a hybrid queuing implementation for MCE supporting normal
168          queues and priority queues from a single module. MCE::Queue
169          exchanges data via the core engine to enable queuing to work for
170          both children (spawned from fork) and threads.
171
172       MCE::Relay
173          Provides workers the ability to receive and pass information orderly
174          with zero involvement by the manager process. This module is loaded
175          automatically by MCE when specifying the "init_relay" MCE option.
176

MCE MODELS

178       The MCE models are sugar syntax on top of the MCE::Core API. Two MCE
179       options (chunk_size and max_workers) are configured automatically.
180       Moreover, spawning workers and later shutdown occur transparently
181       behind the scene.
182
183       Choosing a MCE Model largely depends on the application. It all boils
184       down to how much automation you need MCE to handle transparently. Or if
185       you prefer, constructing the MCE object and running using the core MCE
186       API is fine too.
187
188       MCE::Grep
189          Provides a parallel grep implementation similar to the native grep
190          function.
191
192       MCE::Map
193          Provides a parallel map implementation similar to the native map
194          function.
195
196       MCE::Loop
197          Provides a parallel for loop implementation.
198
199       MCE::Flow
200          Like "MCE::Loop", but with support for multiple pools of workers.
201          The pool of workers are configured transparently via the MCE
202          "user_tasks" option.
203
204       MCE::Step
205          Like "MCE::Flow", but adds a "MCE::Queue" object between each pool
206          of workers. This model, introduced in 1.506, allows one to pass data
207          forward (left to right) from one sub-task into another with little
208          effort.
209
210       MCE::Stream
211          This provides an efficient parallel implementation for chaining
212          multiple maps and greps transparently. Like "MCE::Flow" and
213          "MCE::Step", it too supports multiple pools of workers. The
214          distinction is that "MCE::Stream" passes data from right to left and
215          done for you transparently.
216

MISCELLANEOUS

218       Miscellaneous additions included with the distribution.
219
220       MCE::Examples
221          Describes various demonstrations for MCE including a Monte Carlo
222          simulation.
223
224       MCE::Subs
225          Exports functions mapped directly to MCE methods; e.g. mce_wid. The
226          module allows 3 options; :manager, :worker, and :getter.
227

REQUIREMENTS

229       Perl 5.8.0 or later. PDL::IO::Storable is required in scripts running
230       PDL.
231

SOURCE AND FURTHER READING

233       The source, cookbook, and examples are hosted at GitHub.
234
235       •  <https://github.com/marioroy/mce-perl>
236
237       •  <https://github.com/marioroy/mce-cookbook>
238
239       •  <https://github.com/marioroy/mce-examples>
240

AUTHOR

253       Mario E. Roy, <marioeroy AT gmail DOT com>
254

COPYRIGHT AND LICENSE

256       Copyright (C) 2012-2020 by Mario E. Roy
257
258       MCE is released under the same license as Perl.
259
260       See <http://dev.perl.org/licenses/> for more information.
261
262
263
264perl v5.34.0                      2021-07-22                            MCE(3)