MCE(3pm) - f29

1MCE(3)                User Contributed Perl Documentation               MCE(3)
2
3
4

NAME

6       MCE - Many-Core Engine for Perl providing parallel processing
7       capabilities
8

VERSION

10       This document describes MCE version 1.837
11
12       Many-Core Engine (MCE) for Perl helps enable a new level of performance
13       by maximizing all available cores.
14

DESCRIPTION

16       MCE spawns a pool of workers and therefore does not fork a new process
17       per each element of data. Instead, MCE follows a bank queuing model.
18       Imagine the line being the data and bank-tellers the parallel workers.
19       MCE enhances that model by adding the ability to chunk the next n
20       elements from the input stream to the next available worker.
21

SYNOPSIS

23       This is a simplistic use case of MCE running with 5 workers.
24
25        # Construction using the Core API
26
27        use MCE;
28
29        my $mce = MCE->new(
30           max_workers => 5,
31           user_func => sub {
32              my ($mce) = @_;
33              $mce->say("Hello from " . $mce->wid);
34           }
35        );
36
37        $mce->run;
38
39        # Construction using a MCE model
40
41        use MCE::Flow max_workers => 5;
42
43        mce_flow sub {
44           my ($mce) = @_;
45           MCE->say("Hello from " . MCE->wid);
46        };
47
48       The following is a demonstration for parsing a huge log file in
49       parallel.
50
51        use MCE::Loop;
52
53        MCE::Loop::init { max_workers => 8, use_slurpio => 1 };
54
55        my $pattern  = 'something';
56        my $hugefile = 'very_huge.file';
57
58        my @result = mce_loop_f {
59           my ($mce, $slurp_ref, $chunk_id) = @_;
60
61           # Quickly determine if a match is found.
62           # Process the slurped chunk only if true.
63
64           if ($$slurp_ref =~ /$pattern/m) {
65              my @matches;
66
67              # The following is fast on Unix, but performance degrades
68              # drastically on Windows beyond 4 workers.
69
70              open my $MEM_FH, '<', $slurp_ref;
71              binmode $MEM_FH, ':raw';
72              while (<$MEM_FH>) { push @matches, $_ if (/$pattern/); }
73              close   $MEM_FH;
74
75              # Therefore, use the following construction on Windows.
76
77              while ( $$slurp_ref =~ /([^\n]+\n)/mg ) {
78                 my $line = $1; # save $1 to not lose the value
79                 push @matches, $line if ($line =~ /$pattern/);
80              }
81
82              # Gather matched lines.
83
84              MCE->gather(@matches);
85           }
86
87        } $hugefile;
88
89        print join('', @result);
90
91       The next demonstration loops through a sequence of numbers with
92       MCE::Flow.
93
94        use MCE::Flow;
95
96        my $N = shift || 4_000_000;
97
98        sub compute_pi {
99           my ( $beg_seq, $end_seq ) = @_;
100           my ( $pi, $t ) = ( 0.0 );
101
102           foreach my $i ( $beg_seq .. $end_seq ) {
103              $t = ( $i + 0.5 ) / $N;
104              $pi += 4.0 / ( 1.0 + $t * $t );
105           }
106
107           MCE->gather( $pi );
108        }
109
110        # Compute bounds only, workers receive [ begin, end ] values
111
112        MCE::Flow::init(
113           chunk_size  => 200_000,
114           max_workers => 8,
115           bounds_only => 1
116        );
117
118        my @ret = mce_flow_s sub {
119           compute_pi( $_->[0], $_->[1] );
120        }, 0, $N - 1;
121
122        my $pi = 0.0;  $pi += $_ for @ret;
123
124        printf "pi = %0.13f\n", $pi / $N;  # 3.1415926535898
125

CORE MODULES

127       Three modules make up the core engine for MCE.
128
129       MCE::Core
130          Provides the Core API for Many-Core Engine. The various MCE options
131          are described here. It includes several demonstrations at the end of
132          the page.
133
134       MCE::Signal
135          Temporary directory creation, cleanup, and signal handling.
136
137       MCE::Util
138          Utility functions for Many-Core Engine.
139

MCE EXTRAS

141       There are 4 add-on modules for use with MCE.
142
143       MCE::Candy
144          Provides a collection of sugar methods and output iterators for
145          preserving output order.
146
147       MCE::Mutex
148          Provides a simple semaphore implementation supporting threads and
149          processes.  Two implementations are provided. One via pipes or
150          socket depending on the platform. The other via Fcntl.
151
152       MCE::Queue
153          Provides a hybrid queuing implementation for MCE supporting normal
154          queues and priority queues from a single module. MCE::Queue
155          exchanges data via the core engine to enable queuing to work for
156          both children (spawned from fork) and threads.
157
158       MCE::Relay
159          Enables workers to receive and pass on information orderly with zero
160          involvement by the manager process while running.
161

MCE MODELS

163       The models take Many-Core Engine to a new level for ease of use. Two
164       options (chunk_size and max_workers) are configured automatically as
165       well as spawning and shutdown.
166
167       MCE::Loop
168          Provides a MCE model for building parallel loops.
169
170       MCE::Flow
171          A parallel flow model for building creative applications. This makes
172          use of user_tasks in MCE. The author has full control when utilizing
173          this model.  MCE::Flow is similar to MCE::Loop, but allows for
174          multiple code blocks to run in parallel with a slight change to
175          syntax.
176
177       MCE::Grep
178          Provides a parallel grep implementation similar to the native grep
179          function.
180
181       MCE::Map
182          Provides a parallel map model similar to the native map function.
183
184       MCE::Step
185          Provides a parallel step implementation utilizing MCE::Queue between
186          user tasks. MCE::Step is a spin off from MCE::Flow with a touch of
187          MCE::Stream.  This model, introduced in 1.506, allows one to pass
188          data from one sub-task into the next transparently.
189
190       MCE::Stream
191          Provides an efficient parallel implementation for chaining multiple
192          maps and greps together through user_tasks and MCE::Queue. Like with
193          MCE::Flow, MCE::Stream can run multiple code blocks in parallel with
194          a slight change to syntax from MCE::Map and MCE::Grep.
195

MISCELLANEOUS

197       Miscellaneous additions included with the distribution.
198
199       MCE::Examples
200          Describes various demonstrations for MCE including a Monte Carlo
201          simulation.
202
203       MCE::Subs
204          Exports functions mapped directly to MCE methods; e.g. mce_wid. The
205          module allows 3 options; :manager, :worker, and :getter.
206

REQUIREMENTS

208       Perl 5.8.0 or later. PDL::IO::Storable is required in scripts running
209       PDL.
210

SOURCE AND FURTHER READING

212       The source, cookbook, and examples are hosted at GitHub.
213
214       ·  <https://github.com/marioroy/mce-perl>
215
216       ·  <https://github.com/marioroy/mce-cookbook>
217
218       ·  <https://github.com/marioroy/mce-examples>
219

AUTHOR

229       Mario E. Roy, <marioeroy AT gmail DOT com>
230

COPYRIGHT AND LICENSE

232       Copyright (C) 2012-2018 by Mario E. Roy
233
234       MCE is released under the same license as Perl.
235
236       See <http://dev.perl.org/licenses/> for more information.
237
238
239
240perl v5.28.0                      2018-08-25                            MCE(3)