- Added datastructure retrieval from child processes. (Ken Clarke)
- Using CORE::exit() instead of exit() for mod_perl compatibility.
The new version is available in the project page and (sooner or later) from CPAN.
I did some research about the module publicity and found the following things:
- The most interesting story is that Nathan Patwardhan ported this module to ruby.
- It is mentioned in the perfaq8 man page.
- An introduction article about it in www.perlmonks.org from 2003.
- An article and discussion: Use Parallel Processing For Faster Perl Scripts.
- A blog post about Parallelization in PERL by Michael S. Williamson.
- The following RPM-based Linux distros have packages: Fedora, AltLinux, Redhat EL, Mandrake, Mandriva, PCLinuxOS, PLD.
- The following Linux distros also have packages: Ubuntu, Debian, Gentoo.
- Other unix packages: OpenBSD, FreeBSD.
- Someone actually had a talk about it in Madison, WI.
- There is a module which plugs in P::FM to App:CLI. I don't exactly know what it is, but seems cool. :)
- Still in CPAN, there is a project called Archer, which seems to support it, too.
- Other modules popped up which uses P::FM or mention it in their doc: Parallel::Prefork and Parallel::Forker, Parallel::ForkControl, Proc::Queue, Parallel::Loops, MyCPAN::Indexer::Dispatcher::Parallel, Parallel::Simple (the author even wrote a review of the module, scroll down to the SEE ALSO section).
- A hell lot of other discussion threads in www.perlmonks.org
Enjoy!
Hi Balázs,
ReplyDeleteI was wondering if you could answer a question regarding the perl module :: Parallel::ForkManager
I know ForkManager, lets you specify the max number of processes, but I was wondering how I can specify the min number of processes running?
For example, let’s say I have a total of 200 calculations I need to complete, how can I specify I want a MAX = 30 processes running and a minimum of 20 processes at one time?
(I want to be able to add additional calculations whenever the minimum number of calculations falls below 20).
Any ideas would be greatly appreciated. Also, thanks for contributing to the open source community.
use Parallel::ForkManager;
use LWP::Simple;
my $pm=new Parallel::ForkManager(30);
for my $link (@ARGV)
{
$pm->start and next;
my ($fn)= $link =~ /^.*\/(.*?)$/;
if (!$fn)
{
warn "Cannot determine filename from $fn\n";
}
else
{
$0.=" ".$fn;
print "Getting $fn from $link\n";
my $rc=getstore($link,$fn);
print "$link downloaded. response code: $rc\n";
};
$pm->finish;
};
Hi Alex,
ReplyDeleteYou cannot set minimum here right now. It assume that mimum=maximum, so if the number of processes drops below the maximum, it starts a new process.
I also don't really understand the use case here, I don't know why do you want minimum < maximum...
Balázs
Hi Balázs,
ReplyDeleteFor example, when I set the MAX NUMBER OF PROCESSES to 30, and I have a total of 900 calculations. I am noticing that all 30 initial calculations have to finish before the next 30 start. Has this been your experience? For example, I am not seeing a new process initiate once the MAX drops below 30.
Many Thanks In Advance, Alex
Nope. It should not work like that. It should start the next calculation as soon as the number of processes drops below 30.
ReplyDeleteIf it does not work like that, send me your code in email and let me look at that...
Hi Balazs,
ReplyDeleteHere is a snippet of my code:
my $MAX_PROCESSES = 60;
my $process_manager = new Parallel::ForkManager($MAX_PROCESSES);
my $scenario_batch_cmd;
my @list_of_scenarios= get_list_of_scenarios($SCENARIO_DIR);
my $scenario_num = @list_of_scenarios;
foreach my $scn_file (@list_of_scenarios)
{
my ($scn_file_bname, $scn_file_dir, $scn_file_suffix);
if ($scn_file)
{
($scn_file_bname, $scn_file_dir, $scn_file_suffix) = fileparse($scn_file,qr{\.scn});
$process_manager->start and next;
generate_scenario($scn_file);
$process_manager->finish; #terminate child process
}
}
$process_manager->wait_all_children; #hang out untill all processes have completed
Hmmm... Looks perfectly OK to me...
ReplyDeleteUsing a callback or some other way, how can I see how many concurrent processes are running?
ReplyDeleteThanks,
Alex
The $pm->{processes} hash (in the parent) contains the child process PIDs, so if you count the number of keys, then you get the number of child processes running.
ReplyDelete