Sunday, August 15, 2010

Parallel::ForkManager 0.7.6 (#perl, #forkmanager) and publicity

I've put out a new Parallel::ForkManager release: 0.7.6 with small changes:
  • Added datastructure retrieval from child processes. (Ken Clarke)
  • Using CORE::exit() instead of exit() for mod_perl compatibility.
The new version is available in the project page and (sooner or later) from CPAN.

I did some research about the module publicity and found the following things:
Enjoy!

8 comments:

  1. Hi Balázs,

    I was wondering if you could answer a question regarding the perl module :: Parallel::ForkManager

    I know ForkManager, lets you specify the max number of processes, but I was wondering how I can specify the min number of processes running?

    For example, let’s say I have a total of 200 calculations I need to complete, how can I specify I want a MAX = 30 processes running and a minimum of 20 processes at one time?
    (I want to be able to add additional calculations whenever the minimum number of calculations falls below 20).

    Any ideas would be greatly appreciated. Also, thanks for contributing to the open source community.



    use Parallel::ForkManager;
    use LWP::Simple;
    my $pm=new Parallel::ForkManager(30);
    for my $link (@ARGV)
    {
    $pm->start and next;
    my ($fn)= $link =~ /^.*\/(.*?)$/;
    if (!$fn)
    {
    warn "Cannot determine filename from $fn\n";
    }
    else
    {
    $0.=" ".$fn;
    print "Getting $fn from $link\n";
    my $rc=getstore($link,$fn);
    print "$link downloaded. response code: $rc\n";
    };
    $pm->finish;
    };

    ReplyDelete
  2. Hi Alex,

    You cannot set minimum here right now. It assume that mimum=maximum, so if the number of processes drops below the maximum, it starts a new process.

    I also don't really understand the use case here, I don't know why do you want minimum < maximum...

    Balázs

    ReplyDelete
  3. Hi Balázs,

    For example, when I set the MAX NUMBER OF PROCESSES to 30, and I have a total of 900 calculations. I am noticing that all 30 initial calculations have to finish before the next 30 start. Has this been your experience? For example, I am not seeing a new process initiate once the MAX drops below 30.
    Many Thanks In Advance, Alex

    ReplyDelete
  4. Nope. It should not work like that. It should start the next calculation as soon as the number of processes drops below 30.

    If it does not work like that, send me your code in email and let me look at that...

    ReplyDelete
  5. Hi Balazs,

    Here is a snippet of my code:

    my $MAX_PROCESSES = 60;
    my $process_manager = new Parallel::ForkManager($MAX_PROCESSES);

    my $scenario_batch_cmd;
    my @list_of_scenarios= get_list_of_scenarios($SCENARIO_DIR);
    my $scenario_num = @list_of_scenarios;
    foreach my $scn_file (@list_of_scenarios)
    {
    my ($scn_file_bname, $scn_file_dir, $scn_file_suffix);
    if ($scn_file)
    {
    ($scn_file_bname, $scn_file_dir, $scn_file_suffix) = fileparse($scn_file,qr{\.scn});
    $process_manager->start and next;
    generate_scenario($scn_file);
    $process_manager->finish; #terminate child process
    }
    }
    $process_manager->wait_all_children; #hang out untill all processes have completed

    ReplyDelete
  6. Hmmm... Looks perfectly OK to me...

    ReplyDelete
  7. Using a callback or some other way, how can I see how many concurrent processes are running?

    Thanks,
    Alex

    ReplyDelete
  8. The $pm->{processes} hash (in the parent) contains the child process PIDs, so if you count the number of keys, then you get the number of child processes running.

    ReplyDelete