Gearman::Driver - Manages Gearman workers

NAME

Gearman::Driver - Manages Gearman workers

SYNOPSIS

    package My::Workers::One;

    # Yes, you need to do it exactly this way
    use base qw(Gearman::Driver::Worker);
    use Moose;

    # this method will be registered with gearmand as 'My::Workers::One::scale_image'
    sub scale_image : Job {
        my ( $self, $job, $workload ) = @_;
        # do something
    }

    # this method will be registered with gearmand as 'My::Workers::One::do_something_else'
    sub do_something_else : Job : MinProcesses(2) : MaxProcesses(15) {
        my ( $self, $job, $workload ) = @_;
        # do something
    }

    # this method wont be registered with gearmand at all
    sub do_something_internal {
        my ( $self, $job, $workload ) = @_;
        # do something
    }

    1;

    package My::Workers::Two;

    use base qw(Gearman::Driver::Worker);
    use Moose;

    # this method will be registered with gearmand as 'My::Workers::Two::scale_image'
    sub scale_image : Job {
        my ( $self, $job, $workload ) = @_;
        # do something
    }

    1;

    package main;

    use Gearman::Driver;

    my $driver = Gearman::Driver->new(
        namespaces => [qw(My::Workers)],
        server     => 'localhost:4730,otherhost:4731',
        interval   => 60,
    );

    $driver->run;

DESCRIPTION

Having hundreds of Gearman workers running in separate processes can consume a lot of RAM. Often many of these workers share the same code/objects, like the database layer using the DBIx::Class manpage for example. This is where the Gearman::Driver manpage comes in handy:

You write some base class which inherits from the Gearman::Driver::Worker manpage. Your base class loads your database layer for example. Each of your worker classes inherit from that base class. In the worker classes you can register single methods as jobs with gearmand. It's even possible to control how many workers doing that job/method in parallel. And this is the point where you'll save some RAM: Instead of starting each worker in a separate process the Gearman::Driver manpage will fork each worker from the main process. This will take advantage of copy-on-write on Linux and save some RAM.

There's only one mandatory parameter which has to be set when calling the constructor: namespaces

    use Gearman::Driver;
    my $driver = Gearman::Driver->new( namespaces => [qw(My::Workers)] );

See also: namespaces. If you do not set server (gearmand) attribute the default will be used: localhost:4730

Each module found in your namespaces will be loaded and introspected, looking for methods having the 'Job' attribute set:

    package My::Workers::ONE;

    sub scale_image : Job {
        my ( $self, $job, $workload ) = @_;
        # do something
    }

This method will be registered as job function with gearmand, verify it by doing:

    plu@mbp ~$ telnet localhost 4730
    Trying ::1...
    Connected to localhost.
    Escape character is '^]'.
    status
    My::Workers::ONE::scale_image   0       0       1
    .
    ^]
    telnet> Connection closed.

If you dont like to use the full package name you can also specify a custom prefix:

    package My::Workers::ONE;

    sub prefix { 'foo_bar_' }

    sub scale_image : Job {
        my ( $self, $job, $workload ) = @_;
        # do something
    }

This would register 'foo_bar_scale_image' with gearmand.

See also: prefix

ATTRIBUTES

namespaces

Will be passed to the Module::Find manpage findallmod method to load worker modules. Each one of those modules has to be inherited from the Gearman::Driver::Worker manpage or a subclass of it. It's also possible to use the full package name to load a single module/file. There is also a method get_namespaces which returns a sorted list of all namespaces.

See also: wanted.

wanted

This CodeRef will be called on each of the modules found in your namespace. The first and only parameter to this sub is the name of the module. If a true value is returned, the module will be loaded and checked if it's a valid the Gearman::Driver::Worker manpage subclass.

Let's say you have a namespace called My::Project:

To avoid every module being loaded and inspected being a the Gearman::Driver::Worker manpage subclass you can use wanted to only load classes having Worker in the package name:

    my $driver = Gearman::Driver->new(
        interval   => 0,
        namespaces => [qw(My::Project)],
        wanted     => sub {
            return 1 if /Worker/;
            return 0;
        },
    );

This would only load:

server

A list of Gearman servers the workers should connect to. The format for the server list is: host[:port][,host[:port]]

See also: the Gearman::XS manpage

console_port

Gearman::Driver has a telnet management console, see also:

the Gearman::Driver::Console manpage

Set this to 0 to disable management console at all.

interval

Each n seconds the Net::Telnet::Gearman manpage is used in the Gearman::Driver::Observer manpage to check status of free/running/busy workers on gearmand. This is used to fork more workers depending on the queue size and the MinProcesses/MaxProcesses attribute of the job method. See also: the Gearman::Driver::Worker manpage

logfile

Path to logfile.

loglayout

See also the Log::Log4perl manpage.

loglevel

See also the Log::Log4perl manpage.

lib

This is just for convenience to extend @INC from command line using gearman_driver.pl:

    gearman_driver.pl --lib ./lib --lib /custom/lib --namespaces My::Workers

unknown_job_callback

Whenever the Gearman::Driver::Observer manpage sees a job that isnt handled it will call this CodeRef, passing following arguments:

    my $driver = Gearman::Driver->new(
        namespaces           => [qw(My::Workers)],
        unknown_job_callback => sub {
            my ( $driver, $status ) = @_;
            # notify nagios here for example
        }
    );

$status might look like:

    $VAR1 = {
        'busy'    => 0,
        'free'    => 0,
        'name'    => 'GDExamples::Sleeper::unknown_job',
        'queue'   => 6,
        'running' => 0
    };

INTERNAL ATTRIBUTES

This might be interesting for subclassing the Gearman::Driver manpage.

modules

Every worker module loaded by the Module::Find manpage will be added to this list. There are also two methods: get_modules and has_modules.

jobs

Stores all the Gearman::Driver::Job manpage instances. The key is the name the job gets registered with gearmand. There are also two methods: get_job and has_job.

Example:

    {
        'My::Workers::ONE::scale_image'       => bless( {...}, 'Gearman::Driver::Job' ),
        'My::Workers::ONE::do_something_else' => bless( {...}, 'Gearman::Driver::Job' ),
        'My::Workers::TWO::scale_image'       => bless( {...}, 'Gearman::Driver::Job' ),
    }

observer

Instance of the Gearman::Driver::Observer manpage.

console

Instance of the Gearman::Driver::Console manpage.

METHODS

add_job

There's one mandatory param (hashref) with following keys:

Basically you never really need this method if you use namespaces. But namespaces depend on method attributes which some people do hate. In this case, feel free to setup your $driver this way:

    package My::Workers::One;

    use Moose;
    use JSON::XS;
    extends 'Gearman::Driver::Worker::Base';

    sub scale_image {
        my ( $self, $job, $workload ) = @_;
        # do something
    }

    # this method will be registered with gearmand as 'My::Workers::One::do_something_else'
    sub do_something_else {
        my ( $self, $job, $workload ) = @_;
        # do something
    }

    sub encode_json {
        my ( $self, $result ) = @_;
        return JSON::XS::encode_json($result);
    }

    sub decode_json {
        my ( $self, $workload ) = @_;
        return JSON::XS::decode_json($workload);
    }

    1;

    package main;

    use Gearman::Driver;
    use My::Workers::One;

    my $driver = Gearman::Driver->new(
        server   => 'localhost:4730,otherhost:4731',
        interval => 60,
    );

    my $worker = My::Workers::One->new();

    foreach my $method (qw(scale_image do_something_else)) {
        $driver->add_job(
            decode        => 'decode_json',
            encode        => 'encode_json',
            max_processes => 5,
            method        => $worker->meta->find_method_by_name($method)->body,
            min_processes => 1,
            name          => $method,
            object        => $worker,
        );
    }

    $driver->run;

run

This must be called after the the Gearman::Driver manpage object is instantiated.

shutdown

Sends TERM signal to all child processes and exits Gearman::Driver.

get_namespaces

Returns a sorted list of namespaces.

get_modules

Returns a sorted list of modules.

has_modules

Returns the count of modules.

has_job

Params: $name

Returns true/false if the job exists.

get_job

Params: $name

Returns the job instance.

AUTHOR

Johannes Plunien <plu@cpan.org>

CONTRIBUTORS

Uwe Voelker, <uwe.voelker@gmx.de>

COPYRIGHT AND LICENSE

Copyright 2009 by Johannes Plunien

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

REPOSITORY

http://github.com/plu/gearman-driver/

 Gearman::Driver - Manages Gearman workers