Dear list,

I'm wrinting a perl program that works with different threads. Those
threads depend on each other, not all in the same way. Some threads should
stop when others are finished with their work, and again others are to be
started afterwards. The current architecture I have in mind is that of a
master thread controlling the others. I'm looking for a way to:

1. signal that a thread has finished its work, and find out which thread
that is,
2. stopping another thread on that signal,
3. gathering the results from each of the two; thereby
4. I want to be able to start a new bunch of threads on demand.

More verbose and concrete:

I have six worker machines, M1 to M6, and a controller machine, M0. The
perl program runs on the controller machine. This program is basically a
dispatcher I control from a command line interface. A command causes two
threads to be started, say on M1 and M2. One thread on M1 produces work,
the other on M2 plots the network traffic. When M1 is finished, I want the
thread M0 that caused the workload to signal to the dispatcher that M1 is
finished. The dispatcher then shall signal M2 to stop monitoring.
Afterwards, I want the results from both M1 and M2, preferably in form of
a perl structure.

First of all, is that possible?

I have already looked at threads, threads::shared and the traditional
fork(). With the standard IPC stuff, I'm able to signal and trap that in
the master process with a signal handler. But I cannot, however, get the
PID of the child that emitted the signal, thus I'm not able to send a
SIGTERM to another process. I'm also not sure how I could get my result
data.

threads::shared allows me to share a perl structure I could fill, but I
don't know how to signal the master thread. Passing a subroutine reference
doesn't work, of other options I don't know.

I'd very much appreciate any hints.

Thanks in advance.

# Eric

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

On Feb 26, 2010, at 10:34 PM, Eric Veith1 wrote:

> Dear list,
>
> I'm wrinting a perl program that works with different threads.

Your first sentence already has me worried. Threads are un-fun.

> Those
> threads depend on each other, not all in the same way. Some threads =
should
> stop when others are finished with their work, and again others are to =
be
> started afterwards.

Now getting really complex.

> I have six worker machines, M1 to M6, and a controller machine, M0. =
The
> perl program runs on the controller machine. This program is basically =
a
> dispatcher I control from a command line interface. A command causes =
two
> threads to be started, say on M1 and M2.

At this point, these are really processes. Threads are in the kernel, =
you are just starting a separate process on two different machines.

> One thread on M1 produces work,
> the other on M2 plots the network traffic. When M1 is finished, I want =
the
> thread M0 that caused the workload to signal to the dispatcher that M1 =
is
> finished. The dispatcher then shall signal M2 to stop monitoring.
> Afterwards, I want the results from both M1 and M2, preferably in form =
of
> a perl structure.

These are all separate processes, not threads. Are you sure you need to =
use threads? In general a thread is something that runs _inside_ a =
process. http://en.wikipedia.org/wiki/Thread_(computer_science)

> First of all, is that possible?

Yes. Not that hard. You have exit values from your process so you can =
trap that and do something based on your exit value.
>
> I have already looked at threads, threads::shared and the traditional =

> fork(). With the standard IPC stuff, I'm able to signal and trap that =
in
> the master process with a signal handler. But I cannot, however, get =
the
> PID of the child that emitted the signal, thus I'm not able to send a =

> SIGTERM to another process. I'm also not sure how I could get my =
result
> data.

Well, why not just create something in a perl structure and just use =
something like the Storable module or YAML to dump it to disk and you =
can read it from your other machine.
>
> threads::shared allows me to share a perl structure I could fill, but =
I
> don't know how to signal the master thread. Passing a subroutine =
reference
> doesn't work, of other options I don't know.
>
> I'd very much appreciate any hints.

I would avoid threads. I would have a program / process on machine 0 =
that fires off another program / process on machine 1. Then I guess you =
need to fire off your program on machine 2 to do network monitoring (?). =
When the program on 1 is finished, it dumps its data to disk and call =
machine 0 who in turn calls 2.

Jeremiah



--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

From: Jeremiah Foster
> On Feb 26, 2010, at 10:34 PM, Eric Veith1 wrote:
>>
>> I'm wrinting a perl program that works with different threads.
>
> Your first sentence already has me worried. Threads are un-fun.
>
>> Those
>> threads depend on each other, not all in the same way. Some threads
should
>> stop when others are finished with their work, and again others are
to be
>> started afterwards.
>
> Now getting really complex.
>
>> I have six worker machines, M1 to M6, and a controller machine, M0.
The
>> perl program runs on the controller machine. This program is
basically a
>> dispatcher I control from a command line interface. A command causes
two
>> threads to be started, say on M1 and M2.
>
>At this point, these are really processes. Threads are in the kernel,
you
> are just starting a separate process on two different machines.
>
>> One thread on M1 produces work,
>> the other on M2 plots the network traffic. When M1 is finished, I
want the
>> thread M0 that caused the workload to signal to the dispatcher that
M1 is
>> finished. The dispatcher then shall signal M2 to stop monitoring.
>> Afterwards, I want the results from both M1 and M2, preferably in
form of
>> a perl structure.
>
> These are all separate processes, not threads. Are you sure you need
to
> use threads? In general a thread is something that runs _inside_ a
> process. http://en.wikipedia.org/wiki/Thread_(computer_science)
>
>> First of all, is that possible?
>
> Yes. Not that hard. You have exit values from your process so you can
> trap that and do something based on your exit value.
>>
>> I have already looked at threads, threads::shared and the traditional

>> fork(). With the standard IPC stuff, I'm able to signal and trap that
in
>> the master process with a signal handler. But I cannot, however, get
the
>> PID of the child that emitted the signal, thus I'm not able to send a

>> SIGTERM to another process. I'm also not sure how I could get my
result
>> data.
>
> Well, why not just create something in a perl structure and just use
> something like the Storable module or YAML to dump it to disk and you
> can read it from your other machine.
>>
>> threads::shared allows me to share a perl structure I could fill, but
I
>> don't know how to signal the master thread. Passing a subroutine
reference
>> doesn't work, of other options I don't know.
>>
>> I'd very much appreciate any hints.
>
> I would avoid threads. I would have a program / process on machine 0
> that fires off another program / process on machine 1. Then I guess
> you need to fire off your program on machine 2 to do network
> monitoring (?). When the program on 1 is finished, it dumps its data
> to disk and call machine 0 who in turn calls 2.

Depending on how much control you have over development of this package,
I would suggest a hybrid approach. First, launch an application which
creates multiple threads to start up the individual processes. Open a
communications channel between each thread and the process it called.
Then use the threads to monitor and/or control the individual processes
until they terminate. Once the process is done, terminate the matching
thread.

When processes run on the local machine, any of the IPC options can be
used for the comm channel. On remote machines it is more likely to be a
socket. Interprocess communications can be done either via IPC or
relayed through the control threads.

However, if the application is this complex, is Perl really the best
language to use? It would not be my first choice.

Bob McConnell

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

Jeremiah Foster <jeremiah [at] jeremiahfoster.com> wrote on 03/03/2010 02:09:40 =

PM:
> I would avoid threads. I would have a program / process on machine 0
> that fires off another program / process on machine 1. Then I guess
> you need to fire off your program on machine 2 to do network
> monitoring (?). When the program on 1 is finished, it dumps its data
> to disk and call machine 0 who in turn calls 2.

Jeremiah,

thanks for your answer. I fear my writing was misleading in regards of
threads/subprocesses, sorry for that. One intention of my question was to
find out whether the best way was to use Perl's interpreter threads or its =

fork() implementation.

Your answer pushed me in the direction of using fork(), because I can then =

use the SIGCHLD handler to send a signal to another sub-process to solve
my dependency problem. Dumping the stuff on the disk and reading/parsing
it later also prooved to be good push in the right direction; I'm using
FreezeThaw for that, together with IO::Compress::LZMA in case I expect a
lot of data.

Subprocess control on remote machines is now implemented using a simple
RPC-like architecture, with JSON for data transport.

I designed my Moose classes in a way that they are not aware whether
they've been called locally or remote, a Dispatcher class takes care of
that and also controls my processes.

It all works pretty well now. Thanks!

Bob,

what language would you have used? I admit this is my first Perl project
to involve this kind of child process handling.

--
Eric MSP Veith <eric.veith [at] de.ibm.com>
Hechtsheimer Str. 2
DE-55131 Mainz
Germany

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Erich Clementi
Geschäftsführung: Martin Jetter (Vorsitzender), Reinhard Reschke,
Christoph Grandpierre, Matthias Hartmann, Michael Diemer, Martina
Koederitz
Sitz der Gesellschaft: Stuttgart
Registergericht: Amtsgericht Stuttgart, HRB 14562
WEEE-Reg.-Nr. DE 99369940

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

From: Eric Veith1

> Bob,
>
> what language would you have used? I admit this is my first Perl
project
> to involve this kind of child process handling.

Eric,

The final choice depends very much on A) target environment and B)
resources required. The use of libraries, both private and third party,
will also have a marked impact on how long it takes to complete a
project. Personally, I have a lot of C experience (> 20 years) with
various multi-tasking and multi-processing environments. So on a Posix
platform I would probably write the control program in C with pthreads.
On MS-Windows it would still be C, but using a different API. I know
several programmers that would do it in Java, and a couple that would
even fight through it with C++.

How to write the applications also depends on what they need to do. The
description I gave is a rough outline of the MS-Windows Service Control
Manager and the way it interacts with various services and the display
applet. Again, I have written several services in C, so I have working
templates already available for that model. But, if they have a GUI
component, that is a very different requirement from a console/ daemon/
service process, and requires significantly different tool kits. Also,
the choice of OS will dictate some of those requirements. Unfortunately,
there are not a lot of tool kits that are fully portable between
MS-Windows and the rest of the world.

Bob McConnell

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

On Wed, Mar 3, 2010 at 8:28 AM, Bob McConnell <rvm [at] cbord.com> wrote:
[snip]
>
> However, if the application is this complex, is Perl really the best
> language to use? It would not be my first choice.
>


That is a very strange statement to make on a Perl beginners list, not
least because it's complete bosh.

What better language to simple network control structures? This is
exactly the sort of task that Perl accomplishes better and more easily
than any other language out there, and why it's "the glue of the
internet."


In particular, this sounds like a job for RPC::Lite.

The issue, though, isn't just how processes communicate with each
other--any RPC/IPC implementation can handle that for you
transparently; I suggested RPC::Lite but Moose::Async or something
similar is fine, too, although probably overkill--it's also how to
launch the program on the remote machine. Here you really have two
options: you can set up listener daemons on the worker machines that
wait fro input from the console and launch workers, or you can use
something like Net::SSH2 to login to the worker machine and launch a
new process each time.

Also, be careful with your terminology. You'll get better help, here,
if you ask the right questions.

Network/socket communication and IPC are two very different things.
Most operating systems (in fact all but one that I'm aware of; the
possible exception being Plan9) localize processes to the running
kernel. Processes (whether perl programs or otherwise) can't fork
children on other machines or deliver SIGTERM across the network, and
asking about threads and remote machines in the same breath is likely
to yield strange responses and/or confused silence.

HTH,

-- jay
--------------------------------------------------
This email and attachment(s): [ ] blogable; [ x ] ask first; [ ]
private and confidential

daggerquill [at] gmail [dot] com
http://www.tuaw.com http://www.downloadsquad.com http://www.engatiki.org

values of =E2 will give rise to dom!

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

From: Jay Savage
> On Wed, Mar 3, 2010 at 8:28 AM, Bob McConnell <rvm [at] cbord.com> wrote:
> [snip]
>>
>> However, if the application is this complex, is Perl really the best
>> language to use? It would not be my first choice.
>
> That is a very strange statement to make on a Perl beginners list, not
> least because it's complete bosh.
>
> What better language to simple network control structures? This is
> exactly the sort of task that Perl accomplishes better and more easily
> than any other language out there, and why it's "the glue of the
> internet."

The way I read his problem description, it sounded neither simple nor
easy.

While Perl is very useful for a number of things, it is not the best
hammer to use for everything, and I believe that needs to be pointed out
occasionally. Even though Larry has tried to put the kitchen sink into
the interpreter, there are still tools that can do some tasks better
than Perl can. In particular, all too often I see warnings that threads
are still not handled very well in Perl. Until that changes, I will not
consider Perl for anything that requires multi-threaded code.

Bob McConnell

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

Jeremiah Foster wrote:

> "Unix processes are one of two techniques for achieving reliable concurrency and parallelism in server applications. Threads are out. You can use processes, or async/events, or both processes and async/events, but definitely not threads. Threads are out."

For people who still think threads (in the sense of: subprocesses that
share resources such as state and memory) are tolerable: please start
educating yourselves.

Read what Dijkstra wrote (some of it even 40 years ago) on formal
verification, distributed computing and self-stabilization
(convergency). See also:

http://en.wikipedia.org/wiki/List_of_important_publications_ in_concurrent,_parallel,_and_distributed_computing
http://en.wikipedia.org/wiki/Thread_%28computer_science%29
http://en.wikipedia.org/wiki/Fork_%28operating_system%29

Threads are like Microsoft products: full of problems. Selling them can
make you the richest person on earth, so please don't follow my advice
if you are in it for the money.

Next question: Why do machines exist that let instructions and data
share resources?
http://en.wikipedia.org/wiki/Harvard_architecture

--
Ruud (fork'n'merge)

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

"Bob McConnell" <rvm [at] CBORD.com> wrote on 03/05/2010 08:22:23 PM:
> The way I read his problem description, it sounded neither simple nor
> easy.

Bob, Jay,

I fear, Jay, my explanaitions weren't fully able to depict what my app is
trying to archieve. It is not only a matter of IPC on one machine that
happens to run tasks on other computers; actually, I have both IPC and
Socket operations there. Perhaps that's why you wrote the warning about my =

terminology.

First of all, there is exactly one "master" program on a master machine.
This master program has a master process that is responsible for reading
commands in a not further described way (may be a script, a perl subclass
or an instance of Term::ReadLine). I've a "ProtocolHandler" role and
several concrete protocol classes that take care of parsing the commands.
There's also a class called "Dispatcher" that does the management work,
that is, instanceating classes and calling methods based on what the
Protocol parsed.

At this point, I need to introduce parallel work: Most of the work is done =

on another machine, be it running dd, setting some option or starting
tcpdump. I call them "Commands" and created the Command role and several
classes. This parallelism is archieved by spawning subprocesses (I
refrained from the idea of using interpreter threads for some reasons).
Error/return codes, output and other stuff is neatly wrapped in such a
class performing the Command role. The reason I introduced this role is
not only to gather all stuff in one place, but also because what I call a
"command" can actually be a bunch of real commands or Perl function calls. =

Introducing this layer of abstraction helps me with the management stuff
in the Dispatcher.

Now the same code base lives also on several "slave" machines, except for
a script that does some specific initialization work. Here, too, runs an
instance of a Protocol class, and the Dispatcher. The only difference is
the concrete Protocol class, which now uses Sockets instead of a (e.g.)
Term::ReadLine interface.

Now IPC comes into play at that point: Some sub-processes depend on
others. When I'm creating some kind of load on one machine and monitoring
the network activity on another, I'll want the network monitor stop as
soon as the load generator stops. The Dispatcher takes care of that, using =

a Command called "Kill".

Finally, results get transferred back to the master that analyzes them
using some other scripts I wrote and, for example, puts a nice report
together.

You see, there's IPC on the local machine and possibly sockets to a remote =

machine.

I'm going pretty well with that right now. :-)

There's only one thing that concerns me, although I haven't run into any
problems right now: Safe (aka deferred) vs. immediate signals. When using
Term::ReadLine to read commands, signals won't be handled until somebody
at least presses enter. This, however, means that the whole dependency
system doesn't work as I want it to, because another process isn't killed
when the first one exits, but after somebody pressed enter. I currently
circumvent the problem using the "POSIX" package's RT signals, but from
what I learned from perlipc, these signals aren't safe and can lead to
corrumption. Now "perlipc" doesn't give a concrete example on that topic.
I don't want to do much in the signal handler, but there's some work I
cannot avoid (e.g., flushing command output on SIGTERM, that, in turn, can =

be issued because of a SIGCHLD -- the dependency stuff). Could somebody
please give me an example in which -- using POSIX RT signals -- work in
the signal handler is responsible for corruption, or better yet, outline
when corruption is very likely to happen, and why?

Thanks very much in advance,
Eric

--
Eric MSP Veith <eric.veith [at] de.ibm.com>
Hechtsheimer Str. 2
DE-55131 Mainz
Germany

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Erich Clementi
Geschäftsführung: Martin Jetter (Vorsitzender), Reinhard Reschke,
Christoph Grandpierre, Matthias Hartmann, Michael Diemer, Martina
Koederitz
Sitz der Gesellschaft: Stuttgart
Registergericht: Amtsgericht Stuttgart, HRB 14562
WEEE-Reg.-Nr. DE 99369940

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

On Mon, Mar 8, 2010 at 10:27 AM, Eric Veith1 <eric.veith [at] de.ibm.com> wrote:
> "Bob McConnell" <rvm [at] CBORD.com> wrote on 03/05/2010 08:22:23 PM:
>> The way I read his problem description, it sounded neither simple nor
>> easy.
>
> Bob, Jay,
>

[snip]

> You see, there's IPC on the local machine and possibly sockets to a remot=
e
> machine.
>

Yes, that's a much clearer picture... ;)

> I'm going pretty well with that right now. :-)
>

Great!


> There's only one thing that concerns me, although I haven't run into any
> problems right now: Safe (aka deferred) vs. immediate signals. When using
> Term::ReadLine to read commands, signals won't be handled until somebody
> at least presses enter. This, however, means that the whole dependency
> system doesn't work as I want it to, because another process isn't killed
> when the first one exits, but after somebody pressed enter. I currently
> circumvent the problem using the "POSIX" package's RT signals, but from
> what I learned from perlipc, these signals aren't safe and can lead to
> corrumption. Now "perlipc" doesn't give a concrete example on that topic.
> I don't want to do much in the signal handler, but there's some work I
> cannot avoid (e.g., flushing command output on SIGTERM, that, in turn, ca=
n
> be issued because of a SIGCHLD -- the dependency stuff). Could somebody
> please give me an example in which -- using POSIX RT signals -- work in
> the signal handler is responsible for corruption, or better yet, outline
> when corruption is very likely to happen, and why?
>

It sounds like Term::Readline is using Term::ReadLine::Gnu as the
back-end. The problem there is that, to Perl, the XS call for GNU
readline() looks like a single system call.

Try setting the PERL_RL environment to "Perl" instead of "Gnu":

use Term::ReadLine;
$ENV{PERL_RL} =3D "Perl";
my $term =3D Term::ReadLine->new('my term');

You could also just use <> unless you want some special feature of ReadLine=
..


As for "corruption," you never know. The behavior is undefined and
unpredictable, because you don't know what you're interrupting. The
real danger is that you'll interrupt a malloc call in a way that
causes the perl interpreter to crash and dump core.

HTH,

-- jay
--------------------------------------------------
This email and attachment(s): [ ] blogable; [ x ] ask first; [ ]
private and confidential

daggerquill [at] gmail [dot] com
http://www.tuaw.com http://www.downloadsquad.com http://www.engatiki.org

values of =E2 will give rise to dom!

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

Jay:

Jay Savage <daggerquill [at] gmail.com> wrote on 03/08/2010 08:53:40 PM:
> It sounds like Term::Readline is using Term::ReadLine::Gnu as the
> back-end. The problem there is that, to Perl, the XS call for GNU
> readline() looks like a single system call.
> Try setting the PERL=5FRL environment to "Perl" instead of "Gnu":

That's a good idea, I'll try that.

> As for "corruption," you never know. The behavior is undefined and
> unpredictable, because you don't know what you're interrupting. The
> real danger is that you'll interrupt a malloc call in a way that
> causes the perl interpreter to crash and dump core.

Well, that's as far as I'm right now. But I cannot really gasp it, that's
why I'd like a example, just the way one demonstrates the pitfalls of
malloc() to a C newbie. So if somebody could show me how to trigger such a =

corruption, I'd be really grateful. :-)

Thanks,
Eric

--
Eric MSP Veith <eric.veith [at] de.ibm.com>
Hechtsheimer Str. 2
DE-55131 Mainz
Germany

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Erich Clementi
Geschäftsführung: Martin Jetter (Vorsitzender), Reinhard Reschke,
Christoph Grandpierre, Matthias Hartmann, Michael Diemer, Martina
Koederitz
Sitz der Gesellschaft: Stuttgart
Registergericht: Amtsgericht Stuttgart, HRB 14562
WEEE-Reg.-Nr. DE 99369940



--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/