Hi,

I have a data file with n columns and r row.
The first 3 columns and the first 5 rows are:

2 3 1
1 6 X
4 0 X
X 8 X
5 X 5

The "X" means missing.
How could I write a script to calculate the average by column and
replace "X" with the average?

The output should be like:

2 3 1
1 6 3
4 0 3
3 8 3
5 4.25 5

For column 1, the counts excluding "X" is 4 with sum=12. So the
average for column 1 is 3.
For column 2, the counts excluding "X" is 4 with sum=17. So the
average for column 2 is 4.25.
For column 3, the counts excluding "X" is 2 with sum=6. So the average
for column 3 is 3.

Thanks!!!


--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

On Fri, Jan 7, 2011 at 9:16 PM, S.F. <lotus.sf1208 [at] gmail.com> wrote:
> I have a data file with n columns and r row.
> The first 3 columns and the first 5 rows are:
>
> 2 3 1
> 1 6 X
> 4 0 X
> X 8 X
> 5 X 5
>
> The "X" means missing.
> How could I write a script to calculate the average by column and
> replace "X" with the average?
>
> The output should be like:
>
> 2 =C2=A03 =C2=A0 =C2=A0 =C2=A0 =C2=A01
> 1 =C2=A06 =C2=A0 =C2=A0 =C2=A0 =C2=A03
> 4 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A03
> 3 =C2=A08 =C2=A0 =C2=A0 =C2=A0 =C2=A03
> 5 =C2=A04.25 =C2=A0 5
>
> For column 1, the counts excluding "X" is 4 with sum=3D12. So the
> average for column 1 is 3.
> For column 2, the counts excluding "X" is 4 with sum=3D17. So the
> average for column 2 is 4.25.
> For column 3, the counts excluding "X" is 2 with sum=3D6. So the average
> for column 3 is 3.

This sounds an awful lot like a homework assignment. ;) Why don't you
start by telling us in English how you might accomplish this? Then
convert that to pseudo-code. It's hard to write Perl if you don't know
Perl, but it's harder still to write Perl if you don't know what
you're trying to write. :)

It always helps when coding to figure out the problem before hand, one
step at a time.

--
Brandon McCaig <http://www.bamccaig.com> <bamccaig [at] gmail.com>
V zrna gur orfg jvgu jung V fnl. Vg qbrfa'g nyjnlf fbhaq gung jnl.
Castopulence Software <http://www.castopulence.org/> <bamccaig [at] castopulence=
..org>

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

At 18:16 -0800 07/01/2011, S.F. wrote:

>I have a data file with n columns and r row.
>The first 3 columns and the first 5 rows are:
>
>2 3 1
>1 6 X
>4 0 X
>X 8 X
>5 X 5
>
>The "X" means missing.
>How could I write a script to calculate the average by column and
>replace "X" with the average?

There must be a cleverer way to do it but this works:


#!/usr/local/bin/perl
use strict;
my ( [at] col1, [at] col2, [at] col3);
my ($tot1, $tot2, $tot3, $avg1, $avg2, $avg3);
my ($n, $output);
while (<DATA>){
chomp;
my ($v1, $v2, $v3) = split /\s*/;
push [at] col1, $v1; push [at] col2, $v2; push [at] col3, $v3;
}
#
$n=0; for ( [at] col1) {$n-- if /x/i ; $tot1 += $_}
$avg1 = $tot1/($n + [at] col1);
$n=0; for ( [at] col2) {$n-- if /x/i ; $tot2 += $_}
$avg2 = $tot2/($n + [at] col1);
$n=0; for ( [at] col3) {$n-- if /x/i ; $tot3 += $_}
$avg3 = $tot3/($n + [at] col1);
#
for $n (0.. [at] col1-1){
$_ = $col1[$n]; s~x~$avg1~i;
$output .= "$_\t";
$_ = $col2[$n]; s~x~$avg2~i;
$output .= "$_\t";
$_ = $col3[$n]; s~x~$avg3~i;
$output .= "$_\n";
}
print $output;
__DATA__
2 3 1
1 6 X
4 0 X
X 8 X
5 X 5




--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

On 2011-01-08 03:16, S.F. wrote:

> I have a data file with n columns and r row.
> The first 3 columns and the first 5 rows are:
>
> 2 3 1
> 1 6 X
> 4 0 X
> X 8 X
> 5 X 5
>
> The "X" means missing.
> How could I write a script to calculate the average by column and
> replace "X" with the average?
>
> The output should be like:
>
> 2 3 1
> 1 6 3
> 4 0 3
> 3 8 3
> 5 4.25 5

perl -MData::Dumper -ale'
[at] d=map[do{ [at] t=split;$n=$s=0;$n++,$s+=$_ for grep$_ ne"X", [at] t;$_=$s/$n
for grep$_ eq"X", [at] t; [at] t}],split",",$ARGV[0];
print Dumper\ [at] d
' "2 1 4 X 5,3 6 0 8 X,1 X X X 5"

$VAR1 = [
[
2,
1,
4,
'3',
5
],
[
3,
6,
0,
8,
'4.25'
],
[
1,
'3',
'3',
'3',
5
]
];

--
Ruud

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

Hi all.

I have read the explaination of the Map function and it is still a mystry to
myself on what it is for and when would you use it?

All explainations I have seen in books and blogs don't make it clear.

Sean


--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

On Tue, Jan 11, 2011 at 4:57 AM, Sean Murphy <mhysnm1964 [at] gmail.com> wrote:
> Hi all.
>
> I have read the explaination of the Map function and it is still a mystry to
> myself on what it is for and when would you use it?
>
> All explainations I have seen in books and blogs don't make it clear.
>

my [at] new = map {$_} [at] arr;

is the same as:

foreach ( [at] arr) {
push $_, [at] new;
}

in other words, do something to every element of an array.

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

>>>>> "sw" == shawn wilson <ag4ve.us [at] gmail.com> writes:

sw> On Tue, Jan 11, 2011 at 4:57 AM, Sean Murphy <mhysnm1964 [at] gmail.com> wrote:
>> Hi all.
>>
>> I have read the explaination of the Map function and it is still a mystry to
>> myself on what it is for and when would you use it?
>>
>> All explainations I have seen in books and blogs don't make it clear.
>>

sw> my [at] new = map {$_} [at] arr;

sw> is the same as:

sw> foreach ( [at] arr) {
sw> push $_, [at] new;
sw> }

sw> in other words, do something to every element of an array.

that is a poor explanation IMO.

map transforms one list into another. it is that simple but it is very
useful. you can alway write map in terms of a perl loop but it will be
slower and clunkier in most cases.

map take an input list, it aliases $_ to each element (an implied loop)
and then executes the block or expression. it collects the results of
that code and builds up a list which it returns when the input elements
are exhausted. the key to understanding map is that the expression can
return 0, 1 or more elements EACH time through. so the complete map call
can return ANY number of elements, regardless of the number of input
elements. if the expression returns 0 or 1 elements, it is similar to
perl's grep:

these are equivilent:

my [at] out = grep /foo/, [at] in ;
my [at] out = map { /foo/ ? $_ : () } [at] in ;

but map can do much more than grep. grep can ONLY return its input list
or some elements of it. map can generate more elements:

this builds a hash of 3 keys, each which has the value 1. it can be used
to test if something is one of the keys with a simple boolean.

my %is_a_foo = map { $_ => 1 } qw( foo bar baz) ;

if ( $is_a_foo( $stuff ) { ....

map is very simple when you really get it and very useful. it should be
in all perl hacker's skill sets. for some reason newbies have trouble
with it but that is usually due to poor explanations and not knowing
about functional programming (map comes from lisp which has similar map
funcs). map's name comes from its role in mapping one list into
another. that should help too.

uri

--
Uri Guttman ------ uri [at] stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

Hi Uri

Thanks for that. It makes more sense. I will have to play with it and see
what type of results I can get. It might be useful when I use it when I have
a lot of data being passed from captures of the routers I have to support
and their show commands.

Sean
----- Original Message -----
From: "Uri Guttman" <uri [at] StemSystems.com>
To: "shawn wilson" <ag4ve.us [at] gmail.com>
Cc: "Sean Murphy" <mhysnm1964 [at] gmail.com>; <beginners [at] perl.org>
Sent: Tuesday, January 11, 2011 9:44 PM
Subject: Re: Map function, what does it do?


>>>>>> "sw" == shawn wilson <ag4ve.us [at] gmail.com> writes:
>
> sw> On Tue, Jan 11, 2011 at 4:57 AM, Sean Murphy <mhysnm1964 [at] gmail.com>
> wrote:
> >> Hi all.
> >>
> >> I have read the explaination of the Map function and it is still a
> mystry to
> >> myself on what it is for and when would you use it?
> >>
> >> All explainations I have seen in books and blogs don't make it clear.
> >>
>
> sw> my [at] new = map {$_} [at] arr;
>
> sw> is the same as:
>
> sw> foreach ( [at] arr) {
> sw> push $_, [at] new;
> sw> }
>
> sw> in other words, do something to every element of an array.
>
> that is a poor explanation IMO.
>
> map transforms one list into another. it is that simple but it is very
> useful. you can alway write map in terms of a perl loop but it will be
> slower and clunkier in most cases.
>
> map take an input list, it aliases $_ to each element (an implied loop)
> and then executes the block or expression. it collects the results of
> that code and builds up a list which it returns when the input elements
> are exhausted. the key to understanding map is that the expression can
> return 0, 1 or more elements EACH time through. so the complete map call
> can return ANY number of elements, regardless of the number of input
> elements. if the expression returns 0 or 1 elements, it is similar to
> perl's grep:
>
> these are equivilent:
>
> my [at] out = grep /foo/, [at] in ;
> my [at] out = map { /foo/ ? $_ : () } [at] in ;
>
> but map can do much more than grep. grep can ONLY return its input list
> or some elements of it. map can generate more elements:
>
> this builds a hash of 3 keys, each which has the value 1. it can be used
> to test if something is one of the keys with a simple boolean.
>
> my %is_a_foo = map { $_ => 1 } qw( foo bar baz) ;
>
> if ( $is_a_foo( $stuff ) { ....
>
> map is very simple when you really get it and very useful. it should be
> in all perl hacker's skill sets. for some reason newbies have trouble
> with it but that is usually due to poor explanations and not knowing
> about functional programming (map comes from lisp which has similar map
> funcs). map's name comes from its role in mapping one list into
> another. that should help too.
>
> uri
>
> --
> Uri Guttman ------ uri [at] stemsystems.com --------
> http://www.sysarch.com --
> ----- Perl Code Review , Architecture, Development, Training,
> Support ------
> --------- Gourmet Hot Cocoa Mix ----
> http://bestfriendscocoa.com ---------
>


--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

>> that is a poor explanation IMO.

hell, i was going to give this which dereferences the reference of a
reference DBI's selectall_arrayref returns so that i can later loop
through it again and again with different parameters:

my ( [at] groups) = map { [at] $_} [at] { $dbh->selectall_arrayref("select distinct
org from groups") };

but, i didn't want to confuse the subject. just thought i'd post this
as an example of map's usefulness.

use Data::Dumper to figure out what you're getting returned and then
use map to put it in the format that you prefer.

though, Uri's explanation was much better than mine IMHO :)
but, as he said, whatever you can do with map, you can do with a
foreach loop and vise versa. whenever you find yourself writing a
foreach loop, just remember that you may want to consider map (i don't
know the benchmarks, i just use it for style at this point).

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

Dr.Ruud wrote:
> On 2011-01-08 03:16, S.F. wrote:
>
>> I have a data file with n columns and r row.
>> The first 3 columns and the first 5 rows are:
>>
>> 2 3 1
>> 1 6 X
>> 4 0 X
>> X 8 X
>> 5 X 5
>>
>> The "X" means missing.
>> How could I write a script to calculate the average by column and
>> replace "X" with the average?
>>
>> The output should be like:
>>
>> 2 3 1
>> 1 6 3
>> 4 0 3
>> 3 8 3
>> 5 4.25 5
>
> perl -MData::Dumper -ale'
> [at] d=map[do{ [at] t=split;$n=$s=0;$n++,$s+=$_ for grep$_ ne"X", [at] t;$_=$s/$n for
> grep$_ eq"X", [at] t; [at] t}],split",",$ARGV[0];
> print Dumper\ [at] d
> ' "2 1 4 X 5,3 6 0 8 X,1 X X X 5"

Why the do{} block? Just use map's block. And why include the -a
switch when you are not using [at] F? And you don't really need the -l
switch either because Data::Dumper already provides its own newlines.

perl -MData::Dumper -e'
[at] d=map{my [at] t=split;$n=$s=0;!/X/and$n++,$s+=$_ for [at] t;/X/and$_=$s/$n
for [at] t;\ [at] t}split/,/,$ARGV[0];
print Dumper\ [at] d
' "2 1 4 X 5,3 6 0 8 X,1 X X X 5"



John
--
Any intelligent fool can make things bigger and
more complex... It takes a touch of genius -
and a lot of courage to move in the opposite
direction. -- Albert Einstein

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

shawn wilson wrote:
> On Tue, Jan 11, 2011 at 4:57 AM, Sean Murphy<mhysnm1964 [at] gmail.com> wrote:
>> Hi all.
>>
>> I have read the explaination of the Map function and it is still a mystry to
>> myself on what it is for and when would you use it?
>>
>> All explainations I have seen in books and blogs don't make it clear.
>>
>
> my [at] new = map {$_} [at] arr;
>
> is the same as:
>
> foreach ( [at] arr) {
> push $_, [at] new;
> }

ITYM

foreach ( [at] arr) {
push [at] new, $_;
}

Trying to use a scalar as the first argument to push will result in an
error message.



John
--
Any intelligent fool can make things bigger and
more complex... It takes a touch of genius -
and a lot of courage to move in the opposite
direction. -- Albert Einstein

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

--20cf3054ac85314807049992edb2
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jan 11, 2011 at 10:19 AM, John W. Krahn <jwkrahn [at] shaw.ca> wrote:

> foreach ( [at] arr) {
> push [at] new, $_;
> }
>
> Trying to use a scalar as the first argument to push will result in an
> error message.
>
>
Doesn't make the original any less wrong, but as an aside, that's no longer
true on 5.13.7+ Perls. http://www.effectiveperlprogramming.com/blog/756

Brian.

--20cf3054ac85314807049992edb2--

On 11/01/2011 14:30, Brian Fraser wrote:
>
> On Tue, Jan 11, 2011 at 10:19 AM, John W. Krahn<jwkrahn [at] shaw.ca> wrote:
>
>> foreach ( [at] arr) {
>> push [at] new, $_;
>> }
>>
>> Trying to use a scalar as the first argument to push will result in an
>> error message.
>>
>>
> Doesn't make the original any less wrong, but as an aside, that's no longer
> true on 5.13.7+ Perls. http://www.effectiveperlprogramming.com/blog/756

This being a beginner's list, I imagine John was simply trying to avoid
any confusion that the mistake may have caused. I can see no other
errors in Shawn's post and don't understand how it can otherwise be made
'less wrong'.

The link you supply talks about recent versions of push, pop, etc.
working on array references as well as arrays, so that the code could
also be written

my $new_ref = \ [at] new;

foreach ( [at] arr) {
push $new_ref, $_;
}

That is a long way from being able to provide any scalar value as the
first parameter of push.

Rob

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

--20cf30434548141a050499974be8
Content-Type: text/plain; charset=ISO-8859-1

Rob, I'm going to answer the middle part of your mail first, hopefully you
won't mind:

> On Tue, Jan 11, 2011 at 3:56 PM, Rob Dixon <rob.dixon [at] gmx.com> wrote:


> The link you supply talks about recent versions of push, pop, etc.
> working on array references as well as arrays, so that the code could
> also be written
>
> my $new_ref = \ [at] new;
>
> foreach ( [at] arr) {
> push $new_ref, $_;
> }
>
> That is a long way from being able to provide any scalar value as the
> first parameter of push.
>
>
push my $scalar, {an => 'example'}, [1..10], 'Yeah!!!!!';

The arrayref is autovivified.
Of course, that still breaks down if there's any defined value in the scalar
- But that wasn't the point of my reply.

This being a beginner's list, I imagine John was simply trying to avoid
> any confusion that the mistake may have caused. I can see no other
> errors in Shawn's post and don't understand how it can otherwise be made
> 'less wrong'.
>
He is using a scalar as the first argument of push. In Perls before 5.13.7,
that's illegal. In newer Perls, if the scalar is an arrayref or is
undefined, that actually works, but is otherwise still illegal.
The first is -always- wrong. The second is -occassionally- wrong. shawn's
for example is perfectly legal, working Perl, if [at] arr is an array of
arrayrefs.

Brian.

--20cf30434548141a050499974be8--

On 2011-01-11 11:44, Uri Guttman wrote:

> these are equivilent:
>
> my [at] out = grep /foo/, [at] in ;
> my [at] out = map { /foo/ ? $_ : () } [at] in ;

Indeed equivalent. There is a difference though
in what comes out of map and grep:

perl -wle'
my [at] in = 1 .. 4;
print " [at] in";

++$_ for map { $_ % 2 ? $_ : () } [at] in; # copies
print " [at] in";

++$_ for grep { $_ % 2 } [at] in; # aliases
print " [at] in";
'

1 2 3 4
1 2 3 4
2 2 4 4


--
Ruud

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

On 2011-01-12 18:08, Uri Guttman wrote:
> Ruud writes:
>> On 2011-01-11 11:44, Uri Guttman wrote:

>>> these are equivilent:
>>>
>>> my [at] out = grep /foo/, [at] in ;
>>> my [at] out = map { /foo/ ? $_ : () } [at] in ;
>>
>> Indeed equivalent. There is a difference though
>> in what comes out of map and grep:
>>
>> perl -wle'
>> my [at] in = 1 .. 4;
>> print " [at] in";
>>
>> ++$_ for map { $_ % 2 ? $_ : () } [at] in; # copies
>> print " [at] in";
>>
>> ++$_ for grep { $_ % 2 } [at] in; # aliases
>> print " [at] in";
>> '
>>
>> 1 2 3 4
>> 1 2 3 4
>> 2 2 4 4
>
> interesting and evil. my rule is grep/map should never do side effects
> and do read only work. if you want modifications use a proper loop.

With map and grep I prefer a functional coding style too. Still it helps
to know implementation details like aliasing.

--
Ruud

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

>>>>> "R" == Ruud <rvtol+usenet [at] isolution.nl> writes:

R> On 2011-01-12 18:08, Uri Guttman wrote:
>> Ruud writes:
>>> On 2011-01-11 11:44, Uri Guttman wrote:

>>>> these are equivilent:
>>>>
>>>> my [at] out = grep /foo/, [at] in ;
>>>> my [at] out = map { /foo/ ? $_ : () } [at] in ;
>>>
>>> Indeed equivalent. There is a difference though
>>> in what comes out of map and grep:
>>>
>>> perl -wle'
>>> my [at] in = 1 .. 4;
>>> print " [at] in";
>>>
>>> ++$_ for map { $_ % 2 ? $_ : () } [at] in; # copies
>>> print " [at] in";
>>>
>>> ++$_ for grep { $_ % 2 } [at] in; # aliases
>>> print " [at] in";
>>> '
>>>
>>> 1 2 3 4
>>> 1 2 3 4
>>> 2 2 4 4
>>
>> interesting and evil. my rule is grep/map should never do side effects
>> and do read only work. if you want modifications use a proper loop.

R> With map and grep I prefer a functional coding style too. Still it
R> helps to know implementation details like aliasing.

and it actually makes some odd sense when you know that for modifier
does aliasing too (like for loops). it was added to be the proper way to
do side effects on a list (which map isn't meant to do). too many used
map for loops without using the return list. why grep uses aliases is
not clear but it could be historical and for efficiency reasons. map
wasn't changed to aliases since we have for modifier. anyhow, keeping
their usage functional is proper.

uri

--
Uri Guttman ------ uri [at] stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/

>>>>> "R" == Ruud <rvtol+usenet [at] isolution.nl> writes:

R> On 2011-01-11 11:44, Uri Guttman wrote:
>> these are equivilent:
>>
>> my [at] out = grep /foo/, [at] in ;
>> my [at] out = map { /foo/ ? $_ : () } [at] in ;

R> Indeed equivalent. There is a difference though
R> in what comes out of map and grep:

R> perl -wle'
R> my [at] in = 1 .. 4;
R> print " [at] in";

R> ++$_ for map { $_ % 2 ? $_ : () } [at] in; # copies
R> print " [at] in";

R> ++$_ for grep { $_ % 2 } [at] in; # aliases
R> print " [at] in";
R> '

R> 1 2 3 4
R> 1 2 3 4
R> 2 2 4 4

interesting and evil. my rule is grep/map should never do side effects
and do read only work. if you want modifications use a proper loop.

uri

--
Uri Guttman ------ uri [at] stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe [at] perl.org
For additional commands, e-mail: beginners-help [at] perl.org
http://learn.perl.org/