--------------020303040806020700070005
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi everyone,

I am trying to use Mechanize to scrape information of my account page at
www.attwireless.com. am having problems signing on.

The commands to fill in the form all work without complaint:

$mech->form_name($formname);
$mech->field($usernameField,$loginID);
$mech->field($passwordField,$pswd);

However, when I submit the form, I don't get logged in, I come back to
the Login page (which is the target of the POST). There is no error
message on the login page, but it doesn't let me in either.

When I sign on with Firefox, everything works fine. The only difference
I have been able to determine, compared to the trace from LWP::Debug, is
that a cookie named phonenumberflag is set to "false" in LWP/Mechanize
and is "true" in Firefox. (Although I admit I don't know how to
create/interpret the dumps described in Colin Magee's Thread Viewing
exchange between browser and website.
<http://www.mail-archive.com/libwww [at] perl.org/msg05155.html>)

Anyone have any ideas on what the problem could be, how to find it or
how to fix it? Thanks in advance!

Peter Stevens

Sample Code:

#!/usr/local/bin/perl -w
######### ######### ######### ######### ######### ######### ######### #########
# bug.pl - log into AT_and_T user account
#
use strict;
use LWP::Debug qw(+);
use WWW::Mechanize;

############ start

sub load{

my ($loginID, $pswd) = [at] _ ;

my ($mech) ;

$mech = WWW::Mechanize->new( autocheck => 0 );

my $homepage="http://www.attwireless.com" ;

my $formname="LoginForm"; # from mech_dump/source
my $usernameField="/attws/ocx/userprofiling/OCXProfileFormHandl er.value.login" ;
my $passwordField="/attws/ocx/userprofiling/OCXProfileFormHandl er.value.password" ;

$mech->agent_alias ( "Windows IE 6" );
print STDERR "Get $homepage \n";
$mech->get( $homepage );
die "Can't even get $homepage: " . $mech->response->status_line unless ($mech->success) ;
print STDERR "Got $homepage as ". $mech->uri() . "\n";

print STDERR "fill in form...\n";
$mech->form_name($formname);
$mech->field($usernameField,$loginID);
$mech->field($passwordField,$pswd);
# print STDERR "Login ID is $loginID , Pass is $pswd\n";
print STDERR "submit form...\n";
$mech->submit();

die "Could not submit $homepage: ". $mech->response->status_line unless ($mech->success) ;
print STDERR "Login brought us to ". $mech->uri() . "\n";
if ( $mech->uri() =~ /Login/i ) {
print STDERR "Wrong answer\n";
}

# print $mech->content;
}


my ($parms, $i) ;
$parms = {} ;

foreach $i ( qw (loginID pswd) ) {
print "$i: " ;
$parms->{$i} = readline STDIN ;
chomp $parms->{$i};
}

load($parms->{"loginID"}, $parms->{"pswd"});
# end of bug.pl
######### ######### ######### ######### ######### ######### ######### #########

###########Dump from LWP::Debug (indented, prints are left flush):

LWP::UserAgent::new: ()
Get http://www.attwireless.com
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking www.attwireless.com for cookies
HTTP::Cookies::add_cookie_header: Checking .attwireless.com for cookies
HTTP::Cookies::add_cookie_header: Checking attwireless.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: GET http://www.attwireless.com
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 475 bytes
[ snip ] ...
LWP::Protocol::collect: read 1010 bytes
HTTP::Cookies::extract_cookies: Set cookie DSESSIONID => H0HZKOGKXE4AVB4R0HYSFEY
HTTP::Cookies::extract_cookies: Set cookie phonenumberflag => false
HTTP::Cookies::extract_cookies: Set cookie browserid => 1100175229559H0HZKOGKXE4AVB4R0HYSFEY
LWP::UserAgent::request: Simple response: OK
Got http://www.attwireless.com as http://www.attwireless.com
fill in form...
submit form...
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking www.attwireless.com for cookies
HTTP::Cookies::add_cookie_header: - checking cookie path=/
HTTP::Cookies::add_cookie_header: - checking cookie DSESSIONID=H0HZKOGKXE4AVB4R0HYSFEY
HTTP::Cookies::add_cookie_header: it's a match
HTTP::Cookies::add_cookie_header: - checking cookie phonenumberflag=false
HTTP::Cookies::add_cookie_header: it's a match
HTTP::Cookies::add_cookie_header: - checking cookie browserid=1100175229559H0HZKOGKXE4AVB4R0HYSFEY
HTTP::Cookies::add_cookie_header: it's a match
HTTP::Cookies::add_cookie_header: Checking .attwireless.com for cookies
HTTP::Cookies::add_cookie_header: Checking attwireless.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: POST https://www.attwireless.com/ocs/authenticate/Login.jhtml;dse ssionid=H0HZKOGKXE4AVB4R0HYSFEY?_DARGS=/home/left_frag_b2c.j html
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 658 bytes
LWP::Protocol::collect: read 1 bytes
[ snip ]
LWP::Protocol::collect: read 182 bytes
HTTP::Cookies::extract_cookies: Set cookie browserid => 1100175229559H0HZKOGKXE4AVB4R0HYSFEY
LWP::UserAgent::request: Simple response: OK
Login brought us to https://www.attwireless.com/ocs/authenticate/Login.jhtml;dse ssionid=H0HZKOGKXE4AVB4R0HYSFEY?_DARGS=/hom
e/left_frag_b2c.jhtml
Wrong answer

###########End Dump from LWP::Debug:




--------------020303040806020700070005--

Try using $mech->click instead of $mech->submit just for giggles... I cant
explain why but it often works for me when i find my submits bringing me
back to the exact same page with no errors or explanation.

--
gedanken

I'm still giggling :-) ! Worked like a charm. Thanks! I call that a
super "Gendankensaustauch"

Peter

Gedanken wrote:

>Try using $mech->click instead of $mech->submit just for giggles... I cant
>explain why but it often works for me when i find my submits bringing me
>back to the exact same page with no errors or explanation.
>
>
>

On Thu, 11 Nov 2004, Peter Stevens wrote:

Ive never been able to figure why mech->click and mech->submit are any
different... its like voodoo to me but it works heh. Ive used mechanize
on about 100+ sites now, and its popped up 2-3 times. Took me forever to
'figure it out' i.e pull a guess out of thin air that happens to work.

ged

I'm still giggling :-) ! Worked like a charm. Thanks! I call that a
super "Gendankensaustauch"

Peter

Gedanken wrote:

>Try using $mech->click instead of $mech->submit just for giggles... I cant
>explain why but it often works for me when i find my submits bringing me
>back to the exact same page with no errors or explanation.
>
>
>


--
gedanken

On Thu, Nov 11, 2004 at 10:07:34AM -0600, Gedanken (gedanken [at] io.com) wrote:
> On Thu, 11 Nov 2004, Peter Stevens wrote:
>
> Ive never been able to figure why mech->click and mech->submit are any
> different... its like voodoo to me but it works heh. Ive used mechanize
> on about 100+ sites now, and its popped up 2-3 times. Took me forever to
> 'figure it out' i.e pull a guess out of thin air that happens to work.

click() requires a button to click. submit() just submits the form.
Some sites require a certain button to have been clicked, rather than
just having the form submitted.

xoxo,
Andy

--
Andy Lester => andy [at] petdance.com => www.petdance.com => AIM:petdance