Ruby Forum Ruby > grabbing html from pages that require login

Hi ive searched the forums already but couldnt come up with any concrete
information.

Ive written a script that accesses certain pages on ebay and scapes
information. Its using open-uri to grab html and just standard regexps
to grab the data i want. When building the script I used ebay webpages I
saved to my harddisk. All worked fine.

However when the script attmepts to access the actual online webpages I
get an "file or directory not found ..." error.
What I think is happening is that ebay is automatically redirecting the
script to the user login. Even if im already logged in via firefox i
guess the fact that the request is coming from something else outside of
firefox makes ebay suspect its from another location and thus for
security reasons requests the user to reenter their detials. Im assuming
this is to do with cookies???

So how do get round this issue? Is there a way to get avoid this login
requirement or somehow providing my script with the info it needs to
login and continue from there?

You might be able to use httpclient for this.  It manages a session
much like a browser might and can track cookies that the site might
send.  I was able to use this for doing some pretty ugly manipulation
of some pop-up windows that was clearly intended to be used only by a
human with a browser.

   gem install httpclient

-Rob

On Aug 19, 2008, at 8:20 AM, Adam Akhtar wrote:

> However when the script attmepts to access the actual online  
> this is to do with cookies???
>
> So how do get round this issue? Is there a way to get avoid this login
> requirement or somehow providing my script with the info it needs to
> login and continue from there?
> -- 

Rob Biedenharn    http://agileconsultingllc.com
Rob@AgileConsultingLLC.com

Thank you for that rob, ill look into it now.

On Tue, Aug 19, 2008 at 12:19 PM, Rob Biedenharn
<Rob@agileconsultingllc.com> wrote:
> You might be able to use httpclient for this.  It manages a session much
> like a browser might and can track cookies that the site might send.  I was
> able to use this for doing some pretty ugly manipulation of some pop-up
> windows that was clearly intended to be used only by a human with a browser.
>
>  gem install httpclient

You may also wish to look at: mechanize, scrubyt, firewatir, etc.

Use mechanize, it makes logins very easy