Dear Diary,

It has been a very long time since we last spoke. I have been extremely busy with classes/papers/exams over the last few months.

Luckily, Classes are now over and I can get started on my research, woot woot!

I began today by looking around for a way to automate Google searches using a Java program. I was able to find one method, but it has a few limitations. According to the Google terms and conditions, you are not supposed to automate searches without permission.

Google is pretty good at noticing if a search query was automated so I had to structure my Java program to look like it was creating the request from Mozilla, even though it obviously isn't. This method will work for my preliminary stuff but I will certainly have to find an alternate method later on.

As of right now I have a Java program that takes a string from standard in and uses that string to search Google. The results are returned as html meta-deta. So then, I take this html and save it to a file, named according to the IP of the webserver in our database. This file is saved as .html currently so you can simply double click any of the saved files and it will open on your local machine with the exact results Google returned.

Things to be done: I have to modify this program to query against the database and pull a section of text from each page to use as our search criteria. From this, the results should be saved in a more efficient way. Saving each results as an HTML file is not ideal, it takes up a lot of space. Once this has been completed I need to run tests to determine exactly how long the character string needs to be so that we get accurate results. To begin I will pull an 80 character string from each page and use that as our search criteria. I will then scale it back, by 5 characters at a time, until a good threshold is found.

Yours Always,

Joseph O'Neill.

Internet stats...

http://img229.imageshack.us/img229/9232/internetet3.png

Dear Diary,

I was doing some research on the Google API today and found some very interesting things. After searching through the API for a bit to see if I could do a website lookup via the IP address I decided to switch my focus onto Dr. Trudel's suggestion.

Dr. Trudel originally suggested taking the first 30-40 characters of a website and using that as the base of our search. I did a bit of thinking on this and decided to try searching for websites by their Title block.

I spent an hour or so going through random webpages and running their Title block through Google, to my surprise every website I ran came up as the first hit on Googles search result.

Exact steps I followed:

1)Pick a random website: www.fonora.com (a website I found on the back of a textile package in my house)
2) Right click and hit "View Page Source"
3) Scroll through and copy the [title] section of the source code (I cant show the actual html representation of <> title on this blog)
4) Go to google.com and insert the title section.
5) Encase the title section in quotes
6) Search

I am going to generate a larger list of test cases and try all of them.

Until then,

Joseph O'Neill

Dear Diary,

I met with Dr. Benoit, Dr. Trudel and the other master students again yesterday. I was able to ask Dr. Benoit and Dr. Trudel what they thought about my research topic (Checking Googles Accuracy).

They raised some potential issues, but I still think that it is worth exploring. If I am able to actually get the data I am looking for from Google I could potentially end up with some very interesting results.

Things to be done:

1) Check Google's API to determine if there is a way to search via IP

Yours Always

Joseph O'Neill

Dear Diary,

I spent some time over the weekend looking at the topics I could use for my research. The one that caught my attention was checking the accuracy of Google. When I first asked about the possible research topics Dr. Benoit had mentioned this particular one but said there might be some difficulties.

Things to do for Wednesday:

1) More research on checking Google's accuracy
2)Discuss the possibility and any difficulties with Dr. Benoit and Dr. Trudel.

Always yours

Joseph O'Neill

Dear Diary,

I met with Dr. Benoit and the masters students again today.

I have been going over the choices for my masters research and I am starting to lean towards some sort of analysis of web census data. After managing the Web Census for over a year, doing analysis on the data would be a nice change.

I plan on doing some research on the topics and will update you when I have decided

Love Always,

Joseph O'Neill

Dear Diary,

I had a meeting with Dr. Trudel, Dr. Benoit and all of the students working on the Web Census yesterday. We all talked about what we had done over the past week, unfortunately I had nothing to contribute.

This semester I am concentrating on my classes as opposed to research. Dr. Benoit mentioned that he would forward me a list of research topics related to the Web Census, which I thought was a good idea.

Unfinished Work:

- Choose a topic for my research

Joseph O'Neill

First post!

First post!