Dear Diary,

It has been a very long time since we last spoke. I have been extremely busy with classes/papers/exams over the last few months.

Luckily, Classes are now over and I can get started on my research, woot woot!

I began today by looking around for a way to automate Google searches using a Java program. I was able to find one method, but it has a few limitations. According to the Google terms and conditions, you are not supposed to automate searches without permission.

Google is pretty good at noticing if a search query was automated so I had to structure my Java program to look like it was creating the request from Mozilla, even though it obviously isn't. This method will work for my preliminary stuff but I will certainly have to find an alternate method later on.

As of right now I have a Java program that takes a string from standard in and uses that string to search Google. The results are returned as html meta-deta. So then, I take this html and save it to a file, named according to the IP of the webserver in our database. This file is saved as .html currently so you can simply double click any of the saved files and it will open on your local machine with the exact results Google returned.

Things to be done: I have to modify this program to query against the database and pull a section of text from each page to use as our search criteria. From this, the results should be saved in a more efficient way. Saving each results as an HTML file is not ideal, it takes up a lot of space. Once this has been completed I need to run tests to determine exactly how long the character string needs to be so that we get accurate results. To begin I will pull an 80 character string from each page and use that as our search criteria. I will then scale it back, by 5 characters at a time, until a good threshold is found.

Yours Always,

Joseph O'Neill.