Bay Area Sold Homes, my experience with it, and me

Since November 3^rd my search tips page has been broken. I’m not sure what happened, because it looks like the only code left in the page was some counter code. This is code that I have never used.

I apologize for the lack of search tips, but everything has been fixed now.

Also, I’ve worked out 99% of the bugs in the automated process that automagically updates records each Friday night. For now on you can check late Friday night, or early Saturday morning for that weeks sold properties now. The only exception to this is when the Chronicle forgets to put the data up on their website. (Which did happen a couple weeks ago.)

Posted by rainbow at 09:37 PM | Permalink | Comments (0) | TrackBacks (0)

I added a bunch of error handling and now everything about the data getting processed is automated using scheduled tasks, batch scripts and the lovely firefox.

This is pretty exciting because it means I won’t have to remember anything, and the

data will get updated automagically. It also means that the new data will appear predictably at the same time each week for you. No more wondering when you can check out the latest sales in your neighborhood.

Posted by rainbow at 05:44 PM | Permalink | Comments (0) | TrackBacks (0)

Well, it looks like whomever was sick last friday is there now. The data that was missing is there now!

Our data is updated now. Enjoy. 

Posted by rainbow at 12:49 PM | Permalink | Comments (0) | TrackBacks (0)

Well, that was a real bummer.

Today I used my own site, BayAreaSoldHomes.com, to check out the properties that are selling in my neighborhood, and there’s no data for this past week!

Ugh.

As it turns out, the San Francisco Chronicle never updated their information.

So, my automated process to update my data hummed along, and didn’t get any data because there wasn’t any. I suppose this is a lesson that I really do need better error handling. As it is, this script was thrown together just to get the automation started. And the funny thing is, I had just sat down tonight to fix it up some more.

The point is that there will be no more data until the Chronicle updates theirs.

Posted by rainbow at 06:42 PM | Permalink | Comments (0) | TrackBacks (0)

Data has been updated through 9/3/2006. It’s pretty strange considering I’m not even home, nor have I been logged into a computer for 6 days.

Let’s hope everything worked as expected.

Posted by rainbow at 08:53 AM | Permalink | Comments (0) | TrackBacks (0)

Data is updated through 8/27/2006.

Also, I finished the automation of the weekly update of sold homes. Lucky thing too, considering I’ll be at Burning Man through the 5^th.

Posted by rainbow at 08:51 AM | Permalink | Comments (0) | TrackBacks (0)

Data is updated through August 20^th, 2006.

I have exciting news! I passed the real estate sales person exam. I’m through the hard part of getting a real estate license. It’s pretty exciting. Hopefully, I’ll have access to the MLS soon, and before you know it there’ll be more useful data on the website.

Posted by rainbow at 05:03 PM | Permalink | Comments (0) | TrackBacks (0)

Sorry for the delay on getting the data updated this week. I took the Real Estate Exam this Monday morning and I’ve been studying for it like crazy. I think I did okay, and hopefully pretty soon I’ll have access to the MLS and be able to offer better information and statistics about the houses that I have listed on the site.

As a result of the studying I didn’t make sure the new data got up on Sunday. But it’s up now!

Posted by rainbow at 09:49 AM | Permalink | Comments (0) | TrackBacks (0)

Today I put up the Search Tips page. The content is a little rough, but it does provide some search tips. Also, the main page now has questions marks which link into the Search Tips page and describes certain functionality and uses for the area they clicked from. I hope you like it.

Eventually I will add some images and make it more obvious what I’m talking about.

Posted by rainbow at 10:39 PM | Permalink | Comments (0) | TrackBacks (0)

There’s a new layout that’s been pushed up.

I think it looks much better and it will include search tips and some words about the purpose of this website.

I’ve been working on this design for a while. It’s definitely been fun, though I have to tell you that sometimes it can be pretty hard to get everything just exactly as you want it. I know my CSS pretty well, but I not the super expert that might be required to solve the minor issues that I couldn’t get to work exactly perfectly. And wouldn’t you know it, but the problems were both non-compliance by IE. When are they going to follow the standards?

Why do they get away with this?

I hope you enjoy the new design. I know I do. I think it looks much better than the almost ugly old version.

Also, I changed the way the searching works. Before, the AJAX call was grabbing the results from a PHP page that would first go and count the number of records that were possible to be returned, and then it would hit the database again to get the actual records that were going to be returned to the front end. So, two queries that weren’t running asynchronous. Now they are two different AJAX calls and it runs faster. I find it interesting that sometimes the count query doesn’t beat the record set results query, and as a result the notation of how many pages could possibly be returned isn’t listed. But usually it is. There's something really interesting to me about the fact that sometimes it doesn’t return first and sometimes it does. I like slight randomness in programming. Alot.

Also, I built the ability to page through the record results. So, instead of just knowing that you’re not seeing all the results for a particular query, now you can actually flip through the pages, if you wanted. I’m curious to see how many people actually will do that.

You did know that I store the queries that people search on? I don’t recall if I posted that before. I haven’t had a chance to look at that data yet. But I’m excited to look at it and see where and what folks are searching on.

Below find the first version of BayAreaSoldHomes. Stored here for posterity’s sake.

Continue reading "New Design! Engine Changes!" »

Posted by rainbow at 03:15 PM | Permalink | Comments (0) | TrackBacks (0)

The data has been updated through 7/16/2006.

It was a little later than usual because we were out camping this weekend at Sonoma Lake. Its pretty fun and you can camp where it's boat in only camping. Though, it doesn't prevent the loud, obnoxious types from camping near you.

Hopefully soon I’ll finish getting the weekly process all automated so it won’t matter where I am on the weekends.

Posted by rainbow at 06:48 PM | Permalink | Comments (1) | TrackBacks (0)

I fixed the sorting algorithm on the table of results data for the site. Sometimes it would sort 800 sq ft as a greater number than 1,200 sq ft. I found a couple problems with the regular expressions I was using. Now everything sorts correctly, and I dare say a little faster too!

Also, today is Sunday, and as such the homes sold data for the past week has been posted. Data has been updated for the website. You can now search through 7/9/2006 on your quest for information.

Posted by rainbow at 09:34 AM | Permalink | Comments (0) | TrackBacks (0)

That’s right there’s 42,250 properties listed than there were mere minutes ago.

When I started this project I didn’t have adequate time to invest in the processes that were creating the data for this project. As a result, there wasn’t the best error handling and some of the other code was lacking.

Well, not anymore. The ‘second pass’ has been completed and with it there’s now 14% more sold properties listed and mapped. How cool is that!

That being said I still have 35,991 properties that I haven’t been able to geocode. Even with Yahoo’s geocoder. The addresses are too new, most likely new developments. It’s possible that there are some with erroneous addresses also.

Posted by rainbow at 03:15 PM | Permalink | Comments (0) | TrackBacks (0)

I was having some problems with my process that scrapes and geocodes the records.

So I was a week behind. But today we're all caught up again!

I'm going on a vacation for the rest of this month and am hoping to finish the process to automate the weekly process.

Data is complete through 6/4/2006. 

Posted by rainbow at 08:55 PM | Permalink | Comments (0) | TrackBacks (0)

Yup, the data's been updated again.
It would have happened faster had the family not been visiting. (But that was pretty fun.)

I also fixed a problem with the zip code filter.

Posted by rainbow at 03:29 PM | Permalink | Comments (1) | TrackBacks (0)

That's right! New data is here.

Watch those prices go up! 

Posted by rainbow at 11:59 AM | Permalink | Comments (0) | TrackBacks (0)

Posted by rainbow at 12:06 PM | Permalink | Comments (0) | TrackBacks (0)

The zip code field is now user entered and will move the map to that area.

The city drop down will also move the map centered on the city. I still need to add this to the county field.

The main page has a much better layout. (I’m still on the search for a nice looking template to make it look better.) And I finally figured out some div layout issues I was struggling with.

I’ve been looking for a decent date picker to replace the drop downs for date. But I haven’t found anything yet. That's the next thing to change once I find one.

Posted by rainbow at 11:27 AM | Permalink | Comments (0) | TrackBacks (0)

Posted by rainbow at 01:22 PM | Permalink | Comments (0) | TrackBacks (0)

Today Late last night I finished the first pass of all the data. The data gets ‘released’ each Sunday, so there will be more data today.

Per d$’s request there’s a new feature. When you put your mouse over the address, it’ll change the color to green if it’s in the viewable map, or red if it’s not.

Now it’s possible to be more specific with the range of prices to search on. The price range has been broken out of one restricting drop down to two now. (Another d$ request.)

There’s also a new filter for free form text. You can search on any part of the address:

• Street number, part of the street number
• Street name, part of the street name

I upgraded the Google Maps API to version 2 today. I started having some random problems that weren’t that consistent and seemed to be memory related (lags and such). Most likely caused from all the testing so I decided it was time to use the new. They even mentioned memory leaks on their website.

There were about 20 XML files that were in my log as being in error. (Each XML file was one date and county on the old system, now it’s getting put in a mySQL db.) I went through those and got them set up to processed.

Posted by rainbow at 01:02 PM | Permalink | Comments (0) | TrackBacks (0)

I’m still working on populating the data. There are so many data points. The real estate market around here is pretty crazy.
The data that just went in is the first pass of the following dates:

04.27.03 - 05.30.04
04.03.05 - 11.16.05

Which means we’re missing 05.30.04 - 04.03.05 and 11.16.05 – now (04.07.06)
There’s about 16 months of data to go. It should be done by tomorrow after noon.

What does ‘first pass’ mean? Well, I first geocode the address with a CGI script that I built (instructions can be found on geocoder.us). If the address isn’t found there I try geocoding using Yahoo’s Geocoding Service. However, sometimes I go over the 5000 permitted per day and it’s no longer an option until the next day.
The next pass will be using Yahoo's geocoder on all the addresses that didn't resolve on mine, and then maybe taking a look at some of the data. There was one address that I noticed that said West San Jose as it's city. I don't think that exists. So I might have to work some of that out another way.

Posted by rainbow at 04:47 PM | Permalink | Comments (0) | TrackBacks (0)