I've been paying attention to my web logs lately.
Today I noticed a referring site that actually had a fair amount of people clicking through and guess what I found?
I've been blogged! How exciting is that?
This is way more fun than I imagined.
You can see that post here: http://g-blog.net/user/dsh/entry/31433
I finally started advertising Bay Area Sold Homes and I’ve got some consistent traffic. There have been some times when I haven’t advertised, and I still have consistent traffic!
There are definitely some small things I would still like done to give the site a more ‘polished’ look before I went live. But I started to realize that I needed to get this started already.
The functionality is tight and working perfectly. The interface can be improved more, and it will be soon enough. The layout/design needs to be improved too.
I tried hiring somebody to do GUI design and I didn’t find anybody that conveyed their understanding of flat HTML using CSS. Everybody was using Flash.
I’m about 4 days away from finishing the second pass of data, including better geocoding. The data that is live now is missing about 12% of the entries due to geocoding issues on the first pass.
Well, I guess it’s time to tell a little about the project.
Basically it all started when we were in the process of buying a house and I was dismayed at the tools available to compare and ‘see’ the homes being sold, or those already being sold. I’m a data person and what was available just didn’t work for me.
I eventually ran across SFGates Homesales which lists homes sold data in a tidy, but pretty much useless, list format. I also had been playing with the Google Maps API a bit (mapping skate parks with pictures and directions).
So, I decided to start playing with the data. It started out using Ruby to scrape the data, geocoder.us to geocode the addresses, into XML files that a Javascript front end was pulling and parsing for the map. (Ruby is the bomb, you should check it out sometime) But each XML file consisted of one weeks worth of data for one county. So looking at say a months worth of data for the entire bay area was about 36 XML files to be grabbed, parsed, filtered, and displayed. Which as you can guess was pretty slow, especially considering if you were filtering for a particular price range, you still had to load all that data.
And then I got busy with other parts of my life.
Recently, I acquired a new job. It seemed like a great place, but it was a terrible fit for me. (It had nothing to do with the people there. It was other reasons.) So I quit.
And suddenly had some time to play!
I built a personal copy of the TIGER/line data from the US census. Now I have a CGI script that will geocode fast and I’ve learned some Perl in the process. I have a bunch of PHP that took those XML files, scraped them and put them into a mySQL database which the front end uses.And it’s so much faster. (Like, dah!) I added some filters and have been toying with other parts too. Oh, and I’m using AJAX to get the data to the front end. PHP for everything back end. I fought with bringing XML to the front using AJAX but it basically sucked. And then replaced the XML with JSON and all I have to say is WOW! Stop fighting with XML and just use JSON. It’s so much easier.
Incase you came here another way. The project is mapping sold homes in the Bay Area and can be found at http://www.BayAreaSoldHomes.com