Show HN: Crawling car dealerships for a real time consumer search engine
I have built something very similar to this is the past, here are a couple of observations:
Scraping:
- Dealer web sites are run by a handful of different data brokers, for the most part if you find a good way to scrape one (say dealer who uses http://www.dealer.com/ than you can extend your scraper to get others)
- Dealer web sites, in general, are horrible to view.
- Learn to love VIN explosion/decoding - http://www.researchmaniacs.com/VIN/VIN-Decoder.html the dealers enter features in so many different ways, it is your best chance to normalize the data.
- Normalize the ext. color, create a distinct list of all the crazy colors of car makers give their cars and all the short hand dealers give the colors. Create a map of the colors and apply when scraping
- Scraping is like farming, a lot of initial work, but there is constant upkeep for changing sites
Display:
- User's don't want to search by city as much as what is close to them, you should geocode the dealerships and display distance. For instance if I search West Des Moines, I would expect inventory in Des Moines also to come up
- Add searching by zip code, you can easily find database of the centroids. It can also be a cheap way of geocoding the dealerships
- Switch Mileage to use miles instead of KM, it looks like most of the inventory is in the US an that is what the user will expect.
- Use a ip2geo to set the initial location of the search, right now it looks like it is all over the place, check to see if the browser supports geo location and optionally set the initial search by that
I think this is nice, and I already found a deal I haven't come across yet on a vehicle I'm interested in, so thanks!
A few initial thoughts:
- I can't click the radio buttons themselves within each options group; only the link/label itself is clickable
- I would rather the mileage filters be < 10k, < 20k, < 36k, etc. If I can't have this, I want to be able to select multiple mileage filters at once
- I want to select multiples of other filters too. Year for example. eBay allows me to enter "2009-2011" or "2009-". I can only view one year at a time on this site.
- It took me a minute to realize I had to select a make before a model (yes, I feel stupid for this); showing an empty stub where the options will appear once I have selected a make is not intuitive - how about not showing the not-yet-ready filters until they are relevant?
- Seeing the "Contact Dealer" red button directly below the phone number made me initially think I was going to be calling the dealer. I finally clicked it after seeing no other place to view the dealer's own listing. I'd make getting to the dealer's site a bit more prominent
A few thoughts: - I think CarFax has an affiliate program that might let you link through to their reports, keyed by VIN. These reports are a pretty valuable tool to shoppers as the condition of a used car is driving 50% of the buying decision.
- One data point that is really important to dealers is how long the car has been on the lot. Hypothetically you could just track how long the same car has appeared at the same dealership. The longer the car has been sitting there, the more motivated they'll be to sell it. I remember hearing 30 days is a long time for dealerships to sit on a car. For buyers this could be good information to have.
- Getting the mobile experience right would be a huge win. So often the cars the dealer lists aren't actually the cars they have on the site (I'm not sure how much of this is intentional and how much is a matter of how fast inventory turns over). So when you get to the dealership, you want to quickly and easily be able to comparison shop - get that right and you cover people in that really critical uncomfortable moment.
What about trim levels? Trim levels are crucial in the buying decision. For example, if you search for a Ford F-150 on your site, there is no way to see if it's an XLT, King Ranch, Raptor, etc. The price and equipment difference in those vehicles is huge.
What about options and equipment? Does the car have navigation? Sunroof? Most consumers have specific option packages in mind when searching for car online.
This is why VIN explosion is necessary for any serious automotive shopping site. If a consumer can't narrow the vehicles down to a trim and option package level then it won't get wide adoption.
Suggest a map that shows which dealer and where a car is located. There's a subset of car buyers where the car itself is less important than the garage they bought it from, in case they need to bring it back for repairs / tuning etc. This is especially true of used cars. And very important on mobile
Overall, impressive data, but UI/UX could be improved.
'Refine By' menu shows an arrow on mouseover, but user can only click on words, not arrow or high-lighted area.
'Refine By' pop-out menus show a square, value as hyperlink, count. Square looks like a checkbox (with rounded corners) but clicking on square produces no result (this user expected a checkbox response, & ability to select multiple models, colors, etc.)
Change of other filters does not require a 'go' button, but change of search radius does.
Results appear to have a wide radius, but pull-down for location does not show what default/pre-selected radius is (by experimentation, 10 miles).
Generally, the user should be able to tell what filters are currently active, and what their values are.
Clear Price filter action appeared to clear all filters.
US states and placenames with multi-word names should have all elements capitalized: s/New mexico/New Mexico/ s/San luis obispo/San Luis Obispo/
Price filter should allow either end-point to be absent.
Mileage should allow arbitrary range end-points, like price.
Year should allow a range, or checkboxes.
Consider having Refine by Make hide the less popular makes behind a 'more' button. So, by default display top N makes & 'more', 'more' displays top N2 (or all) makes & 'more', top N4 or all, top N*8 or all, etc. (otherwise the menu may grow to many dozens of obscure makes)
Support 'open in a new tab' on the hyperlinks show below the first search box. Searching <color> <make> <letter> displayed 3 links with counts, but displays nothing - sigh. (perhaps the back-end function is not yet implemented?)
Again, impressive data.
Great job! Nicely done.
A suggestion, also a source of competitive advantage: Allow selecting more than one Make per search. Seems nobody does this. I would like to see all SUVs except those by the big North American manufacturers. To do this I need to execute many separate searches.
Servers couldn't handle it.
Just a weird error. If I accept the location tracking and use this url: http://www.demanjo.com/new/search?3=3670447604&0=3526732...
Then it will show the search, but go to another search afterwards.
Searched for WRX, but got a bunch of Hondas. Looks like Honda is a fallback if model is not recognized...
It's clean and fast. How do you fetch the data and what's the revenue model? Cheers.
Funny, I made something very similar when I was looking to change my car 2 years ago or so, but did't open sourced it. The UI was awful, but it had email notifications when a search query matched
Seems very useful (looking for a car right now), but the filter by price range function seems to return zero results with no regard to the values entered.
How did you manage to come into agreement with car dealers to crawl their sites and use their photos in your aggregator? Congratulations!
I'm seeing a fair few duplicates in the results, probably need to work on the algorithm for filtering these out.
There is a similar site in Russia: http://auto.yandex.ru/
Is there any reference for designing classified websites? In terms of usability and experience.
How come there is no cars in San Francisco ? Yu should may be think of parsing craiglist ?
You should be able to click the image on the homepage to go to the listing.
Great job. Love the idea. Can you tell us which technologies you used ?