Hey I saw your site! Looks like we had the same idea at the same time. I think we put them out only about a week apart. I like what you're doing with the PIDs. That would certainly be helpful with the github format as I sometimes like to compare my values to past values.

Blackouts done pretty well. Only thing I've broken was the top plate. I essentially folded it where the plate for the FPV camera sits. Thanks for fixing the prices! Are you scraping the URLs as people put them in?

Yeah, totally. fpvracing.tv added builds around the same time too. I'm really interested in the social aspect of your site. I'm having trouble getting people to come back to my site now. Would love to compare notes on our implementations/ideas. All of my site is open source here: https://github.com/rcbuild-info including all of the parts I know about (in the parts repo.) I also hang out in #cleanflight on Freenode. Good luck with your site!

Yes, and I need to add rules for each new store that gets added. Sometimes my default rules work, but most of the time these stores don't use conventions to label their content. Fpvracing.tv seems to have a nice system, very similar to what we've got here. Oh boy, I haven't been on IRC for ages. I also run a site that has deep roots in IRC called Hawkee.com.

Interesting! Do you deduplicate them too? The only conventions I've seen are the itemprops for schema.org. Hawkee looks like the roots of this site. :-) They look well designed. My spiders are here: https://github.com/rcbuild-info/scrape/tree/master/rcbi/rcbi/spiders

No, I do not de-duplicate them at the moment. I did build a price comparison engine a few years ago, but I'm not taking that route with this. I don't think I want this to turn into a product engine. I had a look at your scrapers and CrawlSpider looks pretty neat. I've done spiders like that in the past with PHP simple dom parser, but at the moment I'm only using QueryPath for single page scrapes. And yes, I use itemprop, but it doesn't always work.

Yeah, I've gotten pretty fast with writing the scrapers. Splitting out the manufacturer and grooming the data are the hardest parts.

I look forward to seeing what comes next for your site. For me I'm going towards a bit more build details (pics) and then flight details probably. I'm also starting a hardware project too.