For open APIs this is possible but how to do with the closed ones ?
You can either dissect the inner workings of ajax calls (if the website has any) or write a regular full-page scraper which traverses the product catalog.
Popular choices for scraping are casperjs, beautiful soup, mechanize, etc. Each having its strengths and drawbacks - casper for instance can emulate the browser behavior, while the rest are much less resource intensive but often require going through extra hoops when the website is complicated.
Answered 11 years ago
Access 20,000+ Startup Experts, 650+ masterclass videos, 1,000+ in-depth guides, and all the software tools you need to launch and grow quickly.
Already a member? Sign in