How to Use AnyCrawl for Scraping Prices from Websites
How to Use AnyCrawl for Scraping Prices from Websites
If you've ever tried to manually track prices across dozens of websites, you know how tedious it gets. Copy-paste, copy-paste, over and over. That's exactly why price scrapers exist—and why we built AnyCrawl to make scraping prices from websites as painless as possible.
In this guide, we'll walk through building your own price scraper, from a simple single-page extraction to a full-blown price monitoring system.
Why Bother with a Price Scraper?
Here's the thing: prices change constantly. Your competitors adjust theirs, suppliers update quotes, and deals come and go. Keeping up manually just doesn't scale.
A good price scraper tool lets you:
- Track competitors — Know when they drop prices before your customers do
- Find deals — Automatically spot discounts across multiple retailers
- Monitor suppliers — Get alerts when procurement costs change
- Do fare scraping — Track flights, hotels, or rental cars for travel apps
- Power dynamic pricing — Adjust your own prices based on market data
Getting Started
You'll need an AnyCrawl API key. Grab one at anycrawl.dev if you haven't already.
The Simple Approach: JSON Mode
The easiest way to scrape prices is using AnyCrawl's JSON mode. Instead of writing CSS selectors or XPath queries (which break every time a site updates), you just tell the AI what data you want. It figures out the rest.
Here's a basic price scraper in JavaScript:
const extractPrice = async productUrl => {
const response = await fetch('https://api.anycrawl.dev/v1/scrape', {
method: 'POST',
headers: {
Authorization: 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: productUrl,
engine: 'playwright',
formats: ['json'],
json_options: {
schema: {
type: 'object',
properties: {
product_name: { type: 'string' },
current_price: { type: 'number' },
original_price: { type: 'number' },
currency: { type: 'string' },
in_stock: { type: 'boolean' },
},
},
user_prompt:
'Extract the product name, current price, original price if on sale, currency, and whether it is in stock',
},
}),
});
const result = await response.json();
return result.data.json;
};
// Try it out
const data = await extractPrice('https://example.com/product/laptop-pro');
console.log(data);
// { product_name: "Laptop Pro 15", current_price: 999.99, original_price: 1299.99, currency: "USD", in_stock: true }
Same thing in Python:
import requests
def extract_price(product_url):
response = requests.post(
'https://api.anycrawl.dev/v1/scrape',
headers={
'Authorization': 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json'
},
json={
'url': product_url,
'engine': 'playwright',
'formats': ['json'],
'json_options': {
'schema': {
'type': 'object',
'properties': {
'product_name': {'type': 'string'},
'current_price': {'type': 'number'},
'original_price': {'type': 'number'},
'currency': {'type': 'string'},
'in_stock': {'type': 'boolean'}
}
},
'user_prompt': 'Extract the product name, current price, original price if on sale, currency, and whether it is in stock'
}
}
)
return response.json()['data']['json']
data = extract_price('https://example.com/product/laptop-pro')
print(data)
That's it. No DOM parsing, no regex nightmares. The AI handles different price formats ($1,299.99, €999, £1.299,00) automatically.
Monitoring Multiple Products
Scraping one product is nice, but the real value comes from monitoring many at once. Here's how to build a simple price scrapper that checks multiple URLs:
const monitorPrices = async urls => {
const promises = urls.map(url =>
fetch('https://api.anycrawl.dev/v1/scrape', {
method: 'POST',
headers: {
Authorization: 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json',
},
body: JSON.stringify({
url,
engine: 'cheerio', // faster for static pages
formats: ['json'],
json_options: {
schema: {
type: 'object',
properties: {
product_name: { type: 'string' },
price: { type: 'number' },
currency: { type: 'string' },
},
},
user_prompt: 'Extract product name, price, and currency',
},
}),
}).then(res => res.json())
);
const results = await Promise.all(promises);
return results.filter(r => r.success).map(r => r.data.json);
};
// Compare prices across competitors
const competitors = [
'https://store-a.com/widget',
'https://store-b.com/widget',
'https://store-c.com/widget',
];
const prices = await monitorPrices(competitors);
console.log(prices);
Crawling Entire Catalogs
Sometimes you need to scrape prices from an entire product category—not just individual URLs. That's where the Crawl API comes in. It discovers pages automatically and extracts data from each one.
This is especially useful for fare scraping (think flight comparison sites) or building price databases:
const crawlCatalog = async startUrl => {
const response = await fetch('https://api.anycrawl.dev/v1/crawl', {
method: 'POST',
headers: {
Authorization: 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: startUrl,
engine: 'playwright',
strategy: 'same-domain',
max_depth: 3,
limit: 100,
include_paths: ['/products/*', '/shop/*'],
exclude_paths: ['/cart/*', '/checkout/*'],
scrape_options: {
formats: ['json'],
json_options: {
schema: {
type: 'object',
properties: {
product_name: { type: 'string' },
price: { type: 'number' },
category: { type: 'string' },
sku: { type: 'string' },
},
},
user_prompt: 'Extract product name, price, category, and SKU',
},
},
}),
});
const { data } = await response.json();
return data.job_id; // crawl runs async, poll for results
};
Building a Price Monitoring System
Let's put it all together. Here's a more complete price scraping software that tracks prices over time and detects changes:
class PriceMonitor {
constructor(apiKey) {
this.apiKey = apiKey;
}
async scrapePrice(url) {
const response = await fetch('https://api.anycrawl.dev/v1/scrape', {
method: 'POST',
headers: {
Authorization: `Bearer ${this.apiKey}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
url,
engine: 'playwright',
formats: ['json'],
json_options: {
schema: {
type: 'object',
properties: {
product_name: { type: 'string' },
price: { type: 'number' },
currency: { type: 'string' },
in_stock: { type: 'boolean' },
},
},
user_prompt: 'Extract product name, price, currency, and stock status',
},
}),
});
const result = await response.json();
if (!result.success) throw new Error(result.error);
return {
...result.data.json,
url,
scraped_at: new Date().toISOString(),
};
}
async checkAll(urls) {
const results = await Promise.allSettled(urls.map(url => this.scrapePrice(url)));
return results.filter(r => r.status === 'fulfilled').map(r => r.value);
}
detectChanges(previous, current) {
const changes = [];
for (const item of current) {
const prev = previous.find(p => p.url === item.url);
if (prev && prev.price !== item.price) {
changes.push({
product: item.product_name,
url: item.url,
old_price: prev.price,
new_price: item.price,
change: (((item.price - prev.price) / prev.price) * 100).toFixed(1) + '%',
});
}
}
return changes;
}
}
// Usage
const monitor = new PriceMonitor('YOUR_API_KEY');
const urls = ['https://store1.com/headphones', 'https://store2.com/headphones'];
const prices = await monitor.checkAll(urls);
// Save to database, compare with yesterday's prices, send alerts, etc.
Picking the Right Engine
AnyCrawl supports three scraping engines. Which one you use affects speed and cost:
| Engine | Best for | Speed | Cost |
|---|---|---|---|
| Cheerio | Static HTML pages | Fastest | Lowest |
| Playwright | JavaScript-heavy sites | Medium | Medium |
| Puppeteer | Chrome-specific needs | Medium | Medium |
Most e-commerce sites these days use JavaScript to render prices, so you'll usually want Playwright. But if you're scraping a simple static site, Cheerio is faster and cheaper.
// Static site? Use cheerio
{ engine: 'cheerio' }
// Modern e-commerce site? Use playwright
{ engine: 'playwright' }
Handling Edge Cases
Different Price Formats
Websites display prices in all sorts of ways: $1,299.99, €999, £1.299,00, "From $99", "Starting at $49/mo". The AI extraction handles most of these automatically, but you can be more specific:
json_options: {
user_prompt: 'Extract the main price as a number. If there is a range, use the lowest price. Ignore shipping costs.'
}
Geographic Pricing
Some sites show different prices based on your location. Use a proxy to see prices from specific regions:
{
url: 'https://global-store.com/product',
engine: 'playwright',
proxy: 'http://us-proxy:8080'
}
Slow-Loading Pages
Some sites take forever to load prices (looking at you, travel booking sites). Increase the timeout and wait time:
{
url: 'https://slow-site.com/flights',
engine: 'playwright',
wait_for: 5000,
timeout: 60000
}
A Few Things to Keep in Mind
Web price scraping is powerful, but use it responsibly:
- Check robots.txt — Some sites explicitly disallow scraping
- Don't hammer servers — Add delays between requests if you're scraping a lot
- Cache when possible — No need to re-scrape if prices haven't changed
- Respect ToS — Some sites prohibit automated access in their terms
Wrapping Up
Building a price scraper used to mean wrestling with CSS selectors, handling JavaScript rendering, and constantly fixing broken scrapers when sites updated their HTML. With AnyCrawl's JSON mode, you just describe what data you want and let the AI figure out the rest.
Whether you're tracking a handful of competitor products or building a full price scraping software for thousands of SKUs, the approach is the same: define your schema, point at the URLs, and let AnyCrawl do the heavy lifting.
Ready to try it? Sign up at anycrawl.dev and start with a single product URL. Once you see how easy it is, you'll wonder why you ever did it manually.
Questions? Hit us up at [email protected].
