crawlee
by apify
Crawlee is a Node.js library for building web scrapers and browser automation scripts in JavaScript or TypeScript. It handles link queues, retries, proxy rotation and data storage, so you can collect pages, files and data from websites.
- Stars
- 26.3k +8 a day
- Licence
- Apache-2.0
- Runs on
- Your computer
- Setup
- Moderate
- Language
- TypeScript
- Age
- 3716 days
What you can do with it
- Crawl a website by following links and save page titles, prices or other data as JSON
- Control a headless Playwright or Puppeteer browser to scrape pages that rely on JavaScript
- Download files such as PDFs and images from websites, using rotating proxies for larger jobs
Make money with it
A freelance developer could offer custom data-collection or price-monitoring scrapers to clients on a project or subscription basis.
An idea, not financial advice. Check the licence before charging for anything built on it.
Ways a business could use it
- Collect competitor prices or product listings to inform your own pricing without paying for a data service
- Build lead lists or research datasets from public websites for your own use or for clients
Can you use it commercially?
Free to use, change and sell, including in paid products. Keep the copyright notice.
Not legal advice. Read the project's licence file before building a product on it.
Getting it running
Install it with npm (npm install crawlee playwright) or run npx crawlee create my-crawler to generate a starter project; it needs Node.js 22.13 or higher.
Follow the official instructions in the project README.
Things to check
- You need to write code, so it is not suited to non-developers
- Scraping may breach some websites' terms of service or data laws, so check before you collect
- Features that avoid bot detection raise ethical and legal questions
- Good proxies for large crawls are usually a paid extra
Checked 30 Oct 2026. Summary drafted by AI from the project's README.
