crawl4ai
by unclecode
Crawl4AI is a web crawler and scraper that turns web pages into clean Markdown or structured JSON that AI tools can read easily. It is meant for feeding website content into AI agents, search-based question answering (RAG) and data pipelines.
- Stars
- 86.4k +42 a day
- Licence
- Apache-2.0
- Runs on
- Your computer
- Setup
- Moderate
- Language
- Python
- Age
- 901 days
What you can do with it
- Convert any web page into clean Markdown ready for an AI model
- Extract structured JSON data from sites using CSS, XPath or LLM-based rules
- Crawl whole sites in depth, including JavaScript-heavy pages, and take screenshots or PDFs
Make money with it
A freelancer could offer data-collection or website-to-chatbot setup services to clients, using it to gather and clean the source content.
An idea, not financial advice. Check the licence before charging for anything built on it.
Ways a business could use it
- Collect competitor prices, product listings or news into structured data without paying for a scraping service
- Build a searchable knowledge base or chatbot from your own website or documentation
Can you use it commercially?
Free to use, change and sell, including in paid products. Keep the copyright notice.
Not legal advice. Read the project's licence file before building a product on it.
Getting it running
Install with pip (pip install -U crawl4ai, then run crawl4ai-setup to fetch the browser), or run it as a Docker server; a hosted cloud version is also offered.
Follow the official instructions in the project README.
Things to check
- Scraping sites may breach their terms of service or robots rules, so check before crawling
- Stealth and proxy features for getting past bot detection carry legal and ethical risks
- LLM-based extraction needs your own paid API keys unless you use the hosted cloud
- The hosted cloud has usage limits and paid plans from $10 a month
Checked 27 Oct 2026. Summary drafted by AI from the project's README.
