luigi
by spotify
Luigi is a Python package for building pipelines of batch jobs, where each task can depend on other tasks. It handles ordering, failures and progress tracking, and includes a web page to see what has run.
- Stars
- 18.8k
- Licence
- Apache-2.0
- Runs on
- Your computer
- Setup
- Moderate
- Language
- Python
- Age
- 5155 days
What you can do with it
- Chain data tasks such as database exports, scripts and Spark or Hadoop jobs into one automated workflow
- Rerun a failed pipeline so it skips finished steps and carries on from where it stopped
- View task dependencies and progress in a web page and search through all your tasks
Make money with it
A freelancer could build and maintain automated data pipelines and reporting for small clients as a paid service.
An idea, not financial advice. Check the licence before charging for anything built on it.
Ways a business could use it
- Automate nightly reporting, such as pulling sales data from a database and building summary files, without hand-running scripts
- Replace a fragile set of scheduled scripts with one pipeline that retries safely and shows what failed
Can you use it commercially?
Free to use, change and sell, including in paid products. Keep the copyright notice.
Not legal advice. Read the project's licence file before building a product on it.
Getting it running
Install it with "pip install luigi", then write your tasks as Python code and run them from the command line.
Follow the official instructions in the project README.
Things to check
- Requires Python programming knowledge; there is no no-code interface
- Does not schedule runs by itself, so you need cron or a similar tool to trigger it
- Built mainly for large Hadoop-style workloads, so it may be more than a small team needs
Checked 02 Nov 2026. Summary drafted by AI from the project's README.
