ForkFree

Developer tools

opencompass

by open-compass

OpenCompass is a Python platform for testing how well large language models perform. It runs models from providers such as OpenAI, Anthropic and Gemini, or local open-source models, against more than 100 benchmark datasets and summarises the results.

Proof level: ReadWorth a lookApache-2.0

Get it on GitHubProject websiteCompare

No demo yet

Proof level: Read. We read its README and licence and ran our automatic checks. We have not run it yet. So this page is based on its README and our automatic checks.

The catch

Worth a lookEstablished, with a few things to check (listed below).

Check this first: Evaluating hosted models needs paid API keys and can incur significant usage costs.

Things to check

  • Evaluating hosted models needs paid API keys and can incur significant usage costs
  • Running local open-source models usually needs a GPU, though the tool itself does not
  • Aimed at technical users; setup and configuration are not beginner-friendly

Can you use opencompass commercially?

Free to use, change and sell, including in paid products. Keep the copyright notice.

Not legal advice. Read the project's licence file before building a product on it.

What can you do with opencompass?

  • Compare several hosted or local language models on the same benchmarks for knowledge, reasoning, coding or safety
  • Run evaluations using zero-shot, few-shot, chain-of-thought or LLM-as-judge methods and get summarised results
  • Add your own models or datasets and run large evaluations in parallel across several machines

How do you get opencompass running?

It is a Python package installed following the installation guide in the documentation, with configuration files used to choose models and datasets.

Follow the official instructions in the project README.

Make money with it

A freelancer could offer model comparison and evaluation reports to companies choosing or fine-tuning language models.

An idea, not financial advice. Check the licence before charging for anything built on it.

Ways a business could use it

  • Test several AI models on your own tasks to pick the cheapest one that performs well enough before committing to a provider
  • Check quality and safety of a fine-tuned or customised model before releasing it in a product or client project

Who's talking about it

Checked 09 Nov 2026. Summary drafted by AI from the project's README. Proof level 1 of 4, Read: we read its readme and licence and ran our automatic checks. we have not run it yet.

Similar tools in Developer tools

taste-skill Leonxlnx
Worth a look

Taste Skill is an add-on for AI coding agents such as Claude Code, Codex and Cursor. It guides the agent to design more distinctive websites and app screens by choosing an art direction, using credited real photographs and checking its own work in a real browser.

Proof level: Read 119.1k stars +646 a day MIT
gpu-unlock Zagitta
Check the licence

This tool unlocks the FP/BF16 instruction issue rate on Nvidia 3000-series cards, doubling the speed of certain operations.

Proof level: Read 20.5k stars +449 a day Unclear
ui vercel-labs
New and promising

UI SDK is a code library that handles the behaviour of interface parts such as keyboard navigation, focus, popups, drag and drop and accessibility markup. It works with React, Vue, Svelte, Solid and Web Components, and is designed so AI coding agents can build components with it.

Proof level: Read 6.2k stars +485 a day Apache-2.0