Eval runs your model on Arabic and GCC question sets, scores each answer and ranks the models you test.
A team picks a model. Nobody can show the test that supports the choice.
Your services are in Arabic and about GCC government. The test must use the same content.
Each team tests in its own way. You cannot compare the results.
You supply the model under test. Eval reaches it through Anar Gateway.
Asks each question. Scores each answer by exact match, fuzzy match or a judge model.
A score for each model and each benchmark. Compare models side by side.
Anar supplies no models. You choose what to test.
questions in 8 benchmarks. 140 of them contain Arabic text.
Count from the product source, November 2026.
- Arabic language: comprehension, reasoning, summarisation and translation.
- Arabic safety: safety and personal-data detection.
- Government: GCC government knowledge and government service questions.
- Anar wrote these sets. They are not published benchmarks.
Choose a model and a benchmark. Eval runs the questions in the background.
Exact match or a judge model, set for each question. Fuzzy match is the fallback.
Compare models side by side. See the history of each model. Export results as JSON.
A ranking of models by score. After a restart, it can leave out older runs.
If the judge model fails, Eval scores with fuzzy match. The result does not say so.
A failed model call gets a score like an answer. It shows as a low score, not as an error.
The safety run does not reach Anar Guard today. On an error, it marks the answer safe.
Eval reaches any model that Anar Gateway serves. A list of direct endpoints is on the roadmap.
Today you cannot choose the judge model. A judge that you choose and pin is on the roadmap.
Today the question sets ship with the product. Your own question sets are on the roadmap.
| Capability | Status | Note |
|---|---|---|
| Benchmarks, runs, scoring, compare, export | Shipped | 8 sets, 147 questions. |
| Leaderboard | Partial | Can leave out older runs after a restart. |
| Judge and error handling | Partial | Fallbacks and model errors are not flagged. |
| Your own model endpoint | Partial | Through Anar Gateway only. |
| Safety run through Anar Guard | Partial | Does not reach Guard today. |
| Your own judge, your own questions | Roadmap | Planned. Not built. |
- Eight Arabic and GCC benchmarks
- Three scoring modes
- Compare, history and export
- Arabic text normalisation in scoring
- A list of your own model endpoints
- A record of each run that you can repeat
- Repeated runs with confidence intervals
- A judge model that you choose
- Gulf dialect questions
- Adapters for published Arabic benchmarks
Eval can send every model call and every judge call through the gateway when you set its address.
Eval can send its events to the Guard audit trail when you set the Guard address.
A one-month pilot: we deploy in your environment and run it with one team.
anarlabs.ai/contact