Arthur Evals Engine
Arthur Evals Engine is an open-source service you host that applies guardrail rules to LLM prompts and responses and collects traces for evaluation.
What Arthur Evals Engine does
Arthur Evals Engine runs as a service next to an application. The application sends it a prompt, and later the model's response, through validation endpoints, and the engine returns the result of each rule. Rules come in six types: keyword, regular expression, sensitive data, hallucination, prompt injection and toxicity. The hallucination rule compares the response with the context the application supplies, typically its retrieved documents. Default rules apply to every task, where a task is one LLM application or use case, and task rules apply to one.
According to the repository, the engine also collects OpenTelemetry traces in the OpenInference format, runs continuous evaluations on live traces, manages prompt versions and runs experiments comparing them. The hosted Arthur Platform adds dashboards, alerts, webhooks and governance on top; in Arthur's design the engine stays in your environment and the platform stores aggregated metrics, not raw data.
Arthur Evals Engine: owner, license and hosting
| Fact | Value | Source and date checked |
|---|---|---|
| Owner | Arthur. | SOURCE ↗ |
| License | MIT for the engine. The Arthur Platform it can report to is a commercial hosted service. | SOURCE ↗ |
| Self-hosting | Yes. The engine is deployed with Docker Compose, a CloudFormation template for AWS ECS, or a Helm chart for Kubernetes. | SOURCE ↗ |
| Hosted regions | The engine runs in your own environment. The pricing page names no hosting region for the Arthur Platform. | SOURCE ↗ |
Each value was read from the vendor's own page on the date shown. How listings are checked
Arthur Evals Engine plans and prices
Arthur Evals Engine pricing, checked on . Lowest paid plan: Premium, $60 a month, for the hosted Arthur Platform. Free tier: The engine is free under the MIT license. The hosted Arthur Platform has a Free plan at $0 a month for up to 4 use cases. Pricing model: No charge for the open-source engine. The hosted Arthur Platform is sold as flat monthly plans with limits on use cases, spans, inferences and evals.
| Plan | Price | Included usage | Beyond that | Data retention | Users |
|---|---|---|---|---|---|
| Open source engine | $0 | PII, sensitive data, custom LLM and regex rules, deployed by you | – | – | – |
| Free | $0 a month | Up to 4 use cases; 5k jobs, 300k spans, 12k inferences, 3k evals | – | 7 days | Unlimited |
| Premium | $60 a month | Up to 100 use cases; 20k jobs, 1.2M spans, 100k inferences, 75k evals | – | 30 days | Unlimited |
| Enterprise | Custom | Custom jobs, spans, inferences and evals | – | Unlimited | Unlimited |
The Free, Premium and Enterprise plans are for the hosted Arthur Platform; the engine is the open-source row. The pricing page lists jobs, spans, inferences and evals per plan without stating a period. Free allows 2 projects, 3 custom alerts per model and 6 alert checks an hour; Premium allows 10 projects and unlimited alerts. Enterprise adds dedicated and managed VPC options, SSO, SLAs and a BAA.
As stated on the vendor's pricing page on . A dash means the page does not say.
Arthur Evals Engine pricing history
The pricing page has been checked daily since . No price change has been recorded since then.
Where Arthur Evals Engine fits
Teams that want guardrails and stored traces from one self-hosted service under an MIT license. Rules are set per application over an API rather than in code, and the engine can be deployed with Docker Compose, a CloudFormation template for AWS ECS, or a Helm chart.
Limits of Arthur Evals Engine
It is a service to operate, with its own Postgres database. The open-source edition downloads models from Hugging Face Hub; air-gapped deployment, Arthur's proprietary evaluation models and custom guardrails are in the Enterprise version. Dashboards and alerts need the hosted platform, whose free plan keeps data for 7 days. Arthur uses three names for the engine across its pages: Arthur Evals Engine in the docs and on the pricing page, Arthur Engine in the repository and GenAI Engine in the engine's readme.
Alternatives to Arthur Evals Engine
-
Amazon Bedrock Guardrails
Amazon Bedrock Guardrails is a managed AWS service that filters prompts and model responses against content, topic and sensitive-data policies.
-
Google Model Armor
Model Armor is a Google Cloud service that screens LLM prompts and responses for prompt injection, harmful content, sensitive data and malicious URLs.
-
Lakera Guard
Lakera Guard is an API that screens prompts, tool calls and model outputs for prompt attacks, data leakage and unwanted content.
-
NVIDIA NeMo Guardrails
NVIDIA NeMo Guardrails is an open-source Python library for adding programmable input, dialog, retrieval and output checks to LLM applications.
-
OpenAI Guardrails
OpenAI Guardrails is an open-source Python library that wraps the OpenAI client and runs configurable checks on model inputs and outputs.
-
Presidio
Presidio is an open-source Python framework that detects and anonymizes personal data in text, images and structured data.
-
SignalVault
BUILT BY THIS SITE'S EDITOR
SignalVault is a hosted proxy that logs requests to OpenAI and Anthropic models and blocks or redacts sensitive data in them.