CompletionKit
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge
Reviews
No review yet. Used CompletionKit? Be the first to help others choose.What it does
CompletionKit is an MCP server, a connector that lets an AI assistant use it as a tool. It is hosted: an assistant connects to it over the internet, nothing to install.
No capability identified yet from its description.
Facts come from the Official MCP Registry, as declared by the publisher. Capabilities are matched automatically (low confidence). None of this is a test of the tool.
At a glance
How to connect
As declared in the Official MCP Registry · hosted endpoint tested by Electra Index 4h ago (live check)Hosted endpoint
https://completionkit.com/mcpstreamable-http
Required: Authorization (get it from the publisher)
MCP settings (Claude Desktop, Cursor and other MCP clients)
{
"mcpServers": {
"completionkit": {
"url": "https://completionkit.com/mcp",
"headers": {
"Authorization": "<your Authorization>"
}
}
}
}Replace the values in <angle brackets>.
Live check
Electra Index connected to the hosted endpoint 4h agoONLINE CompletionKit answered an MCP connection on 17 Nov 2026 and introduced itself as CompletionKit 0.29.3 (MCP protocol 2025-06-18).
54 tools offered to AI assistants
prompts_list· List all promptsprompts_get· Get a prompt by IDprompts_create· Create a promptprompts_update· Update a prompt. Changing the template or model of the published version, or editing any version that already has runs, creates a new DRAFT version (current=false) instead of changing it, so an agent's edits don't go live without a gate. Ruprompts_delete· Delete a promptprompts_publish· Publish a prompt version, making it the current versionprompts_suggest_improvement· Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it asruns_list· List all runsruns_get· Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without liruns_create· Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.runs_update· Update a runruns_delete· Delete a run
Show 42 more
runs_generate· Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.runs_regrade· Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.runs_rerun· Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.runs_retry_failures· Re-run only the failed responses of a run, optionally limited to specific response ids via "only".responses_list· List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "scoresponses_get· Get a specific responsedatasets_list· List all datasetsdatasets_get· Get a dataset by IDdatasets_create· Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, andatasets_update· Update a datasetdatasets_delete· Delete a datasetdatasets_create_from_url· Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL ismetrics_list· List all metricsmetrics_get· Get a metric by IDmetrics_create· Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (jsometrics_update· Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/ometrics_delete· Delete a metricmetrics_suggest_variants· Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire vimetric_groups_list· List all metric groupsmetric_groups_get· Get a metric group by IDmetric_groups_create· Create a metric groupmetric_groups_update· Update a metric groupmetric_groups_delete· Delete a metric groupmetric_versions_list· List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.metric_versions_publish· Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_metric_versions_dismiss· Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead.provider_credentials_list· List all provider credentials (API keys are not exposed)provider_credentials_get· Get a provider credential by ID (API key is not exposed)provider_credentials_create· Create a provider credentialprovider_credentials_update· Update a provider credential. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint.provider_credentials_delete· Delete a provider credentialtags_list· List all tagstags_get· Get a tag by IDtags_create· Create a tag. Color is auto-assigned.tags_update· Rename a tag.tags_delete· Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.agreements_list· List agreements. Filter by run_id, response_id, metric_id, or created_by.agreements_create· Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.judges_replay· Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can cojudges_compare· Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.promptfoo_import· Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summaryusage_get· Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked
What "online" means: the server answered an MCP connection at the time of the check. It is not a guarantee of security, quality or continuous uptime. How we test: we open an MCP session and ask for the tool list, as an AI assistant would. We never run a tool and never send a key. Checked every few days. Method
Is this your server? Monitoring Pro: a check every 30 minutes and an email within the hour when it stops answering. Launch offer: $39.99 once, for life.
Facts & evidence
Each fact shows where it was seen and when.| Fact | Value | Evidence | Source | Last confirmed |
|---|---|---|---|---|
| Hosted endpoint | https://completionkit.com/mcp | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Local install (package) | no | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Remote access (hosted) | yes | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Website | https://completionkit.com | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Repository | https://github.com/homemade-software-inc/completion-kit | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Registry status | active | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Version | 1.0.0 | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| MCP server | yes | DECLARED | SRC-001 Official MCP Registry | 1d ago |
Change history
Is this page accurate?
Flag it and our robot re-checks the official source. Reports never change a score.Is CompletionKit still online?
Free alerts · no account▲ 0Do you run CompletionKit? Know when it stops answering
For publishers · optionalIts hosted endpoint can be monitored. Monitoring Pro connects every 30 minutes like an AI assistant, emails you within the hour when it stops answering, keeps a 90-day availability history and adds your answer rate to your badge.
Monitoring never changes a score or a rank.
Questions about CompletionKit
How do I connect CompletionKit?
Use the hosted endpoint https://completionkit.com/mcp in your assistant's MCP settings. The exact settings are in "How to connect" above, as declared in the Official MCP Registry.
Does CompletionKit need an API key or token?
Yes. The registry entry lists a required value: Authorization. Get it from the publisher.
What tools does CompletionKit offer?
When Electra Index connected on 17 Nov 2026, it listed 54 tools: prompts_list, prompts_get, prompts_create, prompts_update, prompts_delete, prompts_publish, prompts_suggest_improvement, runs_list and more. The full list is in "Live check" above.
Is CompletionKit online?
Yes. Its hosted endpoint answered an MCP connection from Electra Index 4h ago.