How ChatGPT can run A/B tests on your website

How it works

ChatGPT doesn't split your visitors or count conversions by itself. A testing tool does that: a script on your site splits visitors and shows the variants, and a statistics engine decides what won. An MCP server is how ChatGPT talks to that tool. Once the connector is added, ChatGPT can call the tool's actions in a conversation: check the install, read analytics, propose a test, submit variants, read results.

1. Add the connector in ChatGPT

OpenAI documents the steps for a custom MCP server on ChatGPT on the web (OpenAI):

1. Open chatgpt.com/plugins, select +, then Add custom MCP server.
2. Give it a name (Duelpilot) and enter the server URL https://duelpilot.com/api/mcp.
3. Choose OAuth as the authentication. ChatGPT registers itself with Duelpilot's sign-in automatically; there is no token to paste.
4. Accept the risk warning, create it, and install it before you use it in a conversation.
5. Sign in to Duelpilot when asked and pick the organisation, sites and permissions the connection gets.

OpenAI notes that workspace permissions and security restrictions apply, so on a company workspace an admin may need to allow custom servers. Not every ChatGPT plan offers them; if you don't see the option, use Claude or Cursor instead (the Claude guide).

ChatGPT asks you to confirm write actions by default, and treats every tool not marked read-only as a write action (OpenAI). Duelpilot marks its read and analytics tools as read-only, so ChatGPT reads without asking and asks before any other action, such as drafting an idea or pausing a test. You can let it remember your answer for a tool within one conversation.

2. Install and verify the script

Say "Install Duelpilot on my site". ChatGPT fetches the snippet and the installation guide for your platform. Duelpilot supports three ways in: the agent edits your code, the agent and you work through your CMS together (you log in yourself; it asks before every save or publish), or you follow its step-by-step plan. ChatGPT then verifies the install: Duelpilot checks the HTML (in the head, synchronous, not rewritten by an optimiser or held back by a cookie banner) and loads the page in a real desktop and phone browser.

3. Schedule the three routines

Duelpilot's agents are short prompts that run on a schedule in your own AI: the Analyst weekly (ideas with evidence), the Designer hourly (2–4 variants per idea, with previews and site checks on phone, tablet and desktop) and the Monitor daily (watches live tests, pauses on harm, sends a summary). Each prompt is one line; Duelpilot returns the current instructions at the start of every run.

ChatGPT has scheduled tasks: you can ask for one in the chat or create it from the sidebar. OpenAI's help centre lists at most hourly runs on paid plans and daily on Free, with a limit on active tasks per plan (OpenAI Help Center). Whether a custom connector is available inside a scheduled task isn't documented, and ChatGPT tasks have no API trigger, so a design can wait up to an hour after you move a card to Design.

Duelpilot doesn't assume the schedules work: it shows each agent as running, never ran or late from its actual runs, and emails the site's owners when one is late. If ChatGPT's tasks can't reach Duelpilot, run the same prompts elsewhere: a Claude or Cursor scheduled task, or a headless CLI from cron or a scheduled GitHub Action.

4. What ChatGPT can and can't do with Duelpilot

With autopilot on (Pro and up, a switch per site), Duelpilot's server launches the agent's top-ranked variant when every guardrail the owner set passes. Results come from the server's sequential statistics; ChatGPT reports the verdict and never recomputes it. You can always review and approve on Duelpilot's board instead of in the chat.

Other tools ChatGPT can drive

• Kameleoon lists ChatGPT among the assistants for its MCP server (docs); its lifecycle tool can start experiments, and only stop and delete need a confirmation (docs).
• Intelligems includes an MCP server on every paid plan that works with ChatGPT, for Shopify price, shipping and offer tests (pricing).
• pagent has an MCP server for Claude, ChatGPT, Codex and Cursor; with write scope the agent can start and stop tests without a second confirmation (docs).

Every tool we track, with what its MCP server lets an agent do: A/B testing MCP servers.

Try Duelpilot with your own agent