Methodology
Scoring version AIQ v1This document defines the operating protocol used to collect AI recommendations, resolve product identities, calculate category-relative scores, preserve evidence, and publish weekly snapshots.
Collection Protocol
Collection uses the same authenticated consumer web interfaces available to subscribed users. Cooper operates active subscriptions for ChatGPT, Gemini, Claude, and Perplexity and submits prompts through their user-facing products rather than provider APIs. This captures interface-level search, model routing, answer formatting, and citation behavior that a subscriber is more likely to experience.
Each query runs in a fresh controlled browser session through geographically distributed proxies. The scoring-version regional mix gives the United States the largest weight, Europe the next-largest weight, and the rest of the world a smaller combined weight while keeping it represented. Sessions exclude prior conversation history so geography and the published prompt remain the primary controlled variables.
Cooper runs 6–12 independent queries for every question and assistant in each measured regional cell. Collection begins with six observations. If product-set overlap and answer ordering show material drift, additional observations are collected until consensus stabilizes or the cell reaches twelve. Stable cells stop at six; variable cells receive more measurement without receiving more scoring weight.
Each observation stores its start time, assistant account, interface surface, region, prompt version, rendered answer, citations, completion state, and extraction status. A snapshot is eligible for publication only after every required cell has reached its collection rule and all extracted entities have passed validation.
| Collection field | AIQ v1 value | Operational meaning |
|---|---|---|
| Cadence | Weekly | One dated, immutable measurement edition. |
| Measurement surface | Authenticated subscription interfaces | Prompts run through the user-facing web products. |
| Regional sampling | US primary · Europe secondary · global remainder included | Controlled proxy location is stored with every observation. |
| Repetition | 6–12 independent queries per cell | Additional observations are triggered by answer drift. |
| Completion rule | Consensus stabilizes or 12 queries complete | Each cell is normalized by completed query count before weighting. |
Query Matrix
Every active category is measured against three core buyer intents and one standalone Rising dimension on every assistant. Segment-specific runs retain this query matrix while adding the defined company context, which keeps comparisons internally consistent without treating unrelated buyer environments as equivalent.
| Intent | Measurement objective | Interpretation |
|---|---|---|
| Best | Identify the assistant’s strongest general recommendations. | Captures broad category preference without a price or implementation constraint. |
| Rising | Identify recommendations for newer or emerging tools gaining attention. | Produces standalone Rising AIQ; it is question-based, not an independently verified product-age ranking. |
| Most affordable | Identify recommendations for price-sensitive buyers. | Measures perceived affordability, not a Cooper-authored pricing comparison. |
| Easiest to set up | Identify recommendations associated with lower implementation effort. | Measures assistant recommendations, not independently timed implementation tests. |
Overall AIQ uses only Best, Most affordable, and Easiest to set up. Rising uses the same collection and normalization pipeline but remains a separate one-question scope. It does not affect Overall AIQ, Overall rank, weekly Overall movement, or Overall recognition.
Entity Resolution and Validation
Raw model answers are not scored directly. Every named candidate passes through the same identity pipeline so aliases are merged, ambiguous names are rejected, and each retained position refers to a verified software product.
Extract candidate mentions
Parse each raw answer in response order and retain every explicitly named software product together with its original position, surrounding answer text, assistant, question, category, scope, and run identifier.
Normalize product identity
Resolve spelling variants, abbreviations, renamed products, and vendor-versus-product references to one canonical product record without changing the position observed in the original answer.
Validate the entity
Confirm that the candidate is a real software product with a verifiable official domain. Ambiguous names, generic concepts, service businesses, and unverifiable entities are excluded rather than guessed into the index.
Store an atomic mention
Write one immutable mention containing the run, category, scope, buyer intent, assistant, canonical product, answer position, and source references required to reproduce the derived board.
Claim boundary: A claimed profile may add official descriptive content. It cannot change canonical resolution, raw mentions, answer positions, citations, rankings, or AIQ.
Scoring Pipeline
Scoring begins only after entity validation. The pipeline transforms immutable mentions into one reproducible category board and preserves the unrounded intermediate value used for ordering and tie resolution.
Score each observed answer
Convert every verified product position in each independent answer to 1 divided by position. A product not named in that answer contributes zero for that observation.
Normalize repeated queries
Average the reciprocal-rank contributions inside each question, assistant, and region cell by its completed query count. A cell that requires 12 observations therefore has the same maximum influence as one that stabilizes after 6.
Apply regional weight
Combine the normalized regional cells using the scoring-version geographic mix. The United States carries the largest share, Europe the next-largest share, and the rest of the world remains represented at a lower aggregate weight.
Apply assistant weight
Multiply the region-adjusted contribution by the scoring-version assistant weight: ChatGPT 55, Gemini 25, Claude 15, or Perplexity 5 under AIQ v1.
Aggregate the selected scope
Sum weighted contributions across the three core buyer intents for Overall AIQ, or across the Rising question alone for standalone Rising AIQ. Never mix the standalone dimension into the Overall scope.
Normalize to AIQ
Divide the observed weighted sum by the maximum possible sum for the same question and region mix, multiply by 100, and round once to the nearest whole number.
Order products and resolve ties
Sort by the unrounded weighted score. Equal scores resolve by the strongest single placement, then by total verified appearances; the same deterministic rule is used on current and archived boards.
Formula reference: AIQ normalized formula and worked example.
Evidence Handling
Cooper stores each cited URL, normalized page, source class, measurement cell, and the assistant response that exposed it. Evidence is attached to the run that produced it so later source changes do not rewrite the historical record.
Cited pages for a product are pages the assistant cited in that answer and Cooper verified as naming or representing that product. Aggregate source totals also include cited pages that could not be attributed to a specific product, including inaccessible or unsupported pages.
Duplicate normalized URLs are counted once in each applicable view. A page can support zero, one, or multiple products, but only verified product links contribute to product-level counts. A citation does not expose model training data, prove a causal relationship, or mean that Cooper endorses the cited publisher.
Snapshot and Change Policy
- Publication: only a complete, validated run becomes the current public edition.
- Immutability: completed editions retain their original questions, assistant weights, scoring version, mentions, and positions.
- Version changes: a formula, assistant-weight, or material prompt-policy change creates a new scoring version with a published explanation.
- Scope definition: AIQ v1 keeps the three-question Overall score and the standalone Rising score separate without changing the Overall formula.
- Corrections: verified data errors are recorded explicitly; historical results are not silently recalculated under a newer method.
Quality Controls
Publication controls protect completeness, identity accuracy, deterministic scoring, historical integrity, and independence from commercial relationships. A failed control prevents publication or requires an explicit correction record.
Run completeness
Confirm that every question, assistant, and region cell contains at least six completed observations and that any cell escalated for drift reaches consensus or the 12-query ceiling before publication.
Entity validity
Exclude unresolved or unverifiable names from scoring, preserve the raw answer for audit, and avoid creating placeholder products merely to fill a result set.
Scoring determinism
Derive every home, hub, category, archive, and product board from the same scoring implementation and versioned assistant weights.
Historical integrity
Keep completed editions immutable. Corrections are documented and issued as explicit data changes rather than silently rewriting an earlier weekly result.
Commercial independence
Exclude profile ownership, subscription level, advertising, sponsorship, and vendor relationships from collection, validation, scoring, tie-breaking, and publication decisions.
Operational Confidentiality
Cooper publishes the information required to understand and interpret AIQ without publishing the controls needed to reproduce or manipulate the collection system. Public documentation includes the measurement surface, tracked assistants, buyer-intent families, query range, regional weighting hierarchy, assistant weights, scoring formula, limitations, and material methodology changes.
The exact prompt wording and variants, numerical adaptive consensus thresholds, query timing, proxy vendors and endpoints, account identities, browser and session configuration, retry behavior, anomaly-detection rules, manual-review triggers, internal alias maps, and anti-manipulation controls remain confidential.
Cooper also does not publish the complete raw-response archive. These boundaries protect subscription accounts, collection infrastructure, source licensing, and measurement integrity while reducing the ability of a vendor or third party to target the collection process rather than improve genuine recommendation visibility.
Disclosure rule: Material changes to scoring, coverage, question families, assistant weights, regional weighting policy, or publication controls are public. Implementation details whose disclosure would enable evasion, interference, account compromise, or result manipulation remain confidential.
Public Archive and Audit Records
The public archive preserves each verified weekly edition, but it does not publish every individual query, complete browser session, or raw answer. Public history is designed to prove what Cooper reported and which methodology governed that report, not to expose the operational collection system.
Every public edition identifies its measurement window, methodology and scoring version, included assistants and category scope, collection completion status, aggregate scores and ranks, available source summary, and any published correction log. Earlier editions retain the rules and results originally associated with them.
Query-level responses, session metadata, extraction records, validation decisions, and internal quality-control logs are retained under access controls for investigation, quality assurance, and audit. Where additional verification is warranted, Cooper may provide relevant records to qualified independent auditors under confidentiality rather than release the complete underlying archive publicly.
| Record class | Access | Purpose |
|---|---|---|
| Weekly edition | Public | Historical rankings, coverage, version, sources, and corrections. |
| Aggregate evidence | Public or product access | Interpreting recommendation patterns without exposing collection controls. |
| Raw observations | Restricted | Quality assurance, incident review, and controlled independent audit. |
| Operational controls | Confidential | Protecting accounts, infrastructure, and measurement integrity. |
Questions about AIQ or its methodology? Email us at science@cooperaiq.com.