Methodology

Scoring version AIQ v1

This document defines the operating protocol used to collect AI recommendations, resolve product identities, calculate category-relative scores, preserve evidence, and publish weekly snapshots.

Collection Protocol

Collection uses the same authenticated consumer web interfaces available to subscribed users. Cooper operates active subscriptions for ChatGPT, Gemini, Claude, and Perplexity and submits prompts through their user-facing products rather than provider APIs. This captures interface-level search, model routing, answer formatting, and citation behavior that a subscriber is more likely to experience.

Each query runs in a fresh controlled browser session through geographically distributed proxies. The scoring-version regional mix gives the United States the largest weight, Europe the next-largest weight, and the rest of the world a smaller combined weight while keeping it represented. Sessions exclude prior conversation history so geography and the published prompt remain the primary controlled variables.

Cooper runs 6–12 independent queries for every question and assistant in each measured regional cell. Collection begins with six observations. If product-set overlap and answer ordering show material drift, additional observations are collected until consensus stabilizes or the cell reaches twelve. Stable cells stop at six; variable cells receive more measurement without receiving more scoring weight.

Each observation stores its start time, assistant account, interface surface, region, prompt version, rendered answer, citations, completion state, and extraction status. A snapshot is eligible for publication only after every required cell has reached its collection rule and all extracted entities have passed validation.

Collection fieldAIQ v1 valueOperational meaning
CadenceWeeklyOne dated, immutable measurement edition.
Measurement surfaceAuthenticated subscription interfacesPrompts run through the user-facing web products.
Regional samplingUS primary · Europe secondary · global remainder includedControlled proxy location is stored with every observation.
Repetition6–12 independent queries per cellAdditional observations are triggered by answer drift.
Completion ruleConsensus stabilizes or 12 queries completeEach cell is normalized by completed query count before weighting.

Query Matrix

Every active category is measured against three core buyer intents and one standalone Rising dimension on every assistant. Segment-specific runs retain this query matrix while adding the defined company context, which keeps comparisons internally consistent without treating unrelated buyer environments as equivalent.

IntentMeasurement objectiveInterpretation
BestIdentify the assistant’s strongest general recommendations.Captures broad category preference without a price or implementation constraint.
RisingIdentify recommendations for newer or emerging tools gaining attention.Produces standalone Rising AIQ; it is question-based, not an independently verified product-age ranking.
Most affordableIdentify recommendations for price-sensitive buyers.Measures perceived affordability, not a Cooper-authored pricing comparison.
Easiest to set upIdentify recommendations associated with lower implementation effort.Measures assistant recommendations, not independently timed implementation tests.

Overall AIQ uses only Best, Most affordable, and Easiest to set up. Rising uses the same collection and normalization pipeline but remains a separate one-question scope. It does not affect Overall AIQ, Overall rank, weekly Overall movement, or Overall recognition.

Entity Resolution and Validation

Raw model answers are not scored directly. Every named candidate passes through the same identity pipeline so aliases are merged, ambiguous names are rejected, and each retained position refers to a verified software product.

  1. Extract candidate mentions

    Parse each raw answer in response order and retain every explicitly named software product together with its original position, surrounding answer text, assistant, question, category, scope, and run identifier.

  2. Normalize product identity

    Resolve spelling variants, abbreviations, renamed products, and vendor-versus-product references to one canonical product record without changing the position observed in the original answer.

  3. Validate the entity

    Confirm that the candidate is a real software product with a verifiable official domain. Ambiguous names, generic concepts, service businesses, and unverifiable entities are excluded rather than guessed into the index.

  4. Store an atomic mention

    Write one immutable mention containing the run, category, scope, buyer intent, assistant, canonical product, answer position, and source references required to reproduce the derived board.

Claim boundary: A claimed profile may add official descriptive content. It cannot change canonical resolution, raw mentions, answer positions, citations, rankings, or AIQ.

Scoring Pipeline

Scoring begins only after entity validation. The pipeline transforms immutable mentions into one reproducible category board and preserves the unrounded intermediate value used for ordering and tie resolution.

  1. Score each observed answer

    Convert every verified product position in each independent answer to 1 divided by position. A product not named in that answer contributes zero for that observation.

  2. Normalize repeated queries

    Average the reciprocal-rank contributions inside each question, assistant, and region cell by its completed query count. A cell that requires 12 observations therefore has the same maximum influence as one that stabilizes after 6.

  3. Apply regional weight

    Combine the normalized regional cells using the scoring-version geographic mix. The United States carries the largest share, Europe the next-largest share, and the rest of the world remains represented at a lower aggregate weight.

  4. Apply assistant weight

    Multiply the region-adjusted contribution by the scoring-version assistant weight: ChatGPT 55, Gemini 25, Claude 15, or Perplexity 5 under AIQ v1.

  5. Aggregate the selected scope

    Sum weighted contributions across the three core buyer intents for Overall AIQ, or across the Rising question alone for standalone Rising AIQ. Never mix the standalone dimension into the Overall scope.

  6. Normalize to AIQ

    Divide the observed weighted sum by the maximum possible sum for the same question and region mix, multiply by 100, and round once to the nearest whole number.

  7. Order products and resolve ties

    Sort by the unrounded weighted score. Equal scores resolve by the strongest single placement, then by total verified appearances; the same deterministic rule is used on current and archived boards.

Evidence Handling

Cooper stores each cited URL, normalized page, source class, measurement cell, and the assistant response that exposed it. Evidence is attached to the run that produced it so later source changes do not rewrite the historical record.

Cited pages for a product are pages the assistant cited in that answer and Cooper verified as naming or representing that product. Aggregate source totals also include cited pages that could not be attributed to a specific product, including inaccessible or unsupported pages.

Duplicate normalized URLs are counted once in each applicable view. A page can support zero, one, or multiple products, but only verified product links contribute to product-level counts. A citation does not expose model training data, prove a causal relationship, or mean that Cooper endorses the cited publisher.

Snapshot and Change Policy

  • Publication: only a complete, validated run becomes the current public edition.
  • Immutability: completed editions retain their original questions, assistant weights, scoring version, mentions, and positions.
  • Version changes: a formula, assistant-weight, or material prompt-policy change creates a new scoring version with a published explanation.
  • Scope definition: AIQ v1 keeps the three-question Overall score and the standalone Rising score separate without changing the Overall formula.
  • Corrections: verified data errors are recorded explicitly; historical results are not silently recalculated under a newer method.

Quality Controls

Publication controls protect completeness, identity accuracy, deterministic scoring, historical integrity, and independence from commercial relationships. A failed control prevents publication or requires an explicit correction record.

  1. Run completeness

    Confirm that every question, assistant, and region cell contains at least six completed observations and that any cell escalated for drift reaches consensus or the 12-query ceiling before publication.

  2. Entity validity

    Exclude unresolved or unverifiable names from scoring, preserve the raw answer for audit, and avoid creating placeholder products merely to fill a result set.

  3. Scoring determinism

    Derive every home, hub, category, archive, and product board from the same scoring implementation and versioned assistant weights.

  4. Historical integrity

    Keep completed editions immutable. Corrections are documented and issued as explicit data changes rather than silently rewriting an earlier weekly result.

  5. Commercial independence

    Exclude profile ownership, subscription level, advertising, sponsorship, and vendor relationships from collection, validation, scoring, tie-breaking, and publication decisions.

Operational Confidentiality

Cooper publishes the information required to understand and interpret AIQ without publishing the controls needed to reproduce or manipulate the collection system. Public documentation includes the measurement surface, tracked assistants, buyer-intent families, query range, regional weighting hierarchy, assistant weights, scoring formula, limitations, and material methodology changes.

The exact prompt wording and variants, numerical adaptive consensus thresholds, query timing, proxy vendors and endpoints, account identities, browser and session configuration, retry behavior, anomaly-detection rules, manual-review triggers, internal alias maps, and anti-manipulation controls remain confidential.

Cooper also does not publish the complete raw-response archive. These boundaries protect subscription accounts, collection infrastructure, source licensing, and measurement integrity while reducing the ability of a vendor or third party to target the collection process rather than improve genuine recommendation visibility.

Disclosure rule: Material changes to scoring, coverage, question families, assistant weights, regional weighting policy, or publication controls are public. Implementation details whose disclosure would enable evasion, interference, account compromise, or result manipulation remain confidential.

Public Archive and Audit Records

The public archive preserves each verified weekly edition, but it does not publish every individual query, complete browser session, or raw answer. Public history is designed to prove what Cooper reported and which methodology governed that report, not to expose the operational collection system.

Every public edition identifies its measurement window, methodology and scoring version, included assistants and category scope, collection completion status, aggregate scores and ranks, available source summary, and any published correction log. Earlier editions retain the rules and results originally associated with them.

Query-level responses, session metadata, extraction records, validation decisions, and internal quality-control logs are retained under access controls for investigation, quality assurance, and audit. Where additional verification is warranted, Cooper may provide relevant records to qualified independent auditors under confidentiality rather than release the complete underlying archive publicly.

Record classAccessPurpose
Weekly editionPublicHistorical rankings, coverage, version, sources, and corrections.
Aggregate evidencePublic or product accessInterpreting recommendation patterns without exposing collection controls.
Raw observationsRestrictedQuality assurance, incident review, and controlled independent audit.
Operational controlsConfidentialProtecting accounts, infrastructure, and measurement integrity.

Questions about AIQ or its methodology? Email us at science@cooperaiq.com.