Content Sources

Source Priority

Each source implements the Discoverer interface. The pipeline queries sources in priority order and merges results, with higher-priority sources overriding lower-priority data for the same URL path.

type Discoverer interface {
    Name() IngestionSource
    Priority() int
    Discover(ctx context.Context, tenant *TenantConfig) ([]RawEntry, error)
    SupportsIncremental() bool
}

All sources produce RawEntry structs:

type RawEntry struct {
    URL           string
    Path          string
    Title         string
    WordCount     int
    Tokens        int
    ContentID     string
    PricingModel  PricingModel
    Rate          float64
    Currency      string
    Permissions   []comp.Function
    Prohibitions  []comp.Function
    LastModified  time.Time
    ChangeFreq    string
    Priority      float64
    Authors       []string
    PubDate       string
    SourceType    int32
    Source        IngestionSource
}

Source 1: FORA Sitemap Extension (Priority 1)

The highest-fidelity source. Providers add FORA-specific metadata directly in their sitemap using XML namespace extensions. This follows established practice: Google uses image:, video:, and news: sitemap extensions.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:fora="https://fora.dev/sitemap/1.0">
  <url>
    <loc>https://example.com/premium/ai-infrastructure.html</loc>
    <lastmod>2026-03-14</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>

    <fora:content>
      <fora:tokens>3300</fora:tokens>
      <fora:content-id>article-2026-03-14-ai-infra</fora:content-id>
      <fora:wordcount>2500</fora:wordcount>
      <fora:pricing-model>per_article</fora:pricing-model>
      <fora:rate currency="USD">0.05</fora:rate>
      <fora:permits>ai-input ai-index search</fora:permits>
      <fora:prohibits>ai-train</fora:prohibits>
      <fora:content-type>text</fora:content-type>
      <fora:author>Sarah Chen</fora:author>
      <fora:source-type>human</fora:source-type>
    </fora:content>
  </url>
</urlset>

FORA Sitemap Namespace Definition

Element	Type	Required	Description
fora:content	Container	No	Wraps all FORA extensions for a URL
fora:tokens	int	No	Estimated token count (most accurate when provider-provided)
fora:content-id	string	No	Stable content identifier (survives URL changes)
fora:wordcount	int	No	Word count of main body text
fora:pricing-model	enum	No	Pricing model hint (Exchange may override with private pricing)
fora:rate	decimal	No	Suggested price (fallback if no private pricing configured). Attribute: currency (ISO 4217)
fora:permits	string	No	Space-separated permitted CoMP functions
fora:prohibits	string	No	Space-separated prohibited CoMP functions
fora:content-type	enum	No	Content type (text, video, image, audio)
fora:author	string	No	Author name(s), comma-separated for multiple
fora:source-type	enum	No	human, ai, or hybrid

Go Parser

type FORASitemapSource struct {
    httpClient *http.Client
    logger     *slog.Logger
}

func (s *FORASitemapSource) Name() IngestionSource {
    return IngestionSourceFORASitemap
}

func (s *FORASitemapSource) Priority() int { return 1 }

func (s *FORASitemapSource) SupportsIncremental() bool { return true }

// Namespace-aware XML structs
type sitemapURL struct {
    Loc        string             `xml:"loc"`
    LastMod    string             `xml:"lastmod"`
    ChangeFreq string             `xml:"changefreq"`
    Priority   float64            `xml:"priority"`
    Content    *foraSitemapContent `xml:"https://fora.dev/sitemap/1.0 content"`
}

type foraSitemapContent struct {
    Tokens       int     `xml:"https://fora.dev/sitemap/1.0 tokens"`
    ContentID    string  `xml:"https://fora.dev/sitemap/1.0 content-id"`
    WordCount    int     `xml:"https://fora.dev/sitemap/1.0 wordcount"`
    PricingModel string  `xml:"https://fora.dev/sitemap/1.0 pricing-model"`
    Rate         rateElement `xml:"https://fora.dev/sitemap/1.0 rate"`
    Permits      string  `xml:"https://fora.dev/sitemap/1.0 permits"`
    Prohibits    string  `xml:"https://fora.dev/sitemap/1.0 prohibits"`
    ContentType  string  `xml:"https://fora.dev/sitemap/1.0 content-type"`
    Author       string  `xml:"https://fora.dev/sitemap/1.0 author"`
    SourceType   string  `xml:"https://fora.dev/sitemap/1.0 source-type"`
}

type rateElement struct {
    Currency string  `xml:"currency,attr"`
    Value    float64 `xml:",chardata"`
}

Source 2: RSL rsl.txt (Priority 2)

Parses RSL 1.0 XML to extract permits, prohibits, and payment hints per content path. RSL is the provider’s authoritative declaration of content availability and licensing terms. Pricing in RSL (<payment>, <amount>) is optional — omitting it means “contact the Exchange for pricing.”

Standard attribution for RSL fields:

• RSL usage types (ai-all, ai-input, ai-index, search) map to CoMP Function/SubFunction enums
• RSL payment types (crawl, purchase, subscription) map to FORA PricingModel enum
• RSL content paths map to CoMP Package.scope + Retrieval

RSL Payment Type to FORA PricingModel

RSL Payment Type	FORA Pricing	Notes
crawl	PER_UNIT (unit: "fetches")	implies a FUNCTION restriction crawl
purchase	FLAT (one-time) or PER_UNIT (metered)	per the RSL unit attribute
subscription	FREE + scopes: ["subscription:…"]	settled off-protocol; access scope-gated
free	FREE	no charge (stated explicitly)
attribution	FREE + Obligation{ATTRIBUTION, ON_USE}	attribution is an obligation, not a model
contribution	FREE + Obligation{CONTRIBUTION, ON_USE}	contribution is an obligation, not a model
training	PER_UNIT (unit per RSL — tokens, pages, fetches)	implies a FUNCTION restriction ai-train
use / inference	PER_UNIT (unit: "tokens")	implies a FUNCTION restriction ai-input
(omitted)	PRICING_MODEL_UNSPECIFIED	No <payment> in RSL = “contact Exchange for pricing” (rejected at ingest)

RSL Usage Type to CoMP Function

RSL Usage Type	CoMP Function	Notes
all	FUNCTION_ALL	All uses permitted/prohibited
ai-all	FUNCTION_AI_ALL	All AI uses
ai-train	FUNCTION_AI_TRAIN	Model training
ai-input	FUNCTION_AI_INPUT	RAG, grounding, agent context
ai-index	FUNCTION_AI_INDEX	Indexing for AI search
search	FUNCTION_SEARCH	Traditional search engine indexing

RSL Content Paths to CoMP Objects

RSL Element	CoMP Object	Notes
<content url="/premium/*">	Package.scope	Content addressing via path patterns
<content url="...">	Retrieval.endpoint	Populated at ExecuteTransaction with signed URL

Source 3: Standard Sitemap XML (Priority 3)

Parses standard sitemap.xml for URL discovery. Provides URL inventory and freshness signals but no pricing or token information. Combined with RSL or tenant defaults for complete catalog entries.

The implementation tries the sitemap index first, then direct sitemap, then falls back to checking robots.txt for sitemap location.

type SitemapSource struct {
    httpClient *http.Client
    logger     *slog.Logger
}

func (s *SitemapSource) Name() IngestionSource { return IngestionSourceSitemap }
func (s *SitemapSource) Priority() int          { return 3 }
func (s *SitemapSource) SupportsIncremental() bool { return true }

Source 4: HTML Crawl + Readability (Priority 4)

For URLs without token estimates from higher-priority sources, the pipeline crawls the page and extracts main content using a readability algorithm (equivalent to Mozilla’s Readability.js). This is the most expensive source — it makes HTTP requests to every content URL.

Crawl constraints:

• Respect robots.txt (cached per domain, refreshed daily)
• Rate limit: max 1 request/second per domain (configurable)
• Timeout: 10s per page
• Max body size: 5 MB
• User-Agent: FORA-Exchange/1.0 (content-indexer)
• Crawl only during ingestion window, never on the hot path

type HTMLCrawlSource struct {
    httpClient  *http.Client
    rateLimiter *rate.Limiter
    robotsCache map[string]*robotstxt.RobotsData
    logger      *slog.Logger
}

func (s *HTMLCrawlSource) Name() IngestionSource { return IngestionSourceHTMLCrawl }
func (s *HTMLCrawlSource) Priority() int          { return 4 }
func (s *HTMLCrawlSource) SupportsIncremental() bool { return true }

The EnrichEntry method crawls a single URL, extracts text via readability, counts words, and estimates tokens. It is called by the Enricher for entries missing word count or token estimates.

Source 5: CMS API Integration (Priority 5)

Pluggable adapters for common CMS platforms. Each adapter implements the Discoverer interface and queries the CMS REST API for content metadata.

CMS	API Endpoint	Content Field	Notes
WordPress	/wp-json/wp/v2/posts	content.rendered	Most common. Pagination via X-WP-TotalPages header
Ghost	/ghost/api/content/posts	html	Requires content API key
Contentful	/spaces/{id}/entries	fields.body	Structured content, may need rich text parsing
Strapi	/api/articles	attributes.content	REST or GraphQL
Drupal JSON:API	/jsonapi/node/article	attributes.body.processed	JSON:API spec compliant

Source 6: Manual Configuration (Priority 6)

Provider or Exchange operator configures resource rules via the Exchange admin API (POST /admin/content-rules, PUT /admin/tenants/{id}/pricing). Provides tenant-level defaults and path-level overrides.

type ManualSource struct {
    configStore ConfigStore
    logger      *slog.Logger
}

func (s *ManualSource) Name() IngestionSource { return IngestionSourceManual }
func (s *ManualSource) Priority() int          { return 6 }
func (s *ManualSource) SupportsIncremental() bool { return false }

Source 7: CatalogService API Push

Optional gRPC service for providers, CMS plugins, and third-party content intelligence providers to push content metadata directly to the Exchange, bypassing crawl/poll cycles.

CatalogService is NOT how Exchange operators push their own inventory. Exchange operators own the Exchange and have direct DB access. CatalogService is for external parties: CMS plugins, third-party intelligence providers, resource syndicators.

Authentication and Authorization

PushResources requests must be signed. The caller_id field (field 4) identifies the registered CatalogService client, and the request itself is signed with an RFC 9421 HTTP Message Signature carried in the HTTP headers (Signature / Signature-Input, with a Content-Digest over the body) — there is no in-message caller signature field.

The Exchange verifies the RFC 9421 signature against the caller’s published Ed25519 key in its WBA directory (the JWK Set at {caller domain}/.well-known/http-message-signatures-directory), identified by its RFC 7638 thumbprint (the RFC 9421 keyid), and confirms that caller_id is authorized to push content for the specified tenant_id.

Pushing with the Publisher SDK

The SDK carries the publisher role in all three languages: a catalog client that signs and sends the three CatalogService verbs, and the Exchange’s own two validation tiers as a pre-check. The client is a separate constructor from the agent client because CatalogService is a separate address (the manifest’s catalog_endpoint) and its caller holds a contributor key named by caller_id; the publisher chose the Exchange, so the origin is configuration.

client := connect.NewCatalogClient(catalogEndpoint, connect.WithSigner(contributorSigner))
if v := helpers.ValidateResourceEntry(entry); !v.OK() {   // what the Exchange will say, before signing
    return fmt.Errorf("entry refused: %+v", v.Violations)
}
resp, err := client.PushResources(ctx, &forav1.PushResourcesRequest{
    Exchange: "exchange.example", TenantId: tenantID, CallerId: "publisher.example",
    Entries:  []*forav1.ResourceEntry{entry},
})

import { createCatalogClient } from "@fora-protocol/sdk/client";
import { validateResourceEntry } from "@fora-protocol/sdk/licenseterm";

const client = createCatalogClient(catalogEndpoint, { signer: contributorSigner });
const verdict = validateResourceEntry(entry);
if (!verdict.ok) throw new Error(JSON.stringify(verdict.violations));
const resp = await client.pushResources({
  exchange: "exchange.example", tenant_id: tenantId, caller_id: "publisher.example",
  entries: [entry],
});

from fora_sdk import validate_resource_entry
from fora_sdk.client import CatalogClient, ClientConfig

verdict = validate_resource_entry(entry)
if not verdict.ok:
    raise ValueError(verdict.violations)
async with CatalogClient(ClientConfig(base_url=catalog_endpoint, signer=contributor_signer)) as client:
    resp = await client.push_resources({
        "exchange": "exchange.example", "tenant_id": tenant_id, "caller_id": "publisher.example",
        "entries": [entry],
    })

ver is stamped when the caller leaves it empty; no idempotency key is minted, because the catalog messages carry none — a push is an upsert. A request whose exchange is empty or not a bare domain — the shape the wire rule admits, which is narrower than a usable host — is refused before it is signed. The pre-check is advice about what the Exchange will say; the Exchange re-runs both tiers on every push and its verdict decides — including the ingest tier’s check that a term is still disjoint after its tokens are canonicalised, which is the one refusal a feed written against the registry’s alias spellings is most likely to hit. The JSONL feed, a sitemap, an RSL file, a crawl or a CMS plugin are inputs that converge on the same ResourceEntry; the SDK owns the shape, its validation and the push, never the source.

Third-Party Content Intelligence and Verification Vendors (v1.0)

Companies like GumGum, Peer39, and DoubleVerify already index the web for contextual ad targeting. They can implement CatalogService.PushResources to feed enriched content metadata (topic, token estimates, content categorization) and signed ResourceAttestation entries to Exchanges.

Verification vendors as attestation sources: In v1.0, third-party vendors push ResourceAttestation entries alongside catalog metadata. Each attestation is a signed envelope of claims (token count, language, content hash) from a trusted party. The Exchange verifies the attestation signature against the vendor’s public key published in its WBA directory (the JWK Set at https://{vendor-domain}/.well-known/http-message-signatures-directory).

Authorization: A verification vendor must be listed in the provider’s catalog_contributors (declared in /.well-known/fora.json) to push attestations for that provider’s content. The Exchange checks this authorization on every PushResources call.

RPC	Request	Response	Description
PushResources	PushResourcesRequest	PushResourcesResponse	Push or update resource entries in the Exchange catalog.
RemoveResources	RemoveResourcesRequest	RemoveResourcesResponse	Remove resource entries.
RefreshCatalog	RefreshCatalogRequest	RefreshCatalogResponse	Trigger a full catalog refresh from configured sources.

Field	Type	Number	Description
ver	string	1	FORA protocol version — "1.0". Stamped by the sender from a single constant; advisory on receive. See "Protocol version" in the file header.
tenant_id	string	2	Tenant identifier
entries	repeated ResourceEntry	3	Content entries to push. At least one: an empty push asks for nothing and is refused rather than answered with zero counts. At most 256, the bound a caller-chosen batch carries elsewhere in this contract (see ResourceQuery.uris) — it bounds one submission, so a larger feed is pushed in several. The cap is over entries because a submission is stored or refused whole, and a refusal names each entry that failed; it does not bound the work of checking a submission, which the recipient bounds at the transport.
caller_id	string	4	Identity of the caller (who is pushing this data). The Exchange verifies this matches a registered CatalogService client.
exchange	string	5	REQUIRED. Bare host of the recipient this request is addressed to (e.g. "exchange.example" or "exchange.example:8081"). See "Request recipient" in the file header. Distinct from tenant_id above, which names a publisher tenant WITHIN an Exchange, not the Exchange itself.
ext	Struct	15	Extension point
ext_critical	repeated string	90	Critical extension keys (COSE crit pattern, RFC 9052). Lists keys within ext that the consumer MUST understand. Unknown keys in this list → reject with UNKNOWN_CRITICAL_EXTENSION. Empty (default) → all ext keys are safe to ignore.

Field	Type	Number	Description
domain	string	1	Provider domain — the bare host the resource lives on, in the shape "Request recipient" defines in the file header: a port is allowed, a scheme, path, query or userinfo is not. With path it forms the catalog URI by concatenation, so a value carrying anything but a host would choose the URI rather than merely name the host.
path	string	2	Content path — an absolute URL path such as "/premium/article-42.html": starts with "/", carries no query or fragment delimiter, no whitespace and no control character, and is at most 2048 characters. Characters, not bytes: protovalidate's max_len counts Unicode code points, and the pattern admits non-ASCII, so a conformant path can exceed 2048 bytes.
content_id	optional string	3	Content identifier
title	optional string	4	Content title
word_count	optional int32	5	Word count
estimated_quantity	optional int32	6	Estimated quantity in the metering unit
content_hash	optional string	7	Content hash, carried as the publisher computed it — a bare hex digest or a "method:hexdigest" form; bounded in length, never format-checked, because hash_method names the algorithm.
hash_method	optional string	8	Hash algorithm
source	optional IngestionSource	9	How the entry was discovered
provenance_source	optional string	10	Who provided this resource metadata. Creates audit trail for "where did this catalog entry come from?"
provenance_timestamp	optional Timestamp	11	When this metadata was collected/generated.
attestations	repeated ResourceAttestation	12	Signed attestations about this resource entry. Same semantics as Offer.attestations — see ResourceAttestation message for verification levels and claim vocabulary. Attestations pushed via CatalogService are verified at push time: the Exchange checks that the attestation verifier is authorized to push for this provider (via catalog_contributors in the provider's WellKnownManifest) and validates the attestation signature against the verifier's public key from its WBA directory (the JWK Set at /.well-known/http-message-signatures-directory; the keyid is the key's RFC 7638 thumbprint). The verifier's fora.json carries only its role, determined by the verifier's operator.
terms	repeated LicenseTerm	13	Publisher-declared licensing terms for this resource. See LicenseTerm for the full model. For ENUMERATED terms, Pricing MUST be present. For REFERENCE_ONLY terms, License.uri is authoritative. The Exchange validates ENUMERATED terms at push time and surfaces them in Offer.terms on discovery. At most 32 terms per entry, stated on the wire so every implementation refuses the same size. An over-cap entry refuses the whole submission, as every catalog rejection does; what being a wire rule changes is WHEN — the refusal now happens at the boundary, before any per-entry classification runs, which is why the rejection reason that named this cap can no longer be produced for a push.
resource_mutability	optional ResourceMutability	14	Optional mutability hint. When omitted, the Exchange applies the STATIC default at Offer build; an explicit UNSPECIFIED is rejected. A value in ext is not read — the typed field is authoritative, so an ext-only value is treated as omitted. Mirrors the required Offer-side ResourceIdentity.resource_mutability.
ext	Struct	15	Extension point
ext_critical	repeated string	90	Critical extension keys (COSE crit pattern, RFC 9052). Lists keys within ext that the consumer MUST understand. Unknown keys in this list → reject with UNKNOWN_CRITICAL_EXTENSION. Empty (default) → all ext keys are safe to ignore.

Provenance Tracking

Every catalog entry records who provided it and when, via two fields on ResourceEntry:

• provenance_source — e.g., “gumgum.com”, “ssp-alpha.com”, “wordpress-plugin”
• provenance_timestamp — when the metadata was collected/generated

This creates an audit trail for “where did this catalog entry come from?” — essential when multiple parties can push metadata for the same content.

Source Decision Matrix

Does the provider have a FORA-extended sitemap.xml?
+-- YES -> Source 1 (FORA Sitemap) -- best fidelity
|
+-- NO -> Does the provider have rsl.txt?
         +-- YES -> Source 2 (RSL) + Source 3 (Standard Sitemap) + Source 4 (HTML Crawl)
         |
         +-- NO -> Does the provider have a standard sitemap.xml?
                  +-- YES -> Source 3 (Standard Sitemap) + Source 4 (HTML Crawl)
                  |
                  +-- NO -> Is a CMS API available?
                           +-- YES -> Source 5 (CMS API)
                           |
                           +-- NO -> Source 6 (Manual configuration only)