Written by Brook Schaaf

More battles are beginning, and entire theaters are opening in the under-discussed war that will reorder the commercial internet — and with it, affiliate marketing. 

The latest salvo (opening or responsive, depending on your perspective) came from Reddit, in a lawsuit filed last week against Perplexity and three scraping companies: Oxylabs UAB, AWMProxy, and Austin’s own SerpApi. According to court documents, over a two-week span this summer they collectively scraped “almost three billion” SERPs. 

The defendants variously appealed to the First Amendment and the principle of free access to public data, for which there is already case law. Frankly, the opposite position is probably unrealistic. Practitioners have informed me that anything public can effectively be grabbed without limit, save perhaps by frequency. Testament to this was given last month by none other than Google, which posted about an anti-scraping position after reducing its num=100 parameter to num=10, causing much tumult amongst the SEO throng.

Non-public content is now also accessible through LLM-powered browsers that “ride along” with user activity, free even of the constraints of already data-extracting browser extensions. Immediately after the launch of OpenAI’s Atlas, security researchers found critical vulnerabilities, prompting some companies to block the browser. (Perhaps “Globe” or “Crushing Weight” would have been a better name if it turns out to be more the burden being lifted rather than the mythical figure doing the lifting.) Bankers are used to such restrictions, but how many retailers can afford to block Atlas, Perplexity’s Comet, or Microsoft’s sharpened Edge, not to mention Google’s Gemini Chrome, which appears headed in the same direction? Recall that Amazon, in true hypocritical fashion, tried unsuccessfully to shut down Honey years ago.

Still, one can sympathize with the possible loss of “a lucrative business licensing its data to LLMs.” Per the Ars Technica article, “Reddit alleged that Perplexity feeds off Reddit and Google, claiming to be ‘the world’s first answer engine’ but really doing ‘nothing groundbreaking.’” Harsh words. 

Then again, does this not describe the pattern of all big tech if we broaden the sense of “scraping” to include its near-synonym “peeling?” From Amazon and Google on down, tech companies will advantage themselves as much as possible with others’ data and even users, hence the reference to Hobbes’ famous phrase “war of all against all.” This is distinct from the regular competition of relative small holders because the titans can lay waste to, for example, the flow of commercial dollars. A recent event by Press Gazette attributed an “affiliate revenue drop of 20–40% at some publishers” to AI Overviews, themselves associated with traffic drops of up to 50%.  

Absent protective regulations enforced by court action, critical roles on the commercial internet may find themselves ruined — not just content creators but even retailers and small platforms. We may all fall under the shadow of a digital Leviathan. Little comfort if it turns out to be a few, and not just one.

The post Excoriato Omnium Contra Omnes appeared first on FMTC.