curatebot — the ZETO crawler

ZETO shows people the sale items, in their size, from the online stores they follow. To do that it reads those stores' public product pages with a crawler that identifies itself, follows robots.txt, and goes slowly.

Identity

Product token: curatebot
User-Agent: curatebot/<version> (+https://shopcur8.com/crawler)
Operator: ZETO
Contact: tcmprog@gmail.com
Egress IPs: Not published yet. Write to the address above and we will confirm whether a specific request was ours.

One User-Agent, always over HTTPS, never varied per site. We never rotate addresses or disguise curatebot as a browser to get around a block.

What it fetches

Public product pages, product JSON (such as Shopify's /products.json), sitemaps and robots.txt of online retailers that a ZETO user has chosen to follow.
Nothing behind a login, no forms, no carts or checkout, no search boxes.
Never marketplaces or classifieds (eBay, Facebook Marketplace and the like) — those are reached only through their own APIs and data providers.
Conditional requests wherever the site supports them (ETag / If-Modified-Since), so an unchanged page costs the site a 304 and no body.
Product images, from wherever the store serves them — usually a platform image CDN rather than the store's own site. They are read to work out what a garment looks like, so a shopper can find it from a photo. That CDN's own robots.txt governs those requests.
A store’s own icon, once every thirty days: the home page’s <head>, its web-app manifest if it links one, and at most two of the icons they name (the manifest icon, the Apple touch icon, the favicon, or the logo in the page’s structured data). It is shown beside the store’s name on its card, linking to the store, exactly as published — never altered, and never copied: ZETO keeps the icon’s address, its size and a fingerprint, and the icon itself is served from wherever the store serves it. These reads honour robots.txt and happen only on a store whose terms have been reviewed.

What it stores

The product facts a shopper compares: name, brand, price and sale price, currency, sizes and their availability, image URLs, and the page's structured product markup.
Not the full page, and never reviews, reviewer names or ratings — those are stripped before anything is written.
Prices are kept as a history so a follower can see when an item went on sale.
Never the image itself. A product photo is read in memory, turned into a list of numbers that describes how it looks, and discarded — ZETO’s systems keep that description and two fingerprints of the original, not a copy of your photography. (One exception, recorded in our legal notes: while we are measuring how well that description works, a copy is held on an engineer’s own machine and deleted when the measurement ends.)

Opting out

Add this to your robots.txt and the crawler stops within 24 hours (it re-reads robots.txt at least that often):

User-agent: curatebot
Disallow: /

Disallow rules for specific paths are honoured the same way, as is Crawl-delay. For an immediate stop, or to have already-collected product data removed, email tcmprog@gmail.com.

One exception, stated here so it is never a surprise: before ZETO reads anything else from a store, and about once a month after that, it reads the store's public terms page (on Shopify, /policies/terms-of-service) even where robots.txt disallows that path, because ZETO cannot honour terms it is not allowed to read. What that page says is read, recorded and reviewed by a person; what the crawler obeys is robots.txt. A robots rule, a 403, a challenge, or an email to the address above stops it — even that one page. The crawl is disclosed here because a crawl found later is the exposure, not the read itself.

For retailers

What a store can do to be listed on ZETO — a product feed, or a robots.txt allowance plus a written yes — and what ZETO reads first and honours, is on the retailers page.

How fast it goes

One request at a time per site, at least 2 seconds apart by default.
A Crawl-delay in robots.txt is honoured up to 60 seconds; above that the site is treated as opted out.
Retry-After is honoured, and the crawler backs off on 429 and 5xx responses; repeated blocks or an anti-bot challenge stop it for that site without any retry storm.
Images come from a CDN shared by thousands of stores rather than from your own servers, so they go a little faster — at most 4 per second, and never more than one at a time to any one host. A Crawl-delay on that CDN still wins. Your own site keeps the 2-second pace above.
Response bodies are capped by type, and feeds are read once, streamed, never re-pulled unchanged.