Common Crawl for Google SERPs
What is the RTA?
The Rank Tracker Alliance is a joint effort by interested parties to share the responsibilities and cost of SERP retrieval for the most common keywords across the web.
Why is the RTA important?
Every rank tracker wanting to provide a global index of common SERP information is currently required to build their own keword list and ultimately collect the same SERP HTML as every other rank tracker.
- Google Search HTML is a commodity
- Collection is expensive and difficult at scale
- Storing the HTML is expensive
- Product differentiation is generally not based on collection, but instead parsing, querying and reporting
How does it work?
The RTA maintains a curated list of billions of keywords that are sorted by search volume each month. Depending on a keywords search volume and various other factors, it is scheduled either daily, weekly, monthly or even at longer intervals for retrieval by an edge-distributed cloud function task queue.
When a keyword is scheduled:
- Its HTML is requested and downloaded via one of our SERP providers
- The HTML is normalized and processed for consistency and QA
- The previous entry's HTML is diffed as a list of delta changes
- The HTML is compressed and stored for cost-efficiency and high availability
- PubSub/SNS notifications are dispatched to subscribers
All keyword data and HTML is stored long-term and made available to subscribers through a lightning fast edge-distributed API.
Who runs the RTA?
The RTA is a not-for profit LLC jointly operated by all members/parties. Current members include:
Ready to dig in?
More details are available in our presentation on the spec below. It covers:
- Technical implementation details
- Cost structures and billing agreements
- FAQ
- Questions and Discussion Points
© 2024 Rank Tracker Alliance LLC