Our crawlers follow internal and external links on the Internet looking for new pages with links. So, we can only find a page if there is an inbound link to it.
Looking at our old system, we found that there was an opportunity to increase overall crawlability and find better content – the content that website owners would want us to crawl and index.
So what did we do?
- We tripled the number of crawlers (from 10 to 30).
- We stopped crawling pages with URL parameters that don’t affect the page content (&sessionid, UTM, etc.).
- We have increased the frequency of reading websites’ robots.txt files and following the guidelines they contain.
Crawler Improvement Technique
- More crawlers (30 now!).
- Clean data without garbage or duplicate links.
- Better ability to find the most relevant content.
- Crawl speed of 25 billion pages per day.
Storage
Storage is where we keep all the links you can mexico telegram data see as a Semrush user. This storage shows you the links in the tool and offers filters you can apply to find what you are looking for.
The main concern we had with our old storage system was that it could only be completely rewritten during an upgrade. This meant that every 2-3 weeks it was rewritten and the process started again.
So, during the update, new links were piling up in the staging, creating a delay in the visibility of the tool to users. We wanted to see if we could improve the speed of this step.
So what did we do?
To improve this, we rewrote the architecture from scratch. To eliminate the need for temporary storage, we more than quadrupled the number of our servers.
It took over 30,000 hours of neil kimber vp development engineering work to implement the latest technologies. Now we have a scalable system that will not encounter any limits, today or tomorrow.
Storage improvement technique
- Over 500 servers in total.
- 287 TB of RAM.
- 16,128 cores.
- 30 PB of total storage space.
- Lightning-fast filtering and reporting.
- INSTANT UPDATE – no more temporary storage.
We conducted a two-part study to compare the speed of our Backlink Analysis to that of Moz, Ahrefs, and Majestic.
To see exactly how much faster
our tool works than other SEO tools on the market, read this article .
We’re so proud of our new Backlink Analysis database d database that we want everyone to be able to take advantage of everything it has to offer.
Get FREE access by signing up for a free Semrush account and the Backlink Analysis section will be fully open to you.
Give it a try and let us know what you think!
Welcome to the future of dynamic backlink management!