When creating Devsnap I was pretty naive. I used create-react-app for my frontend and Go with GraphQL for my backend. A classic SPA with client side rendering.

I knew for that kind of site I would have Google to index a lot of pages, but I wasn't worried, since I knew Google Bot is rendering JavaScript by now and would index it just fine

Oh boy, was I wrong.

At first, everything was fine. Google was indexing the pages bit by bit and I got the first organic traffic.

But somehow indexing was really slow. Google was only indexing ~2 pages per minute. I thought it would speed up after some time, but I it wasn't, so I needed to do something, since my website relies on Google to index a lot of pages fast.

1. Enter SSR

I started by implementing SSR, because I stumbled across some quote from a Googler, stating that client side rendered websites have to get indexed twice. The Google Bot first looks at the initial HTML and immediately follows all the links it can find. The second time, after it has sent everything to the renderer, which returns the final HTML. That is not only very costly for Google, but also slow. That's why I decided I wanted Google Bot to have all the links in the initial HTML.

I was doing that, by following this fantastic guide. I thought it would take me days to implement SSR, but it actually only took a few hours and the result was very nice.

Without SSR I was stuck at around 20k pages indexed, but now it was steadily growing to >100k.

But it was still not fast enough

Google was not indexing more pages, but it was still too slow. If I ever wanted to get those 250k pages indexed and new job postings discovered fast, I needed to do more.

2. Enter dynamic Sitemap

With a site of that size, I figured I'd have to guide Google somehow. I couldn't just rely on Google to crawl everything bit by bit. That's why I created a small service in Go that would create a new Sitemap two times a day and upload it to my CDN.

Since sitemaps are limited to 50k pages, I had to split it up and focused on only the pages that had relevant content.

After submitting it, Google instantly started to crawl faster.

But it was still not fast enough

I noticed the Google Bot was hitting my site faster, but it was still only 5-10 times per minute. I don't really have an indexing comparison to #1 here, since I started implementing #3 just a day later.

3. Enter removing JavaScript

I was thinking why it was still so slow. I mean, there are other websites out there with a lot of pages as well and they somehow managed too.

That's when I thought about the statement of #1. It is reasonable that Google only allocates a specific amount of resources to each website for indexing and my website was still very costly, because even though Google was seeing all the links in the initial HTML, it still had to send it to the renderer to make sure there wasn't anything to index left. It simply doesn't know everything was already in the initial HTML when there is still JavaScript left.

So all I did was removing the JavaScript for Bots.

if(isBot(req)) {
    completeHtml = completeHtml.replace(/<script[^>]*>(?:(?!<\/script>)[^])*<\/script>/g, "")
} 

Immediately after deploying that change the Google Bot went crazy. It was now crawling 5-10 pages - not per minute - per second.

And this is the current result. Since I implemented the changes just recently, there are still a lot of pages that have been crawled, but haven't been indexed yet (grey bars). Nevertheless I'm more than happy with the result.

After having Google index all the pages, the performance went up a lot as well.

Conclusion

If you want to have Google index a big website, only feed it the final HTML and remove all the JavaScript (except for inline Schema-JS of course).

If you mention meditation to people here in Germany you usually get weird looks. It's kind of a pity really. I've always thought differently about it. I started to meditate 10 years ago, since thought it was cool. Maybe because I loved Dragon Ball, where Goku and friends got stronger when they meditated? Anyways, I bought some books and started doing basic breathing meditation for 10 minutes a day. It was kind of hard at first, but got easier over time. After doing it for several weeks I noticed that I was becoming less stressed, more happy and less anxious. I was able to concentrate better and became more self aware. It was fantastic and I was wondering why not anyone would do this? Why didn't they teach it at school? It's like one of the greatest things to do to become more happy.

Some time later I found this study. It says, that doing meditation even irreversibly rewires your brain to achieve the results I mentioned. That's pretty cool actually. You don't even have to do meditation all the time, just doing it once for some time in your life is enough to become a better person for the rest of your life.

2. Yoga

Again, if you mention this to people here in Germany, you get weird looks. As a male even more so. And again it's such a pity. So many people I know complain about back and neck pain, stress and all this stuff. They take pain killers or other meds to get rid of it. Why not work on the cause instead? We are mostly sitting in offices all day long these days. That causes our muscles and tendons to shrink, which causes the aforementioned pains.

Yoga is for exactly these things. It stretches your tendons, strengthens your back and calms your mind. It's not even as exhausting as a full workout in the gym, but much more beneficial. It's something anyone can and should do, regardless of age or other circumstances. On top of that you learn to do headstand. How cool is that?

3. Making

I started coding when I was 12, because I wanted to play games that didn't exist. I didn't accept the fact that whatever I wanted to have wasn't there. I needed to have that so the only way was to create it myself.

The first game was in Visual Basic. It was like Space Invader. It was thrilling to for me to have total control over how the game works and what I could do with it. Usually you'd have to to cope with it if a game didn't work like you wanted.

When I was 14 I created games in Java. One custom version of Tetris and a game where I could shoot down a band I didn't like. A nice side effect was that my computer class teacher let me play my games during class, since he really couldn't teach me anything anymore.

This passion of mine still hasn't stopped. I created many things throughout the years. Some with more success, some with less. But I always learned a lot.

4. Music

I have been playing the piano since I was 6 and I'm convinced it helped me a great deal in developing my brain into what it is today. While playing music you need to concentrate. A lot. You have to repeat things over and over again, become totally engrossed in a task and overcome your limits.

5. Games

I love playing games. It's the best thing to detach yourself and relax. These days I usually play a few games League of Legends after coming home. It's always a good break.