Solving Baidu's Inability to Index Personal Blogs Hosted on GitHub

# Solving Baidu's Inability to Index Static Blogs Hosted on GitHub

Warning

If you're looking for this blog's setup documentation, I recommend checking the README (opens new window) of this repository.

# Background

Since GitHub blocks Baidu's web crawlers, blogs hosted on GitHub Pages cannot be indexed by Baidu. This can be reproduced through Baidu Webmaster's Fetch as Baidu tool, which consistently returns 403 Forbidden errors.

# Solution

Host the blog simultaneously on both GitHub Pages and Coding Pages (opens new window) to solve the Baidu indexing problem. I discovered that Coding Pages loads particularly fast within China and gets indexed by Baidu, so I made the Coding Pages site my primary site and the GitHub Pages one a secondary site.

Steps:

Register a Coding (opens new window) account, create a repository, push your code to the Coding repository, and enable the Pages service.

The git operations are similar to GitHub. If you're unfamiliar with git operations, see my other article: Git User Manual (opens new window)

My blog project is built with VuePress, using the following auto-deployment script to push code to both GitHub and Coding simultaneously.

#!/usr/bin/env sh

# Abort on errors
set -e

# Build static files
npm run build

# Enter the built files directory
cd docs/.vuepress/dist

# github
echo 'b.xugaoyi.com' > CNAME
git init
git add -A
git commit -m 'deploy'
git push -f git@github.com:xugaoyi/blog.git master:gh-pages # Deploy to GitHub

# coding
echo 'xugaoyi.com' > CNAME
git add -A
git commit -m 'deploy'
git push -f git@git.dev.tencent.com:xugaoyi/xugaoyi.git master # Deploy to Coding

cd - # Return to the original directory
rm -rf docs/.vuepress/dist

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

I created separate CNAME files because I wanted different custom domains for the two platforms.

If you have a custom domain, you can also bind it in Coding Pages. Just add a CNAME record in your DNS settings pointing to the Coding Pages site address. (If you don't have a custom domain, skip this and remove the CNAME file creation from the deployment script.)

Finally, using Baidu Webmaster's Fetch tool, I confirmed that fetching succeeded! Then I used Baidu Webmaster's Link Submit (opens new window) feature to submit links to Baidu. After some time, the pages may appear in Baidu search results.