Generate Sitemap Automatically with Git Hooks Using Python

Background

If you are a web developer and creating websites, you know how cumbersome it is to generate a new sitemap whenever you update the site/ add a new page.

This is doe to upload the sitemap to Google Search Console, so that your new/ updated pages will be indexed/ refreshed correctly.

The process goes like this:

1. Update site
2. Update sitemap.xml
3. Push changes & Deploy site
4. Upload the new sitemap to Google Search Console

In this blogpost, I'd like to propose a way to automate step 2 - sitemap generation - using git hooks and Pyhon, so that whenever we commit the changes using Git, the sitemap will be automatically refreshed.

We will use a Git pre-commit hook to achieve this, and once this is done, we don't need to worry about updating sitemap.xml manually because it will always be up to date automatically.

For your information, I'm using SvelteKit, so if you are using the same framework, the code I'll be sharing below can be used as is. With a different framework, I guess you can use the code as is, but no guarantee is given.

Method

1. Find the .git folder

I assume you already have git installed on your machinem but if you don't, don't worry. User instructions here to install git.

Once you have it installed, go to the folder where you are developing your website. If you haven't initialized a git repo already, start one using git init command on your terminal. Make sure that you are in this folder in the terminal when initializing.

Once git is initialized, you will have a folder that's named .git. If you can't see it don't worry, because files that start with a period are usually hidden in file explorer. On a MacBook you can use Command + Shift + . to see the folder. Below image shows the contents of this file:

2. Find the hooks folder in .git folder

Once you can access .git folder, you will see a subfolder named hooks. This is where git hooks are saved.

Git hooks are scripts that run automatically every time a particular event occurs in a Git repository. They let you customize Git's internal behavior and trigger customizable actions at key points in the development life cycle.

Initially, there are no hooks, and there are only sample hooks found within this folder:

Each file in this subfolder correspond to a particular event when you are using Git. For example, actions determined in pre-commit run right before a commit is made. This is exactly when we want to create our sitemap.

3. Code in pre-commit file

Firstly, let's remove .sample suffix in .git/hooks/pre-commit.sample file and change its contents with below:

#!/usr/local/bin/python3

# Above line tells git that we'd like to use Python
# Your python executable might be in another location such as /usr/bin/python
# Please adjust accordingly

# Import necessary libraries
import os
from bs4 import BeautifulSoup
from datetime import datetime

# Build the website first to identify links
os.system("npm run build")

# Define variables
root = "www.danyelkoca.com"
today = datetime.today().strftime('%Y-%m-%d')

# Create a list that will keep all links in the website
links =  []

# Get all links in your pages using BeautifulSoup
# Exclude https:// sites, as they link to other pages
for path, subdirs, files in os.walk("path/to/your/build/folder"):
 for name in files:
  if name.endswith(".html"):
   with open(os.path.join(path, name)) as fp:
    soup = BeautifulSoup(fp, 'html.parser')
    page_links = [link["href"] for link in soup.find_all("a") if link["href"] and not link["href"].startswith("https") ]
    links += page_links
					

# Remove links that are repeated
links = list(set(links))

# Add one more item for the root (www.danyelkoca.com)
links += [""]

# Sort based on length (Shorter links usually are higher in hieararchy)
# E.g. /en
# /en/works
# /en/works/machine
links.sort(key=lambda s: len(s))

# Initialize xml with the links
xml = ""
for link in links:
    xml += f"   <url><loc>{root}{link}</loc><lastmod>{today}</lastmod></url>\n"


# Add parent elements
xml = f"""<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
{xml}</urlset>"""

# Save to your static folder
with open("path/to/your/static/sitemap.xml", "w") as file:
	file.write(xml)

Note: if you are getting Permission Denied error when committing, that means Git doesn't have the right permission to run code/ alter your files. Run below to give permission to Git:

chmod +x .git/hooks/pre-commit
chmod +x /usr/local/bin/python3

You are basically giving permission to your git hooks file and to your python executable. If you python executable is somewhere else, change the path accordingly.

Result

Once you stage the changes with git add . and commit your changes with git commit -m "message", the code that is defined above in .git/hooks/pre-commit will run and as a result, will save sitemap.xml in your static folder.

The code works and it creates a file like below (with prettify) as expected:

Looks neat right?

Once the sitemap.xml is generated, don't forrget to stage & commit the changes again as that file is created after the original commit is done.

Conclusion

In this blogpost we have devised a way to automate sitemap generation as a Git pre-commit hook using Python.

Using the code shared above, I hope you can save yourself some workload and have beer instead :)

Happy hacking!