Obsidian Publishing Workflow
By utilizing all the great and publicly available open-source tools, meticulously designing, and persevering despite the hardships, technical limitations, and lack of expertise, I’ve been able to automate my Obsidian publishing workflow. After 3 whole days of constant (and quite frankly back-breaking) development, I’m now able to update my site from within Obsidian with a single Hotkey. I’ll now break down how I’ve achieved this and how it’s all connected.
# Tools
- Obsidian: For note-taking and note-making.
- Obsidian Git: A plugin that sync my vault with the GitHub repo.
- obsidian-export: A CLI written in Rust that processes and exports notes from Obsidian.
- hugo-obsidian: A CLI written in Go that creates indexes that can later be used by Quartz to show graph etc.
- Quartz: A theme for Hugo SSG with extra features like graph view, backlinks, latex support, etc.
- Hugo: A static-site-generator written in Go.
- GitHub Actions: Allows us to automate our workflow based on GitHub events.
- Netlify: Deploys our website.
# How it’s all connected
# Obsidian
My obsidian vault is backed up in a private GitHub repository. This has been the case from the start. This allows me to have a reliable, free, and private backup of all my notes. I can also revert to a previous time, if need be.
Obsidian Git
Plugin allows me to either set an interval to automatically back up my vault or do it manually using a Hotkey. On each backup, it first performs pull
(download from repo if any changes have been made there) and then push
(upload all the changes to the GitHub repo).
# GitHub Action
Here’s the link to the actual configuration file,
GH Action configuration file to automatically publish my Obsidian Vault using Quartz (github.com). On each successful push
to the repository, a GitHub Action gets triggered. It has all the steps listed one after another to build and deploy the website.
- First, it checks out my
notes
and theQuartz
theme. - It then downloads
obsidian-export
to process the notes. - After that, it uses
hugo-obsidian
to generate link indices. - It then builds the
Hugo
website using theQuartz
theme and the processed notes. - Lastly, it pushes the built website onto the
gh-pages
branch.
# obsidian-export
Here’s a detailed overview of what obsidian-export
does. It is a rust CLI written by
Nick Groenen to process notes from obsidian.
It has several helpful features by default, such as:
- Reference Embedding: If you have a reference to a file/section, it will embed that into the actual document.
- It doesn’t currently work with
Block-level References
.
- It doesn’t currently work with
- Notes Filtering: You can specify all the files that you don’t want exported in a
.export-ignore
file.- This allows us to have more control over which file and directories we want to make public.
- Markdown Links: It will convert all the WikiLinks (
[[]]
) into Markdown Links ([]()
). It will also make those links relative to the root path.
To make it work for my particular workflow, I modified it to include the following features:
- Retain WikiLinks: I’ve added an option to retain the WikiLinks format, as Quartz and many others are able to parse those easily.
- Hugo Frontmatter: Add new or rename existing
yaml
properties, so they work with Hugo.- If there’s no
title
property, generate one from the file name. - Remove
alias/aliases
as they have a different functionality in Hugo. - If
created
ormodified
properties are missing, calculate them using file creation and modification time. Rename them todate
andlastmod
respectively. - If
summary
is empty, remove it. - If there’s a
publish
property, rename it to draft. This allows us to exclude certain pages from publishing. - Rename
id
property tourl
.- All of my notes have an
id
property in theYAML
frontmatter. - It is a
nanoid
which ensures that each note will have a unique URL-friendly string.
- All of my notes have an
- If there’s no
- Flat Hierarchy: Add option to generate a flat hierarchy.
- It replaces
.
and\
with-
. - If the
path
beforehand isSome.Dir/Some.File.md
, then it would becomeSome-Dir-Some-File.md
. - Hugo doesn’t work well with files and directories having a
.
in their path. - This option won’t result in any name collisions.
- It replaces
- Embed Info: Add some info regarding the embedding in surrounding
<div>
tags.- By default,
obsidian-export
has no indication that the content was embedded or not. - I’ve added
<div>
tags around the embedded content, so we can later style it however we want. - I’ve also added a link to the file that was embedded.
- This would allow us to create an experience like in Obsidian, where we can go to the original location of the embedded content.
- By default,
# hugo-obsidian
It scrapes all the links from vault and generates link and content indexes. These indexes are later used by Quartz to generate graph and to enable full-text search.
I’ve modified it to my liking as well:
- URL based: Use
url
property from the frontmatter for links.- By default, it uses file names to create
Source
andDestination
links. - I’ve modified it so that the
Source
andDestination
point to theurl
values instead of the file names.
- By default, it uses file names to create
- Strip Comments: Remove obsidian comments.
- This gives us more fine-grained control over what we want to share.
- If there’s a section within a file that we don’t want to share, we can simply wrap it inside Obsidian Comments
%%
. - I use this around
Dataview
tables and properties, as they don’t provide much value when published.- In Live Preview mode in Obsidian, the
Dataview
tables will remain visible.
- In Live Preview mode in Obsidian, the
# Cloudflare Pages
Whenever gh-pages
branch is updated, a Cloudflare Pages
deployment action gets triggered. It looks at the content in the gh-pages
branch and deploys it to the web. I have to use this because GitHub Pages doesn’t work with Private Repo
.
# Failed Experiments
# Netlify
First, I used Netlify
to publish my site in the end, instead of Cloudflare Pages
. But soon, I discovered a problem with that.
# Problem
Whenever I refreshed my page, the URL would instantly become lowercase. For example, if the URL was https://notes.aadam.dev/SBYNtPHqsTW9Ck1Kuoxsu
, then after page reload, it would automatically turn into https://notes.aadam.dev/sbyntphqstw9ck1kuoxsu/
.
This is a major issue, as this doesn’t guarantee that all the IDs will be unique now. Before, AbC
and aBC
would’ve been treated as two different entities, but with this bug, they would resort to a single entity, resulting in a naming collision.
Also, even though my page was being loaded, the graph at the end of the page wasn’t working. This is because it also expects the IDs to be case-sensitive. So, when the URL turns to lowercase, it isn’t able to find that ID.
# Solution
I spent a couple of hours trying to fix this problem. I wasn’t even sure where to look. There wasn’t much information about this on Hugo Forum, Stackoverflow, Google, etc.
At first, I thought that there’s some issue with Hugo and how I deploy it. I changed my GitHub Action workflow and tried different things. I tried moving the building process to Netlify instead of GitHub, but that had its own issues. I also tried moving to Vercel, but that didn’t work out either. I tried changing Hugo configuration files but still, no luck.
In the end, I came across an article which pinpointed the issue for me:
Gotcha: Netlify Makes All Your Filenames Case-Insensitive · Jamie Tanna | Software Engineer (jvt.me). Lo and behold, it was Netlify
all along. I was surprised because the website was working fine locally on my PC, but it was acting up when published.
Because of this, I moved my deployment workflow to Cloudflare Pages. Everything else is just the same, only the last part of the website deployment is shifted.