RIP Gatsby

I ditched Gatsby.

I decided to ditch Gatsby because after an update to how Gatsby's resources were loaded, my site wouldn't build anymore. I could just use an old version of Gatsby forever, but even with the old version things were breaking because of dependencies of Gatsby's dependencies in its package.json (welcome to the NPM ecosystem). I didn't want to my entire project's to hinge on a single package-lock.json, so I decided to write my own framework.

Death to Webpack, too

Webpack is a little black box everyone depends on.

I like Webpack, so I experimented with building my site using prerender-loader or prerender-SPA-plugin paired with Preact-router. I never want to actually use a router since I have to update it every post, but it was bearable and I could dynamically generate routes based on globbing my local directories.

There were a lot of small annoying problems I ran into with prerendering (like needing React AND Preact). The real problem ended up being I had to use the same bloated JS to generate the static HTML as I bundled in the page. This includes things I never want to ship in my bundle like preact-helmet. There was the issue of dynamically templating some pages to NOT include JS...

Suffice it to say it got complicated FAST (I'd have to run webpack twice -- once to prerender, and once to bundle) so I gave up on a Webpack approach. I briefly checked out Parcel but it seems that the approach is flawed more than the tools. Bundlers are made for SPAs.

What I actually want

Since my pages don't include JS, I really just want a templating framework, like hexo or hugo to render my site's HTML. However, I want to write React components in JSX instead of EJS or %-style PHP templates. For example, I already had the post template, article previews, and the breadcrumb written in React,.

{posts.map(post => ArticlePreview(post))}

Using React as my templating engine makes it easy for when I want to write JS for things like my retirement calculator. I can just keep using React.

Back to the basics

I needed to render my components to HTML using something like preact-render-to-string. Rendering React components per-markdown file seems like such a simple use case, yet no other frameworks seemed to have me covered.

To get started using preact-render-to-string, I needed a bit of hackery using @babel/register:

const path = require('path')
const fs = require('fs-extra')
const mime = require('mime')
const render = require('preact-render-to-string')
const { h } = require('preact')
// Hack to hijack all require()s and babel them
require('@babel/register')({
  ignore: [/node_modules/],
  plugins: [
    ['@babel/plugin-transform-react-jsx', { pragma: 'h' }, ],
    ['@babel/plugin-proposal-class-properties']
  ],
  presets: ['@babel/preset-env']
})

// All further `require()`s go through Babel (so they can 
// include JSX, class properties, and `import` statements)
const { myComponent } = require('some-component');
console.log(render(h(myComponent)))

After figuring out preact-render-to-string was something I could use with JSX without having to have extra intermediate files, I was SOLD!!

Let's commit to this renderer

I need to extract some metadata from Markdown files (like the post's title and date) to include in my templated site components (like the post template). Unified is THE natural choice for this in NodeJS land.

Enter: MDX

I also wanted to be able to include React components in my Markdown files so dynamic components can be statically rendered like the retirement calculator. Enter MDX, which has a plugin for Unified called remark-mdx.

I created a Unified pipeline that used remark-mdx like this:

const markdownPipe = require('unified')()
    .use(require('remark-parse'))
    // Frontmatter
    .use(require('remark-frontmatter'))
    .use(require('remark-parse-yaml'))
    .use(() => (ast, file) => {
        visit(ast, 'yaml', item => {
            file.data.frontmatter = item.data.parsedValue
            const { frontmatter } = file.data
            frontmatter.dateShort = moment(frontmatter.date).format('YYYY-MM-DD')
            frontmatter.dateLong = moment(frontmatter.date).format('MMMM DD, YYYY')
        });
    })
    .use(() => (ast, file) => {
        let { frontmatter } = file.data
        let excerpt = ''
        visit(ast, 'text', item => {
            excerpt += item.value + ' '
        })

        frontmatter.excerpt = excerpt.substr(0, 150).trim()
        // Assume 300wpm reading speed
        // Round to nearest .5
        frontmatter.timeToRead = Math.round(excerpt.split(' ').length / 300 * 2) / 2
    })
    // Render to JSX
    .use(require('remark-mdx'))
    .use(require('./mdx-ast-to-mdx-hast'))
    .use(require('./mdx-hast-to-jsx'))

MDX's default way of rendering to JSX wasn't exactly what I wanted, so I wrote my own ast-to-mdx-hast and mdx-hast-to-jsx built on top of remark-mdx.

Now for each one of my Markdown files, I have a .md.js file that looks like this:

import { h, Fragment } from 'preact'

export default () => <Fragment>
  {/* Content here */}
</Fragment>;

Now I just require() that file and use preact-render-to-string on it! I also collect some metadata on all the posts rendered in order to build index pages that list all the posts.

But you need some JavaScript

For interactive pages, I do need some JavaScript. Rollup played the role of Babelling, bundling, and minifying JS files like this:

import { h, render } from 'preact'
import { Logo } from '../../../src/components/logo'

const root = document.getElementById('preact-logo')
render(h(Logo), root, root)

The config is a little overly verbose, but it's not terrible:

rollup.rollup({
  input: file,
  plugins: [
    resolve(),
    babel({ exclude: 'node_modules/**' }),
    terser(),
  ],
}).then(bundle => bundle.generate({
    format: 'iife', // immediately invoked function expression
    name: slug.split('/').pop().replace(/-/g, ''), // global variable name representing your bundle
    compact: true,
  }).then(({ output }) => console.log(output[0].code))
}))

The more annoying part is injecting the CSS and JS file names dynamically into the HTML templates. I just kept track of where I wrote out the CSS and JS files. For development, I don't bother changing the filename to include a hash because that's a lot of work.

Orchestrating it all

I used to have a single NodeJS script orchestrating all the tools, but I just decided to use Gulp, which is a nice task runner to read and write files at certain paths. The default task of my Gulpfile:

• Creates assets with no dependencies: parallel(copyStaticAssets, copyPostAssets, css, js)
  • Populates cssFileNames = [] and jsFileNames =
• Renders posts (and populates posts = to pass to index pages)
• Renders Pages
• Cleans up a file that all posts that are drafts and shouldn't be rendered get pointed to

Creating the start task

This is what Gulp made easy for me since I didn't use a pre-built framework with nice hot-reloading for me. Gulp wraps chokidar's watch function:

function start() {
    watch(paths.postAssets.src, { ignoreInitial: false }, copyPostAssets)
    watch(paths.staticAssets.src, { ignoreInitial: false }, copyStaticAssets)

    // TODO: parallel css/js
    css()
        .on('end', () => js() // TODO: rollup.watch
            .on('end', () => {
                watch(paths.sass.src, css)
                watch(jsWatchFiles, js)
                watch(paths.posts.src, { ignoreInitial: false }, series(renderPosts, renderPages))
                watch(paths.pages.src, { ignoreInitial: false }, renderPages)
            })
        )
}

I do still have to reload my page manually after my site rebuilds, but I could look into using browser-sync to auto-reload it. That would also save me having http-server open in another terminal serving my dist folder.

Reflection

All this work took me upwards of 25 hours. The hardest part was writing the remark plugins for MDX. A lot of it was wasted researching prerendering, and I did write a few posts in the meantime.

It was well worth it to control every tool that creates the HTML, CSS, and JS my readers read. I'm (fairly) framework agnostic and learned a ton of new things!

Future work

My hot-reloading development workflow still needs a few changes:

• Node-sass as well as Rollup have their own watch functions, so I should use those.
• On changes to my component templates, all pages should reload and we need to re-require the module. Currently nothing happens.

My production environment also needs a few fixes:

• I need to minimize my images and wrap them in a container to avoid the content changing height once loaded. I also need to use srcset for responsive images. I'd also like a nicer way to create centered images for posts than having to write all this each time:

<figure>
  <img src="shut-up-and-take-my-money.jpg" alt="Take my money" />
  <figcaption>Shut up and take my soul.</figcaption>
</figure>

• I need a way to separate out my Preact dependency so it doesn't have to reload 4kb on every JS page.