How to build an Astro collection loader

A few days ago Astro released version 1.14 which introduced a new feature called the “Content Layer API”. This feature builds on the existing content collections feature and instead expands it so you can use data from sources other than local files within the Astro project repository.

Now, to get started the Astro team has already published a number of loaders to deal with the most common possible cases you may need. Including “feeds” like RSS feeds, CSV files, etc. But what if you wanted to build your own? Well it’s actually quite simple.

Setting up Astro

First things first, you need to set up a new Astro project. You can do this by running the following command:

npm create astro@latest

This will walk you through a very fancy CLI guide to set up your project.

Once you have done that, you can run the following command to start your project:

npm run dev

With this up and running you can visit http://localhost:4321 to see your new Astro project.

Enable the experiment

Next, you will need to make one slight tweak to the Astro configuration file to enable the Content Layer API as it’s currently still experimental.

To do this, open the astro.config.mjs file and add a experimental object with a contentLayer key set to true like so:

import { defineConfig } from 'astro/config';
 
// https://astro.build/config
export default defineConfig({
	experimental: {
		contentLayer: true,
	},
});

The basic structure

With the experimental API enabled we can now actually start to build out the basic structure for a collection loader.

Now, for the sake of example I’ll be using TypeScript but it’s not strictly required.

First up, create a new file in your project. I’ll call mine src/loaders/jokes.ts. This file will be responsible for loading in a collection of posts.

import type { Loader } from 'astro/loaders';
 
export const jokesLoader: Loader = {
	name: 'jokes',
	load: async (context) => {},
};

At its most basic premise a loader is an object with a 2 required properties & one optional property:

• name (Required): This will be the name of the loader and how it will show up in logs and such.
• load (Required): Here is the actual “loader logic” function where you actually fetch the data from whatever source you need. This function exposes a loader context parameter which can be used to access things like the store where the data will actually get stored, logger for logging, etc.
• schema (Optional): If you want to provide a Zod schema to validate the data after it’s been fetched you can do so here.

Fetch some data

Now we need some data, let’s use this dad joke API to get a bunch of random dad jokes.

To do this in the load function all we need to do is make a fetch request. For this API we just need to make sure we set the Accept header to application/json to get the data in JSON format.

import type { Loader } from 'astro/loaders';
 
export const jokesLoader: Loader = {
	name: 'jokes',
	load: async (context) => {
		const response = await fetch('https://icanhazdadjoke.com/', {
			headers: {
				Accept: 'application/json',
			},
		});
 
		const json = await response.json();
 
		context.logger.info(JSON.stringify(json));
	},
};

I’ve added a console.log statement for now just so we can check it works. But how do we do that?

Using the loader

With some actual logic in the loader we can hook it up to a collection to see what data we get back from the API.

To do this create the following file src/content/config.ts:

import { defineCollection } from 'astro:content';
 
import { jokesLoader } from '../loaders/jokes';
 
const jokes = defineCollection({
	loader: jokesLoader,
});
 
export const collections = {
	jokes,
};

With this we should now have a new collection called jokes that we can access in our Astro project.

If we run a build of the project with npm run build we should see the loader being run and the data being logged to the console.

21:32:37 [content] Syncing content
21:32:37 [jokes] {"id":"mWS7hVKRSnb","joke":"Past, present, and future walked into a bar.... It was tense.","status":200}
21:32:37 [content] Synced content

Perfect, we can see our loader ran by the [jokes] log message and the data being returned from the API. Additionally it confirms the data structure we get back.

Store the data

Now we have some data we need to actually store it so we can access it in our Astro project.

To do this we can use the store scoped data store that is available on the context parameter. This ScopedDataStore appears to be some form of a superset of a regular Map object.

When actually setting the data you need to provide a unique id for the data, along with the data itself. This works quite well for us here as the API gives us an id field for each joke.

import type { Loader } from 'astro/loaders';
 
export const jokesLoader: Loader = {
	name: 'jokes',
	load: async (context) => {
		const response = await fetch('https://icanhazdadjoke.com/', {
			headers: {
				Accept: 'application/json',
			},
		});
		
		const json = await response.json();
		
		context.store.set({
			id: json.id,
			data: json,
		});
	},
};

Accessing the data

Since we removed the logging builds will still succeed if you run npm run build but we won’t see the data being logged anymore.

To actually access the data, you can access it like you would any other collection in Astro.

For example, in an Astro file you can use the getCollection function to get the data.

---
import { getCollection } from 'astro:content';
 
const jokes = await getCollection('jokes');
console.log('jokes', jokes);
---

Running npm run build with the above added you should see something along the lines of:

 generating static routes
22:05:09 ▶ src/pages/index.astro
22:05:09   └─ /index.htmljokes [
  {
    id: '9prWnjyImyd',
    data: {
      id: '9prWnjyImyd',
      joke: 'Why do bears have hairy coats? Fur protection.',
      status: 200
    },
    collection: 'jokes'
  }
]
 (+6ms)
22:05:09 ✓ Completed in 9ms.

And just like that it works! Now every time you build your Astro project it will fetch a new dad joke from the API and store it in the collection.

Conclusion

And that’s it! You’ve now built your own collection loader for Astro. This is a very basic example but it should give you a good starting point to build more complex loaders.

Later down the line this can be easily adapted into a function to return a loader so you could say take in some user options and publish it as a package for others to use.

I’m excited to see what other loaders people come up with now and in the future.