@sdgindex/data

Package for the SDG Index. Helpers for managing the data for our data visualizations.

Full helper documentation can be found at: https://code.sdgindex.org/

Installation

Just run npm install @sdgindex/data.

Usage

Parsing Data

You can use the @sdgindex/data/parse helpers for transforming data from Excel into JSON format. The recommended approach is to create a data/ folder with a prepare.js file:

// data/prepare.js
const { writeStoreToJson } = require("@sdgindex/data/parse");
const setUpAssessments = require("./setUpAssessments");
const setUpRegions = require("./setUpRegions");
const setUpObservations = require("./setUpObservations");
const setUpTimeseries = require("./setUpTimeseries");

// Get data from source Excel files
setUpAssessments();
setUpRegions();
setUpObservations();
setUpTimeseries();

// Write data
writeStoreToJson();

Each setUp-script takes care of one aspect of the data. The writeStoreToJson() function will write all parsed data as JSON files in public/data/.

Fetching Data on Mount

All data should be loaded after the page has finished rendering on the client. This way, platform navigation will be fast and smooth.

Include the following hook in the _app.js page:

// pages/_app.js
import { useEffect } from 'react';
import { loadData } from "@sdgindex/data";

function MyApp() {
  // ...

  // Load data
  useEffect(() => {
    loadData({ timeseries: true })
  }, [])

  return (
    // ...
  )
};

Fetching Data on Page Navigation

Every page that needs to access some data from the store should load the data in its getStaticProps. Explicitly map the results from the data store into your own custom object. This reduces the chance for errors later. Any @sdgindex/data helpers should be used in getStaticProps only. You should not be using any helpers in the rendering of the page.

// pages/profiles.js

const Profiles = () => {
  /* ... */
};

import { loadData, getRegions } from "@sdgindex/data";
import { isCountry } from "@sdgindex/data/regions";

export async function getStaticProps() {
  await loadData();

  // Sort country by name
  const regions = getRegions()
    .filter(isCountry)
    .sort((a, b) => a.name.localeCompare(b.name));

  return {
    regions: regions.map((region) => ({
      id: region.id,
      name: region.name,
      slug: region.slug,
      type: region.type,
    })),
  };
}

If a page has Dynamic Routes and uses getStaticProps, it needs to define a list of paths to be statically generated.

Therefore, on this page both getStaticProps and getStaticPaths are necessary.

When you export a function called getStaticPaths (Static Site Generation) from a page that uses dynamic routes, Next.js will statically pre-render all the paths specified by getStaticPaths.

If you are loading data from @sdgindex/data, you will need to use the same approach as for getStaticProps to generate the paths to pre-render.

Building on top of the previous example, let's say we are now building a page for individual country profiles. We're not going to create 193 pages for each country if they all share the same structure. We'll create [slug].js where slug represents the country slug variable. However, we need to tell Next.js what pages to statically pre-render at build time. In this case, slug can be one of 193 different options.

The changes made to the code are:

• the addition of getStaticPaths to create the 193 possible paths.
• the refactoring of getStaticProps that now has a context as a parameter in which we find the a property called params. params contains the slug (slug because this is how we named the file) that has been passed to the getStaticProps.

// pages/profiles/[slug].js

const IndividualCountryProfile = () => {
  /* ... */
};

// Add findRegionBySlug to get the data for specific region based on slug
import { loadData, getRegions, findRegionBySlug } from "@sdgindex/data";
import { isCountry } from "@sdgindex/data/regions";

// This function gets called at build time
export async function getStaticPaths() {
  await loadData();

  // Get the paths we want to pre-render based on regions
  const regions = getRegions().filter(isCountry);
  const paths = regions.map(({ slug }) => ({
    params: { slug },
  }));

  // We'll pre-render only these paths at build time.
  // { fallback: false } means other routes should 404.
  return { paths, fallback: false };
}

export async function getStaticProps({ params }) {
  // Country slug
  const { slug } = params;
  // Ensure data is loaded
  await loadData();

  // Get the data
  const region = findRegionBySlug(slug);

  return {
    region: {
      id: region.id,
      name: region.name,
      slug: region.slug,
      type: region.type,
    },
  };
}

Note

If the objects in the array of objects loaded by loadData don't all have the same properties (ie: if some regions have type : country and at least one region has no type type : undefined), as getStaticProps passes the data in json, instead of accepting undefined, you have to use null as the value for your variables.

The value undefined denotes that a variable has been declared, but hasn't been assigned any value. So, the value of the variable is undefined. On the other hand, null refers to a non-existent object, which basically means 'empty' or 'nothing'.

Using the above example: we need to delare in getStaticProps that if type doesn't exist, return null.

return {
  region: {
    id: region.id,
    name: region.name,
    slug: region.slug,
    type: region?.type,
    // or...
    // type: region.type || null
  },
};

Fetching Data in Component

You may need to access additional data from the store when the user interacts with the page (e.g., side menu showing timeseries data). In those cases, you should use the useDataStore hook and ensure that data is fully loaded before making use of any of the @sdgindex/data helpers.

Again, this should only be used in components that are rendered client-side only after some sort of user interaction. For data that is rendered by default, use getInitialProps to fetch all necessary data.

// components/MyComponent.js
import { getRegions, useDataStore } from "@sdgindex/data";

const MyComponent = () => {
  // useDataStore uses useState internally.
  // The value of isLoaded will automatically update
  // when the data has fully loaded.
  const { isLoaded } = useDataStore();

  if (!isLoaded) return "Loading...";

  // You can now use any @sdgindex/data helpers
  const regions = getRegions();

  return <div>{regions.map((region) => region.name)}</div>;
};

Development

Sometimes, you may want to use your local copy of @sdgindex/data in your local Next.js app. For example, to test a new feature that you're developing on @sdgindex/data. In those cases, you can navigate to your Next.js app and run npm link ../path/to/@sdgindex/data to use the local version of @sdgindex/data.

Remember to disconnect for this local copy by running npm unlink