Implement decoding library in 60 lines

I was explaining to a friend how decoding libraries like io-ts or zod work and why need them. To do so I wrote this code for him and as it was helpful sharing here as well.

Use of decoding libs is important in most real world applications. Reason is that you usually interact with various sources of data that are not hardcoded or generated by your program but instead are generated by other programs or humans. And generally we shouldn’t trust other programs or humans in our program. When making some api request or reading some json file, you have some expectation of what the shape of loaded data should be but your expectations sometimes could be far from reality. Maybe you are absolutely sure. But times go by and you the shape of data could change and because of you being absolute sure now you are debugging why null has no property "id" or something.

I.e. Doing this is not great:

const user: User = JSON.parse(res.body);

because return type of JSON.parse is any this compiles just fine. Even if type was correct (like returning unknown or some type representing json values). Then you would try this, which is also not great:

const user = JSON.parse(res.body) as User;

What you really want is to check in runtime if the resulting json value is indeed of proper shape. But ideally you also want to remove all unnecessary fields too. But let’s start from basic approach of using guard functions:

type User = {
  name: string;
  pass: string;
  age: number;
  admin: boolean;
  meta: string | number | boolean;
};

function isUser(val: unknown): val is User {
  return (
    typeof val === "object" &&
    val !== null &&
    "name" in val &&
    typeof val.name === "string" &&
    "pass" in val &&
    typeof val.pass === "string" &&
    "age" in val &&
    typeof val.age === "number" &&
    "admin" in val &&
    typeof val.admin === "boolean" &&
    "meta" in val &&
    (typeof val.meta === "string" ||
      typeof val.meta === "number" ||
      typeof val.meta === "boolean")
  );
}

this looks fine but at some point you would realize that guard functions are as unsafe as unsafe casts i.e. If you had written following, typescript compiler would have been happy:

function isUser(val: unknown): val is User {
  return true;
}

You can read more about it here.

Because of that this is error prone. Plus it’s not that nice to write this kind of huge conditional, and if the conditional failed you have no idea why. To solve this problem there are various decoding libs (like io-ts or zod) and here we will write one which will communicate the meta api of such libs and than you can pick whatever you like base on your needs. This one will not have nice error reporting but you could image how one could improve it, by then actually trying out proper decoding libs.

Fundamentally decoder is a function. It takes value of some type that is usually bigger (i.e. many possible values could be in it like unknown) and returns value of some type that is usually smaller (number, User etc) i.e. It rejects some values from input. If it get’s invalid input it fails. This failing part in this example will be done by just throwing exceptions but in practice such libs will return some kind of Result/Either type where you have decoding error or decoded result.

type Decoder<I, O> = (_: I) => O;

from this type we can write small utility types that would return input of decoder and Resulting type of decoder:

type TypeOf<D> = D extends Decoder<any, infer R> ? R : never;
type InputOf<D> = D extends Decoder<infer R, any> ? R : never;

//  Test1 : boolean
type Test1 = TypeOf<Decoder<unknown, boolean>>;
//  Test3 : unknown
type Test3 = InputOf<Decoder<unknown, boolean>>;

Now let’s get into writing some basic decoders of various JS’s primitive types.

const string: Decoder<unknown, string> = (x) => {
  if (typeof x === "string") return x;
  throw new Error("Expected string");
};

const number: Decoder<unknown, number> = (x) => {
  if (typeof x === "number") return x;
  throw new Error("Expected number");
};

const boolean: Decoder<unknown, boolean> = (x) => {
  if (typeof x === "boolean") return x;
  throw new Error("Expected boolean");
};

const _null: Decoder<unknown, null> = (x) => {
  if (x === null) return x;
  throw new Error("Expected null");
};

As you we have basic types now let’s get into composite types, i.e. Types which consist of other types

function array<T>(decoder: Decoder<unknown, T>): Decoder<unknown, T[]> {
  return (input: unknown) => {
    if (Array.isArray(input)) return input.map((x) => decoder(x));
    throw new Error("Expected array");
  };
}

function object<T extends Record<string, Decoder<unknown, unknown>>>(
  spec: T
): Decoder<unknown, { [P in keyof T]: TypeOf<T[P]> }> {
  return (input) => {
    const res = {} as any;
    for (const key in spec) {
      if (key in (input as any)) {
        res[key] = spec[key]((input as any)[key]) as any;
      } else {
        throw new Error(`Expected ${key} to be present`);
      }
    }
    return res;
  };
}

function union<MS extends [Decoder<any, any>, ...Array<Decoder<any, any>>]>(
  ...decoders: MS
): Decoder<InputOf<MS[number]>, TypeOf<MS[number]>> {
  return (input) => {
    for (const decoder of decoders) {
      try {
        return decoder(input);
      } catch (_) {}
    }
    throw new Error(
      `Expected to decode using ${decoders.length} decoders but all failed`
    );
  };
}

now we can use those decoders to write decoder of the User:

// userDecoder: Decoder<unknown, {
//     name: string;
//     pass: string;
//     age: number;
//     admin: boolean;
//     meta: string | number | boolean;
//   }>
const userDecoder = object({
  name: string,
  pass: string,
  age: number,
  admin: boolean,
  meta: union(string, number, boolean),
});

Now this is much readable imo, and easy to maintain and trust. Note that it has type of User in it. So we can extract that instead of writing User type by hand.

type User = TypeOf<typeof userDecoder>;

And we can continue and build up more decoders with existing once. For example let’s say some endpoint returns not one but multiple users? np:

const usersDecoder = array(userDecoder);

So go ahead and play with this, try decoding valid values, try decoding invalid values, try implementing function record<T>(decoder: Decoder<unknown, T>): Decoder<unknown, Record<string, T>>;, if you are brave try to use some Result type that returns more informative error than what we have.