nim-serde

Easy-to-use json serialization capabilities, and a drop-in replacement for std/json.

Quick examples

Opt-in serialization by default:

import pkg/serde/json

type MyType = object
  field1 {.serialize.}: bool
  field2: bool

assert MyType(field1: true, field2: true).toJson == """{"field1":true}"""

Opt-out deserialization by default:

import pkg/serde/json

# All fields deserialized, as none are ignored
type MyType1 = object
  field1: bool
  field2: bool

let jsn1 = """{
                "field1": true,
                "field2": true
              }"""
assert !MyType1.fromJson(jsn1) == MyType1(field1: true, field2: true)

# Don't deserialize ignored fields in OptOut mode
type MyType2 = object
  field1 {.deserialize(ignore=true).}: bool
  field2: bool

let jsn2 = """{
                "field1": true,
                "field2": true,
                "extra": "extra fields don't error in OptOut mode"
              }"""
assert !MyType2.fromJson(jsn2) == MyType2(field1: false, field2: true)

# Note, the ! operator is part of https://github.com/codex-storage/questionable, which retrieves a value if set

Serialize all fields of a type (OptOut mode):

import pkg/serde/json

type MyType {.serialize.} = object
  field1: int
  field2: int

assert MyType(field1: 1, field2: 2).toJson == """{"field1":1,"field2":2}"""

Alias field names in both directions!

import pkg/serde/json

type MyType {.serialize.} = object
  field1 {.serialize("othername"),deserialize("takesprecedence").}: int
  field2: int

assert MyType(field1: 1, field2: 2).toJson == """{"othername":1,"field2":2}"""
let jsn = """{
                "othername":       1,
                "field2":          2,
                "takesprecedence": 3
              }"""
assert !MyType.fromJson(jsn) == MyType(field1: 3, field2: 2)

Supports strict mode, where type fields and json fields must match

import pkg/serde/json

type MyType {.deserialize(mode=Strict).} = object
  field1: int
  field2: int

let jsn = """{
                "field1": 1,
                "field2": 2,
                "extra":  3
              }"""

let res = MyType.fromJson(jsn)
assert res.isFailure
assert res.error of SerdeError
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"

Serde modes

nim-serde uses three different modes to control de/serialization:

OptIn
OptOut
Strict

Modes can be set in the {.serialize.} and/or {.deserialize.} pragmas on type definitions. Each mode has a different meaning depending on if the type is being serialized or deserialized. Modes can be set by setting mode in the serialize or deserialize pragma annotation, eg:

type MyType {.serialize(mode=Strict).} = object
  field1: bool
  field2: bool

Modes reference

	serialize	deserialize
SerdeMode.OptOut	All object fields will be serialized, except fields marked with {.serialize(ignore=true).}.	All json keys will be deserialized, except fields marked with {.deserialize(ignore=true).}. No error if extra json fields exist.
SerdeMode.OptIn	Only fields marked with {.serialize.} will be serialized. Fields marked with {.serialize(ignore=true).} will not be serialized.	Only fields marked with {.deserialize.} will be deserialized. Fields marked with {.deserialize(ignore=true).} will not be deserialized. A SerdeError error is raised if the field is missing in json.
SerdeMode.Strict	All object fields will be serialized, regardless if the field is marked with {.serialize(ignore=true).}.	Object fields and json fields must match exactly, otherwise a SerdeError is raised.

Default modes

nim-serde will de/serialize types if they are not annotated with serialize or deserialize, but will assume a default mode. By default, with no pragmas specified, serde will always serialize in OptIn mode, meaning any fields to b Additionally, if the types are annotated, but a mode is not specified, serde will assume a (possibly different) default mode.

# Type is not annotated
# A default mode of OptIn (for serialize) and OptOut (for deserialize) is assumed.

type MyObj = object
  field1: bool
  field2: bool

# Type is annotated, but mode not specified
# A default mode of OptOut is assumed for both serialize and deserialize.

type MyObj {.serialize, deserialize.} = object
  field1: bool
  field2: bool

Default mode reference

	serialize	deserialize
Default (no pragma)	OptIn	OptOut
Default (pragma, but no mode)	OptOut	OptOut

Serde field options

Type fields can be annotated with {.serialize.} and {.deserialize.} and properties can be set on these pragmas, determining de/serialization behavior.

For example,

import pkg/serde/json

type
  Person {.serialize(mode=OptOut), deserialize(mode=OptIn).} = object
    id {.serialize(ignore=true), deserialize(key="personid").}: int
    name: string
    birthYear: int
    address: string
    phone: string

let person = Person(
              name: "Lloyd Christmas",
              birthYear: 1970,
              address: "123 Sesame Street, Providence, Rhode Island  12345",
              phone: "555-905-justgivemethedamnnumber!⛽️🔥")

let createRequest = """{
  "name": "Lloyd Christmas",
  "birthYear": 1970,
  "address": "123 Sesame Street, Providence, Rhode Island  12345",
  "phone": "555-905-justgivemethedamnnumber!⛽️🔥"
}"""
assert person.toJson(pretty=true) == createRequest

let createResponse = """{
  "personid": 1,
  "name": "Lloyd Christmas",
  "birthYear": 1970,
  "address": "123 Sesame Street, Providence, Rhode Island  12345",
  "phone": "555-905-justgivemethedamnnumber!⛽️🔥"
}"""
assert !Person.fromJson(createResponse) == Person(id: 1)

key

Specifying a key, will alias the field name. When seriazlizing, json will be written with key instead of the field name. When deserializing, the json must contain key for the field to be deserialized.

ignore

Specifying ignore, will prevent de/serialization on the field.

Serde field options reference

	serialize	deserialize
key	aliases the field name in json	deserializes the field if json contains key
ignore	OptOut: field not serialized OptIn: field not serialized Strict: field serialized	OptOut: field not deserialized OptIn: field not deserialized Strict: field deserialized

Deserialization

serde deserializes using fromJson, and in all instances returns Result[T, CatchableError], where T is the type being deserialized. For example:

type MyType = object
 field1: bool
 field2: bool

let jsn1 = """{
               "field1": true,
               "field2": true
             }"""

assert !MyType.fromJson(jsn1) == MyType(field1: true, field2: true)

If there was an error during deserialization, the result of fromJson will contain it:

import pkg/serde/json

type MyType {.deserialize(mode=Strict).} = object
  field1: int
  field2: int

let jsn = """{
                "field1": 1,
                "field2": 2,
                "extra":  3
              }"""

let res = MyType.fromJson(jsn)
assert res.isFailure
assert res.error of SerdeError
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"

Custom types

If serde can't de/serialize a custom type, de/serialization can be supported by overloading % and fromJson. For example:

type
  Address* = distinct array[20, byte]
  SerializationError* = object of CatchableError

func `%`*(address: Address): JsonNode =
  %($address)

func fromJson(_: type Address, json: JsonNode): ?!Address =
  expectJsonKind(Address, JString, json)
  without address =? Address.init(json.getStr), error:
    return failure newException(SerializationError,
      "Failed to convert '" & $json & "' to Address: " & error.msg)
  success address

Serializing to string (toJson)

toJson is a shortcut for serializing an object into its serialized string representation:

import pkg/serde/json

type MyType {.serialize.} = object
  field1: string
  field2: bool

let mt = MyType(field1: "hw", field2: true)
assert mt.toJson == """{"field1":"hw","field2":true}"""

This comes in handy, for example, when sending API responses:

let availability = getAvailability(...)
return RestApiResponse.response(availability.toJson,
                                contentType="application/json")

std/json drop-in replacment

nim-serde can be used as a drop-in replacement for the standard library's json module, with a few notable improvements.

Instead of importing std/json into your application, pkg/serde/json can be imported instead:

- import std/json
+ import pkg/serde/json

As with std/json, % can be used to serialize a type into a JsonNode:

import pkg/serde/json

assert %"hello" == newJString("hello")

And %* can be used to serialize objects:

import pkg/serde/json

let expected = newJObject()
expected["hello"] = newJString("world")
assert %*{"hello": "world"} == expected

As well, serialization of types can be overridden, and serialization of custom types can be introduced. Here, we are overriding the serialization of int:

import pkg/serde/json

func `%`(i: int): JsonNode =
  newJInt(i + 1)

assert 1.toJson == "2"

parseJson and exception tracking

Unfortunately, std/json's parseJson can raise an Exception, so proper exception tracking breaks, eg

## Fails to compile:
## Error: parseJson(me, false, false) can raise an unlisted exception: Exception

import std/json

{.push raises:[].}

type
  MyAppError = object of CatchableError

proc parseMe(me: string): JsonNode =
  try:
    return me.parseJson
  except CatchableError as error:
    raise newException(MyAppError, error.msg)

assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }

This is due to std/json's parseJson incorrectly raising Exception. This can be worked around by instead importing serde and calling its JsonNode.parse routine. Note that serde's JsonNode.parse returns a Result[JsonNode, CatchableError] instead of just a plain JsonNode object as in std/json's parseJson:

import pkg/serde/json

{.push raises:[].}

type
  MyAppError = object of CatchableError

proc parseMe(me: string): JsonNode {.raises: [MyAppError].} =
  without parsed =? JsonNode.parse(me), error:
    raise newException(MyAppError, error.msg)
  parsed

assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }

Known issues

There is a known issue when using mixins with generic overloaded procs like fromJson. At the time of mixin call, only the fromJson overloads in scope of the called mixin are available to be dispatched at runtime. There could be other fromJson overloads declared in other modules, but are not in scope at the time the mixin was called. Therefore, anytime fromJson is called targeting a declared overload, it may or may not be dispatchable. This can be worked around by forcing the fromJson overload into scope at compile time. For example, in your application where the fromJson overload is defined, at the bottom of the module add:

static: MyType.fromJson("")

This will ensure that the MyType.fromJson overload is dispatchable.

The basic types that serde supports should already have their overloads forced in scope in the deserializer module.

For an illustration of the problem, please see this narrow example by @gmega.