v0.4.1

Optimisers v0.4.1

Diff since v0.4.0

Merged pull requests:

• Add Duplicated methods (#192) (@mcabbott)

v0.4.0

Optimisers v0.4.0

Diff since v0.3.4

Merged pull requests:

• make docstrings consistent (#187) (@CarloLucibello)
• Add the option couple to AdamW and set the default to match pytorch (#188) (@CarloLucibello)
• fix epsilon for Float16 (#190) (@CarloLucibello)
• docs for nothing behavior and for walking a tree with keypath (#191) (@CarloLucibello)

Closed issues:

• Stable docs will 404 until a new version is tagged (#25)
• Allow keyword arguments for optimisers (#74)
• doc improvement: working with custom model types (#84)
• Rename or outsource iswriteable (#99)
• Split out the rules.jl as a sub-package (or a separate package) ? (#108)
• Wrong model update for BatchNorm for some specific synthax (#123)
• Use OptChain as an alias for OptimiserChain? (#138)
• nothing does not correspond to updating the state with a zero gradient. (#140)
• Utility for walking a tree (e.g. gradients) w.r.t. a model (#143)
• Adam optimizer can produce NaNs with Float16 due to small epsilon (#167)
• mark as public any non-exported but documented interface (#189)

v0.3.4

Optimisers v0.3.4

Diff since v0.3.3

Merged pull requests:

• Fix macro signature in docstring (#184) (@abhro)
• fix docs (#185) (@CarloLucibello)
• allow Functor 0.5 (#186) (@CarloLucibello)

Closed issues:

• Design (#1)
• convenience constructors (#11)
• Restructure is not type stable but could be made stable? (#177)
• AdamW optimizer implemented incorrectly - weight decay does not incorporate learning rate (#182)

v0.3.3

Optimisers v0.3.3

Diff since v0.3.2

Merged pull requests:

• add missing SignDecay doc reference (#168) (@CarloLucibello)
• fix broken doc links (#170) (@CarloLucibello)
• add trainables (#171) (@CarloLucibello)
• fix broken documentation (#172) (@CarloLucibello)
• add path option to trainables (#174) (@CarloLucibello)

Closed issues:

• Documenter CI is failing (#169)

v0.3.2

Optimisers v0.3.2

Diff since v0.3.1

Merged pull requests:

• WeightDecay for L1 norm (#159) (@mcabbott)
• Add all-keyword constructors, much like @kwdef (#160) (@mcabbott)
• CompatHelper: add new compat entry for Statistics at version 1, (keep existing compat) (#164) (@github-actions[bot])
• Don't load Yota at all (#166) (@mcabbott)

v0.3.1

Optimisers v0.3.1

Diff since v0.3.0

Closed issues:

• Error in update! for Metal arrays and Adam optimiser (#150)

Merged pull requests:

• Fix Float64 beta in Adam etc. (#158) (@mcabbott)

v0.3.0

Optimisers v0.3.0

The major change is #151, which removes type parameterisation from the structs. This should not break straightforward user code, but may break break loading via BSON etc. It also adds errors on negative learning rates, and will in some cases change the default regulator from eps(Float32) to 1e-8.

Diff since v0.2.20

Closed issues:

• Adam(0) fails (#119)
• Error in update! for Metal arrays and Adam optimiser (#150)

Merged pull requests:

• Use eltype(x) everywhere, ignore typeof(η) (#151) (@mcabbott)

v0.2.20

Optimisers v0.2.20

Diff since v0.2.19

Closed issues:

• Implement Lion, up to 5x faster than Adam, and more accurate (#156)

Merged pull requests:

• Add Lion to docs (#157) (@ToucheSir)

v0.2.19

Optimisers v0.2.19

Diff since v0.2.18

Closed issues:

• OptimiserChain(..., ClipNorm) fails on GPU (#127)

Merged pull requests:

• Don't use state anywhere for the whole state tree (#136) (@mcabbott)
• Update TagBot config (#142) (@ToucheSir)
• Make ClipNorm work on GPU Broadcasted (#144) (@mcabbott)

v0.2.18

Optimisers v0.2.18

Diff since v0.2.17

Closed issues:

• Interface for gradient accumulation (#130)
• Optimisers.update fails with gradient of type CUDA.CUSPARSE.CuSparseMatrixCSC (#141)

Merged pull requests:

• Rule for gradient accumulation (#137) (@CarloLucibello)