Add tools and atomic datastructures to upstream-extra #699
Conversation
Signed-off-by: Edwin Török <[email protected]>
It was missing a dependency on rrd-transport. Signed-off-by: Edwin Török <[email protected]>
This can be used to create a `.mli` file (or at least a starting point for one) when `dune` is used and you have a lot of dependencies (so that invoking `ocamlc` by hand would be too tedious). It is recommended by the dune docs: https://dune.readthedocs.io/en/latest/faq.html#how-can-i-generate-an-mli-file-from-an-ml-file I had this pinned in my local opam switch for a long time. Signed-off-by: Edwin Török <[email protected]>
This can be used to keep `dune-project` and `*.opam.template` files up-to-date by finding dependencies that are declared in `dune` files, but not in the `opam` package. It doesn't yet find all such bugs (ocurrent/opam-dune-lint#71), but it is a starting point. It also adds its dependencies for parsing opam files: * ocamlgraph.2.1.0 (I'm surprised we didn't already have this) * opam-core, opam-file-format, opam-format, opam-repository-opam-state 2.2.1 * spdx_licenses.1.2.0, swhid_core.0.1 I had this pinned in my local opam repo for a while too, this is the latest version. Signed-off-by: Edwin Török <[email protected]>
And dependencies: * backoff.0.1.0 * containers.3.9 * domain-local-await.1.0.1 * domain-local-timeout.1.0.1 * domain_shims.0.1.0 * dscheck.0.5.0 * kcas.0.7.0 * mdx.2.4.1 * multicore-bench.0.1.5 * multicore-magic-dscheck.2.3.0 * multicore-magic.2.3.0 * oseq.0.5.1 * qcheck-multicoretest-util.0.4 * qcheck-stm-0.4 * saturn_lockfree.0.5.0 * thread-table.1.0.0 * tsort.2.1.0 This isn't used in the product yet, but having it in xs-opam allows us to start experimenting, and perhaps add some feature flagged code to XAPI that uses these. Signed-off-by: Edwin Török <[email protected]>
|
Have we moved to collect the libraries that we package into an RPM based on dependencies? Previously packaging was based on the tree hierarchy. @psafont |
|
We can't move to a dependency-bases system yet until all the issues surrounding tarball generation andpackage retrieval from it are fixed in opam. They are fixed in master, and should be available in the next major version (2.3.0, 2025.2, ???) |
|
I'm not too happy with the amount of dependencies being added here, we should do some trimming beforehand. For example goblint pulls a lot of dependencies. And while it's packages, it's still unused |
I'll try to find some time on the next OIL day to update the analyzer so we can start using it with XAPI. You are right it has now been more than a year since I originally wrote the analyzer and I haven't been able to find the time to finish the CI integration. I think it'll be easier to do the parallelisation of the analysis inside the tool itself rather than generating dune (and relying on its caching). Lets keep this PR open meanwhile (I'll likely need to update |
Add some useful tools (that I had pinned in my local opam switch for a while):
Also add some parallelism-safe data structures that work on both OCaml 4 and OCaml 5:
saturnandsaturn-lockfreecontains very efficient, domain and thread-safe imperative data structures. It is well tested, and the algorithms are quite well known and used in other languages (property testing and fuzzing with qcheck, and qcheck-stm; model checking withdscheck, and some in-progress formal proofs)kcasandkcas_datawhich provide composable lock-free data structures. Where composability is not neededsaturnis likely faster and uses less memory, but where composability or transactions are needed this is more flexible. This could be useful for XAPI's main database (if we thread through the 'Xt' transaction log). The algorithms are similar to optimistic concurrency control: try to perform all operations assuming you'd succeed, and keep track of which values you've read inside a transaction, and then commit or rollback and restart depending on whether any of those values have changed; and is quite similar to how the transactions work inoxenstored. This allows for transactions that don't interfere with each-other to proceed in parallel. Some attention to detail is still needed, as this doesn't eliminate all concurrency bugs (read skew and write skew are still possible, but there are mechanisms for avoiding them, see the tutorial in the https://github.com/ocaml-multicore/kcas?tab=readme-ov-file#beware-of-torn-reads)Adding these tools and libraries should be safe, as nothing in the product would use them yet. And we can gradually introduce them into XAPI behind a feature flag, only switching to them when we are confident that they work well for our use case.
We also gain some testing tools that are generally useful for XAPI (I've used some of these in various branches already):
Some other useful libraries that get brought in as dependencies:
Obj.magic)These libraries also bring in
multicore-magic, which sounds .. dangerous ... although from its description perhaps most of this will get integrated into future OCaml 5 releases. With careful use ofkcasandsaturnwe might be able to avoid most of the unsafe operations though (they are mostly meant as internal performance optimizations), although I'm still somewhat worried about its presence in the dependencies. We'll need to reevaluate this when we have some actual code that useskcas/saturnthough.