Is having"" as the fallbackname for the case when there's nodev-repo the same as what opam-monorepo does?

opam-monorepo filters out 'virtual' packages (without a dev-repo or url) here:https://github.com/tarides/opam-monorepo/blob/9262e7f71d749520b7e046fbd90a4732a43866e9/lib/lockfile.ml#L104

I'm still thinking about the best way of doing this with opam-nix.

This isn't the only filtering opam-monorepo does: it also filters packages that aren't build with dune (which is the reason foropam-overlays). But AFAIK we can't do this withopam admin list. Maybe modifying the repo to remove any packages not built with dune is the best way to go.

I'm not sure if we need to filter packages built with dune if out of this holds true:

Package versions from earlier repositories take precedence
over package versions from later repositories.

But if this could take a higher version from a later repository (if the higher version doesn't have an overlay yet) it would cause trouble.

if the higher version doesn't have an overlay yet

Hm, yes, good point. The filtering can probably be done by doinglistRepo on theopam-overlays and removing those packages fromopam-repository entirely, before passing the final repo toopam admin list. This can be quite slow though.

I've implemented thishere, but unfortunately it doesn't consider the case when the newer versiondoes build with dune but an older version (which is overlayed) doesn't.

For example,bheap.1.0.0 doesn't build with dune and therefore has an overlay, butbheapversion.2.0.0 does build with dune so doesn't have an overlay.

When buildingmirage-hello usingopamRepositoryModuloOverlay we get the error

       > [ERROR] No solution including optional dependencies for hello.dev:   * Missing dependency:       >             - bheap >= 2.0.0       >             no matching version

I think this is whyopam-monorepo filters (if it depends on dune) by package version rather than package name.

One option I can see is parsing every package in the repo and filtering out only those that don't depend on dune. But I imagine this would be quite expensive (there are ~24k packages inopam-repository[0]) even if it would only have to be done once.

Another option would be to useopam monorepo lock to resolve dependency versions.

[0]

opam-repository $ find packages -mindepth 2 -maxdepth 2 | wc -l24767

Why is this needed here? If callers of this function want to remove some packages, it's really easy for them to just callremoteAttrs themselves.

My thinking was this allows packages to be removed based on their name (rather than their dev repo name). For example, the unikernel package we're building has a `dummy' dev-repo which we need to remove.

This is quite useful when we want to exclude dependencies from the monorepo that are in the same source as the unikernel (e.g.https://github.com/mirage/mirage-www) inbuildOpamMonorepo with:

filterPkgs = ... (attrNames (latestVersions // query)) ...;

Also,hasInfix might be a bit flaky here, and quite slow.

On a more general note, maybe there's a way to just make a customjoinRepos that would overwrite entirepackage's instead of just package versions. That would remove the slowdown, but will probably require modifying thesymlinkJoin from nixpkgs.

That just made me realise thatjoinRepos is not quite correct ATM, since it doesn't actually do the version overriding properly. I think I'll fix it and implement this at the same time.

That sounds like a good idea, but I should warn you I think it will suffer from the same issues I described in the other open comment.

Essentially there's two possibilities when there's a package in opam-overlay with a newer in opam-repository without an overlay. One, an overlay hasn't been written yet. Two, the new version now supports dune and an overlay isn't required. In the former we want to remove the new version from opam-repository, in the latter we don't.

I don't think there's a way to differentiate these cases without parsing the new version's opam file to check for a dune dependency which suggests (although I guess doesn't guarantee, but it seems to work fine for opam-monorepo) dune support.

I'm ironing out the bugs on a function to do this now. It's not fast, but I think it'll work. I'll be offline for the next couple of hours but will push what I have now.

Okay it seems to be working! (Although it's still building... I might lose network connection soon so pushing preemptively)

Ahh, ok, sorry for not reading thoroughly. I still made thejoinRepos a bit better (and quite a lot faster actually) by patchinglndir (which comes from... Xorg of all places?)

#20

Why not just useimportOpam? (maybe it's a good idea to rewrite it and other functions to use yourpkgNameVer thing, though)

Purely aesthetics. It's hard to judge progress when building ~24k derivations with the same name 😄 I can add this toimportOpam if you think it's nice!

No, there's not. IFD is the only way.

(it might be a bit faster to do all theopam2json-ning in one derivation and then read the combined output, though).

Yeah, I was thinking the same thing. I'll give it a shot.

If this works, I'm wondering if materialization might be a way of speeding up deployments...

An aside: it would be nice if we could use just the version solving functionality of opam without pulling the whole program in. I might look into the feasibility of a refactor of opam to support this.

Okay, I've hacked something together that buildsopamRepositoryFiltered in one derivation:

  # remove non-dune dependant packages  opamRepositoryFiltered =    let      keepPackages = [ "ocaml" "dune" "ocaml-solo5" "ocaml-src" "solo5"      "ocaml-config" "ocaml-system" "ocaml-variants" "base-bytes" ];    in bootstrapPackages.stdenv.mkDerivation {      name = "opamRepositoryFiltered";      src = opamRepository;      buildInputs = with bootstrapPackages; [ jq ];      installPhase = ''        # look at package version directories        # (e.g. /nix/store/<hash>-opam-repository-modulo-overlay/packages/<name>/<name>.<version>)        for f in $(find packages -mindepth 2 -maxdepth 2); do          echo $f          # special cases as we will filter them before creating monorepo          # just needed so `opam admin list` doesn't complain          # TODO parse queries opam files to a list of dependancies with           # build=false so only pick up monorepo deps and we remove this workarounds          # TODO or just use opam monorepo solver...          if $(echo "$f" | awk 'BEGIN {              split("${concatStringsSep " " keepPackages}", parts);              for (i in parts) dict[parts[i]]=""            } { exit !($1 in dict) }'); then            # do nothing            echo          # if package depends on dune          # TODO properly recursively parse vals of depends          elif $(${opam2json}/bin/opam2json "$f/opam" |\            jq '[ .depends | flatten | .[] | if type=="string" then . else .val end ] |              if index("dune") == null then halt_error(1) else halt end'); then              mkdir -p "$out/$f"              cp -r "$f" "$out/$f"          fi        done      '';    };

But it's alsovery slow (waiting on exact timings now). I think usingopam monorepo's solver would be a better solution.

Okay, this creates a very IO bound process that takes ~half an hour to complete 😄 I'm going to look at usingopam monorepo lock and parsing the lockfile to solver versions.

That sounds reasonable.