Directly call in-library functions to build packages

[sheaf]

This is a rebase of #9871. I had to create a new PR as Matthew is no longer active (and that PR was made from his personal fork).

---

Template B: This PR does not modify behaviour or interface.

This is a major refactoring in how cabal-install works, with a few knock-on internal changes to some Cabal library functionality.
It should not cause any breakage for users (I tested clc-stackage against previous versions of this PR, and it all builds unchanged with this branch).

---

This PR modifies cabal-install to directly go through the Cabal library when building packages, instead of the Setup interface (except of course for build-type: Custom packages).

In particular, to build a package with build-type: Hooks, cabal-install will compile the package's SetupHooks file into an external hooks executable using the new hooks-exe package. All hooked operations are then performed by communicating with this external executable, instead of going through an opaque Setup executable. The hooks-exe package provides both functionality for compiling an external hooks executable and for interfacing with it. This makes use of the CommunicationHandle API that I added to process.

The main change is to SetupWrapper: the old InternalMethod becomes LibraryMethod, which builds the package by calling Cabal library functions. This is used to build all packages unless build-type: Custom or there is a Cabal library version mismatch which requires us to fall back to compiling Setup.hs and running that. The SelfExec method as well as forceExternalSetupMethod are removed: they no longer have any purpose, as builds can now be carried out concurrently thanks to 7b90583 and edb808a.

The new Distribution.Client.InLibrary module contains all the necessary framework for taking information that cabal-install has and invoking Cabal library functions to perform the appropriate action (configure, build, test, bench, ...). Most of these are pretty simple; the main difficulty is with configure as we need to jump to the part of the Cabal configure code that continues after figuring out information about the system (e.g. compiler information), to avoid wasting work with Cabal rediscovering things that cabal-install already knows. This required a bit of refactoring in Distribution.Simple.Configure.

TODO:

• Update bootstrap plans.
• Rebase over Implement recompilation checking for pre-build rules #11731 once that lands.
• Validate this version of the patch against clc-stackage and update the PR description accordingly.

[sheaf]

I now consider this patch ready for review. Significant changes since #9871:

1. I further refactored the Cabal library configure function so that all the logic that pertains to adjusting BuildOptions based on compiler and toolchain capabilities (e.g. support for bytecode libraries, profiling dynamic, etc) can be shared. This avoids duplicating logic inside cabal-install and prevents the codepaths from going out of sync.
2. I continued refactoring SetupWrapper, taking into account suggestions by @andreabedini on the original PR (e.g. moving more functions to the top level, see e.g. cabalLibFromOptions and cabalLibVersionToUse). I also took the opportunity to more clearly demarcate v1-only logic. On top of that, I was able to get rid of the self-exec setup method, now that Cabal library: allow setting the logging handle #11077 has landed. So there is only "external" (via Setup executable) and "in library" now.
3. The test I recently added for custom-setup builds fail when constraining dependencies in the Cabal-hooks package stack #11331 revealed that the way I had set up the hooks-exe implicit dependency in the previous iteration locked out users from using a different Cabal version in setup-depends entirely, as opposed to falling back to the external setup method. So I moved the necessary logic for compiling an external hooks executable into the Cabal library, which I now realise is the only place where it makes sense for it to live. The logic for invoking the external hooks executable remains separate.

I still need to check stackage compilation. With the changes in (1) implemented, I am now significantly more confident that we are not changing behaviour.