Refinement of Providers into Providers and ExternProviders

src/query.md

@@ -69,22 +69,24 @@ are cheaply cloneable; insert an `Rc` if necessary).
 
 If, however, the query is *not* in the cache, then the compiler will
 call the corresponding **provider** function. A provider is a function
-implemented in a specific module and **manually registered** into the
-[`Providers`][providers_struct] struct during compiler initialization.
-The macro system generates the [`Providers`][providers_struct] struct,
-which acts as a function table for all query implementations, where each
+implemented in a specific module and **manually registered** into either
+the [`Providers`][providers_struct] struct (for local crate queries) or
+the [`ExternProviders`][extern_providers_struct] struct (for external crate queries)
+during compiler initialization. The macro system generates both structs,
+which act as function tables for all query implementations, where each
 field is a function pointer to the actual provider.
 
-**Note:** The `Providers` struct is generated by macros and acts as a function table for all query implementations.
-It is **not** a Rust trait, but a plain struct with function pointer fields.
+**Note:** Both the `Providers` and `ExternProviders` structs are generated by macros and act as function tables for all query implementations.
+They are **not** Rust traits, but plain structs with function pointer fields.
 
 **Providers are defined per-crate.** The compiler maintains,
 internally, a table of providers for every crate, at least
-conceptually. Right now, there are really two sets: the providers for
-queries about the **local crate** (that is, the one being compiled)
-and providers for queries about **external crates** (that is,
-dependencies of the local crate). Note that what determines the crate
-that a query is targeting is not the *kind* of query, but the *key*.
+conceptually. There are two sets of providers:
+- The `Providers` struct for queries about the **local crate** (that is, the one being compiled)
+- The `ExternProviders` struct for queries about **external crates** (that is,
+dependencies of the local crate)
+
+Note that what determines the crate that a query is targeting is not the *kind* of query, but the *key*.
 For example, when you invoke `tcx.type_of(def_id)`, that could be a
 local query or an external query, depending on what crate the `def_id`
 is referring to (see the [`self::keys::Key`][Key] trait for more
@@ -117,59 +119,136 @@ they define both a `provide` and a `provide_extern` function, through
 
 ### How providers are set up
 
-When the tcx is created, it is given the providers by its creator using
-the [`Providers`][providers_struct] struct. This struct is generated by
-the macros here, but it is basically a big list of function pointers:
-
-[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html
+When the tcx is created, it is given both the local and external providers by its creator using
+the `Providers` struct from `rustc_middle::util`. This struct contains both the local and external providers:
 
 ```rust,ignore
-struct Providers {
-    type_of: for<'tcx> fn(TyCtxt<'tcx>, DefId) -> Ty<'tcx>,
-    // ... one field for each query
+pub struct Providers {
+    pub queries: crate::query::Providers,        // Local crate providers
+    pub extern_queries: crate::query::ExternProviders,  // External crate providers
+    pub hooks: crate::hooks::Providers,
 }
 ```
 
+Each of these provider structs is generated by the macros and contains function pointers for their respective queries.
+
 #### How are providers registered?
 
-The `Providers` struct is filled in during compiler initialization, mainly by the `rustc_driver` crate.  
-But the actual provider functions are implemented in various `rustc_*` crates (like `rustc_middle`, `rustc_hir_analysis`, etc).
+The `util::Providers` struct is filled in during compiler initialization, by the `rustc_interface` crate from the `DEFAULT_QUERY_PROVIDERS` static. 
+The actual provider functions are defined across various `rustc_*` crates (like `rustc_middle`, `rustc_hir_analysis`, etc).
 
 To register providers, each crate exposes a [`provide`][provide_fn] function that looks like this:
 
 [provide_fn]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/hir/fn.provide.html
 
 ```rust,ignore
-pub fn provide(providers: &mut Providers) {
-    *providers = Providers {
+pub fn provide(providers: &mut query::Providers) {
+    *providers = query::Providers {
         type_of,
         // ... add more providers here
         ..*providers
     };
 }
 ```
 
-- This function takes a mutable reference to the `Providers` struct and sets the fields to point to the correct provider functions.
-- You can also assign fields individually, e.g. `providers.type_of = type_of;`.
+Note that this function accepts `query::Providers` not `util::Providers`. It is exceedingly rare to need a `provide` function that doesn't just accept `query::Providers`. If more than the `queries` field of `util::Providers`  is being updated then `util::Providers` can be accepted instead:
+```rust,ignore
+pub fn provide(providers: &mut rustc_middle::util::Providers) {
+    providers.queries.type_of = type_of;
+    // ... add more local providers here
+
+    providers.extern_queries.type_of = extern_type_of;
+    // ... add more external providers here
+
+    providers.hooks.some_hook = some_hook;
+    // ... add more hooks here
+}
+```
+
+Note that `util::Providers` implements `DerefMut` to `query::Providers` so callers of the `provide` functions can pass in a `util::Providers` and it will just work for provider functions that accept `query::Providers` too
+
+- This function takes a mutable reference to the `query::Providers` struct and sets the fields to point to the correct provider functions.
+- You can also assign queries individually, e.g. `providers.type_of = type_of;`.
+- You can assign fields individually for each provider type (local, external, and hooks).
 
 #### Adding a new provider
 
-Suppose you want to add a new query called `fubar`. You would:
+Suppose you want to add a new query called `fubar`. This section focuses on wiring up the providers; for how to declare the query itself in the big `rustc_queries!` macro, see [Adding a new query](#adding-a-new-query) below.
+
+In practice you usually:
 
-1. Implement the provider function:
+1. Decide which crate "owns" the query (for example `rustc_hir_analysis`, `rustc_mir_build`, or another `rustc_*` crate).
+2. In that crate, look for an existing `provide` function:
+    ```rust,ignore
+    pub fn provide(providers: &mut query::Providers) {
+        // existing assignments
+    }
+    ```
+   If it exists, you will extend it to set the field for your new query. If the crate does not yet have a `provide` function, add one and make sure it is included in `DEFAULT_QUERY_PROVIDERS` in the `rustc_interface` crate so that it actually gets called during initialization (see the discussion above).
+3. Implement the provider function itself:
     ```rust,ignore
-    fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: DefId) -> Fubar<'tcx> { ... }
+    fn fubar<'tcx>(tcx: TyCtxt<'tcx>, key: LocalDefId) -> Fubar<'tcx> { ... }
     ```
-2. Register it in the `provide` function:
+4. Register it in the crate's `provide` function:
     ```rust,ignore
-    pub fn provide(providers: &mut Providers) {
-        *providers = Providers {
+    pub fn provide(providers: &mut query::Providers) {
+        *providers = query::Providers {
             fubar,
             ..*providers
         };
     }
     ```
 
+### How queries interact with external crate metadata
+
+When a query is made for an external crate (i.e., a dependency), the query system needs to load the information from that crate's metadata. 
+This is handled by the [`rustc_metadata` crate][rustc_metadata], which is responsible for decoding and providing the information stored in the `.rmeta` files.
+
+The process works like this:
+
+1. When a query is made, the query system first checks if the `DefId` refers to a local or external crate by checking if `def_id.krate == LOCAL_CRATE`.
+   This determines whether to use the local provider from [`Providers`][providers_struct] or the external provider from [`ExternProviders`][extern_providers_struct].
+
+2. For external crates, the query system will look for a provider in the [`ExternProviders`][extern_providers_struct] struct.
+   The `rustc_metadata` crate registers these external providers through the `provide_extern` function in `rustc_metadata/src/rmeta/decoder/cstore_impl.rs`. Just like:
+    ```rust
+    pub fn provide_extern(providers: &mut ExternProviders) {
+        providers.foo = |tcx, def_id| {
+            // Load and decode metadata for external crate
+            let cdata = CStore::from_tcx(tcx).get_crate_data(def_id.krate);
+            cdata.foo(def_id.index)
+        };
+        // Register other external providers...
+    }
+    ```
+
+3. The metadata is stored in a binary format in `.rmeta` files that contains pre-computed information about the external crate, such as types, function signatures, trait implementations, and other information needed by the compiler. When an external query is made, the `rustc_metadata` crate:
+   - Loads the `.rmeta` file for the external crate
+   - Decodes the metadata using the `Decodable` trait
+   - Returns the decoded information to the query system
+
+This approach avoids recompiling external crates, allows for faster compilation of dependent crates, and enables incremental compilation to work across crate boundaries.
+Here is a simplified example, when you call `tcx.type_of(def_id)` for a type defined in an external crate, the query system will:
+1. Detect that the `def_id` refers to an external crate by checking `def_id.krate != LOCAL_CRATE`
+2. Call the appropriate provider from `ExternProviders` which was registered by `rustc_metadata`
+3. The provider will load and decode the type information from the external crate's metadata
+4. Return the decoded type to the caller
+
+This is why most `rustc_*` crates only need to provide local providers - the external providers are handled by the metadata system. 
+The only exception is when a crate needs to provide special handling for external queries, in which case it would implement both local and external providers.
+
+When we define a new query that should work across crates, it does not automatically become cross-crate just because it is listed in `rustc_queries!`. You will typically need to:
+
+- Add the query to `rustc_queries!` with appropriate modifiers (for example whether it is cached on disk).
+- Implement a local provider in the owning crate and register it via that crate's `provide` function.
+- Add an external provider in `rustc_metadata` via `provide_extern`, and ensure the query's result is encoded and decoded in the crate metadata.
+
+An example of introducing such a cross-crate query can be found in commit [`996a185`](https://github.com/rust-lang/rust/commit/996a185) in the `rust-lang/rust` repository.
+
+[rustc_metadata]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/index.html
+[providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.Providers.html
+[extern_providers_struct]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/query/struct.ExternProviders.html
+
 ---
 
 ## Adding a new query
@@ -263,4 +342,3 @@ More discussion and issues:
 [GitHub issue #42633]: https://github.com/rust-lang/rust/issues/42633
 [Incremental Compilation Beta]: https://internals.rust-lang.org/t/incremental-compilation-beta/4721
 [Incremental Compilation Announcement]: https://blog.rust-lang.org/2016/09/08/incremental.html
-