Struct HttpCache

Source

pub struct HttpCache { /* private fields */ }

Expand description

HTTP cache with ETag and Last-Modified validation.

Implements RFC 7232 conditional requests to minimize network traffic. All responses are cached with their validation headers, and subsequent requests use If-None-Match (ETag) or If-Modified-Since headers to check for updates.

The cache uses Bytes for response bodies, enabling efficient sharing of cached data across multiple consumers without copying. Bytes is an Arc-like type optimized for network I/O.

§Examples

use deps_core::cache::HttpCache;

let cache = HttpCache::new();

// First request - fetches from network
let data1 = cache.get_cached("https://index.crates.io/se/rd/serde").await?;

// Second request - uses conditional GET (304 Not Modified if unchanged)
let data2 = cache.get_cached("https://index.crates.io/se/rd/serde").await?;

§Cache key

Entries are keyed by URL alone (see Self::cache_key, private) — extra_headers (see HttpCache::get_cached_with_headers) play no part in the cache key. This is safe only as long as “same URL” implies “same representation”: a content-negotiating header (e.g. a per-request Accept) that can vary the response body for an otherwise-identical URL requires giving each distinct representation its own URL (e.g. a query parameter or distinct path), not just a distinct header value, or callers requesting different representations of the same URL will silently share one cache entry.

Likewise, the key doesn’t encode which client (and so which redirect policy) produced an entry — HttpCache::get_cached and HttpCache::get_cached_trusted_origin share one entry map. No caller today requests the same URL through both, but one that did could observe the other’s cached (and differently redirect-validated) body.

Self::get_cached_workspace is the one exception: it is namespaced under a distinct, policy-scoped key prefix (see Self::cache_key, private) so a body fetched under a looser crate::net_policy::WorkspaceRegistryAccess can never be served back once the policy tightens.

Implementations§

Source §

impl HttpCache

Source

pub fn new() -> Self

Creates a new HTTP cache with default configuration and the default crate::net_policy::WorkspaceRegistryAccess policy (PublicOnly).

The cache uses a configurable timeout for all requests and identifies itself with an auto-versioned user agent.

Source

pub fn with_policy(policy: Arc<RegistryAccessPolicy>) -> Self

Creates a new HTTP cache whose Self::get_cached_workspace requests are governed by policy’s live value.

A later Self::set_registry_policy call rebuilds the workspace transport (and its cache-key namespace) in place, so every caller holding this HttpCache sees the new policy take effect immediately, with no need to reconstruct the cache.

§Examples

use deps_core::HttpCache;
use deps_core::net_policy::RegistryAccessPolicy;
use std::sync::Arc;

let policy = Arc::new(RegistryAccessPolicy::default());
let cache = HttpCache::with_policy(Arc::clone(&policy));
assert!(cache.is_empty());

Source

pub fn set_offline(&self, value: bool)

Sets whether outbound network requests are permitted (issue #483).

Enforced by Self::ensure_online (private) at every one of this module’s 4 send sites — effective for every call after this returns. While value is true, this also overrides cache_enabled (see Self::set_cache_enabled) to behave as true on both the read and write path in get_cached_with_headers_via: without this, a warm entry fetched before going offline could never have been stored in the first place if caching was disabled, leaving the offline warm-cache path with nothing to serve — the exact combination cache.enabled: false + network.offline: true is meant to survive.

Source

pub fn is_offline(&self) -> bool

Returns whether outbound network requests are currently blocked.

Source

pub fn set_cache_enabled(&self, value: bool)

Sets whether the entry-map cache is used (issue #482). See Self::set_offline’s docs for the override offline has on this flag while set.

Source

pub async fn get_cached(&self, url: &str) -> Result<Bytes>

Retrieves data from URL with intelligent caching.

On first request, fetches data from the network and caches it. On subsequent requests, performs a conditional GET request using cached ETag or Last-Modified headers. If the server responds with 304 Not Modified, returns the cached data. Otherwise, fetches and caches the new data.

If the conditional request fails due to network errors, falls back to the cached data (stale-while-revalidate pattern).

§Returns

Returns Bytes containing the response body. Multiple calls for the same URL return cheap clones (reference counting) without copying data.

§Errors

Returns DepsError::RegistryError if the initial fetch fails and no cached data exists, DepsError::HttpStatus if the server returns a non-2xx status on that initial fetch, or DepsError::ResponseTooLarge if the response body exceeds the configured size cap.

§Examples

let cache = HttpCache::new();
let data = cache.get_cached("https://example.com/api/data").await?;
println!("Fetched {} bytes", data.len());

Source

pub fn peek_cached(&self, url: &str) -> Option<Bytes>

Returns the cached body for url without making any network request.

Unlike get_cached’s own stale-while-revalidate fallback (the Err arm of conditional_request_with_headers’s match in get_cached_with_headers_via), this is reachable even when a caller wraps get_cached in a short outer timeout: a hung conditional request that never resolves within that timeout gets its whole future cancelled, so get_cached’s internal fallback logic never runs and the caller sees a timeout instead of stale data. A caller in that position can call this instead — a synchronous map lookup, no I/O — to serve the last known-good body itself. Returns None if url has never been successfully cached.

The returned body carries no age bound: this bypasses get_cached’s own freshness/revalidation logic entirely, so a caller that surfaces this body to the user (e.g. inserting it into a manifest edit) should treat it as arbitrarily stale, not just-expired.

Reads the baseline (unprefixed) cache-key namespace only (see Self::cache_key, private) — a body fetched via Self::get_cached_workspace is never visible through this method.

Source

pub async fn get_cached_with_headers( &self, url: &str, extra_headers: &[(HeaderName, &str)], ) -> Result<Bytes>

Fetches a URL with additional request headers, using the cache.

Works the same as get_cached but injects extra headers (e.g., Authorization) into every request. Useful for APIs that require authentication tokens.

§Errors

Returns DepsError::RegistryError if the initial fetch fails and no cached data exists, DepsError::HttpStatus if the server returns a non-2xx status on that initial fetch, or DepsError::ResponseTooLarge if the response body exceeds the configured size cap.

Source

pub async fn get_cached_trusted_origin( &self, url: &str, trusted_origin: &str, ) -> Result<Bytes>

Like Self::get_cached, but additionally stops any redirect hop whose target no longer starts with trusted_origin (e.g. https://api.nuget.org/v3/registration5-gz/).

For a caller that already validated the initial request URL against a trusted prefix (NuGet’s registration-hive paging validates page.id this way) and needs that guarantee to hold through any redirect too, not just the first hop — Self::get_cached’s own policy deliberately does not enforce this, since cross-host redirects are legitimate for the other registry clients sharing this cache; the stricter check is opt-in per call rather than global.

The block only surfaces as an error on a cold cache: like Self::get_cached’s own stale-while-revalidate fallback, a warm entry for url still returns the last known-good body (itself already fetched and origin-validated on a prior call) instead of propagating a blocked-redirect HttpStatus from a revalidation attempt.

§Errors

Same as Self::get_cached.

Source

pub async fn get_cached_trusted_origin_with_headers( &self, url: &str, trusted_origin: &str, extra_headers: &[(HeaderName, &str)], ) -> Result<Bytes>

Like Self::get_cached_trusted_origin, but additionally injects extra_headers (e.g. an Authorization bearer token) into every request — the authenticated counterpart to Self::get_cached_with_headers, composed with the same origin-pinned redirect policy Self::get_cached_trusted_origin uses.

This exists specifically so a header carrying a credential can never survive a cross-origin redirect hop: Self::get_cached_with_headers attaches extra_headers to the initial request only and follows reqwest’s default (same-scheme, any-host) redirect policy for every hop after that, which is exactly the shape a hostile or misconfigured redirect on the resolved index itself could exploit to exfiltrate a bearer token to an attacker-controlled host. Composing Self::transport_for_origin’s (private) pinned-origin transport with header injection closes that by construction — no empirical redirect test is needed to prove the header cannot leak, since the client stops following before a cross-origin hop would ever be sent.

§Errors

Same as Self::get_cached_trusted_origin.

§Examples

use deps_core::cache::HttpCache;
use reqwest::header;

let cache = HttpCache::new();
let data = cache
    .get_cached_trusted_origin_with_headers(
        "https://index.mycorp.dev/se/rd/serde",
        "https://index.mycorp.dev/",
        &[(header::AUTHORIZATION, "Bearer secret-token")],
    )
    .await?;
println!("Fetched {} bytes", data.len());

Source

pub async fn get_cached_pinned( &self, url: &str, trusted_origin: &str, authenticated: bool, auth_id: Option<u64>, ) -> Result<Bytes>

Like Self::get_cached_trusted_origin_with_headers, but for an origin-pinned, connect-address-guarded CacheTier::Pinned transport (issue #561/#562) instead of the baseline-guarded Self::get_cached_trusted_origin one — the only sanctioned way to send a credential to a workspace-declared host. Delegates to Self::get_cached_pinned_with_headers with no extra headers.

§Errors

Same as Self::get_cached.

Source

pub async fn get_cached_pinned_with_headers( &self, url: &str, trusted_origin: &str, authenticated: bool, auth_id: Option<u64>, extra_headers: &[(HeaderName, &str)], ) -> Result<Bytes>

Like Self::get_cached_pinned, but additionally injects extra_headers (e.g. an Authorization header carrying a credential) into every request — composed with the same origin-pinned, connect-address-guarded transport Self::get_cached_pinned uses, so a credential header can never survive a cross-origin redirect hop, exactly like Self::get_cached_trusted_origin_with_headers’s identical closure argument for the baseline-guarded tier.

auth_id (FR-014) — a caller-computed, salted digest of the credential actually being attached (None for an unauthenticated #562 fetch) — is folded into the cache key only, never into CacheTier/the transport-pool key, so a rotated or distinct credential against the same origin never reads back a body fetched under a different one.

§Errors

Same as Self::get_cached.

Source

pub async fn get_cached_workspace(&self, url: &str) -> Result<Bytes>

Like Self::get_cached, but for Cargo’s workspace-declared-registry requests: routes through the workspace transport field, whose guard enforces the live crate::net_policy::WorkspaceRegistryAccess policy on both the resolved connect-time address (issue #455) and any redirect hop, and keys the entry under a policy-scoped namespace (see Self::cache_key, private) distinct from every other method on this cache.

This gives the resolved address the same policy scrutiny deps_cargo::config::RegistryIndex::new already gives the declared URL string at parse time — it does not re-check the initial request URL itself: a caller passing an IP-literal url whose class the policy would reject connects anyway, since hyper-util’s connector parses an IP literal directly and never calls the configured resolver (see BlockedAddrResolver’s docs, private). RegistryIndex::new is the sole, by-design gate for that residual — every caller of this method already went through it.

If an entry is already cached, a revalidation failure — including a guard rejection from a since-rebound or since-tightened-policy address — falls back to serving the cached body, logging only a tracing::warn! (pre-existing behavior, unrelated to this method). This is not a bypass — no new connection to the blocked address is made — but it means such a block is invisible to the caller whenever an entry already exists for that URL.

§Errors

Same as Self::get_cached.

§Examples

use deps_core::HttpCache;
use deps_core::net_policy::RegistryAccessPolicy;
use std::sync::Arc;

let policy = Arc::new(RegistryAccessPolicy::default());
let cache = HttpCache::with_policy(policy);
let data = cache
    .get_cached_workspace("https://index.mycorp.dev/se/rd/serde")
    .await?;
println!("Fetched {} bytes", data.len());

Source

pub async fn get_cached_workspace_with_headers( &self, url: &str, extra_headers: &[(HeaderName, &str)], ) -> Result<Bytes>

Like Self::get_cached_workspace, but additionally forwards extra_headers to the underlying request — the headered form needed by a registry client whose workspace-declared fetch requires a non-default header (e.g. deps-npm’s abbreviated- packument Accept header for an alternate npm registry).

§Security

extra_headers are attached to the initial request only. The workspace transport pins by crate::net_policy::HostClass, not origin — unlike Self::get_cached_trusted_origin_with_headers, which exists precisely to close this gap for a caller that needs it — so a cross-origin redirect hop to any other policy-permitted host is followed with extra_headers re-sent by reqwest’s default redirect policy. This method must never carry a credential. Harmless for its current sole caller (a fixed Accept header), but directly load-bearing for any future auth-wiring work: reach for Self::get_cached_trusted_origin_with_headers instead if a header ever needs to stay pinned to one origin.

Source

pub fn set_registry_policy(&self, value: WorkspaceRegistryAccess)

Updates the policy governing Self::get_cached_workspace, rebuilding the workspace transport field (and so its cache-key namespace and guard together) when value actually differs from the current setting — a no-op call does not rebuild, so a caller that re-applies an unchanged configuration does not pay for a fresh Client and its connection pool.

Effective for every Self::get_cached_workspace call after this returns. Note this only gates future fetches: an All -> PublicOnly/Off tightening does not purge already-registered deps-cargo alternate-registry clients resolved under the looser policy (pre-existing, documented on crate::net_policy::RegistryAccessPolicy::set).

Unlike that pre-existing gap, every CacheTier::Pinned cache entry (issue #561/#562) is purged on every actual policy transition, along with every pinned-tier pooled Transport — substantially narrowing the All -> PublicOnly -> All round-trip hole for credential-carrying entries specifically (NFR-004): re-namespacing alone (as the pre-existing workspace-tier digit prefix does) would leave an old-era authenticated body reachable once the policy round-trips back to a value whose digest happens to collide again. Not an absolute close: a fetch already in flight when the transition happens can still land its response and re-insert an old-era key after the purge — harmless (readable only under the era it was legitimately fetched in), just not prevented by this purge alone.

Source

pub async fn post_json<T: Serialize + ?Sized>( &self, url: &str, body: &T, ) -> Result<Bytes>

POSTs body as JSON and returns the response body.

Deliberately does not cache: the OSV batch endpoint is a POST with a request-body-dependent response and sends no ETag/Last-Modified validators, so entry-map caching would be meaningless here — every call reuses the client, HTTPS enforcement, size cap, and timeout (via read_body_capped) without touching the entry map or Self::total_bytes.

§Errors

Returns DepsError::HttpStatus if the server returns a non-2xx status, DepsError::RegistryError if the request fails, or DepsError::ResponseTooLarge if the response body exceeds the configured size cap.

Source

pub async fn get_transport_only(&self, url: &str) -> Result<Bytes>

GETs url and returns the response body, bypassing the entry-map cache entirely — reuses the client, HTTPS enforcement, size cap, and timeout, exactly like Self::post_json, but for a plain GET.

For a caller whose own values are already cached elsewhere (e.g. OsvClient’s record cache, validated by a modified timestamp rather than ETag/Last-Modified): reusing Self::get_cached there would double-cache every fetched record in this cache’s byte budget too, competing with registry responses for it even though nothing here ever reads that cached copy back.

§Errors

Returns DepsError::HttpStatus if the server returns a non-2xx status, DepsError::RegistryError if the request fails, or DepsError::ResponseTooLarge if the response body exceeds the configured size cap.

Source

pub async fn get_transport_only_with_headers( &self, url: &str, extra_headers: &[(HeaderName, &str)], ) -> Result<Bytes>

Same as Self::get_transport_only, but injects extra request headers (e.g. a content-negotiating Accept) — mirrors how Self::get_cached_with_headers relates to Self::get_cached.

§Errors

Returns DepsError::HttpStatus if the server returns a non-2xx status, DepsError::RegistryError if the request fails, or DepsError::ResponseTooLarge if the response body exceeds the configured size cap.

Source

pub async fn get_transport_only_with_headers_limited( &self, url: &str, extra_headers: &[(HeaderName, &str)], limit: BodyLimit, ) -> Result<Bytes>

Same as Self::get_transport_only_with_headers, but takes an explicit BodyLimit instead of the BodyLimit::DEFAULT (MAX_RESPONSE_BYTES) cap.

For a caller whose response is legitimately larger than every other registry payload — e.g. deps-pypi’s full Simple API project index — without weakening the cap every other caller of this cache relies on. BodyLimit clamps at construction, so this can never be widened past ABSOLUTE_MAX_RESPONSE_BYTES regardless of what the caller passes in.

§Errors

Same as Self::get_transport_only_with_headers.

Source

pub async fn get_transport_only_with_headers_limited_trusted_origin( &self, url: &str, extra_headers: &[(HeaderName, &str)], limit: BodyLimit, trusted_origin: &str, ) -> Result<Bytes>

Same as Self::get_transport_only_with_headers_limited, but additionally stops any redirect hop whose target no longer starts with trusted_origin (see Self::get_cached_trusted_origin, which applies the identical policy to the entry-cached path). For a caller carrying a materially larger BodyLimit than BodyLimit::DEFAULT — the bigger the budget, the more worth pinning the origin an arbitrary cross-host redirect could point it at.