dataset

Github repo dataset related

afwf_github.dataset.make_repo_dataset(dir_user: Path, downloader: Callable[[], list[dict[str, Any]]], cache_expire: int) → DataSet

Low-level DataSet constructor. Pure function — no GitHub or config dependencies. The caller is responsible for providing the per-user directory, a ready-made downloader, and the cache expiry.

afwf_github.dataset.make_downloader(gh: Github, cache: Cache, username: str, cache_expire: int, force: bool = False) → Callable[[], list[dict[str, Any]]]

Build the downloader callable used by the repo DataSet.

Wraps download_data() with a cache-check: if repos are already stored under the per-user cache key, they are returned directly without hitting the GitHub API.

When force is True the diskcache check is skipped and fresh data is always fetched from GitHub. Pass force=True when rebuilding the index on demand so the downloader is not fooled by a still-valid diskcache entry that predates newly created repositories.

afwf_github.dataset.create_repo_dataset(config: Config, force: bool = False) → DataSet

High-level factory. Derives the GitHub client, username, per-user directory, and cache entirely from config.

Set force to True to bypass the diskcache and always pull fresh data from GitHub when the downloader is invoked (i.e. when the caller subsequently calls dataset.search(refresh=True)).