sc_crawler

sc_crawler

The Crawler of Spare Cores Navigator: collect and standardize cloud server offerings data.

Modules:

Name	Description
alembic_helpers	Helpers functions for the Alembic database migrations.
cli	The sc-crawler CLI functions.
description_ingestor	Load the LLM-generated descriptions of the servers from the sc-navigator-descriptions repository.
insert	Helpers to validate and bulk insert per-vendor records into the database.
inspector	Helpers to load the hardware discovery and benchmarking data from the sc-inspector-data repository.
inspector_helpers	Helpers to parse the hardware details of servers from the sc-inspector-data repository.
inspector_rules	Rules for which servers and managed databases the Inspector runs its tasks on.
logger	Logging and progress bar helpers.
lookup	Static lookup data: countries, compliance frameworks and benchmark definitions.
sentry	Report errors to Sentry when the SENTRY_DSN environment variable is set.
str_utils	String helpers, e.g. for wrapping, pluralizing and converting to snake case.
table_bases	Tiny helper classes for the most commonly used fields to be inherited by sc_crawler.tables.
table_fields	Enumerations, JSON nested data objects & other helper classes used in sc_crawler.tables.
tables	Table definitions for vendors, regions, zones, and other cloud resources.
tables_scd	SCD version of the table definitions in sc_crawler.tables.
utils	Utility functions, e.g. for hashing, database engines, and looking up tables and rows.
vendor_helpers	Helpers shared by the vendor modules of sc_crawler.vendors, e.g. fetching the servers of all regions in parallel.
vendors	Helper methods for each cloud compute resource provider.
workload_profile_scores	Workload profile score computation and database persistence.
workload_profiles	Workload profile definitions for compound benchmark scoring.