tables_scd

sc_crawler.tables_scd #

SCD version of the table definitions in sc_crawler.tables.

Scd #

Bases: ScModel

Override the observed_at column to be primary key in SCD tables.

Source code in sc_crawler/tables_scd.py

class Scd(ScModel):
    """Override the `observed_at` column to be primary key in SCD tables."""

    observed_at: datetime = Field(
        primary_key=True,
        default_factory=datetime.utcnow,
        sa_column_kwargs={"onupdate": datetime.utcnow},
        description="Timestamp of the last observation.",
    )

CountryScd #

Bases: Scd, CountryBase

Country and continent mapping (SCD Type 2).

Attributes:

Name	Type	Description
`country_id`	`str`	Country code by ISO 3166 alpha-2.
`continent`	`str`	Continent name.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class CountryScd(Scd, CountryBase, table=True):
    """SCD version of .tables.Country."""

    pass

VendorComplianceLinkScd #

Bases: Scd, VendorComplianceLinkBase

List of known Compliance Frameworks paired with vendors (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`compliance_framework_id`	`str`	Reference to the Compliance Framework.
`comment`	`Optional[str]`	Optional references, such as dates, URLs, and additional information/evidence.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class VendorComplianceLinkScd(Scd, VendorComplianceLinkBase, table=True):
    """SCD version of .tables.VendorComplianceLink."""

    pass

ComplianceFrameworkScd #

Bases: Scd, ComplianceFrameworkBase

List of Compliance Frameworks, such as HIPAA or SOC 2 Type 1 (SCD Type 2).

Attributes:

Name	Type	Description
`compliance_framework_id`	`str`	Unique identifier.
`name`	`str`	Human-friendly name.
`abbreviation`	`Optional[str]`	Short abbreviation of the Framework name.
`description`	`Optional[str]`	Description of the framework in a few paragrahs, outlining key features and characteristics for reference.
`logo`	`Optional[str]`	Publicly accessible URL to the image of the Framework's logo.
`homepage`	`Optional[str]`	Public homepage with more information on the Framework.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class ComplianceFrameworkScd(Scd, ComplianceFrameworkBase, table=True):
    """SCD version of .tables.ComplianceFramework."""

    pass

VendorScd #

Bases: Scd, VendorBase

Compute resource vendors, such as cloud and server providers (SCD Type 2).

Examples:
    >>> from sc_crawler.tables import Vendor
    >>> from sc_crawler.lookup import countries
    >>> aws = Vendor(vendor_id='aws', name='Amazon Web Services', homepage='https://aws.amazon.com', country=countries["US"], founding_year=2002)
    >>> aws
    Vendor(vendor_id='aws'...
    >>> from sc_crawler import vendors
    >>> vendors.aws
    Vendor(vendor_id='aws'...

Attributes:

Name	Type	Description
`vendor_id`	`str`	Unique identifier.
`name`	`str`	Human-friendly name.
`logo`	`Optional[str]`	Publicly accessible URL to the image of the Vendor's logo.
`homepage`	`Optional[str]`	Public homepage of the Vendor.
`country_id`	`str`	Reference to the Country, where the Vendor's main headquarter is located.
`state`	`Optional[str]`	Optional state/administrative area of the Vendor's location within the Country.
`city`	`Optional[str]`	Optional city name of the Vendor's main location.
`address_line`	`Optional[str]`	Optional address line of the Vendor's main location.
`zip_code`	`Optional[str]`	Optional ZIP code of the Vendor's main location.
`founding_year`	`int`	4-digit year when the Vendor was founded.
`status_page`	`Optional[str]`	Public status page of the Vendor.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class VendorScd(Scd, VendorBase, table=True):
    """SCD version of .tables.Vendor."""

    pass

RegionScd #

Bases: Scd, RegionBase

Regions of Vendors (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`region_id`	`str`	Unique identifier, as called at the Vendor.
`name`	`str`	Human-friendly name.
`api_reference`	`str`	How this resource is referenced in the vendor API calls. This is usually either the id or name of the resource, depening on the vendor and actual API endpoint.
`display_name`	`str`	Human-friendly reference (usually the id or name) of the resource.
`aliases`	`List[str]`	List of other commonly used names for the same Region.
`country_id`	`str`	Reference to the Country, where the Region is located.
`state`	`Optional[str]`	Optional state/administrative area of the Region's location within the Country.
`city`	`Optional[str]`	Optional city name of the Region's location.
`address_line`	`Optional[str]`	Optional address line of the Region's location.
`zip_code`	`Optional[str]`	Optional ZIP code of the Region's location.
`lon`	`Optional[float]`	Longitude coordinate of the Region's known or approximate location.
`lat`	`Optional[float]`	Latitude coordinate of the Region's known or approximate location.
`founding_year`	`Optional[int]`	4-digit year when the Region was founded.
`green_energy`	`Optional[bool]`	If the Region is 100% powered by renewable energy.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class RegionScd(Scd, RegionBase, table=True):
    """SCD version of .tables.Region."""

    pass

ZoneScd #

Bases: Scd, ZoneBase

Availability zones of Regions (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`region_id`	`str`	Reference to the Region.
`zone_id`	`str`	Unique identifier, as called at the Vendor.
`name`	`str`	Human-friendly name.
`api_reference`	`str`	How this resource is referenced in the vendor API calls. This is usually either the id or name of the resource, depening on the vendor and actual API endpoint.
`display_name`	`str`	Human-friendly reference (usually the id or name) of the resource.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class ZoneScd(Scd, ZoneBase, table=True):
    """SCD version of .tables.Zone."""

    pass

StorageScd #

Bases: Scd, StorageBase

Flexible storage options that can be attached to a Server (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`storage_id`	`str`	Unique identifier, as called at the Vendor.
`name`	`str`	Human-friendly name.
`description`	`Optional[str]`	Short description.
`storage_type`	`StorageType`	High-level category of the storage, e.g. HDD or SDD.
`max_iops`	`Optional[int]`	Maximum Input/Output Operations Per Second.
`max_throughput`	`Optional[int]`	Maximum Throughput (MiB/s).
`min_size`	`Optional[int]`	Minimum required size (GiB).
`max_size`	`Optional[int]`	Maximum possible size (GiB).
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class StorageScd(Scd, StorageBase, table=True):
    """SCD version of .tables.Storage."""

    pass

ServerScd #

Bases: Scd, ServerBase

Server types (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`server_id`	`str`	Unique identifier, as called at the Vendor.
`name`	`str`	Human-friendly name.
`api_reference`	`str`	How this resource is referenced in the vendor API calls. This is usually either the id or name of the resource, depening on the vendor and actual API endpoint.
`display_name`	`str`	Human-friendly reference (usually the id or name) of the resource.
`description`	`Optional[str]`	Short description.
`family`	`Optional[str]`	Server family, e.g. General-purpose machine (GCP), or M5g (AWS).
`vcpus`	`int`	Default number of virtual CPUs (vCPU) of the server.
`hypervisor`	`Optional[str]`	Hypervisor of the virtual server, e.g. Xen, KVM, Nitro or Dedicated.
`cpu_allocation`	`CpuAllocation`	Allocation of CPU(s) to the server, e.g. shared, burstable or dedicated.
`cpu_cores`	`Optional[int]`	Default number of CPU cores of the server. Equals to vCPUs when HyperThreading is disabled.
`cpu_speed`	`Optional[float]`	Vendor-reported maximum CPU clock speed (GHz).
`cpu_architecture`	`CpuArchitecture`	CPU architecture (arm64, arm64_mac, i386, or x86_64).
`cpu_manufacturer`	`Optional[str]`	The manufacturer of the primary processor, e.g. Intel or AMD.
`cpu_family`	`Optional[str]`	The product line/family of the primary processor, e.g. Xeon, Core i7, Ryzen 9.
`cpu_model`	`Optional[str]`	The model number of the primary processor, e.g. 9750H.
`cpu_l1_cache`	`Optional[int]`	L1 cache size (byte).
`cpu_l2_cache`	`Optional[int]`	L2 cache size (byte).
`cpu_l3_cache`	`Optional[int]`	L3 cache size (byte).
`cpu_flags`	`List[str]`	CPU features/flags.
`cpus`	`List[Cpu]`	JSON array of known CPU details, e.g. the manufacturer, family, model; L1/L2/L3 cache size; microcode version; feature flags; bugs etc.
`memory_amount`	`int`	RAM amount (MiB).
`memory_generation`	`Optional[DdrGeneration]`	Generation of the DDR SDRAM, e.g. DDR4 or DDR5.
`memory_speed`	`Optional[int]`	DDR SDRAM clock rate (Mhz).
`memory_ecc`	`Optional[bool]`	If the DDR SDRAM uses error correction code to detect and correct n-bit data corruption.
`gpu_count`	`int`	Number of GPU accelerator(s).
`gpu_memory_min`	`Optional[int]`	Memory (MiB) allocated to the lowest-end GPU accelerator.
`gpu_memory_total`	`Optional[int]`	Overall memory (MiB) allocated to all the GPU accelerator(s).
`gpu_manufacturer`	`Optional[str]`	The manufacturer of the primary GPU accelerator, e.g. Nvidia or AMD.
`gpu_family`	`Optional[str]`	The product family of the primary GPU accelerator, e.g. Turing.
`gpu_model`	`Optional[str]`	The model number of the primary GPU accelerator, e.g. Tesla T4.
`gpus`	`List[Gpu]`	JSON array of GPU accelerator details, including the manufacturer, name, and memory (MiB) of each GPU.
`storage_size`	`int`	Overall size (GB) of the disk(s).
`storage_type`	`Optional[StorageType]`	Primary disk type, e.g. HDD, SSD, NVMe SSD, or network).
`storages`	`List[Disk]`	JSON array of disks attached to the server, including the size (MiB) and type of each disk.
`network_speed`	`Optional[float]`	The baseline network performance (Gbps) of the network card.
`inbound_traffic`	`float`	Amount of complimentary inbound traffic (GB) per month.
`outbound_traffic`	`float`	Amount of complimentary outbound traffic (GB) per month.
`ipv4`	`int`	Number of complimentary IPv4 address(es).
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class ServerScd(Scd, ServerBase, table=True):
    """SCD version of .tables.Server."""

    pass

ServerPriceScd #

Bases: Scd, ServerPriceBase

Server type prices per Region and Allocation method (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`region_id`	`str`	Reference to the Region.
`zone_id`	`str`	Reference to the Zone.
`server_id`	`str`	Reference to the Server.
`operating_system`	`str`	Operating System.
`allocation`	`Allocation`	Allocation method, e.g. on-demand or spot.
`unit`	`PriceUnit`	Billing unit of the pricing model.
`price`	`float`	Actual price of a billing unit.
`price_upfront`	`float`	Price to be paid when setting up the resource.
`price_tiered`	`List[PriceTier]`	List of pricing tiers with min/max thresholds and actual prices.
`currency`	`str`	Currency of the prices.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class ServerPriceScd(Scd, ServerPriceBase, table=True):
    """SCD version of .tables.ServerPrice."""

    pass

StoragePriceScd #

Bases: Scd, StoragePriceBase

Flexible Storage prices in each Region (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`region_id`	`str`	Reference to the Region.
`storage_id`	`str`	Reference to the Storage.
`unit`	`PriceUnit`	Billing unit of the pricing model.
`price`	`float`	Actual price of a billing unit.
`price_upfront`	`float`	Price to be paid when setting up the resource.
`price_tiered`	`List[PriceTier]`	List of pricing tiers with min/max thresholds and actual prices.
`currency`	`str`	Currency of the prices.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class StoragePriceScd(Scd, StoragePriceBase, table=True):
    """SCD version of .tables.StoragePrice."""

    pass

TrafficPriceScd #

Bases: Scd, TrafficPriceBase

Extra Traffic prices in each Region (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`region_id`	`str`	Reference to the Region.
`direction`	`TrafficDirection`	Direction of the traffic: inbound or outbound.
`unit`	`PriceUnit`	Billing unit of the pricing model.
`price`	`float`	Actual price of a billing unit.
`price_upfront`	`float`	Price to be paid when setting up the resource.
`price_tiered`	`List[PriceTier]`	List of pricing tiers with min/max thresholds and actual prices.
`currency`	`str`	Currency of the prices.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class TrafficPriceScd(Scd, TrafficPriceBase, table=True):
    """SCD version of .tables.TrafficPrice."""

    pass

Ipv4PriceScd #

Bases: Scd, Ipv4PriceBase

Price of an IPv4 address in each Region (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`region_id`	`str`	Reference to the Region.
`unit`	`PriceUnit`	Billing unit of the pricing model.
`price`	`float`	Actual price of a billing unit.
`price_upfront`	`float`	Price to be paid when setting up the resource.
`price_tiered`	`List[PriceTier]`	List of pricing tiers with min/max thresholds and actual prices.
`currency`	`str`	Currency of the prices.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class Ipv4PriceScd(Scd, Ipv4PriceBase, table=True):
    """SCD version of .tables.Ipv4Price."""

    pass

BenchmarkScd #

Bases: Scd, BenchmarkBase

Benchmark scenario definitions (SCD Type 2).

Attributes:

Name	Type	Description
`benchmark_id`	`str`	Unique identifier of a specific Benchmark.
`name`	`str`	Human-friendly name.
`description`	`Optional[str]`	Short description.
`framework`	`str`	The name of the benchmark framework/software/tool used.
`config_fields`	`dict`	A dictionary of descriptions on the framework-specific config options, e.g. {"bandwidth": "Memory amount to use for compression in MB."}.
`measurement`	`Optional[str]`	The name of measurement recoreded in the benchmark.
`unit`	`Optional[str]`	Optional unit of measurement for the benchmark score.
`higher_is_better`	`bool`	If higher benchmark score means better performance, or vica versa.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class BenchmarkScd(Scd, BenchmarkBase, table=True):
    """SCD version of .tables.Benchmark."""

    pass

BenchmarkScoreScd #

Bases: Scd, BenchmarkScoreBase

Results of running Benchmark scenarios on Servers (SCD Type 2).

Attributes:

Name	Type	Description
`vendor_id`	`str`	Reference to the Vendor.
`server_id`	`str`	Reference to the Server.
`benchmark_id`	`str`	Reference to the Benchmark.
`config`	`HashableDict \| dict`	Dictionary of config parameters of the specific benchmark, e.g. {"bandwidth": 4096}
`score`	`float`	The resulting score of the benchmark.
`note`	`Optional[str]`	Optional note, comment or context on the benchmark score.
`status`	`Status`	Status of the resource (active or inactive).
`observed_at`	`datetime`	Timestamp of the last observation.

Source code in sc_crawler/tables_scd.py

class BenchmarkScoreScd(Scd, BenchmarkScoreBase, table=True):
    """SCD version of .tables.BenchmarkScore."""

    pass

tables_scd `module-attribute` #

tables_scd = [o for o in values() if is_table(o)]

List of all SCD SQLModel (table) models.

tables_scd

sc_crawler.tables_scd #

Scd #

CountryScd #

VendorComplianceLinkScd #

ComplianceFrameworkScd #

VendorScd #

RegionScd #

ZoneScd #

StorageScd #

ServerScd #

ServerPriceScd #

StoragePriceScd #

TrafficPriceScd #

Ipv4PriceScd #

BenchmarkScd #

BenchmarkScoreScd #

tables_scd module-attribute #

tables_scd `module-attribute` #