Skip to content

table_fields

sc_crawler.table_fields #

Enumerations, JSON nested data objects & other helper classes used in sc_crawler.tables.

Classes:

Name Description
HashableDict

A dict that can be hashed by its JSON representation.
HashableJSON

Alternative JSON SQLAlchemy column representation, which can be hashed.
Json

Custom base SQLModel class that supports dumping as JSON.
Status

Last known status of a resource, e.g. active or inactive.
Cpu

CPU details.
Gpu

GPU accelerator details.
StorageType

Type of a storage, e.g. HDD or SSD.
Disk

Disk definition based on size and storage type.
TrafficDirection

Direction of the network traffic.
CpuAllocation

CPU allocation methods at cloud vendors.
CpuArchitecture

CPU architectures.
DdrGeneration

Generation of the DDR SDRAM.
Allocation

Server allocation options.
PriceUnit

Supported units for the price tables.
PriceTier

Price tier definition.

HashableDict #

Bases: dict

A dict that can be hashed by its JSON representation.

Useful for typehinting dict-type table columns that are primary keys (which need to be hashable for SQLAlchemy ORM). See sc_crawler.table_fields.HashableJSON class for the related sa_type.

Source code in sc_crawler/table_fields.py 
12
13
14
15
16
17
18
19
20
21
class HashableDict(dict):
    """A dict that can be hashed by its JSON representation.

    Useful for typehinting dict-type table columns that are primary
    keys (which need to be hashable for SQLAlchemy ORM). See
    [sc_crawler.table_fields.HashableJSON][] class for the related `sa_type`.
    """

    def __hash__(self):
        return hash(dumps(self, sort_keys=True))

HashableJSON #

Bases: TypeDecorator

Alternative JSON SQLAlchemy column representation, which can be hashed.

Source code in sc_crawler/table_fields.py 
24
25
26
27
28
29
30
31
32
class HashableJSON(TypeDecorator):
    """Alternative JSON SQLAlchemy column representation, which can be hashed."""

    impl = JSON

    def process_result_value(self, value: str, dialect: Any) -> Any:
        if value is None:
            return None
        return HashableDict(value)

Json #

Bases: BaseModel

Custom base SQLModel class that supports dumping as JSON.

Methods:

Name Description
__json__

Call self.model_dump to serialize into JSON.
Source code in sc_crawler/table_fields.py 
35
36
37
38
39
40
class Json(BaseModel):
    """Custom base SQLModel class that supports dumping as JSON."""

    def __json__(self):
        """Call `self.model_dump` to serialize into JSON."""
        return dict(sorted(self.model_dump().items()))

__json__ #

__json__()

Call self.model_dump to serialize into JSON.

Source code in sc_crawler/table_fields.py 
38
39
40
def __json__(self):
    """Call `self.model_dump` to serialize into JSON."""
    return dict(sorted(self.model_dump().items()))

Status #

Bases: str, Enum

Last known status of a resource, e.g. active or inactive.

Attributes:

Name Type Description
ACTIVE

Active and available resource.
INACTIVE

Inactive resource that is not available anymore.
Source code in sc_crawler/table_fields.py 
43
44
45
46
47
48
49
class Status(str, Enum):
    """Last known status of a resource, e.g. active or inactive."""

    ACTIVE = "active"
    """Active and available resource."""
    INACTIVE = "inactive"
    """Inactive resource that is not available anymore."""

ACTIVE class-attribute instance-attribute #

ACTIVE = 'active'

Active and available resource.

INACTIVE class-attribute instance-attribute #

INACTIVE = 'inactive'

Inactive resource that is not available anymore.

Cpu #

Bases: Json

CPU details.

Attributes:

Name Type Description
manufacturer Optional[str]

The manufacturer of the processor, e.g. Intel or AMD.
family Optional[str]

The product line/family of the processor, e.g. Xeon, Core i7, Ryzen 9.
model Optional[str]

The model number of the processor, e.g. 9750H.
cores Optional[int]

Number of CPU cores.
threads Optional[int]

Number of CPU threads.
l1_cache_size Optional[int]

L1 cache size in bytes.
l2_cache_size Optional[int]

L2 cache size in bytes.
l3_cache_size Optional[int]

L3 cache size in bytes.
microcode Optional[str]

Microcode version.
capabilities List[str]

List of CPU flag/features/capabilities, e.g. MMX, Intel SGX etc.
bugs List[str]

List of known bugs, e.g. cpu_meltdown spectre_v1.
bogomips Optional[float]

BogoMips value.
Source code in sc_crawler/table_fields.py 
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
class Cpu(Json):
    """CPU details."""

    manufacturer: Optional[str] = None
    """The manufacturer of the processor, e.g. Intel or AMD."""
    family: Optional[str] = None
    """The product line/family of the processor, e.g. Xeon, Core i7, Ryzen 9."""
    model: Optional[str] = None
    """The model number of the processor, e.g. 9750H."""
    cores: Optional[int] = None
    """Number of CPU cores."""
    threads: Optional[int] = None
    """Number of CPU threads."""
    l1_cache_size: Optional[int] = None
    """L1 cache size in bytes."""
    l2_cache_size: Optional[int] = None
    """L2 cache size in bytes."""
    l3_cache_size: Optional[int] = None
    """L3 cache size in bytes."""
    microcode: Optional[str] = None
    """Microcode version."""
    capabilities: List[str] = []
    """List of CPU flag/features/capabilities, e.g. MMX, Intel SGX etc."""
    bugs: List[str] = []
    """List of known bugs, e.g. cpu_meltdown spectre_v1."""
    bogomips: Optional[float] = None
    """BogoMips value."""

manufacturer class-attribute instance-attribute #

manufacturer = None

The manufacturer of the processor, e.g. Intel or AMD.

family class-attribute instance-attribute #

family = None

The product line/family of the processor, e.g. Xeon, Core i7, Ryzen 9.

model class-attribute instance-attribute #

model = None

The model number of the processor, e.g. 9750H.

cores class-attribute instance-attribute #

cores = None

Number of CPU cores.

threads class-attribute instance-attribute #

threads = None

Number of CPU threads.

l1_cache_size class-attribute instance-attribute #

l1_cache_size = None

L1 cache size in bytes.

l2_cache_size class-attribute instance-attribute #

l2_cache_size = None

L2 cache size in bytes.

l3_cache_size class-attribute instance-attribute #

l3_cache_size = None

L3 cache size in bytes.

microcode class-attribute instance-attribute #

microcode = None

Microcode version.

capabilities class-attribute instance-attribute #

capabilities = []

List of CPU flag/features/capabilities, e.g. MMX, Intel SGX etc.

bugs class-attribute instance-attribute #

bugs = []

List of known bugs, e.g. cpu_meltdown spectre_v1.

bogomips class-attribute instance-attribute #

bogomips = None

BogoMips value.

Gpu #

Bases: Json

GPU accelerator details.

Attributes:

Name Type Description
manufacturer str

The manufacturer/brand of the GPU accelerator, e.g. Nvidia or AMD.
family Optional[str]

The model family/architecture of the GPU accelerator.
model Optional[str]

The model number of the GPU accelerator.
memory int

Memory (MiB) allocated to the GPU accelerator.
firmware_version Optional[str]

Firmware version.
bios_version Optional[str]

Video BIOS version.
graphics_clock Optional[int]

GPU core clock speed (Mhz).
sm_clock Optional[int]

Streaming Multiprocessor clock speed (Mhz).
mem_clock Optional[int]

Memory clock speed (Mhz).
video_clock Optional[int]

Video clock speed (Mhz).
Source code in sc_crawler/table_fields.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
class Gpu(Json):
    """GPU accelerator details."""

    manufacturer: str
    """The manufacturer/brand of the GPU accelerator, e.g. Nvidia or AMD."""
    family: Optional[str] = None
    """The model family/architecture of the GPU accelerator."""
    model: Optional[str] = None
    """The model number of the GPU accelerator."""
    memory: int
    """Memory (MiB) allocated to the GPU accelerator."""
    firmware_version: Optional[str] = None
    """Firmware version."""
    bios_version: Optional[str] = None
    """Video BIOS version."""
    graphics_clock: Optional[int] = None
    """GPU core clock speed (Mhz)."""
    sm_clock: Optional[int] = None
    """Streaming Multiprocessor clock speed (Mhz)."""
    mem_clock: Optional[int] = None
    """Memory clock speed (Mhz)."""
    video_clock: Optional[int] = None
    """Video clock speed (Mhz)."""

manufacturer instance-attribute #

manufacturer

The manufacturer/brand of the GPU accelerator, e.g. Nvidia or AMD.

family class-attribute instance-attribute #

family = None

The model family/architecture of the GPU accelerator.

model class-attribute instance-attribute #

model = None

The model number of the GPU accelerator.

memory instance-attribute #

memory

Memory (MiB) allocated to the GPU accelerator.

firmware_version class-attribute instance-attribute #

firmware_version = None

Firmware version.

bios_version class-attribute instance-attribute #

bios_version = None

Video BIOS version.

graphics_clock class-attribute instance-attribute #

graphics_clock = None

GPU core clock speed (Mhz).

sm_clock class-attribute instance-attribute #

sm_clock = None

Streaming Multiprocessor clock speed (Mhz).

mem_clock class-attribute instance-attribute #

mem_clock = None

Memory clock speed (Mhz).

video_clock class-attribute instance-attribute #

video_clock = None

Video clock speed (Mhz).

StorageType #

Bases: str, Enum

Type of a storage, e.g. HDD or SSD.

Attributes:

Name Type Description
HDD

Magnetic hard disk drive.
SSD

Solid-state drive.
NVME_SSD

NVMe based solid-state drive.
NETWORK

Storage over network, e.g. using NFS.
Source code in sc_crawler/table_fields.py 
106
107
108
109
110
111
112
113
114
115
116
class StorageType(str, Enum):
    """Type of a storage, e.g. HDD or SSD."""

    HDD = "hdd"
    """Magnetic hard disk drive."""
    SSD = "ssd"
    """Solid-state drive."""
    NVME_SSD = "nvme ssd"
    """NVMe based solid-state drive."""
    NETWORK = "network"
    """Storage over network, e.g. using NFS."""

HDD class-attribute instance-attribute #

HDD = 'hdd'

Magnetic hard disk drive.

SSD class-attribute instance-attribute #

SSD = 'ssd'

Solid-state drive.

NVME_SSD class-attribute instance-attribute #

NVME_SSD = 'nvme ssd'

NVMe based solid-state drive.

NETWORK class-attribute instance-attribute #

NETWORK = 'network'

Storage over network, e.g. using NFS.

Disk #

Bases: Json

Disk definition based on size and storage type.

Attributes:

Name Type Description
size int

Storage size in GiB.
storage_type StorageType

Type of the storage.
description Optional[str]

Optional description of the storage, e.g. temp disk.
Source code in sc_crawler/table_fields.py 
119
120
121
122
123
124
125
126
127
class Disk(Json):
    """Disk definition based on size and storage type."""

    size: int = 0
    """Storage size in GiB."""
    storage_type: StorageType
    """[Type][sc_crawler.table_fields.StorageType] of the storage."""
    description: Optional[str] = None
    """Optional description of the storage, e.g. temp disk."""

size class-attribute instance-attribute #

size = 0

Storage size in GiB.

storage_type instance-attribute #

storage_type

Type of the storage.

description class-attribute instance-attribute #

description = None

Optional description of the storage, e.g. temp disk.

TrafficDirection #

Bases: str, Enum

Direction of the network traffic.

Attributes:

Name Type Description
IN

Inbound traffic.
OUT

Outbound traffic.
Source code in sc_crawler/table_fields.py 
130
131
132
133
134
135
136
class TrafficDirection(str, Enum):
    """Direction of the network traffic."""

    IN = "inbound"
    """Inbound traffic."""
    OUT = "outbound"
    """Outbound traffic."""

IN class-attribute instance-attribute #

IN = 'inbound'

Inbound traffic.

OUT class-attribute instance-attribute #

OUT = 'outbound'

Outbound traffic.

CpuAllocation #

Bases: str, Enum

CPU allocation methods at cloud vendors.

Attributes:

Name Type Description
SHARED

Shared CPU with other virtual server tenants.
BURSTABLE

CPU that can temporarily burst above its baseline performance.
DEDICATED

Dedicated CPU with known performance.
Source code in sc_crawler/table_fields.py 
139
140
141
142
143
144
145
146
147
class CpuAllocation(str, Enum):
    """CPU allocation methods at cloud vendors."""

    SHARED = "Shared"
    """Shared CPU with other virtual server tenants."""
    BURSTABLE = "Burstable"
    """CPU that can temporarily burst above its baseline performance."""
    DEDICATED = "Dedicated"
    """Dedicated CPU with known performance."""

SHARED class-attribute instance-attribute #

SHARED = 'Shared'

Shared CPU with other virtual server tenants.

BURSTABLE class-attribute instance-attribute #

BURSTABLE = 'Burstable'

CPU that can temporarily burst above its baseline performance.

DEDICATED class-attribute instance-attribute #

DEDICATED = 'Dedicated'

Dedicated CPU with known performance.

CpuArchitecture #

Bases: str, Enum

CPU architectures.

Attributes:

Name Type Description
ARM64

64-bit ARM architecture.
ARM64_MAC

Apple 64-bit ARM architecture.
I386

32-bit x86 architecture.
X86_64

64-bit x86 architecture.
X86_64_MAC

Apple 64-bit x86 architecture.
Source code in sc_crawler/table_fields.py 
150
151
152
153
154
155
156
157
158
159
160
161
162
class CpuArchitecture(str, Enum):
    """CPU architectures."""

    ARM64 = "arm64"
    """64-bit ARM architecture."""
    ARM64_MAC = "arm64_mac"
    """Apple 64-bit ARM architecture."""
    I386 = "i386"
    """32-bit x86 architecture."""
    X86_64 = "x86_64"
    """64-bit x86 architecture."""
    X86_64_MAC = "x86_64_mac"
    """Apple 64-bit x86 architecture."""

ARM64 class-attribute instance-attribute #

ARM64 = 'arm64'

64-bit ARM architecture.

ARM64_MAC class-attribute instance-attribute #

ARM64_MAC = 'arm64_mac'

Apple 64-bit ARM architecture.

I386 class-attribute instance-attribute #

I386 = 'i386'

32-bit x86 architecture.

X86_64 class-attribute instance-attribute #

X86_64 = 'x86_64'

64-bit x86 architecture.

X86_64_MAC class-attribute instance-attribute #

X86_64_MAC = 'x86_64_mac'

Apple 64-bit x86 architecture.

DdrGeneration #

Bases: str, Enum

Generation of the DDR SDRAM.

Attributes:

Name Type Description
DDR3

DDR3 SDRAM.
DDR4

DDR4 SDRAM.
DDR5

DDR5 SDRAM.
Source code in sc_crawler/table_fields.py 
165
166
167
168
169
170
171
172
173
class DdrGeneration(str, Enum):
    """Generation of the DDR SDRAM."""

    DDR3 = "DDR3"
    """DDR3 SDRAM."""
    DDR4 = "DDR4"
    """DDR4 SDRAM."""
    DDR5 = "DDR5"
    """DDR5 SDRAM."""

DDR3 class-attribute instance-attribute #

DDR3 = 'DDR3'

DDR3 SDRAM.

DDR4 class-attribute instance-attribute #

DDR4 = 'DDR4'

DDR4 SDRAM.

DDR5 class-attribute instance-attribute #

DDR5 = 'DDR5'

DDR5 SDRAM.

Allocation #

Bases: str, Enum

Server allocation options.

Attributes:

Name Type Description
ONDEMAND

On-demand server.
RESERVED

Reserved server.
SPOT

Spot/preemptible server.
Source code in sc_crawler/table_fields.py 
176
177
178
179
180
181
182
183
184
class Allocation(str, Enum):
    """Server allocation options."""

    ONDEMAND = "ondemand"
    """On-demand server."""
    RESERVED = "reserved"
    """Reserved server."""
    SPOT = "spot"
    """Spot/preemptible server."""

ONDEMAND class-attribute instance-attribute #

ONDEMAND = 'ondemand'

On-demand server.

RESERVED class-attribute instance-attribute #

RESERVED = 'reserved'

Reserved server.

SPOT class-attribute instance-attribute #

SPOT = 'spot'

Spot/preemptible server.

PriceUnit #

Bases: str, Enum

Supported units for the price tables.

Attributes:

Name Type Description
YEAR

Price per year.
MONTH

Price per month.
HOUR

Price per hour.
GIB

Price per gibibyte (GiB).
GB

Price per gigabyte (GB).
GB_MONTH

Price per gigabyte (GB)/month.
Source code in sc_crawler/table_fields.py 
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
class PriceUnit(str, Enum):
    """Supported units for the price tables."""

    YEAR = "year"
    """Price per year."""
    MONTH = "month"
    """Price per month."""
    HOUR = "hour"
    """Price per hour."""
    GIB = "GiB"
    """Price per gibibyte (GiB)."""
    GB = "GB"
    """Price per gigabyte (GB)."""
    GB_MONTH = "GB/month"
    """Price per gigabyte (GB)/month."""

YEAR class-attribute instance-attribute #

YEAR = 'year'

Price per year.

MONTH class-attribute instance-attribute #

MONTH = 'month'

Price per month.

HOUR class-attribute instance-attribute #

HOUR = 'hour'

Price per hour.

GIB class-attribute instance-attribute #

GIB = 'GiB'

Price per gibibyte (GiB).

GB class-attribute instance-attribute #

GB = 'GB'

Price per gigabyte (GB).

GB_MONTH class-attribute instance-attribute #

GB_MONTH = 'GB/month'

Price per gigabyte (GB)/month.

PriceTier #

Bases: Json

Price tier definition.

As standard JSON does not support Inf, NaN etc values, those should be passed as string, e.g. for the upper bound.

See float_inf_to_str for converting an infinite numeric value into "Infinity".

Attributes:

Name Type Description
lower Union[float, str]

Lower bound of pricing tier, e.g. 100 GB. Unit is defined in the parent object.
upper Union[float, str]

Upper bound of pricing tier, e.g. 1 TB. Unit is defined in the parent object.
price float

Price in the pricing tier. Currency is defined in the parent object.
Source code in sc_crawler/table_fields.py 
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
class PriceTier(Json):
    """Price tier definition.

    As standard JSON does not support Inf, NaN etc values,
    those should be passed as string, e.g. for the upper bound.

    See [float_inf_to_str][sc_crawler.utils.float_inf_to_str] for
    converting an infinite numeric value into "Infinity"."""

    lower: Union[float, str]
    """Lower bound of pricing tier, e.g. 100 GB. Unit is defined in the parent object."""
    upper: Union[float, str]
    """Upper bound of pricing tier, e.g. 1 TB. Unit is defined in the parent object."""
    price: float
    """Price in the pricing tier. Currency is defined in the parent object."""

lower instance-attribute #

lower

Lower bound of pricing tier, e.g. 100 GB. Unit is defined in the parent object.

upper instance-attribute #

upper

Upper bound of pricing tier, e.g. 1 TB. Unit is defined in the parent object.

price instance-attribute #

price

Price in the pricing tier. Currency is defined in the parent object.