Vendor support#
Each file in the src/sc_crawler/vendors folder provides the required helpers for a given Vendor, named as the id of the vendor prefixed with an underscore. For example, _aws.py provides functions to be used by its Vendor instance, called aws.
First steps#
- Define the new Vendor instance in
src/sc_crawler/vendors/vendors.py. - Copy the below template file as a starting point to
src/sc_crawler/vendors/_<vendor_id>.py. - Update
src/sc_crawler/vendors/__init__.pyto include the new vendor. - Update
docs/add_vendor.mdwith the credential requirements for the new vendor. - Implement the
inventorymethods. - Preferably also implement the related cloud-discovery tool in the
resource-trackerpackage. - Add the new vendor to
_boot_from_attached_network_drivein_find_storage_disks_from_lsblk(src/sc_crawler/inspector.py). The value (TrueorFalse) must match whether the vendor's Pulumi provisioning script in the sc-runner repo attaches a network block volume as the boot drive: checksrc/sc_runner/resources/<vendor_id>.py.
Inventory methods#
Each vendor module should provide the below functions:
inventory_compliance_frameworks: DefineVendorComplianceLinkinstances to describe which frameworks the vendor complies with. Optionally include references in thecommentfield. To avoid duplicatingComplianceFrameworkinstances, easiest is to use thecompliance_framework_idfield instead of thecompliance_frameworkrelationship, preferably via sc_crawler.lookup.map_compliance_frameworks_to_vendor.inventory_regions: DefineRegioninstances with location, energy source etc for each region the vendor has.inventory_zones: Define aZoneinstance for each availability zone of the vendor in each region.inventory_servers: DefineServerinstances for the vendor's server/instance types.inventory_server_prices: Define theServerPriceinstances for the standard/ondemand (or optionally also for the reserved) pricing of the instance types per region and zone. When applicable, include the monthly cap for tiered pricing in theprice_tieredfield.inventory_server_prices_spot: Similar to the above, defineServerPriceinstances but theallocationfield set toAllocation.SPOT. Very likely to see different spot prices per region/zone.inventory_storage_prices: DefineStoragePriceinstances to describe the available storage options that can be attached to the servers.inventory_databases: DefineDatabaseinstances for managed database instances (e.g. PostgreSQL).inventory_database_prices: DefineDatabasePriceinstances for compute pricing per region.haandha_strategyare scalar primary-key fields (one price row per HA deployment). Use hourlypricewith optional monthly cap inprice_tiered.inventory_database_storages: DefineDatabaseStorageinstances for decoupled or add-on storage products.inventory_database_storage_prices: DefineDatabaseStoragePriceinstances for storage pricing per region.inventory_traffic_prices: DefineTrafficPriceinstances to describe the pricing of ingress/egress traffic.inventory_ipv4_prices: DefineIpv4Priceinstances on the price of an IPv4 address.
Each function will be picked up as the related Vendor instance's instance methods, so each function should take a single argument, that is the Vendor instance. E.g. sc_crawler.vendors._aws.inventory_regions is called by sc_crawler.tables.Vendor.inventory_regions.
The functions should return an array of dict representing the related objects. The vendor's inventory method will pass the array to sc_crawler.insert.insert_items along with the table object.
Other functions and variables must be prefixed with an underscore to suggest those are internal tools.
Progress bars#
To create progress bars, you can use the Vendor's progress_tracker attribute with the below methods:
The start_task will register a task in the "Current tasks"
progress bar list with the provided name automatically prefixed by the vendor name, and the provided number of expected
steps. You should call advance_task after each step finished,
which will by default update the most recently created task's progress bar. If making updates in parallel, store the
TaskID returned by start_task and pass
to advance_task
and hide_task explicitly. Make sure to call hide_task when the
progress bar is not to be shown anymore. It's a good practice to log the number of fetched/synced objects afterwards
with logger.info. See the manual of VendorProgressTracker for more
details.
Basic example:
def inventory_zones(vendor):
zones = range(5)
vendor.progress_tracker.start_task(name="Searching zones", total=len(zones))
for zone in zones:
# do something
vendor.progress_tracker.advance_task()
vendor.progress_tracker.hide_task()
return zones
Template file for new vendors#
def inventory_compliance_frameworks(vendor):
return map_compliance_frameworks_to_vendor(
vendor.vendor_id,
[
# "hipaa",
# "soc2t2",
# "iso27001",
],
)
def inventory_regions(vendor):
items = []
# for region in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": "",
# "name": "",
# "api_reference": "",
# "display_name": "",
# "aliases": [],
# "country_id": "",
# "state": None,
# "city": None,
# "address_line": None,
# "zip_code": None,
# "lon": None,
# "lat": None,
# "founding_year": None,
# "green_energy": None,
# }
# )
return items
def inventory_zones(vendor):
items = []
# for zone in []:
# items.append({
# "vendor_id": vendor.vendor_id,
# "region_id": "",
# "zone_id": "",
# "name": "",
# "api_reference": "",
# "display_name": "",
# })
return items
def inventory_servers(vendor):
items = []
# for server in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "server_id": ,
# "name": ,
# "api_reference": ,
# "display_name": ,
# "description": None,
# "family": None,
# "vcpus": ,
# "hypervisor": None,
# "cpu_allocation": CpuAllocation....,
# "cpu_cores": None,
# "cpu_speed": None,
# "cpu_architecture": CpuArchitecture....,
# "cpu_manufacturer": None,
# "cpu_family": None,
# "cpu_model": None,
# "cpu_l1d_cache": None,
# "cpu_l1d_cache_total": None,
# "cpu_l1i_cache": None,
# "cpu_l1i_cache_total": None,
# "cpu_l2_cache": None,
# "cpu_l2_cache_total": None,
# "cpu_l3_cache": None,
# "cpu_l3_cache_total": None,
# "cpu_flags": [],
# "cpus": [],
# "memory_amount": ,
# "memory_generation": None,
# "memory_speed": None,
# "memory_ecc": None,
# "gpu_count": 0,
# "gpu_memory_min": None,
# "gpu_memory_total": None,
# "gpu_manufacturer": None,
# "gpu_family": None,
# "gpu_model": None,
# "gpus": [],
# "storage_size": 0,
# "storage_type": None,
# "storages": [],
# "network_speed": None,
# "inbound_traffic": 0,
# "outbound_traffic": 0,
# "ipv4": 0,
# }
# )
return items
def inventory_server_prices(vendor):
items = []
# for server in []:
# items.append({
# "vendor_id": ,
# "region_id": ,
# "zone_id": ,
# "server_id": ,
# "operating_system": ,
# "allocation": Allocation....,
# "unit": PriceUnit.HOUR,
# "price": ,
# "price_upfront": 0,
# "price_tiered": [
# {"lower": 0, "upper": monthly_cap, "price": hourly_price},
# {"lower": monthly_cap + 1, "upper": "Infinity", "price": 0},
# ],
# "currency": "USD",
# })
return items
def inventory_server_prices_spot(vendor):
return []
def inventory_storages(vendor):
items = []
# for storage in []:
# items.append(
# {
# "storage_id": ,
# "vendor_id": vendor.vendor_id,
# "name": ,
# "description": None,
# "storage_type": StorageType....,
# "max_iops": None,
# "max_throughput": None,
# "min_size": None,
# "max_size": None,
# }
# )
return items
def inventory_storage_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "storage_id": ,
# "unit": PriceUnit.GB_MONTH,
# "price": ,
# "currency": "USD",
# }
# )
return items
def inventory_traffic_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "price": ,
# "price_tiered": [],
# "currency": "USD",
# "unit": PriceUnit.GB_MONTH,
# "direction": TrafficDirection....,
# }
# )
return items
def inventory_ipv4_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "price": ,
# "currency": "USD",
# "unit": PriceUnit.MONTH,
# }
# )
return items
def inventory_databases(vendor):
items = []
# for database in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "database_id": ,
# "name": ,
# "api_reference": ,
# # Named API args for provisioning (e.g. Pulumi/Terraform).
# "api_reference_object": None,
# "display_name": ,
# "description": None,
# "family": None,
# "server_id": None,
# "vcpus": None,
# "memory_amount": None, # MiB
# "engine": DatabaseEngine.POSTGRESQL,
# "wire_protocol": DatabaseWireProtocol.POSTGRESQL,
# "engine_versions": ["16", "17"],
# "auto_upgrade_versions": None,
# # Ordered JSON lists (highest tier first); defaults to [NONE].
# "ha": [DatabaseHaLevel.NONE],
# "ha_strategy": [DatabaseHaStrategy.NONE],
# "max_read_replicas": None,
# "custom_config": None,
# "custom_extensions": None,
# "storage_size": None, # bundled GB
# "storage_extra_min": None, # GB
# "storage_extra_max": None, # GB
# "storage_extra_autosize": None,
# "disk_encryption": None,
# "scheduled_backups": None,
# "continuous_backups": None, # max PITR retention (days)
# "connection_pool": None,
# "system_monitoring": None,
# "database_monitoring": None,
# "autotuning_advice": None,
# "autotuning_apply": None,
# "sla": None, # e.g. 99.95
# # Supported security capabilities for this db instance.
# "security_features": [
# # DatabaseSecurityFeature.IP_FILTERING,
# # DatabaseSecurityFeature.PRIVATE_NETWORK,
# # DatabaseSecurityFeature.NETWORK_PEERING,
# # DatabaseSecurityFeature.IDENTITY_BASED_AUTH,
# # DatabaseSecurityFeature.CLIENT_CERT_AUTH,
# # DatabaseSecurityFeature.ENFORCED_TLS,
# # DatabaseSecurityFeature.CUSTOMER_MANAGED_KEYS,
# # DatabaseSecurityFeature.AUDIT_LOGGING,
# ],
# }
# )
return items
def inventory_database_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "database_id": ,
# "allocation": Allocation.ONDEMAND,
# # Scalar PK fields: one row per HA deployment (e.g. Single-AZ vs Multi-AZ).
# "ha": DatabaseHaLevel.NONE,
# "ha_strategy": DatabaseHaStrategy.NONE,
# "unit": PriceUnit.HOUR,
# "price": ,
# "price_upfront": 0,
# "price_tiered": [],
# "currency": "USD",
# }
# )
return items
def inventory_database_storages(vendor):
items = []
# for storage in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "database_storage_id": ,
# "name": ,
# "description": None,
# "scope": DatabaseStorageScope.DATA,
# "min_size": None,
# "max_size": None,
# }
# )
return items
def inventory_database_storage_prices(vendor):
items = []
# for price in []:
# items.append(
# {
# "vendor_id": vendor.vendor_id,
# "region_id": ,
# "database_storage_id": ,
# "unit": PriceUnit.GB_MONTH,
# "price": ,
# "currency": "USD",
# }
# )
return items