workload_profiles

sc_crawler.workload_profiles

Workload profile definitions for compound benchmark scoring.

Each workload profile is a weighted combination of benchmark scores that represents a specific real-world usage pattern. Scores are aggregated as a weighted average (geometric mean) of benchmark scores compared to their medians. A score of 1.0 represents a synthetic baseline server with the median performance of each component benchmark.

Weights within each workload sum to 1.0.

Classes:

Name	Description
BenchmarkEntry	A single benchmark component contributing to a workload profile score.
CompoundSource
Workload	A named workload profile composed of weighted benchmark entries.

Attributes:

Name	Type	Description
WORKLOADS	dict[str, Workload]	Workload profile definitions keyed by workload ID.

BenchmarkEntry

Bases: BaseModel

A single benchmark component contributing to a workload profile score.

Methods:

Name	Description
effective_penalty	Return the penalty floor used when on_missing is PENALIZE.

Attributes:

Name	Type	Description
benchmark_id	str	The benchmark ID of a BenchmarkScore.
weight	float	Relative weight of this component. Weights within a workload sum to 1.0.
label	str	Human-readable description of what this component measures.
config_filter	dict[str, Any] | None	Optional filter applied to the benchmark's config JSON column.
on_missing	BenchmarkComponentMissingPolicy	How to handle a missing or invalid measurement for this component.
penalty	float | None	Substituted normalized ratio when on_missing is PENALIZE.

Source code in sc_crawler/workload_profiles.py

class BenchmarkEntry(BaseModel):
    """A single benchmark component contributing to a workload profile score."""

    benchmark_id: str
    """The benchmark ID of a BenchmarkScore."""
    weight: float
    """Relative weight of this component. Weights within a workload sum to 1.0."""
    label: str
    """Human-readable description of what this component measures."""
    config_filter: dict[str, Any] | None = None
    """Optional filter applied to the benchmark's config JSON column."""
    on_missing: BenchmarkComponentMissingPolicy = BenchmarkComponentMissingPolicy.IGNORE
    """How to handle a missing or invalid measurement for this component."""
    penalty: float | None = None
    """Substituted normalized ratio when on_missing is PENALIZE."""

    model_config = ConfigDict(frozen=True)

    def effective_penalty(self) -> float:
        """Return the penalty floor used when on_missing is PENALIZE."""
        return self.penalty if self.penalty is not None else _DEFAULT_COMPONENT_PENALTY

    def __json__(self):
        data = dict(sorted(self.model_dump(mode="json").items()))
        if data.get("on_missing") == BenchmarkComponentMissingPolicy.PENALIZE.value:
            if data.get("penalty") is None:
                data["penalty"] = _DEFAULT_COMPONENT_PENALTY
        else:
            data.pop("penalty", None)
        return data

benchmark_id instance-attribute

benchmark_id

The benchmark ID of a BenchmarkScore.

weight instance-attribute

weight

Relative weight of this component. Weights within a workload sum to 1.0.

label instance-attribute

label

Human-readable description of what this component measures.

config_filter class-attribute instance-attribute

config_filter = None

Optional filter applied to the benchmark's config JSON column.

on_missing class-attribute instance-attribute

on_missing = BenchmarkComponentMissingPolicy.IGNORE

How to handle a missing or invalid measurement for this component.

penalty class-attribute instance-attribute

penalty = None

Substituted normalized ratio when on_missing is PENALIZE.

effective_penalty

effective_penalty()

Return the penalty floor used when on_missing is PENALIZE.

Source code in sc_crawler/workload_profiles.py

def effective_penalty(self) -> float:
    """Return the penalty floor used when on_missing is PENALIZE."""
    return self.penalty if self.penalty is not None else _DEFAULT_COMPONENT_PENALTY

CompoundSource

Bases: Json

Attributes:

Name	Type	Description
aggregation	BenchmarkComponentAggregationMethod	How component benchmark scores are combined into one composite score.
normalization	BenchmarkComponentNormalizationMethod	How each raw benchmark value is scaled to be comparable across benchmarks.
components	list[BenchmarkEntry]	The components of the workload profile.
impact_formula	str | None	Human-friendly explanation of how to read per-component impact on scores.

Source code in sc_crawler/workload_profiles.py

class CompoundSource(Json):
    kind: Literal["compound"] = "compound"
    aggregation: BenchmarkComponentAggregationMethod
    """How component benchmark scores are combined into one composite score."""
    normalization: BenchmarkComponentNormalizationMethod
    """How each raw benchmark value is scaled to be comparable across benchmarks."""
    components: list[BenchmarkEntry]
    """The components of the workload profile."""
    impact_formula: str | None = None
    """Human-friendly explanation of how to read per-component ``impact`` on scores."""

    def __json__(self):
        data = dict(sorted(self.model_dump(mode="json").items()))
        data["components"] = [component.__json__() for component in self.components]
        return data

aggregation instance-attribute

aggregation

How component benchmark scores are combined into one composite score.

normalization instance-attribute

normalization

How each raw benchmark value is scaled to be comparable across benchmarks.

components instance-attribute

components

The components of the workload profile.

impact_formula class-attribute instance-attribute

impact_formula = None

Human-friendly explanation of how to read per-component impact on scores.

Workload

Bases: BaseModel

A named workload profile composed of weighted benchmark entries.

Attributes:

Name	Type	Description
name	str	Short human-readable name, e.g. 'Web server'.
version	str	Workload profile version.
rationale	str	Explanation of which benchmarks were chosen and why.
benchmarks	list[BenchmarkEntry]	Ordered list of benchmark components with weights.

Source code in sc_crawler/workload_profiles.py

class Workload(BaseModel):
    """A named workload profile composed of weighted benchmark entries."""

    name: str
    """Short human-readable name, e.g. 'Web server'."""
    version: str
    """Workload profile version."""
    rationale: str
    """Explanation of which benchmarks were chosen and why."""
    benchmarks: list[BenchmarkEntry]
    """Ordered list of benchmark components with weights."""

name instance-attribute

name

Short human-readable name, e.g. 'Web server'.

version instance-attribute

version

Workload profile version.

rationale instance-attribute

rationale

Explanation of which benchmarks were chosen and why.

benchmarks instance-attribute

benchmarks

Ordered list of benchmark components with weights.

WORKLOADS module-attribute

WORKLOADS = {'web': Workload(name='Web Server', version='2.0', rationale='Primary workloads drivers are single-process static HTTP serving speed and throughput, text processing, TLS termination, and asset compression.', benchmarks=[BenchmarkEntry(benchmark_id='static_web:rps-extrapolated', weight=0.3, label='Static web RPS (1 KiB, 8 conn/vCPU)', config_filter={'size': '1k', 'connections_per_vcpus': 8.0}), BenchmarkEntry(benchmark_id='static_web:rps-extrapolated', weight=0.2, label='Static web RPS (64 KiB, 8 conn/vCPU)', config_filter={'size': '64k', 'connections_per_vcpus': 8.0}), BenchmarkEntry(benchmark_id='static_web:throughput-extrapolated', weight=0.2, label='Static web throughput (256 KiB, 8 conn/vCPU)', config_filter={'size': '256k', 'connections_per_vcpus': 8.0}), BenchmarkEntry(benchmark_id='openssl', weight=0.2, label='OpenSSL AES-256-CBC (16 kB blocks)', config_filter={'algo': 'AES-256-CBC', 'block_size': 16384}), BenchmarkEntry(benchmark_id='compression_text:compress', weight=0.05, label='Gzip compression (multi-core, level 5)', config_filter={'algo': 'gzip', 'compression_level': 5, 'cores': 'multi'}), BenchmarkEntry(benchmark_id='passmark:cpu_string_sorting_test', weight=0.05, label='PassMark string sorting')]), 'compute': Workload(name='Compute Heavy Applications', version='2.0', rationale='Number-crunching workload augmenting raw CPU performance stressing, general CPU performance benchmarks, memory bandwidth, and pure math computation speed like floating point, integer, SIMD (AVX/SSE/FMA) operations.', benchmarks=[BenchmarkEntry(benchmark_id='stress_ng:bestn', weight=0.15, label='stress-ng div16 best-N cores'), BenchmarkEntry(benchmark_id='stress_ng:best1', weight=0.1, label='stress-ng div16 single core'), BenchmarkEntry(benchmark_id='passmark:cpu_mark', weight=0.2, label='PassMark CPU Mark (composite)'), BenchmarkEntry(benchmark_id='bw_mem', weight=0.1, label='Memory bandwidth (read, 64 MB)', config_filter={'operation': 'rd', 'size': 64.0}), BenchmarkEntry(benchmark_id='passmark:cpu_floating_point_maths_test', weight=0.15, label='PassMark floating point'), BenchmarkEntry(benchmark_id='passmark:cpu_extended_instructions_test', weight=0.15, label='PassMark AVX/SSE/FMA (SIMD)'), BenchmarkEntry(benchmark_id='passmark:cpu_integer_maths_test', weight=0.1, label='PassMark integer math'), BenchmarkEntry(benchmark_id='passmark:cpu_physics_test', weight=0.05, label='PassMark physics simulation')]), 'cache': Workload(name='Cache Intensive', version='2.0', rationale='In-memory key-value store workload, mixing direct Redis performance metrics with memory speed and latency benchmarks, and single-core CPU performance profiles.', benchmarks=[BenchmarkEntry(benchmark_id='redis:rps-extrapolated', weight=0.5, label='Redis RPS (pipeline=1, SET)', config_filter={'operation': 'SET', 'pipeline': 1.0}), BenchmarkEntry(benchmark_id='redis:rps-extrapolated', weight=0.2, label='Redis RPS (pipeline=16, SET)', config_filter={'operation': 'SET', 'pipeline': 16.0}), BenchmarkEntry(benchmark_id='passmark:memory_mark', weight=0.1, label='PassMark Memory Mark (composite)'), BenchmarkEntry(benchmark_id='bw_mem', weight=0.1, label='Memory bandwidth (read, 16 MB ~ L3)', config_filter={'operation': 'rd', 'size': 16.0}), BenchmarkEntry(benchmark_id='passmark:cpu_single_threaded_test', weight=0.1, label='PassMark single-thread CPU')]), 'data_analysis': Workload(name='Data Analysis', version='2.0', rationale='Data analysis and ETL workloads are memory-bandwidth-bound and CPU-throughput-driven. The profile combines general CPU performance and memory bandwidth/latency as the primary drivers, supplemented by single-core compression speed as a proxy for serialisation-heavy ETL tasks.', benchmarks=[BenchmarkEntry(benchmark_id='passmark:cpu_mark', weight=0.7, label='PassMark CPU Mark (composite)'), BenchmarkEntry(benchmark_id='compression_text:compress', weight=0.1, label='Gzip compression (single-core, level 5)', config_filter={'algo': 'gzip', 'compression_level': 5, 'cores': 'single'}), BenchmarkEntry(benchmark_id='bw_mem', weight=0.1, label='Memory bandwidth (read, 64 MB)', config_filter={'operation': 'rd', 'size': 64.0}), BenchmarkEntry(benchmark_id='passmark:memory_mark', weight=0.1, label='PassMark Memory Mark (composite)')]), 'llm': Workload(name='LLM Inference', version='2.0', rationale='VRAM and memory-bandwidth-bound LLM inference workload, using direct LLM speed benchmarks at three model sizes, and supplementing with raw memory bandwidth and SIMD performance benchmarks.', benchmarks=[BenchmarkEntry(benchmark_id='llm_speed:text_generation', weight=0.15, label='LLM text generation (SmolLM-135M, 128 tok)', config_filter={'model': 'SmolLM-135M.Q4_K_M.gguf', 'tokens': 128}, on_missing=BenchmarkComponentMissingPolicy.REQUIRE), BenchmarkEntry(benchmark_id='llm_speed:prompt_processing', weight=0.15, label='LLM prompt processing (SmolLM-135M, 512 tok)', config_filter={'model': 'SmolLM-135M.Q4_K_M.gguf', 'tokens': 512}, on_missing=BenchmarkComponentMissingPolicy.REQUIRE), BenchmarkEntry(benchmark_id='llm_speed:text_generation', weight=0.15, label='LLM text generation (Llama 7B, 128 tok)', config_filter={'model': 'llama-7b.Q4_K_M.gguf', 'tokens': 128}, on_missing=BenchmarkComponentMissingPolicy.PENALIZE, penalty=0.0001), BenchmarkEntry(benchmark_id='llm_speed:prompt_processing', weight=0.15, label='LLM prompt processing (Llama 7B, 512 tok)', config_filter={'model': 'llama-7b.Q4_K_M.gguf', 'tokens': 512}, on_missing=BenchmarkComponentMissingPolicy.PENALIZE, penalty=0.0001), BenchmarkEntry(benchmark_id='llm_speed:text_generation', weight=0.15, label='LLM text generation (Llama-3.3 70B, 128 tok)', config_filter={'model': 'Llama-3.3-70B-Instruct-Q4_K_M.gguf', 'tokens': 128}, on_missing=BenchmarkComponentMissingPolicy.PENALIZE, penalty=0.01), BenchmarkEntry(benchmark_id='llm_speed:prompt_processing', weight=0.15, label='LLM prompt processing (Llama-3.3 70B, 512 tok)', config_filter={'model': 'Llama-3.3-70B-Instruct-Q4_K_M.gguf', 'tokens': 512}, on_missing=BenchmarkComponentMissingPolicy.PENALIZE, penalty=0.01), BenchmarkEntry(benchmark_id='bw_mem', weight=0.05, label='Memory bandwidth (read, 256 MB)', config_filter={'operation': 'rd', 'size': 256.0}), BenchmarkEntry(benchmark_id='passmark:cpu_extended_instructions_test', weight=0.025, label='PassMark AVX/SSE/FMA (SIMD)'), BenchmarkEntry(benchmark_id='passmark:cpu_floating_point_maths_test', weight=0.025, label='PassMark floating point')]), 'cicd': Workload(name='CI/CD Build', version='2.0', rationale='Build performance is mainly driven by multi-core compilation throughput, but also bundles single-core compilation speed and general CPU performance, multi-core compression and text/scripting processing.', benchmarks=[BenchmarkEntry(benchmark_id='geekbench:clang', weight=0.5, label='Geekbench Clang compilation (multi-core)', config_filter={'cores': 'multi'}), BenchmarkEntry(benchmark_id='geekbench:clang', weight=0.1, label='Geekbench Clang compilation (single-core)', config_filter={'cores': 'single'}), BenchmarkEntry(benchmark_id='stress_ng:bestn', weight=0.2, label='stress-ng div16 best-N cores'), BenchmarkEntry(benchmark_id='passmark:cpu_integer_maths_test', weight=0.05, label='PassMark integer math'), BenchmarkEntry(benchmark_id='passmark:cpu_compression_test', weight=0.05, label='PassMark compression'), BenchmarkEntry(benchmark_id='compression_text:compress', weight=0.05, label='Brotli compression (multi-core, level 0)', config_filter={'algo': 'brotli', 'compression_level': 0, 'cores': 'single'}), BenchmarkEntry(benchmark_id='passmark:cpu_string_sorting_test', weight=0.05, label='PassMark string sorting')])}

Workload profile definitions keyed by workload ID.