gBench: A Comprehensive Open‑Source Benchmarking Suite for LLMs and VLMs

(≈ 4,000 words)

1. Why gBench Exists

Open‑source language and vision‑language models (LLMs & VLMs) have exploded in both popularity and variety. Researchers now ship everything from tiny 125‑M parameter models to gigantic 100‑B parameter behemoths. At the same time, hardware vendors release new GPUs, TPUs, and specialized inference chips every month. The gap between model development and real‑world deployment is wide:

| Gap | Existing Tools | What gBench Adds | |-----|----------------|------------------| | Serving performance | Benchmarks usually focus on accuracy or perplexity | gBench measures latency, throughput, concurrency, and cost in a realistic serving setup | | Hardware profiling | Many tools ignore GPU memory, CPU utilisation, or power draw | gBench profiles hardware utilisation on the exact machine running the benchmark | | Model agnosticism | Some suites lock to specific frameworks (TensorFlow, PyTorch) | gBench is framework‑agnostic: it supports PyTorch, ONNX, HuggingFace Transformers, and can be extended to other back‑ends | | Extensibility | Few open‑source benchmarks allow community‑driven task or model additions | gBench uses a plugin architecture that lets anyone drop in new datasets, metrics, or model adapters |

gBench therefore fills a critical niche: it is the first comprehensive, open‑source suite that lets you measure how a model performs when you actually run it on the hardware you own, rather than just how well it performs on a paper benchmark.

2. The Challenges of Benchmarking Modern Models

Before diving into gBench’s design, it helps to understand why such a suite is necessary. Modern LLMs & VLMs present several unique challenges:

1. Scale and Complexity

• Models can contain billions of parameters, distributed across multiple GPUs.
• Running them requires careful handling of model sharding, parallelism, and distributed inference.

1. Serving vs. Training

• Training metrics (loss, perplexity) are useful during development, but serving metrics (latency, throughput) dictate user experience.
• Deployment can involve batching, queuing, or multi‑tenant inference, all of which affect latency.

1. Hardware Heterogeneity

• GPUs differ dramatically: A100 vs. RTX 3090 vs. A10G vs. edge TPUs.
• CPU‑backed inference (e.g., on mobile devices) is a separate scenario.

1. Optimization Techniques

• Quantisation (INT8, INT4), pruning, knowledge distillation, and model sparsification all alter the speed‑accuracy trade‑off.
• Benchmarks must measure the impact of each optimisation path.

1. Real‑world Workloads

• Chatbots, summarisation engines, or image captioners often receive bursty traffic.
• Benchmarks must support concurrency and load‑testing scenarios, not just single‑request latency.

1. Reproducibility

• Many existing benchmarks are tightly coupled to a specific environment or dataset.
• gBench emphasises determinism (fixed seeds, consistent dataset splits) and open‑source reproducibility.

By tackling these issues head‑on, gBench provides the industry with a single tool that can be applied to any model‑hardware combination.

3. gBench’s Architecture

gBench’s design is intentionally modular. Think of it as a pipeline consisting of:

1. Core Engine – orchestrates benchmark runs, manages concurrency, records metrics.
2. Adapters – model‑specific wrappers that translate gBench’s API into the underlying framework (PyTorch, ONNX, TensorFlow).
3. Dataset Loaders – responsible for fetching and preprocessing the input data for each task.
4. Metrics Engine – computes evaluation scores (e.g., BLEU, ROUGE, accuracy, perplexity).
5. Reporting Layer – produces JSON, CSV, and interactive dashboards.

(Figure: The modular pipeline of gBench. Each component can be swapped or extended via plugins.)

3.1 Core Engine

At the heart of gBench is a lightweight scheduler that can:

• Run benchmarks sequentially or in parallel.
• Accept batch sizes, sequence lengths, and max tokens as configuration options.
• Dispatch requests to multiple worker processes for multi‑GPU setups.
• Measure warm‑up time versus steady‑state latency.

The engine logs each request’s timestamp, output, and any hardware counters, feeding them into the metrics engine.

3.2 Adapter Layer

gBench does not ship with a monolithic model loader. Instead, adapters are tiny Python modules that expose a common interface:

class ModelAdapter:
    def __init__(self, config): ...
    def forward(self, inputs): ...

This abstraction means you can plug in a PyTorch Llama adapter or an ONNX BLIP adapter with no changes to the rest of the system.

3.3 Dataset Loaders

Each supported task (e.g., question‑answering, image captioning, translation) has its own loader that:

• Downloads the official dataset (SQuAD, MSCOCO, GLUE, etc.).
• Handles tokenisation (via HuggingFace tokenisers) and image preprocessing (ResNet‑style).
• Splits into validation and test sets deterministically.

Loaders are also lazy: they stream examples to avoid memory overhead.

3.4 Metrics Engine

Accuracy‑related metrics are computed after the benchmark finishes:

| Task | Metric | Description | |------|--------|-------------| | LLM | Perplexity | Log‑probability of gold outputs | | LLM | BLEU / ROUGE | N‑gram overlap for summarisation | | VLM | Accuracy | Exact match on VQA datasets | | VLM | CIDEr | Consensus‑based caption quality |

For latency and throughput, the engine uses Python’s timeit and psutil modules, coupled with NVML for GPU utilisation.

3.5 Reporting Layer

After the run, gBench generates:

1. CSV – raw timings, memory usage, and per‑example outputs.
2. JSON – structured data for downstream analysis.
3. HTML Dashboard – interactive charts (latency percentiles, throughput curves, GPU utilisation).
4. PDF Summary – ready‑to‑share report for stakeholders.

4. Supported Models & Tasks

gBench was built with extensibility in mind. The initial release already covers dozens of models and tasks; the plugin system makes it trivial to add more.

4.1 LLMs

| Model | Parameters | Framework | Notes | |-------|------------|-----------|-------| | GPT‑NeoX | 20B | PyTorch | 8‑bit quantisation support | | Llama‑2 | 7B / 13B | PyTorch / ONNX | Supports chat and generation modes | | Mistral | 7B | PyTorch | 16‑bit floating‑point (FP16) default | | Bloom | 176B | HuggingFace | Requires distributed inference |

Adapters for each model implement token‑injection for streaming outputs, essential for real‑time chat services.

4.2 VLMs

| Model | Backbone | Task | Framework | |-------|----------|------|-----------| | Stable Diffusion | Diffusion | Text‑to‑Image | PyTorch | | BLIP‑2 | CLIP‑ViT | Image Captioning | PyTorch | | OpenAI CLIP | ViT | Zero‑shot Image Classification | ONNX | | VQA‑GPT | GPT + Vision | Visual QA | PyTorch |

VLM adapters expose image preprocessing pipelines and cross‑modal attention handling, enabling end‑to‑end inference with minimal code.

4.3 Model Formats

| Format | Strengths | gBench Support | |--------|-----------|----------------| | PyTorch (.pt) | Native training API | Full support | | ONNX | Inter‑framework portability | Full support | | TensorFlow | Legacy models | Partial (experimental) | | HuggingFace Transformers | Unified API | Full support |

Because gBench relies on the HuggingFace transformers library under the hood, most models are immediately compatible.

5. Benchmarking Scenarios

gBench is not limited to accuracy; it shines in serving scenarios that mirror real deployments.

5.1 Serving Performance

| Metric | Definition | Why It Matters | |--------|------------|----------------| | Latency (ms) | Time from request to first token | Determines user perceived speed | | Throughput (tokens/s) | Total tokens produced per second | Maximal API utilisation | | Latency Percentiles | P50, P90, P99 | Captures tail behaviour for SLA | | Concurrency | Simultaneous requests | Stress‑tests queue management |

gBench exposes a simple CLI:

gbench run \
  --model llama-2-13b \
  --dataset sft \
  --batch-size 8 \
  --concurrency 32 \
  --output results/llama2-13b.yaml

The resulting report includes P90 latency of 110 ms and throughput of 240 tokens/s on an A100.

5.2 Hardware Profiling

Using NVIDIA’s NVML or AMD’s ROCm, gBench reports:

• GPU Memory Peak – useful for determining batch‑size limits.
• GPU Utilisation (%) – indicates whether the GPU is the bottleneck.
• CPU Usage (%) – shows whether the host CPU is a constraint.
• Power Draw (W) – helps calculate energy cost per inference.

For edge devices (e.g., Jetson Nano), gBench records thermal throttling events automatically.

5.3 Quantisation & Optimisation

Quantisation dramatically reduces model size and inference time. gBench can benchmark:

1. Post‑Training Quantisation (PTQ) – INT8/INT4 weights and activations.
2. Dynamic Quantisation – dynamic activation scaling at runtime.
3. Sparse Attention – pruning attention heads.

Results often look like:

| Optimisation | Latency (ms) | Throughput (tokens/s) | Accuracy (BLEU) | |--------------|--------------|-----------------------|-----------------| | FP16 (baseline) | 120 | 200 | 33.2 | | INT8 PTQ | 85 | 300 | 32.9 | | INT4 PTQ | 65 | 400 | 32.0 |

Thus, a 30 % latency reduction comes at a 2 % BLEU drop—a typical trade‑off for high‑traffic services.

5.4 Concurrency & Load Testing

Real services rarely handle a single request at a time. gBench can emulate burst traffic:

gbench run \
  --model stable-diffusion-v1-5 \
  --dataset text2image \
  --concurrency 16 \
  --burst 200

The resulting dashboard shows P99 latency climbing from 450 ms (steady state) to 800 ms under burst, giving developers a realistic view of SLA risk.

6. Implementation Details

Below is an inside look at gBench’s codebase, with a focus on the most critical modules.

6.1 Data Loading & Preprocessing

from datasets import load_dataset
from transformers import AutoTokenizer, AutoFeatureExtractor

def load_text_task(dataset_name, split='validation'):
    ds = load_dataset(dataset_name, split=split)
    tokenizer = AutoTokenizer.from_pretrained("gpt-2")
    return ds.map(lambda x: tokenizer(x["text"], truncation=True, padding="max_length", max_length=512))

Key points:

• Uses HuggingFace Datasets for efficient streaming.
• Supports dynamic batching: collects a set number of examples until the target token budget is reached.

6.2 Inference Engine

class BaseAdapter:
    def __init__(self, model_name):
        self.model = AutoModelForCausalLM.from_pretrained(model_name)
        self.model.eval()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.model.to(self.device)

    def forward(self, input_ids, attention_mask):
        with torch.no_grad():
            return self.model.generate(input_ids, attention_mask=attention_mask, max_new_tokens=128)

• The adapter wraps generate() which supports streaming and sampling options.
• For VLMs, an extra vision_encoder is loaded and a cross‑modal attention module is called.

6.3 Multi‑Threading & Async

gBench uses Python’s asyncio to dispatch multiple requests concurrently. For each worker:

async def serve(request_queue, results_queue):
    while True:
        request = await request_queue.get()
        start = time.perf_counter()
        output = await adapter.forward(request.input_ids, request.attention_mask)
        latency = (time.perf_counter() - start) * 1000
        await results_queue.put(Result(output, latency))

This design eliminates the Global Interpreter Lock (GIL) bottleneck by delegating heavy tensor work to the GPU.

6.4 Profiling Tools

• NVML (NVIDIA Management Library) for GPU stats.
• psutil for CPU usage and memory.
• pyinstrument for fine‑grained Python trace‑level profiling.

The profiler runs in a background thread, writing logs to a CSV that is later merged with the main results.

7. Real‑World Use Cases

7.1 Research Labs

• Scenario: A group at MIT wants to evaluate a new LoRA‑fine‑tuned version of Llama‑2.
• Action:

1. Install gBench via pip install gbench.
2. Load the LoRA adapters and run gbench run --model llama-2-13b-lora.
3. Examine accuracy vs. latency trade‑offs in the PDF report.

7.2 Cloud Providers

• Scenario: AWS wants to benchmark the cost‑effectiveness of its GPU instances for LLM inference.
• Action:

1. Spin up an EC2 g5.12xlarge (4×A10G).
2. Run gbench run --model mistral-7b --concurrency 64 on each node.
3. Compute tokens per watt to compare against other instance types.

7.3 Hardware Vendors

• Scenario: NVIDIA seeks to showcase the new Grace Hopper chip’s inference performance.
• Action:

1. Run the VLM benchmark on the chip with gBench’s torchscript adapter.
2. Collect latency tail metrics for marketing.

8. gBench vs. Existing Benchmarks

| Benchmark | Focus | Strength | Limitation | |-----------|-------|----------|------------| | HuggingFace Hub | Accuracy (perplexity, BLEU) | Huge model zoo | No latency/throughput | | MLPerf Inference | End‑to‑end inference | Industry‑standard | Complex setup, limited to specific workloads | | CBench | LLM serving | Simple, Python‑based | Single‑threaded, no hardware profiling | | gBench | Accuracy + Serving + Hardware | Open‑source, modular, real‑time profiling | Still growing ecosystem, may need adaptation for niche models |

gBench’s unique selling proposition is its dual focus: it gives you both the quality of a model and the cost of deploying it. That makes it the perfect tool for both academic and production teams.

9. Community & Ecosystem

9.1 GitHub Repository

gBench lives at https://github.com/gbench-org/gbench. Key sections:

• docs/ – full documentation and tutorials.
• examples/ – ready‑to‑run scripts for each supported model.
• plugins/ – directories for community‑submitted adapters and datasets.

9.2 Contribution Guidelines

• Pull Requests: Must include unit tests, update docs, and a benchmark run.
• Issue Template: For reporting bugs or proposing new features.
• Release Cycle: Semantic versioning with quarterly releases.

9.3 Documentation & Tutorials

The docs are split into:

1. Getting Started – installing, first run.
2. Core Concepts – adapters, datasets, metrics.
3. Advanced Use Cases – multi‑GPU, custom optimisations.
4. Developer Guide – extending the plugin system.

9.4 Ecosystem Partnerships

• HuggingFace – gBench integrates seamlessly with the 🤗 ecosystem.
• NVIDIA – Provided hardware profiling libraries.
• Intel AI – Contributed CPU‑specific adapters for openvino inference.

10. Future Directions

gBench is already powerful, but its roadmap points to exciting new capabilities.

10.1 Multimodal & Cross‑Domain Benchmarks

• Extending beyond LLM/VLM to audio‑to‑text, video‑caption, and AR/VR inference.
• Supporting joint datasets like LAION or OpenAI’s multimodal corpora.

10.2 Edge & Tiny Models

• Optimised adapters for EdgeTPU, Apple Neural Engine, and RISC‑V inference.
• Benchmarking TinyBERT and DistilBERT on ARM Cortex‑A55.

10.3 Real‑Time Streaming Benchmarks

• Measuring latency for streaming output (e.g., long‑form chat).
• Evaluating token‑level throughput vs. turn‑level latency.

10.4 Fairness & Bias Metrics

• Incorporating dataset bias checks (e.g., gender, race).
• Adding counterfactual evaluation suites to assess model fairness.

10.5 Cloud‑Native Integration

• Docker images and Kubernetes operators for auto‑scaling benchmarks.
• Integration with Prometheus and Grafana for live monitoring.

11. Final Takeaway

gBench is more than a benchmark – it is a benchmarking ecosystem that bridges the gap between research and production. By measuring both accuracy and real‑world serving performance, it empowers teams to:

• Make data‑driven decisions about model choice and optimisation.
• Quantify cost per token or energy per inference, aiding sustainability.
• Provide transparent, reproducible results that stakeholders can trust.

Whether you are a researcher pushing the boundaries of LLMs, a cloud provider setting pricing tiers, or a hardware vendor showcasing a new chip, gBench offers a unified, open‑source foundation to evaluate the next generation of AI models. Its modular design and active community ensure that as the AI landscape evolves, gBench will keep pace—delivering reliable, actionable insights for all who rely on large language and vision‑language models.