mbpp

Benchmark suite for evaluating LLMs and SLMs on coding and SE tasks. Features HumanEval, MBPP, SWE-bench, and BigCodeBench with an interactive Streamlit UI. Supports cloud APIs (OpenAI, Anthropic, Google) and local models via Ollama. Tracks pass rates, latency, token usage, and costs.

LLM Collaboration for Code Generation

A lightweight, configuration-driven evaluation framework for LLM code generation & reasoning tasks (MBPP, HumanEval, GSM8K). Supports multi-provider (DeepSeek, OpenAI, ZhipuAI) and concurrent execution.

Lightweight LLM evaluation tool for OpenAI-compatible endpoints (Chat Completions / Responses) with CLI and local Web UI. MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag — bring your own BASE_URL + MODEL_NAME.

Reference implementation of Double Ratchet: co-evolving an inspectable evaluation metric with a lifecycle-managed skill library for self-improving LLM agents (arXiv:2607.12790)

Autonomous code agent built entirely from scratch (agent frameworks prohibited): a Thought → Code → Observation loop where the LLM writes and executes real Python to solve programming tasks, using code-based tool calling instead of JSON.

Reference implementation of Ratchet: a minimal hygiene recipe for self-evolving LLM agents (arXiv:2605.22148)

LLM code-agent framework — Thought→Code→Observation loop, process-isolated sandbox, MBPP + SWE-bench tracks, MCP tools, multi-model OpenRouter benchmarks. 42 School project.

🧪 Automated LLM coding benchmarks with Ollama - HumanEval & MBPP evaluation suite with safe execution, comprehensive logging, and detailed analysis tools

Dissertation: a multi-agent (Programmer→Tester) pipeline benchmarking self-hosted small language models against Claude for automated unit-test generation on HumanEval and MBPP, with coverage and mutation metrics.

Fine-tuning CodeT5 for Python code generation on the MBPP dataset. Features custom TensorFlow training loops, mixed precision, XLA optimization, and distributed multi-GPU strategies.

Controlled comparison of boosted SFT vs. GRPO at matched compute. i.e., which training loop escapes the problems a base model never solves. Reproduces Amin et al. (NeurIPS 2025) as the baseline. In progress.

Diagnostic toolkit for self-hosted LLM inference: failure-taxonomic stress harness + 4-phase orchestrator + parametric vLLM launchers

Study #4: FP8+MTP{3,5} speed on repne/vllm:v2 + max_tokens=8192 quality re-runs for BF16+DFlash n=8 and FP8+MTP=3. Follow-up to studies #2 and #3.

Stress-validation of Qwen3.6-27B inference configurations on dual RTX PRO 6000 Blackwell. 5 configs x 4 phases (gates, throughput matrix, HumanEval, MBPP) = 2,105 hard coding problems, zero crashes. Headline: FP8+MTP=3 wins HumanEval (79.3%), BF16+DFlash wins MBPP (89.5%). MTP=5 dominated on correctness despite faster raw tok/s.