DEV Community: kohei matsunobu

Before You Burn GPU Hours, Smoke-Test One Task on a Cheap CPU Runtime

kohei matsunobu — Sat, 31 Oct 2026 07:34:56 +0000

This article was originally published in Japanese on Zenn.

Introduction

Hi, I'm Kohei Matsunobe, a second-year student in the online (correspondence) program at ZEN University. I'm working toward becoming an AI engineer, and on my own I run experiment pipelines that evaluate LLM outputs and code on notebooks and cloud runtimes.

GPU runtimes are great, but they're expensive per hour, and there's a cap on how many compute units you can use. I've made the same mistakes more than once: "I grabbed a GPU, launched the real job, and it crashed in the first few minutes," or "it crashed, I didn't notice, and the runtime just sat there running."

Looking back, almost all of those failures could have been caught without a GPU. In this post, I'll walk through the idea of a "smoke test" — a minimal check you run on a cheap CPU runtime before moving up to the expensive one — plus a small script you can use right away.

What tends to go wrong

In pipelines that evaluate LLMs or generated code, these problems come up a lot:

Missing test dependencies: Your setup grades generated code with pytest, but the production runtime doesn't have pytest, or the libraries the tested code needs.
Cache or data not found: The path where you thought you'd put model weights or dataset caches is empty on the fresh runtime. So downloads run every time, or you get errors because offline mode is still on.
Mismatched environment variables or paths: It worked locally, but on a different runtime the working directory or cache location has changed.
An exception on the very first task: Simple mistakes, like a missing output directory or a misspelled argument name.

None of these have anything to do with GPU performance. But when they happen on a GPU runtime, you lose all the time spent on setup and model loading too. Retry the job enough times and you might suddenly find you've used up your compute units.

The idea behind the smoke test

It boils down to three points:

Keep dependencies identical to production: Build the environment from the same definition as production — the same requirements file, lock file, or container image. If you maintain a separate dependency list for CPU, you hide the very differences you want to catch. (The only exception: packages that need a GPU-specific build get handled separately.)
Run just one task on the cheap runtime: Don't run the whole eval set — just one item. Inference may be slow on CPU, but all you need to know is "does it make it to the end?"
Promote to the expensive runtime only if it passes: Only when the smoke test succeeds do you run the real job on the GPU runtime, with the same code and the same config.

If you think of it as a CI pipeline, it's a staged gate: put a light job in front of the heavy one, and don't move on unless the light one passes.

The minimum checklist

Here's what I check every time:

Can the key packages be imported? (Including pytest for evaluation)
Do the cache and data directories exist, and are they non-empty?
Are the required environment variables set?
Can I write to the output directory?
Does at least one pytest test actually pass?
Can the same entry point as production run to completion as a dry run with just one task?

You don't need to do all of these perfectly. Just adding "anything that has bitten me even once" to the list goes a long way.

A small script

Here's a script that bundles the checks above. Adjust REQUIRED_MODULES, REQUIRED_PATHS, and run_task to fit your own pipeline.

#!/usr/bin/env python3
# smoke_test.py
"""Smoke test to run on a cheap runtime before moving to an expensive one."""
from __future__ import annotations

import argparse
import importlib
import json
import os
import subprocess
import sys
from pathlib import Path

# Modules to verify with the same dependency stack as production
REQUIRED_MODULES = ["numpy", "pytest", "transformers"]
# Cache/data paths that should exist and have contents
REQUIRED_PATHS = [
    Path(os.environ.get("HF_HOME", "~/.cache/huggingface")).expanduser(),
    Path("data"),
]
REQUIRED_ENV = ["HF_HOME"]


def check_imports(modules: list[str]) -> list[str]:
    errors = []
    for name in modules:
        try:
            importlib.import_module(name)
        except Exception as e:  # also catch init errors, not just ImportError
            errors.append(f"import failed: {name} ({type(e).__name__}: {e})")
    return errors


def check_paths(paths: list[Path]) -> list[str]:
    errors = []
    for p in paths:
        if not p.exists():
            errors.append(f"path missing: {p}")
        elif p.is_dir() and not any(p.iterdir()):
            errors.append(f"directory is empty: {p}")
    return errors


def check_env(names: list[str]) -> list[str]:
    return [f"env var not set: {n}" for n in names if not os.environ.get(n)]


def check_writable(out_dir: Path) -> list[str]:
    try:
        out_dir.mkdir(parents=True, exist_ok=True)
        probe = out_dir / ".smoke_write_test"
        probe.write_text("ok", encoding="utf-8")
        probe.unlink()
    except OSError as e:
        return [f"not writable: {out_dir} ({e})"]
    return []


def run_one_pytest(test_id: str) -> list[str]:
    # e.g. tests/test_eval.py::test_single_case
    result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", test_id])
    # Exit code 5 means "no tests were collected", so treat it as a failure too
    if result.returncode != 0:
        return [f"pytest failed: {test_id} (exit={result.returncode})"]
    return []


def load_tasks(path: Path) -> list[dict]:
    with path.open(encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]


def run_task(task: dict) -> dict:
    # Replace this with the same processing as production (inference, grading)
    return {"id": task.get("id"), "status": "ok"}


def main() -> int:
    parser = argparse.ArgumentParser(description="cheap runtime smoke test")
    parser.add_argument("--tasks", type=Path, help="JSONL file, one task per line")
    parser.add_argument("--out", type=Path, default=Path("outputs"))
    parser.add_argument("--pytest", metavar="TEST_ID", help="run a single test ID")
    parser.add_argument("--dry-run", action="store_true", help="run only one task")
    args = parser.parse_args()

    errors: list[str] = []
    errors += check_imports(REQUIRED_MODULES)
    errors += check_paths(REQUIRED_PATHS)
    errors += check_env(REQUIRED_ENV)
    errors += check_writable(args.out)
    if args.pytest:
        errors += run_one_pytest(args.pytest)

    if errors:
        print("[NG] Pre-flight checks found problems:")
        for e in errors:
            print("  -", e)
        return 1

    if args.tasks:
        tasks = load_tasks(args.tasks)
        if args.dry_run:
            tasks = tasks[:1]
        for task in tasks:
            result = run_task(task)
            print(json.dumps(result, ensure_ascii=False))

    print("[OK] Smoke test passed. Safe to promote to the expensive runtime.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

On the cheap CPU runtime, run it like this:

python smoke_test.py --tasks data/tasks.jsonl --dry-run \
  --pytest tests/test_eval.py::test_single_case

If you only start the real run when the exit code is 0, the same pattern works in notebooks and in CI:

python smoke_test.py --tasks data/tasks.jsonl --dry-run && python run_full_eval.py

Wrap-up

Most expensive GPU job failures come from things that have nothing to do with the GPU: dependencies, caches, and path settings.
Keep the same dependency stack as production, push just one task through on a cheap CPU runtime, and promote to the expensive runtime only when it passes.
For the checklist, adding things you've personally been bitten by, little by little, is plenty.

A few minutes of smoke testing protects both your compute units and your motivation. I hope this helps anyone else running experiments on their own!

Summarize Kaggle Notebook Status and Logs in ~7 Lines with the Kaggle CLI (Goodbye, .bat + Notepad)

kohei matsunobu — Sat, 31 Oct 2026 07:34:55 +0000

This article was originally published in Japanese on Qiita.

Introduction

Hi, I'm Kohei Matsunobe. I'm a second-year student in the online (correspondence) program at ZEN University, studying to become an AI engineer.

Once I started running Kaggle notebooks (kernels) from my local machine with kaggle kernels push, one small thing kept eating my time: checking what was going on. Is it still running? Did it finish? Did it fail? And if it failed, where? Checking that in the browser or digging through log files every single time got old fast.

In this post, I'll share a script that calls kaggle kernels status and kaggle kernels output from Python and boils the status and log down to a ~7-line summary plus one CSV row. I'll also cover how I fixed garbled text (mojibake) on Windows, and how I made the same script usable both from a .bat file and from an AI agent.

Note: this post is not about submissions. It focuses purely on monitoring run status and summarizing logs.

How I used to do it

At first, I got by with a batch file like this:

@echo off
kaggle kernels status koheip/my-notebook
kaggle kernels output koheip/my-notebook -p .\out -o
notepad .\out\my-notebook.log

It worked, but it had a few problems:

The logs are long, and scrolling through Notepad to find the error is painful
Non-ASCII output (Japanese, in my case) got garbled in the console
There was no record of "when, which notebook, and what status"
When I asked an AI agent to check the log, it had to read the entire thing, which is wasteful

That last point was the big one. If you're handing something to an agent, a short text with only the information you need is much easier to work with.

What the CLI gives you

You only need two commands (official docs: Kaggle CLI documentation, kaggle.com/docs/api).

`kaggle kernels status`

Shows the status of the latest run. You pass the kernel as owner/kernel-slug.

kaggle kernels status koheip/my-notebook

The output is a single line (on failure, an extra Failure message: "..." line is added):

koheip/my-notebook has status "KernelWorkerStatus.COMPLETE"

Depending on the CLI version, you may see lowercase complete or the KernelWorkerStatus. prefix, so the script takes only the last word and lowercases it.

`kaggle kernels output`

Downloads the output files from the latest run. The main options:

Option	Meaning
`-p, --path`	Destination folder
`-o, --force`	Overwrite existing files
`-q, --quiet`	Suppress progress output
`--file-pattern`	Only download files matching a regex

The key point: along with the output files, it also saves the execution log as <kernel-slug>.log. In most cases, it's a JSON array of entries with stream_name (stdout / stderr), time, and data. That's what we'll read and summarize.

If your notebook produces large output files, narrowing things down with --file-pattern speeds it up.

Authentication

The Kaggle CLI can authenticate via the KAGGLE_API_TOKEN environment variable, ~/.kaggle/access_token, or the classic ~/.kaggle/kaggle.json, among others. I went with reading the token from a dedicated file and passing it as an environment variable only to the child process. That way, whether the script is called from a .bat file or from an agent, the token is loaded the same way.

The summary script

Save this as kaggle_status_summary.py. It only uses the standard library.

# kaggle_status_summary.py
import argparse
import csv
import json
import os
import re
import subprocess
import sys
from datetime import datetime
from pathlib import Path

# Force stdout to UTF-8, even on Windows (Python 3.7+)
if hasattr(sys.stdout, "reconfigure"):
    sys.stdout.reconfigure(encoding="utf-8", errors="replace")

DEFAULT_TOKEN_FILE = Path.home() / ".config" / "kaggle_token.txt"


def build_env(token_file: Path) -> dict:
    """Read the token from a file and build env vars for the child process."""
    env = os.environ.copy()
    env["PYTHONIOENCODING"] = "utf-8"
    env["PYTHONUTF8"] = "1"
    if token_file.exists():
        env["KAGGLE_API_TOKEN"] = token_file.read_text(encoding="utf-8").strip()
    return env


def run(cmd: list[str], env: dict) -> str:
    r = subprocess.run(
        cmd, env=env, capture_output=True,
        text=True, encoding="utf-8", errors="replace",
    )
    if r.returncode != 0:
        raise RuntimeError(f"{' '.join(cmd)} failed: {r.stderr.strip()[:300]}")
    return r.stdout


def get_status(kernel: str, env: dict) -> tuple[str, str]:
    out = run(["kaggle", "kernels", "status", kernel], env)
    m = re.search(r'has status "([^"]+)"', out)
    status = m.group(1).split(".")[-1].lower() if m else "unknown"
    f = re.search(r'Failure message: "(.*)"', out)
    return status, (f.group(1) if f else "")


def read_log(log_path: Path) -> list[tuple[str, str]]:
    """Return a list of (stream_name, text). Fall back to plain text if it isn't JSON."""
    raw = log_path.read_text(encoding="utf-8", errors="replace")
    try:
        items = json.loads(raw)
        return [(i.get("stream_name", ""), str(i.get("data", ""))) for i in items]
    except (json.JSONDecodeError, AttributeError, TypeError):
        return [("", line) for line in raw.splitlines()]


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("kernel", help="owner/kernel-slug")
    ap.add_argument("--out", default="kaggle_out")
    ap.add_argument("--csv", default="kaggle_status_log.csv")
    ap.add_argument("--token-file", type=Path, default=DEFAULT_TOKEN_FILE)
    args = ap.parse_args()

    env = build_env(args.token_file)
    status, failure = get_status(args.kernel, env)

    slug = args.kernel.split("/")[1]
    out_dir = Path(args.out) / slug
    entries, files = [], []
    if status in ("complete", "error", "cancelacknowledged"):
        out_dir.mkdir(parents=True, exist_ok=True)
        run(["kaggle", "kernels", "output", args.kernel,
             "-p", str(out_dir), "-o", "-q"], env)
        files = [p.name for p in out_dir.iterdir() if p.is_file()]
        log_path = out_dir / f"{slug}.log"
        if log_path.exists():
            entries = read_log(log_path)

    err_pat = re.compile(r"(Error|Exception|Traceback)")
    stderr_n = sum(1 for s, _ in entries if s == "stderr")
    err_lines = [t.strip() for _, t in entries if err_pat.search(t)]
    last_line = next((t.strip() for _, t in reversed(entries) if t.strip()), "")
    now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    # The "~7 line" summary
    print(f"[kernel]  {args.kernel}")
    print(f"[checked] {now}")
    print(f"[status]  {status}" + (f" / {failure}" if failure else ""))
    print(f"[log]     {len(entries)} entries (stderr: {stderr_n})")
    print(f"[errors]  {len(err_lines)} lines")
    print(f"[last_err] {err_lines[-1][:120] if err_lines else '-'}")
    print(f"[tail]    {last_line[:120] or '-'}  / files: {len(files)}")

    # Append one row to a CSV history (with BOM so Excel opens it cleanly)
    csv_path = Path(args.csv)
    new = not csv_path.exists()
    with csv_path.open("a", newline="", encoding="utf-8-sig") as f:
        w = csv.writer(f)
        if new:
            w.writerow(["checked_at", "kernel", "status", "log_entries",
                        "stderr", "error_lines", "last_error", "files"])
        w.writerow([now, args.kernel, status, len(entries), stderr_n,
                    len(err_lines), err_lines[-1][:200] if err_lines else "",
                    len(files)])
    return 0 if status != "error" else 1


if __name__ == "__main__":
    sys.exit(main())

Running it looks like this (values are just an example):

[kernel]  koheip/my-notebook
[checked] 2026-10-31 15:42:10
[status]  complete
[log]     214 entries (stderr: 12)
[errors]  0 lines
[last_err] -
[tail]    Done.  / files: 3

Three things I was deliberate about:

Exit code: it returns 1 when the status is error, so it's easy to branch on with if errorlevel 1 in a .bat file or on the agent side
Status normalization: to absorb differences between CLI versions, KernelWorkerStatus.COMPLETE becomes complete
No download while running: for running / queued, it only checks the status and exits, so it stays lightweight even if you call it often

That said, the Kaggle API has rate limits. If you monitor in a loop, leave a gap of a few minutes between checks.

Setting up the token file

Generate a token on your Kaggle settings page and save it as a one-line text file.

# PowerShell: create the token file (it should contain only the token, on one line)
New-Item -ItemType Directory -Force "$HOME\.config" | Out-Null
Set-Content -Path "$HOME\.config\kaggle_token.txt" -Value "YOUR_TOKEN_HERE" -Encoding ascii -NoNewline

I'm strict about two rules: never put this file under Git, and never paste it into an article or a chat. When an agent uses the script, all it needs is the script's path; it never has to see the token itself.

Fixing garbled text on Windows

On Windows, the console code page often isn't UTF-8 by default (on Japanese Windows it's CP932 / Shift_JIS). Kaggle logs, on the other hand, are UTF-8. If you do nothing, non-ASCII characters get garbled, or the script dies with a UnicodeEncodeError.

The fix is to switch to UTF-8 on both the console side and the Python side.

Calling from a .bat file

@echo off
chcp 65001 >nul
set PYTHONIOENCODING=utf-8
set PYTHONUTF8=1
python "%~dp0kaggle_status_summary.py" koheip/my-notebook
if errorlevel 1 echo [!] The run ended with an error

If your .bat file itself contains non-ASCII text, save it as UTF-8 so it displays correctly after chcp 65001.

Calling from PowerShell

chcp 65001 > $null
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
$OutputEncoding = [System.Text.Encoding]::UTF8
$env:PYTHONIOENCODING = "utf-8"
$env:PYTHONUTF8 = "1"

python .\kaggle_status_summary.py koheip/my-notebook
if ($LASTEXITCODE -ne 0) { Write-Host "[!] The run ended with an error" }

What the Python side does

Inside the script, I also do three things just to be safe:

sys.stdout.reconfigure(encoding="utf-8") to make stdout UTF-8
subprocess.run(..., encoding="utf-8", errors="replace") to read the child process output as UTF-8
Pass PYTHONIOENCODING=utf-8 to the child process (the Kaggle CLI) as well

There are cases where chcp 65001 alone, or PYTHONIOENCODING alone, only fixes half of the problem. That tripped me up at first. Set both and you're good.

Wrap-up

kaggle kernels status gives you the status; kaggle kernels output gives you the output files and <slug>.log
Reading the log in Python and condensing it into a ~7-line summary plus one CSV row makes checking much easier
Loading the token from a file and passing it to the child process means .bat files and agents use the same path
On Windows, use both chcp 65001 and PYTHONIOENCODING=utf-8 to avoid garbled text

Compared to the days of scrolling through logs in Notepad, I now spend far less time checking runs, and I hand far less text to my agents. I hope this helps anyone else running Kaggle notebooks from their local machine!

Eliminating Accidental Submissions in Kaggle Work: The Kaggle CLI, a Confirmation Wrapper, and Measuring on Production Hardware

kohei matsunobu — Sat, 17 Oct 2026 10:56:10 +0000

This article was originally published in Japanese on Zenn.

Introduction

I've been working on Kaggle with help from LLM agents and scripts. Automation makes things easier, but it also nearly caused accidents, like the workflow going all the way to submission without anyone checking.

In this article, I'll cover how to work with notebooks using the Kaggle CLI, a wrapper that always puts a human confirmation step in the loop, and what I learned from the differences between my validation environment and the production environment.

1. Checking notebook status with the Kaggle CLI

Setting up the token

With the newer token method, you put your token in the KAGGLE_API_TOKEN environment variable.

export KAGGLE_API_TOKEN="(your token issued by Kaggle)"

Getting notebook status and output

You can check the status of a notebook running on Kaggle and fetch its output from the CLI.

# Check whether it's running, finished, or failed
kaggle kernels status <username>/<notebook-name>

# Download the output files locally
kaggle kernels output <username>/<notebook-name>

Since you can see the status without opening a browser, it's much easier to handle from scripts and agents.

2. A confirmation wrapper to prevent accidental submissions

I once had an automated batch go all the way to submission without any confirmation. Submissions are often limited in number, and they can't be undone.

So I built a wrapper that splits the path to submission into these steps:

Show which file will be submitted
Confirm with y/N (pressing just Enter means "No")
Run it
Validate the output (whether the file exists, whether the format is correct, etc.)
Only display the submit command and stop there

For the final submission, a human runs the displayed command themselves. This way, even if something breaks somewhere in the automation, a human always takes a look right before submission.

The confirmation part can be written in Python in just a few lines, for example:

def confirm(message: str) -> bool:
    answer = input(f"{message} [y/N]: ").strip().lower()
    return answer == "y"

if not confirm("Run with this file?"):
    raise SystemExit("Aborted")

The key is making "No" the default. Even when it's called from an agent or script with no input, it won't proceed on its own.

3. Measure time-limited workloads on the same hardware as production

When I was doing local validation on a fast GPU, the trends in the results sometimes differed from those on the slower GPU used in production.

When a workload has a time limit, slower hardware alone can cause it to be cut off partway, or change what you can get done within the limit. Just because something "fits in time" on a fast environment doesn't mean it will in production.

I learned that for time-limited workloads, it's important to measure on the same hardware as production.

Summary

Once you set KAGGLE_API_TOKEN, the Kaggle CLI lets you track your notebooks with kernels status and kernels output, no browser needed
Even with automation, always put a human confirmation step right before submission (default to "No", and only display the submit command)
Measure time-limited workloads on the same hardware as production

The more I automate, the more I feel it's important to decide up front "where a human steps in to stop things."

I Stopped Feeding Whole Jupyter Notebooks to LLMs and Cut Tokens by Over 95% (Claude Code / Codex CLI / PowerShell)

kohei matsunobu — Sat, 17 Oct 2026 10:56:09 +0000

This article was originally published in Japanese on Qiita.

Introduction

When I work on Kaggle, I write code in Jupyter notebooks and ask Claude Code or Codex CLI for reviews and advice along the way. The problem: passing a whole notebook to an LLM came to about 30k tokens per notebook, and after asking a few times, the cost added up fast.

In this article, I'll share the tricks I used to save tokens, plus the gotchas I hit when calling these CLIs from Windows PowerShell.

1. Pass only the "code cell diff" of the notebook

An .ipynb file is JSON under the hood, and it's packed with cell outputs (logs and images) and metadata. In most cases, the code is all I actually want the LLM to look at, so I wrote a script that does the following:

Drops outputs and metadata, and extracts only the code cells
Outputs only the diff against the previous version
Replaces long lines over 300 characters with their "length" and a "hash" (all I need to know is whether the content changed)

With this, comparing versions fits in about 1–1.6k tokens (a reduction of over 95%).

Also, long prompt strings inside a notebook make it hard to read, so I create a "reading view" where those strings are split out into separate .txt files. The notebook I actually run stays as a single file, and I pass the reading view to the LLM or for reviews. That's how I keep the two roles separate.

2. Calling Claude Code / Codex CLI as commands from another program

Claude Code

With claude -p, you can ask a single question and get an answer back without opening the interactive UI.

claude -p --max-turns 1 --disallowedTools "Bash,Read,..."

The key is to disallow tools like file reading and pass only the summary you've prepared yourself. Since it won't go off and read files in your repository on its own, token usage stays cheap.

Codex CLI

I installed it with npm.

npm i -g @openai/codex

When logging in through the browser failed, authenticating with a URL and a code worked.

codex login --device-auth

I pass the question via standard input.

echo "Your question here" | codex exec --skip-git-repo-check -

3. Japanese text gets garbled in Windows PowerShell

When I piped a Japanese prompt from PowerShell into the CLI, the text came out garbled (mojibake). Setting the following three to UTF-8 fixes it.

[Console]::InputEncoding  = [System.Text.Encoding]::UTF8
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
$OutputEncoding           = [System.Text.Encoding]::UTF8

Also, PowerShell doesn't let you chain commands with & the way cmd does; you get an AmpersandNotAllowed error. If you write commands with a cmd mindset, this will trip you up, so watch out.

Summary

Instead of passing the whole notebook, passing only the code cell diff cut tokens by over 95%
With claude -p, disallowing tools and passing only a summary keeps it cheap
If Codex CLI login fails, try codex login --device-auth
When passing Japanese text in PowerShell, set the three encodings to UTF-8

I hope this helps anyone else who's going back and forth between LLMs and notebooks like I am.