tinyeval for R

Find out when a prompt edit or a model swap makes the AI replies of your R code worse. Your test cases sit in a short bash file, and your own R functions answer them.

Install

From R-universe:

install.packages("tinyeval", repos = c("https://evalship-hq.r-universe.dev", getOption("repos")))

In an renv project, run renv::snapshot() after, so renv.lock records it.

Example

evals/support.sh holds the test cases. Its worker line starts R once per worker, not once per test case:

worker Rscript evals/worker.R

test_case "refund after the window" "Can I still return my shoes? They arrived 40 days ago."
  matches "30[- ]day"
  judge "Says a refund isn't possible anymore, politely"

evals/worker.R loads your code and hands it each input:

library(tinyeval) # before your code, so its printing stays out of the replies
source("R/support.R")

worker(function(input) {
  if (Sys.getenv("ANTHROPIC_API_KEY") == "") couldnt_run("ANTHROPIC_API_KEY isn't set")
  answer(input)
})

Run it

From the project root:

Rscript -e 'quit(status = tinyeval::run())'            # every file in evals/
Rscript -e 'quit(status = tinyeval::run("-n", "3"))'   # each test case 3 times; it passes when 2 do

tinyeval::run() returns tinyeval's exit code, and quit(status = ...) hands it to CI, so a failing eval fails the job. Ctrl-C stops the run.

tinyeval needs bash 3.2 or later: macOS, Linux, or Git for Windows, whose bash run() finds on its own.

Errors

Output

In a worker tinyeval starts, library(tinyeval) points R's output at stderr with sink(), so print() and cat() can't end up in a reply. The replies go to the process's stdout. Base R can't move a file descriptor, so programs started with system(), and C code that writes to stdout directly, still reach the replies.

R reads .Rprofile before your worker file, so what it prints lands in the first reply. tinyeval::run() turns off renv's out-of-sync note for its workers.

Updating

install.packages("tinyeval", repos = "https://evalship-hq.r-universe.dev")

tinyeval::run("update") prints that line, and the run's summary says when a new release is out. Dependabot and Renovate don't read renv.lock yet.

Links