History investigation

$ python3 -B -m scorekit.runner data/smoke.jsonl
rows=10 exact_match=0.400 token_f1=0.444 rouge_l=0.489
[exit status: 1]
$ git describe
v0.2.0-5-ge376e5b

$ git log --graph --oneline --decorate --all
* e376e5b (HEAD -> main) Mention the nightly run in the README
* 2652768 Add a separate pass mark for the nightly set
| * a10f9a1 (feat/report) Write a report when --report <path> is given
| * f37a7d8 Mention the nightly run in the README
| * a8e8550 Add per-row report writer
|/  
* ca7e2b7 Fail fast on an empty reference
* d9d075d Remove the experimental BLEU scorer
* ec4fad7 Simplify token overlap in token_f1
* c0c9a30 (tag: v0.2.0) Add nightly evaluation set
* 7823232 Add ROUGE-L metric
| * 2f2883f (feat/rouge-l) Report ROUGE-L in the runner
| * 80ee55f Add ROUGE-L F-measure
| * 1cbe38a Add longest-common-subsequence helper
|/  
* 9c8df98 Reformat sources: four-space indent, double quotes
* d926d3c Speed up normalize with a precompiled pattern
* be9ad1b Fix crash when --limit is not given
* 074d492 Send warnings to stderr
* 8dc82cb Add --limit option to the runner
* 72134f3 Document the runner's exit status
*   8657273 Merge branch 'feat/text-utils'
|\  
| * c0d33a5 Add a tokens helper and use it in the metrics
| * dd70d9e Move normalize into scorekit/text.py
* | 389337a Raise the pass mark to 0.75
* | 4da82b5 Document the metrics
* | b4b066b Skip rows with an empty reference
|/  
* 8e2cac1 Strip punctuation before comparing
* 682bc71 (tag: v0.1.0) Add config module with the pass mark
* 18bb23e Rename the scorer module to metrics
* d38a5aa Move sources into the scorekit package
* d369eac Add experimental BLEU scorer
* 633e3e6 Add evaluation runner and smoke set
* bb5eb57 Add token-level F1 scorer
* b250238 Add exact-match scorer
* 5015359 Add project skeleton

$ git log --graph --oneline --decorate -6 main feat/report
* e376e5b (HEAD -> main) Mention the nightly run in the README
* 2652768 Add a separate pass mark for the nightly set
| * a10f9a1 (feat/report) Write a report when --report <path> is given
| * f37a7d8 Mention the nightly run in the README
| * a8e8550 Add per-row report writer
|/  
* ca7e2b7 Fail fast on an empty reference
$ git merge-base main feat/report
ca7e2b7fcc44f8d5a494107b68440c3d4ec23b55

$ git diff --stat main feat/report
 scorekit/config.py | 1 -
 scorekit/report.py | 7 +++++++
 scorekit/runner.py | 3 +++
 3 files changed, 10 insertions(+), 1 deletion(-)
$ git diff --stat main..feat/report
 scorekit/config.py | 1 -
 scorekit/report.py | 7 +++++++
 scorekit/runner.py | 3 +++
 3 files changed, 10 insertions(+), 1 deletion(-)

$ git diff --stat main...feat/report
 README.md          | 2 ++
 scorekit/report.py | 7 +++++++
 scorekit/runner.py | 3 +++
 3 files changed, 12 insertions(+)
$ git diff --stat $(git merge-base main feat/report) feat/report
 README.md          | 2 ++
 scorekit/report.py | 7 +++++++
 scorekit/runner.py | 3 +++
 3 files changed, 12 insertions(+)

# Two endpoints: the line that main added after the fork shows up as a deletion.
$ git diff main feat/report -- scorekit/config.py
diff --git a/scorekit/config.py b/scorekit/config.py
index 12c95c2..5efef40 100644
--- a/scorekit/config.py
+++ b/scorekit/config.py
@@ -1,3 +1,2 @@
 # Thresholds used by the runner.
 PASS_MARK = 0.75
-NIGHTLY_PASS_MARK = 0.6
# Three dots: the branch never touched the file.
$ git diff main...feat/report -- scorekit/config.py

# The README paragraph was cherry-picked to main. Both tips have it, the merge base does not.
$ git diff --stat main feat/report -- README.md
$ git diff main...feat/report -- README.md
diff --git a/README.md b/README.md
index bfae817..08f9933 100644
--- a/README.md
+++ b/README.md
@@ -5,3 +5,5 @@ Scores model answers against reference answers.
     python3 -m scorekit.runner data/smoke.jsonl
 
 The runner prints one score line and exits with status 1 when exact_match is below the pass mark.
+
+The nightly job runs the same command on data/nightly.jsonl.

                    a8e8550---f37a7d8---a10f9a1   feat/report
                   /
  ...---ca7e2b7---2652768---e376e5b               main   (HEAD -> main)
        (merge base)

  git diff main feat/report      compares  e376e5b  with  a10f9a1   (two tips)
  git diff main...feat/report    compares  ca7e2b7  with  a10f9a1   (merge base and right tip)
  git diff feat/report...main    compares  ca7e2b7  with  e376e5b   (merge base and the other tip)

Observed behavior : "git diff main feature" shows a setting being deleted that the feature never touched
Git state         : main gained commits after the branch forked; the feature's tip does not contain them
Mechanism         : A B (and A..B) compare the trees of the two tips, in the direction A to B
Root cause        : what main added after the fork is absent from the feature's tree, so it prints as "-"
Why Git does this : diff takes two endpoints; it does not look at ancestry unless asked (three dots)
Correct fix       : git diff main...feature, or git diff --merge-base main feature
Prevention        : three dots for review; in scripts the explicit --merge-base spelling

$ git diff --stat v0.1.0:scorekit main:scorekit
 __init__.py |  2 +-
 bleu.py     | 15 ---------------
 config.py   |  3 ++-
 metrics.py  | 24 ++++++++++--------------
 rouge.py    | 23 +++++++++++++++++++++++
 runner.py   | 31 +++++++++++++++++++++++--------
 text.py     | 12 ++++++++++++
 7 files changed, 71 insertions(+), 39 deletions(-)
$ git diff v0.1.0:scorekit/config.py main:scorekit/config.py
diff --git a/scorekit/config.py b/scorekit/config.py
index c22ffa6..12c95c2 100644
--- a/scorekit/config.py
+++ b/scorekit/config.py
@@ -1,2 +1,3 @@
 # Thresholds used by the runner.
-PASS_MARK = 0.7
+PASS_MARK = 0.75
+NIGHTLY_PASS_MARK = 0.6

$ git diff --stat v0.1.0 v0.2.0
 README.md            |  2 ++
 data/nightly.jsonl   | 14 ++++++++++++++
 docs/metrics.md      |  8 ++++++++
 scorekit/__init__.py |  2 +-
 scorekit/bleu.py     | 18 +++++++++---------
 scorekit/config.py   |  2 +-
 scorekit/metrics.py  | 22 ++++++++++------------
 scorekit/rouge.py    | 23 +++++++++++++++++++++++
 scorekit/runner.py   | 32 ++++++++++++++++++++++++--------
 scorekit/text.py     | 12 ++++++++++++
 10 files changed, 104 insertions(+), 31 deletions(-)
$ git diff --shortstat v0.1.0 v0.2.0
 10 files changed, 104 insertions(+), 31 deletions(-)

$ git diff --name-only v0.1.0 v0.2.0
README.md
data/nightly.jsonl
docs/metrics.md
scorekit/__init__.py
scorekit/bleu.py
scorekit/config.py
scorekit/metrics.py
scorekit/rouge.py
scorekit/runner.py
scorekit/text.py
$ git diff --name-status v0.1.0 main
M	README.md
A	data/nightly.jsonl
A	docs/metrics.md
M	scorekit/__init__.py
D	scorekit/bleu.py
M	scorekit/config.py
M	scorekit/metrics.py
A	scorekit/rouge.py
M	scorekit/runner.py
A	scorekit/text.py

$ git diff --name-status --diff-filter=A v0.1.0 main
A	data/nightly.jsonl
A	docs/metrics.md
A	scorekit/rouge.py
A	scorekit/text.py
$ git diff --name-status --diff-filter=D v0.1.0 main
D	scorekit/bleu.py
# Lowercase letters exclude: everything except additions and deletions.
$ git diff --name-status --diff-filter=ad v0.1.0 main
M	README.md
M	scorekit/__init__.py
M	scorekit/config.py
M	scorekit/metrics.py
M	scorekit/runner.py

$ git diff --stat v0.1.0 main -- '*.py' ':(exclude)scorekit/runner.py'
 scorekit/__init__.py |  2 +-
 scorekit/bleu.py     | 15 ---------------
 scorekit/config.py   |  3 ++-
 scorekit/metrics.py  | 24 ++++++++++--------------
 scorekit/rouge.py    | 23 +++++++++++++++++++++++
 scorekit/text.py     | 12 ++++++++++++
 6 files changed, 48 insertions(+), 31 deletions(-)

$ git diff --name-status d369eac v0.1.0
M	README.md
A	scorekit/__init__.py
R090	bleu.py	scorekit/bleu.py
A	scorekit/config.py
R100	scorer.py	scorekit/metrics.py
R082	run_eval.py	scorekit/runner.py

$ git diff --name-status --no-renames d369eac v0.1.0
M	README.md
D	bleu.py
D	run_eval.py
A	scorekit/__init__.py
A	scorekit/bleu.py
A	scorekit/config.py
A	scorekit/metrics.py
A	scorekit/runner.py
D	scorer.py

$ git diff --name-status -M90% d369eac v0.1.0
M	README.md
D	run_eval.py
A	scorekit/__init__.py
R090	bleu.py	scorekit/bleu.py
A	scorekit/config.py
R100	scorer.py	scorekit/metrics.py
A	scorekit/runner.py
$ git diff --name-status -M d369eac v0.1.0 -- run_eval.py scorekit/runner.py
R082	run_eval.py	scorekit/runner.py

$ git diff d369eac v0.1.0 -- run_eval.py scorekit/runner.py
diff --git a/run_eval.py b/scorekit/runner.py
similarity index 82%
rename from run_eval.py
rename to scorekit/runner.py
index 9f41591..d846906 100644
--- a/run_eval.py
+++ b/scorekit/runner.py
@@ -1,9 +1,8 @@
 import json
 import sys
 
-from scorer import exact_match, token_f1
-
-PASS_MARK = 0.7
+from scorekit.config import PASS_MARK
+from scorekit.metrics import exact_match, token_f1
 
 
 def main(path):

$ git show --stat --format='%h %s' c0c9a30
c0c9a30 Add nightly evaluation set

 data/nightly.jsonl | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
$ git show --stat --format='%h %s' -C c0c9a30
c0c9a30 Add nightly evaluation set

 data/nightly.jsonl | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
$ git show --stat --format='%h %s' --find-copies-harder c0c9a30
c0c9a30 Add nightly evaluation set

 data/{smoke.jsonl => nightly.jsonl} | 4 ++++
 1 file changed, 4 insertions(+)
$ git show --name-status --format='%h %s' --find-copies-harder c0c9a30
c0c9a30 Add nightly evaluation set

C071	data/smoke.jsonl	data/nightly.jsonl

# docs/metrics.md was renamed to docs/scoring.md and half of its text rewritten, in one step.
$ git status --short
D  docs/metrics.md
A  docs/scoring.md
$ git diff --cached --name-status -M40%
R043	docs/metrics.md	docs/scoring.md

$ git show --stat --format='%h %an: %s' 9c8df98
9c8df98 Asha Rao: Reformat sources: four-space indent, double quotes

 scorekit/__init__.py |  2 +-
 scorekit/bleu.py     | 16 ++++++++--------
 scorekit/metrics.py  | 18 +++++++++---------
 scorekit/runner.py   | 36 ++++++++++++++++++------------------
 scorekit/text.py     |  8 ++++----
 5 files changed, 40 insertions(+), 40 deletions(-)
$ git show --stat --format='%h %an: %s' -w 9c8df98
9c8df98 Asha Rao: Reformat sources: four-space indent, double quotes

 scorekit/__init__.py |  2 +-
 scorekit/runner.py   | 14 +++++++-------
 scorekit/text.py     |  6 +++---
 3 files changed, 11 insertions(+), 11 deletions(-)

$ git show --format='%h %s' -w 9c8df98 -- scorekit/text.py
9c8df98 Reformat sources: four-space indent, double quotes

diff --git a/scorekit/text.py b/scorekit/text.py
index 3075545..d404d86 100644
--- a/scorekit/text.py
+++ b/scorekit/text.py
@@ -1,11 +1,11 @@
 import re
 
-_PUNCTUATION = re.compile(r'[^\w\s]')
+_PUNCTUATION = re.compile(r"[^\w\s]")
 
 
 def normalize(text):
-  text = _PUNCTUATION.sub('', text)
-  return ' '.join(text.split())
+    text = _PUNCTUATION.sub("", text)
+    return " ".join(text.split())
 
 
 def tokens(text):

$ git diff -- docs/metrics.md
diff --git a/docs/metrics.md b/docs/metrics.md
index 7b59436..9bd401e 100644
--- a/docs/metrics.md
+++ b/docs/metrics.md
@@ -1,7 +1,7 @@
 # Metrics
 
 - exact_match: 1 when prediction and reference are equal after normalization, else 0.
-- token_f1: harmonic mean of token precision and recall.
+- token_f1: harmonic mean of token-level precision and recall.
 - rouge_l: F-measure of the longest common subsequence of tokens.
 
-Normalization lowercases the text, strips punctuation and collapses whitespace.
+Normalization strips punctuation and collapses whitespace.

$ git diff --word-diff -- docs/metrics.md
diff --git a/docs/metrics.md b/docs/metrics.md
index 7b59436..9bd401e 100644
--- a/docs/metrics.md
+++ b/docs/metrics.md
@@ -1,7 +1,7 @@
# Metrics

- exact_match: 1 when prediction and reference are equal after normalization, else 0.
- token_f1: harmonic mean of [-token-]{+token-level+} precision and recall.
- rouge_l: F-measure of the longest common subsequence of tokens.

Normalization[-lowercases the text,-] strips punctuation and collapses whitespace.

# The two functions of scorekit/overlap.py were swapped. Nothing else changed.
$ git diff --diff-algorithm=myers
diff --git a/scorekit/overlap.py b/scorekit/overlap.py
index 1394725..49c341c 100644
--- a/scorekit/overlap.py
+++ b/scorekit/overlap.py
@@ -1,12 +1,12 @@
-def token_precision(pred, ref):
+def token_recall(pred, ref):
     overlap = len(set(pred) & set(ref))
-    if not pred:
+    if not ref:
         return 0.0
-    return overlap / len(pred)
+    return overlap / len(ref)
 
 
-def token_recall(pred, ref):
+def token_precision(pred, ref):
     overlap = len(set(pred) & set(ref))
-    if not ref:
+    if not pred:
         return 0.0
-    return overlap / len(ref)
+    return overlap / len(pred)

$ git diff --diff-algorithm=histogram
diff --git a/scorekit/overlap.py b/scorekit/overlap.py
index 1394725..49c341c 100644
--- a/scorekit/overlap.py
+++ b/scorekit/overlap.py
@@ -1,12 +1,12 @@
-def token_precision(pred, ref):
-    overlap = len(set(pred) & set(ref))
-    if not pred:
-        return 0.0
-    return overlap / len(pred)
-
-
 def token_recall(pred, ref):
     overlap = len(set(pred) & set(ref))
     if not ref:
         return 0.0
     return overlap / len(ref)
+
+
+def token_precision(pred, ref):
+    overlap = len(set(pred) & set(ref))
+    if not pred:
+        return 0.0
+    return overlap / len(pred)

$ git diff --shortstat --diff-algorithm=myers
 1 file changed, 6 insertions(+), 6 deletions(-)
$ git diff --shortstat --diff-algorithm=histogram
 1 file changed, 7 insertions(+), 7 deletions(-)
$ git diff --shortstat --patience
 1 file changed, 7 insertions(+), 7 deletions(-)

# A line with two trailing spaces in config.py, a forgotten conflict marker in docs/metrics.md.
$ git diff --check
docs/metrics.md:8: leftover conflict marker
scorekit/config.py:4: trailing whitespace.
+SMOKE_LIMIT = 10  
[exit status: 2]

$ git add -A
$ git diff --check
[exit status: 0]
$ git diff --cached --check
docs/metrics.md:8: leftover conflict marker
scorekit/config.py:4: trailing whitespace.
+SMOKE_LIMIT = 10  
[exit status: 2]

# 8657273 is the merge of feat/text-utils. ^1 and ^2 choose a parent; ~1 always follows the first parent.
$ git rev-parse 8657273^1 8657273~1 8657273^2 8657273^2~1
389337aa37f6f099b27867f23fdf426b5c0307bd
389337aa37f6f099b27867f23fdf426b5c0307bd
c0d33a57b14ce8f345d5cf625ec8500fd30d85a5
dd70d9e494fe1312ef1f62b858ada29744f9edbe
$ git show -s --format='%h %s' 8657273^1 8657273^2 8657273^2~1
389337a Raise the pass mark to 0.75
c0d33a5 Add a tokens helper and use it in the metrics
dd70d9e Move normalize into scorekit/text.py

$ git show -s --format='%h %s' 'HEAD@{1}' 'main@{2}'
e376e5b Mention the nightly run in the README
2652768 Add a separate pass mark for the nightly set
$ git show -s --format='%h %cd %s' --date=format:'%a %H:%M' 'main@{yesterday}'
d9d075d Mon 12:31 Remove the experimental BLEU scorer

$ git show -s --format='%h %s' ':/Reformat sources'
9c8df98 Reformat sources: four-space indent, double quotes
$ git show -s --format='%h %s' 'v0.2.0^{/pass mark}'
389337a Raise the pass mark to 0.75
$ git show -s --format='%h %s' ':/nightly run'
e376e5b Mention the nightly run in the README

$ git cat-file -t v0.1.0
tag
$ git rev-parse v0.1.0 v0.1.0^{} v0.1.0^{commit} v0.1.0^{tree}
2b135670afee8cdfbb883de84523a5c97a25d914
682bc717015f0ab43b873d1df1ac9da0ba01c393
682bc717015f0ab43b873d1df1ac9da0ba01c393
2d7515ae10f361f36e63b9496cd27f732f656c01
$ git rev-parse v0.1.0^0
682bc717015f0ab43b873d1df1ac9da0ba01c393

$ git rev-parse main:scorekit/config.py v0.1.0:scorekit/config.py :scorekit/config.py
12c95c24c2ec9a14f3b6770dc3cf21994a1fd054
c22ffa6cffb75db7b502a54f0e0d158689414374
12c95c24c2ec9a14f3b6770dc3cf21994a1fd054
$ git show v0.1.0:scorekit/config.py
# Thresholds used by the runner.
PASS_MARK = 0.7
$ git cat-file -t main:scorekit
tree

$ git rev-parse --abbrev-ref @{u}
origin/main
$ git show -s --format='%h %s' @{u}
e376e5b Mention the nightly run in the README
$ git log --oneline @{u}..
fd9fa62 Start the 0.3 cycle

$ git log --oneline main..feat/report
a10f9a1 Write a report when --report <path> is given
f37a7d8 Mention the nightly run in the README
a8e8550 Add per-row report writer
$ git log --oneline ^main feat/report
a10f9a1 Write a report when --report <path> is given
f37a7d8 Mention the nightly run in the README
a8e8550 Add per-row report writer
$ git log --oneline feat/report --not main
a10f9a1 Write a report when --report <path> is given
f37a7d8 Mention the nightly run in the README
a8e8550 Add per-row report writer

$ git log --oneline feat/report..main
e376e5b Mention the nightly run in the README
2652768 Add a separate pass mark for the nightly set

$ git log --oneline main...feat/report
e376e5b Mention the nightly run in the README
2652768 Add a separate pass mark for the nightly set
a10f9a1 Write a report when --report <path> is given
f37a7d8 Mention the nightly run in the README
a8e8550 Add per-row report writer
$ git log --oneline --left-right main...feat/report
< e376e5b Mention the nightly run in the README
< 2652768 Add a separate pass mark for the nightly set
> a10f9a1 Write a report when --report <path> is given
> f37a7d8 Mention the nightly run in the README
> a8e8550 Add per-row report writer
$ git log --oneline --graph --boundary main...feat/report
* e376e5b Mention the nightly run in the README
* 2652768 Add a separate pass mark for the nightly set
| * a10f9a1 Write a report when --report <path> is given
| * f37a7d8 Mention the nightly run in the README
| * a8e8550 Add per-row report writer
|/  
o ca7e2b7 Fail fast on an empty reference

$ git rev-parse 8657273^@
389337aa37f6f099b27867f23fdf426b5c0307bd
c0d33a57b14ce8f345d5cf625ec8500fd30d85a5
$ git log --oneline 8657273^!
8657273 Merge branch 'feat/text-utils'
$ git log --oneline 8657273^-
8657273 Merge branch 'feat/text-utils'
c0d33a5 Add a tokens helper and use it in the metrics
dd70d9e Move normalize into scorekit/text.py
$ git log --oneline 8657273^1..8657273
8657273 Merge branch 'feat/text-utils'
c0d33a5 Add a tokens helper and use it in the metrics
dd70d9e Move normalize into scorekit/text.py

$ git diff --stat ec4fad7^!
 scorekit/metrics.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
$ git diff --stat ec4fad7~1 ec4fad7
 scorekit/metrics.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

                    a8e8550---f37a7d8---a10f9a1   feat/report
                   /
  ...---ca7e2b7---2652768---e376e5b               main

  main..feat/report    = { a8e8550, f37a7d8, a10f9a1 }                    what the branch has that main lacks
  feat/report..main    = { 2652768, e376e5b }                             what main has that the branch lacks
  main...feat/report   = { 2652768, e376e5b, a8e8550, f37a7d8, a10f9a1 }  both of the above
  git diff main..feat/report    compares the trees of e376e5b and a10f9a1
  git diff main...feat/report   compares the trees of ca7e2b7 and a10f9a1

$ git log --oneline main..v0.2.0
$ git merge-base --is-ancestor v0.2.0 main && echo "v0.2.0 is an ancestor of main"
v0.2.0 is an ancestor of main

# Everything that is on some branch and not yet in main, in one query:
$ git log --oneline --branches --not main
a10f9a1 Write a report when --report <path> is given
f37a7d8 Mention the nightly run in the README
a8e8550 Add per-row report writer
2f2883f Report ROUGE-L in the runner
80ee55f Add ROUGE-L F-measure
1cbe38a Add longest-common-subsequence helper
$ git rev-list --count v0.1.0..main
20

$ git log --graph --oneline 8e2cac1~1..8657273
*   8657273 Merge branch 'feat/text-utils'
|\  
| * c0d33a5 Add a tokens helper and use it in the metrics
| * dd70d9e Move normalize into scorekit/text.py
* | 389337a Raise the pass mark to 0.75
* | 4da82b5 Document the metrics
* | b4b066b Skip rows with an empty reference
|/  
* 8e2cac1 Strip punctuation before comparing

$ git log --oneline --first-parent 8e2cac1~1..8657273
8657273 Merge branch 'feat/text-utils'
389337a Raise the pass mark to 0.75
4da82b5 Document the metrics
b4b066b Skip rows with an empty reference
8e2cac1 Strip punctuation before comparing

$ git log --oneline --merges
8657273 Merge branch 'feat/text-utils'
$ git rev-list --count v0.1.0..main
20
$ git rev-list --count --first-parent v0.1.0..main
18
$ git rev-list --count --no-merges v0.1.0..main
19

$ git log --graph --oneline --decorate --simplify-by-decoration --all
* e376e5b (HEAD -> main) Mention the nightly run in the README
| * a10f9a1 (feat/report) Write a report when --report <path> is given
|/  
* c0c9a30 (tag: v0.2.0) Add nightly evaluation set
| * 2f2883f (feat/rouge-l) Report ROUGE-L in the runner
|/  
* 682bc71 (tag: v0.1.0) Add config module with the pass mark
* 5015359 Add project skeleton

$ git log --oneline --author=Ravi
e376e5b Mention the nightly run in the README
d9d075d Remove the experimental BLEU scorer
d926d3c Speed up normalize with a precompiled pattern
074d492 Send warnings to stderr
8e2cac1 Strip punctuation before comparing
d369eac Add experimental BLEU scorer
$ git log --format='%h %an (committed by %cn) %s' --author=Ravi --committer='Lab User'
e376e5b Ravi Menon (committed by Lab User) Mention the nightly run in the README

$ git log --oneline --grep=crash
be9ad1b Fix crash when --limit is not given
$ git log --oneline -i --grep=rouge
d9d075d Remove the experimental BLEU scorer
7823232 Add ROUGE-L metric
$ git log --oneline -i --grep=rouge --all
d9d075d Remove the experimental BLEU scorer
7823232 Add ROUGE-L metric
2f2883f Report ROUGE-L in the runner
80ee55f Add ROUGE-L F-measure
$ git log --oneline --grep=limit --grep=crash
be9ad1b Fix crash when --limit is not given
8dc82cb Add --limit option to the runner
$ git log --oneline --grep=limit --grep=crash --all-match
be9ad1b Fix crash when --limit is not given

$ git log --oneline --since='2026-09-10 00:00' --until='2026-09-11 00:00'
d926d3c Speed up normalize with a precompiled pattern
be9ad1b Fix crash when --limit is not given
074d492 Send warnings to stderr
8dc82cb Add --limit option to the runner
72134f3 Document the runner's exit status
$ git log --oneline --since='2 days ago'
e376e5b Mention the nightly run in the README
2652768 Add a separate pass mark for the nightly set
ca7e2b7 Fail fast on an empty reference
d9d075d Remove the experimental BLEU scorer
ec4fad7 Simplify token overlap in token_f1

# --since and --until test the committer date. The newest commit was authored at 10:46 and committed at 12:41:
$ git log --format='%h authored %ad, committed %cd' --date=format:'%a %H:%M' --since='2026-09-15 12:30'
e376e5b authored Tue 10:46, committed Tue 12:41

# A date without a time takes the current time of day. The lab clock reads 12:51.
$ git log --oneline --since=2026-09-10 --until=2026-09-11
9c8df98 Reformat sources: four-space indent, double quotes
d926d3c Speed up normalize with a precompiled pattern

Observed behavior : --since=2026-09-10 --until=2026-09-11 omits four commits made on the morning of the 10th
                    and includes one made on the morning of the 11th
Git state         : nothing unusual; the commits exist and carry those dates
Mechanism         : Git's date parser took the time of day, which the argument left out, from the clock
Root cause        : "2026-09-10" was read as "10 September at the present time of day", here 12:51
Why Git does this : the same parser serves "yesterday" and "2 days ago", which are meant relative to now;
                    the manual notes only that "today" means the last midnight
Correct fix       : write the time: --since='2026-09-10 00:00' --until='2026-09-11 00:00'
Prevention        : in scripts and incident notes always give a full timestamp, and state the time zone

$ git log --oneline -- scorekit/config.py
2652768 Add a separate pass mark for the nightly set
389337a Raise the pass mark to 0.75
682bc71 Add config module with the pass mark
$ git log --oneline v0.2.0..main -- docs scorekit/metrics.py
d9d075d Remove the experimental BLEU scorer
ec4fad7 Simplify token overlap in token_f1

$ git log --stat --format='%h %s' -2 -- scorekit/config.py
2652768 Add a separate pass mark for the nightly set

 scorekit/config.py | 1 +
 1 file changed, 1 insertion(+)
389337a Raise the pass mark to 0.75

 scorekit/config.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git log -p --format='%h %s' -1 -- scorekit/config.py
2652768 Add a separate pass mark for the nightly set

diff --git a/scorekit/config.py b/scorekit/config.py
index 5efef40..12c95c2 100644
--- a/scorekit/config.py
+++ b/scorekit/config.py
@@ -1,2 +1,3 @@
 # Thresholds used by the runner.
 PASS_MARK = 0.75
+NIGHTLY_PASS_MARK = 0.6

$ git log --oneline -- scorekit/metrics.py
ec4fad7 Simplify token overlap in token_f1
9c8df98 Reformat sources: four-space indent, double quotes
c0d33a5 Add a tokens helper and use it in the metrics
dd70d9e Move normalize into scorekit/text.py
8e2cac1 Strip punctuation before comparing
18bb23e Rename the scorer module to metrics

$ git log --oneline --follow -- scorekit/metrics.py
ec4fad7 Simplify token overlap in token_f1
9c8df98 Reformat sources: four-space indent, double quotes
c0d33a5 Add a tokens helper and use it in the metrics
dd70d9e Move normalize into scorekit/text.py
8e2cac1 Strip punctuation before comparing
18bb23e Rename the scorer module to metrics
d38a5aa Move sources into the scorekit package
bb5eb57 Add token-level F1 scorer
b250238 Add exact-match scorer

$ git log --follow --diff-filter=AR --name-status --format='%h %s' -- scorekit/metrics.py
18bb23e Rename the scorer module to metrics

R100	scorekit/scorer.py	scorekit/metrics.py
d38a5aa Move sources into the scorekit package

R100	scorer.py	scorekit/scorer.py
b250238 Add exact-match scorer

A	scorer.py

$ git log --oneline --follow -- scorekit/metrics.py scorekit/text.py
fatal: --follow requires exactly one pathspec
[exit status: 128]

# Without --follow, the old names can be given as extra paths, if you know them:
$ git log --oneline -- scorekit/metrics.py scorekit/scorer.py scorer.py
ec4fad7 Simplify token overlap in token_f1
9c8df98 Reformat sources: four-space indent, double quotes
c0d33a5 Add a tokens helper and use it in the metrics
dd70d9e Move normalize into scorekit/text.py
8e2cac1 Strip punctuation before comparing
18bb23e Rename the scorer module to metrics
d38a5aa Move sources into the scorekit package
bb5eb57 Add token-level F1 scorer
b250238 Add exact-match scorer

$ git log --oneline -S'PASS_MARK'
2652768 Add a separate pass mark for the nightly set
682bc71 Add config module with the pass mark
633e3e6 Add evaluation runner and smoke set

$ git log --oneline -G'PASS_MARK'
2652768 Add a separate pass mark for the nightly set
9c8df98 Reformat sources: four-space indent, double quotes
389337a Raise the pass mark to 0.75
682bc71 Add config module with the pass mark
633e3e6 Add evaluation runner and smoke set

$ git show --format='%h %s' 389337a
389337a Raise the pass mark to 0.75

diff --git a/scorekit/config.py b/scorekit/config.py
index c22ffa6..5efef40 100644
--- a/scorekit/config.py
+++ b/scorekit/config.py
@@ -1,2 +1,2 @@
 # Thresholds used by the runner.
-PASS_MARK = 0.7
+PASS_MARK = 0.75
$ git grep -c PASS_MARK 389337a~1 389337a
389337a~1:scorekit/config.py:1
389337a~1:scorekit/runner.py:2
389337a:scorekit/config.py:1
389337a:scorekit/runner.py:2

$ git log --format='%h %ad %<(10)%an %s' --date=short -S'.lower()'
d926d3c 2026-09-10 Ravi Menon Speed up normalize with a precompiled pattern
dd70d9e 2026-09-09 Asha Rao   Move normalize into scorekit/text.py
b250238 2026-09-07 Lab User   Add exact-match scorer
$ git log --format='%h %ad %<(10)%an %s' --date=short -G'\.lower\(\)'
d926d3c 2026-09-10 Ravi Menon Speed up normalize with a precompiled pattern
dd70d9e 2026-09-09 Asha Rao   Move normalize into scorekit/text.py
8e2cac1 2026-09-09 Ravi Menon Strip punctuation before comparing
b250238 2026-09-07 Lab User   Add exact-match scorer

$ git log -p --format='%h %s' -S'.lower()' -1
d926d3c Speed up normalize with a precompiled pattern

diff --git a/scorekit/text.py b/scorekit/text.py
index 48081af..3075545 100644
--- a/scorekit/text.py
+++ b/scorekit/text.py
@@ -1,10 +1,10 @@
-import string
+import re
 
-_PUNCTUATION = str.maketrans('', '', string.punctuation)
+_PUNCTUATION = re.compile(r'[^\w\s]')
 
 
 def normalize(text):
-  text = text.lower().translate(_PUNCTUATION)
+  text = _PUNCTUATION.sub('', text)
   return ' '.join(text.split())
 
 

  occurrences of ".lower()" per commit along main
  b250238   0 -> 1    reported by -S and -G     (added)
  8e2cac1   1 -> 1    reported by -G only       (line edited, call kept)
  dd70d9e   1 -> 1    reported by both          (metrics.py 1 -> 0, text.py 0 -> 1: counted per file)
  d926d3c   1 -> 0    reported by -S and -G     (removed)

$ git log --oneline -S'write_report'
$ git log --oneline --all -S'write_report'
a10f9a1 Write a report when --report <path> is given
a8e8550 Add per-row report writer

$ git log --oneline -s -L :token_f1:scorekit/metrics.py
ec4fad7 Simplify token overlap in token_f1
9c8df98 Reformat sources: four-space indent, double quotes
c0d33a5 Add a tokens helper and use it in the metrics
bb5eb57 Add token-level F1 scorer

$ git log --format='%h %an: %s' -L 2,2:scorekit/config.py
389337a Lab User: Raise the pass mark to 0.75

diff --git a/scorekit/config.py b/scorekit/config.py
index c22ffa6..5efef40 100644
--- a/scorekit/config.py
+++ b/scorekit/config.py
@@ -2,1 +2,1 @@
-PASS_MARK = 0.7
+PASS_MARK = 0.75
682bc71 Lab User: Add config module with the pass mark

diff --git a/scorekit/config.py b/scorekit/config.py
new file mode 100644
index 0000000..c22ffa6
--- /dev/null
+++ b/scorekit/config.py
@@ -0,0 +2,1 @@
+PASS_MARK = 0.7

$ git log --format='%h %an: %s' -L :normalize:scorekit/text.py -2
9c8df98 Asha Rao: Reformat sources: four-space indent, double quotes

diff --git a/scorekit/text.py b/scorekit/text.py
index 3075545..d404d86 100644
--- a/scorekit/text.py
+++ b/scorekit/text.py
@@ -6,5 +6,5 @@
 def normalize(text):
-  text = _PUNCTUATION.sub('', text)
-  return ' '.join(text.split())
+    text = _PUNCTUATION.sub("", text)
+    return " ".join(text.split())
 
 
d926d3c Ravi Menon: Speed up normalize with a precompiled pattern

diff --git a/scorekit/text.py b/scorekit/text.py
index 48081af..3075545 100644
--- a/scorekit/text.py
+++ b/scorekit/text.py
@@ -6,5 +6,5 @@
 def normalize(text):
-  text = text.lower().translate(_PUNCTUATION)
+  text = _PUNCTUATION.sub('', text)
   return ' '.join(text.split())
 
 

$ git log --oneline -L 40,45:scorekit/config.py
fatal: file scorekit/config.py has only 3 lines
[exit status: 128]
$ git log --oneline -L :no_such_function:scorekit/text.py
fatal: -L parameter 'no_such_function' starting at line 1: no match
[exit status: 128]
$ git log --oneline -L 2,2:scorekit/config.py -- scorekit/config.py
fatal: -L<range>:<file> cannot be used with pathspec
[exit status: 128]

$ git log --diff-filter=D --name-status --format="%h %ad %an: %s" --date=short
d9d075d 2026-09-14 Ravi Menon: Remove the experimental BLEU scorer

D	scorekit/bleu.py

$ git log --oneline -- scorekit/bleu.py
d9d075d Remove the experimental BLEU scorer
9c8df98 Reformat sources: four-space indent, double quotes
c0d33a5 Add a tokens helper and use it in the metrics
dd70d9e Move normalize into scorekit/text.py
18bb23e Rename the scorer module to metrics
d38a5aa Move sources into the scorekit package
$ git log --oneline -- '*bleu*'
d9d075d Remove the experimental BLEU scorer
9c8df98 Reformat sources: four-space indent, double quotes
c0d33a5 Add a tokens helper and use it in the metrics
dd70d9e Move normalize into scorekit/text.py
18bb23e Rename the scorer module to metrics
d38a5aa Move sources into the scorekit package
d369eac Add experimental BLEU scorer

$ git show --stat d9d075d
commit d9d075d63dcc94ce04c8e404e06b5cc3173180c2
Author: Ravi Menon <ravi@example.com>
Date:   Mon Sep 14 12:31:00 2026 +0530

    Remove the experimental BLEU scorer
    
    Nothing imports it, and for one reference its unigram precision duplicates token_f1.
    ROUGE-L covers the use case it was added for.

 docs/metrics.md  |  1 -
 scorekit/bleu.py | 15 ---------------
 2 files changed, 16 deletions(-)

$ git show d9d075d:scorekit/bleu.py
fatal: path 'scorekit/bleu.py' does not exist in 'd9d075d'
[exit status: 128]

$ git restore --source=d9d075d~1 -- scorekit/bleu.py
$ git status --short
?? scorekit/bleu.py

$ git log --oneline -- scorekit/report.py
$ git log --oneline --all -- scorekit/report.py
a8e8550 Add per-row report writer
$ git branch --contains $(git log --all --format=%h -1 -- scorekit/report.py)
  feat/report

$ git log --oneline --left-right main...feat/report
< e376e5b Mention the nightly run in the README
< 2652768 Add a separate pass mark for the nightly set
> a10f9a1 Write a report when --report <path> is given
> f37a7d8 Mention the nightly run in the README
> a8e8550 Add per-row report writer
$ git log --oneline --left-right --cherry-pick main...feat/report
< 2652768 Add a separate pass mark for the nightly set
> a10f9a1 Write a report when --report <path> is given
> a8e8550 Add per-row report writer

# dd70d9e moved normalize() into its own module, on a side branch. Everything v0.2.0 has that dd70d9e lacks:
$ git log --oneline dd70d9e..v0.2.0
c0c9a30 Add nightly evaluation set
7823232 Add ROUGE-L metric
9c8df98 Reformat sources: four-space indent, double quotes
d926d3c Speed up normalize with a precompiled pattern
be9ad1b Fix crash when --limit is not given
074d492 Send warnings to stderr
8dc82cb Add --limit option to the runner
72134f3 Document the runner's exit status
8657273 Merge branch 'feat/text-utils'
389337a Raise the pass mark to 0.75
4da82b5 Document the metrics
b4b066b Skip rows with an empty reference
c0d33a5 Add a tokens helper and use it in the metrics

# Only the commits that are descendants of dd70d9e as well: the path the change travelled.
$ git log --oneline --ancestry-path dd70d9e..v0.2.0
c0c9a30 Add nightly evaluation set
7823232 Add ROUGE-L metric
9c8df98 Reformat sources: four-space indent, double quotes
d926d3c Speed up normalize with a precompiled pattern
be9ad1b Fix crash when --limit is not given
074d492 Send warnings to stderr
8dc82cb Add --limit option to the runner
72134f3 Document the runner's exit status
8657273 Merge branch 'feat/text-utils'
c0d33a5 Add a tokens helper and use it in the metrics

$ git log --oneline --merges --ancestry-path dd70d9e..main | tail -n 1
8657273 Merge branch 'feat/text-utils'

$ git log --oneline --all --grep='nightly run'
f37a7d8 Mention the nightly run in the README
$ git log --oneline --reflog --grep='nightly run'
e376e5b Mention the nightly run in the README
f37a7d8 Mention the nightly run in the README
$ git log --oneline -g -2
87f090b HEAD@{0}: commit (amend): Describe the nightly job in the README
e376e5b HEAD@{1}: cherry-pick: Mention the nightly run in the README

$ git log -4 --format='%h %ad %<(10,trunc)%an %s' --date=short
e376e5b 2026-09-15 Ravi Menon Mention the nightly run in the README
2652768 2026-09-15 Lab User   Add a separate pass mark for the nightly set
ca7e2b7 2026-09-14 Asha Rao   Fail fast on an empty reference
d9d075d 2026-09-14 Ravi Menon Remove the experimental BLEU scorer
$ git log -3 --format='%h %ar%x09%s'
e376e5b 2 hours ago	Mention the nightly run in the README
2652768 32 minutes ago	Add a separate pass mark for the nightly set
ca7e2b7 22 hours ago	Fail fast on an empty reference

$ git log -1 --format='%h %s%n  author: %an%n  with:   %(trailers:key=Co-authored-by,valueonly,separator=%x2C )' 7823232
7823232 Add ROUGE-L metric
  author: Lab User
  with:   Asha Rao <asha@example.com>, Ravi Menon <ravi@example.com>

$ git shortlog -sn HEAD
    14	Lab User
     8	Asha Rao
     6	Ravi Menon
$ git shortlog -sn --no-merges --group=author --group=trailer:co-authored-by HEAD
    13	Lab User
     9	Asha Rao
     7	Ravi Menon

$ git shortlog --no-merges v0.1.0..v0.2.0
Asha Rao (4):
      Move normalize into scorekit/text.py
      Add a tokens helper and use it in the metrics
      Reformat sources: four-space indent, double quotes
      Add nightly evaluation set

Lab User (7):
      Skip rows with an empty reference
      Document the metrics
      Raise the pass mark to 0.75
      Document the runner's exit status
      Add --limit option to the runner
      Fix crash when --limit is not given
      Add ROUGE-L metric

Ravi Menon (3):
      Strip punctuation before comparing
      Send warnings to stderr
      Speed up normalize with a precompiled pattern

# Without a revision, and with standard input that is not a terminal, shortlog reads a log from standard input:
$ git shortlog -sn < /dev/null
$ git log v0.2.0..main | git shortlog -sn
     2	Asha Rao
     2	Ravi Menon
     1	Lab User

$ git blame scorekit/text.py
d926d3c5 (Ravi Menon 2026-09-10 13:36:00 +0530  1) import re
dd70d9e4 (Asha Rao   2026-09-09 11:01:00 +0530  2) 
9c8df982 (Asha Rao   2026-09-11 10:16:00 +0530  3) _PUNCTUATION = re.compile(r"[^\w\s]")
dd70d9e4 (Asha Rao   2026-09-09 11:01:00 +0530  4) 
dd70d9e4 (Asha Rao   2026-09-09 11:01:00 +0530  5) 
dd70d9e4 (Asha Rao   2026-09-09 11:01:00 +0530  6) def normalize(text):
9c8df982 (Asha Rao   2026-09-11 10:16:00 +0530  7)     text = _PUNCTUATION.sub("", text)
9c8df982 (Asha Rao   2026-09-11 10:16:00 +0530  8)     return " ".join(text.split())
c0d33a57 (Asha Rao   2026-09-09 13:21:00 +0530  9) 
c0d33a57 (Asha Rao   2026-09-09 13:21:00 +0530 10) 
c0d33a57 (Asha Rao   2026-09-09 13:21:00 +0530 11) def tokens(text):
9c8df982 (Asha Rao   2026-09-11 10:16:00 +0530 12)     return normalize(text).split()

$ git blame -s -L 6,8 scorekit/text.py
dd70d9e4 6) def normalize(text):
9c8df982 7)     text = _PUNCTUATION.sub("", text)
9c8df982 8)     return " ".join(text.split())
$ git blame -s -L :normalize scorekit/text.py
dd70d9e4  6) def normalize(text):
9c8df982  7)     text = _PUNCTUATION.sub("", text)
9c8df982  8)     return " ".join(text.split())
c0d33a57  9) 
c0d33a57 10) 
$ git blame --date=short -L '/^def load/,+3' scorekit/runner.py
b4b066b4 (Lab User 2026-09-09  9) def load(path):
9c8df982 (Asha Rao 2026-09-11 10)     rows = []
9c8df982 (Asha Rao 2026-09-11 11)     for line in open(path):

$ git blame -s v0.1.0 -- scorekit/metrics.py
bb5eb57c scorer.py  1) from collections import Counter
bb5eb57c scorer.py  2) 
bb5eb57c scorer.py  3) 
b2502383 scorer.py  4) def normalize(text):
b2502383 scorer.py  5)   return ' '.join(text.lower().split())
b2502383 scorer.py  6) 
b2502383 scorer.py  7) 
b2502383 scorer.py  8) def exact_match(prediction, reference):
b2502383 scorer.py  9)   return normalize(prediction) == normalize(reference)
bb5eb57c scorer.py 10) 
bb5eb57c scorer.py 11) 
bb5eb57c scorer.py 12) def token_f1(prediction, reference):
bb5eb57c scorer.py 13)   pred = normalize(prediction).split()
bb5eb57c scorer.py 14)   ref = normalize(reference).split()
bb5eb57c scorer.py 15)   overlap = sum((Counter(pred) & Counter(ref)).values())
bb5eb57c scorer.py 16)   if overlap == 0:
bb5eb57c scorer.py 17)     return 0.0
bb5eb57c scorer.py 18)   precision = overlap / len(pred)
bb5eb57c scorer.py 19)   recall = overlap / len(ref)
bb5eb57c scorer.py 20)   return 2 * precision * recall / (precision + recall)

$ git blame -s v0.1.0..main -- scorekit/config.py
^682bc71 1) # Thresholds used by the runner.
389337aa 2) PASS_MARK = 0.75
26527689 3) NIGHTLY_PASS_MARK = 0.6

$ git blame --line-porcelain -L 7,7 scorekit/text.py
9c8df98229e009115b3044edb1181d11075ecbbb 7 7 1
author Asha Rao
author-mail <asha@example.com>
author-time 1789101960
author-tz +0530
committer Asha Rao
committer-mail <asha@example.com>
committer-time 1789101960
committer-tz +0530
summary Reformat sources: four-space indent, double quotes
previous d926d3c5a77b586297eb25335c33fed4aee6f053 scorekit/text.py
filename scorekit/text.py
	    text = _PUNCTUATION.sub("", text)

# The line that lowercased the text is gone. Blame annotates lines that exist; it has no row for it.
$ git blame -s d926d3c~1 -L 7,7 -- scorekit/text.py
dd70d9e4 7)   text = text.lower().translate(_PUNCTUATION)
$ git blame -s -L 7,7 scorekit/text.py
9c8df982 7)     text = _PUNCTUATION.sub("", text)

  line 7 on main:   text = _PUNCTUATION.sub("", text)

  9c8df98  formatter: quotes changed             <- plain blame stops here
  d926d3c  Ravi: line replaced, .lower() gone    <- blame --ignore-rev 9c8df98 stops here
  dd70d9e  Asha: function moved into text.py     <- blame d926d3c~1 stops here (the older line 7)
  8e2cac1  Ravi: older line written, metrics.py  <- blame -C dd70d9e reaches this

$ git blame --date=short -L '/^def load/,+7' scorekit/runner.py
b4b066b4 (Lab User 2026-09-09  9) def load(path):
9c8df982 (Asha Rao 2026-09-11 10)     rows = []
9c8df982 (Asha Rao 2026-09-11 11)     for line in open(path):
9c8df982 (Asha Rao 2026-09-11 12)         row = json.loads(line)
9c8df982 (Asha Rao 2026-09-11 13)         if not row["reference"].strip():
ca7e2b7f (Asha Rao 2026-09-14 14)             raise ValueError("empty reference in row %d" % row["id"])
9c8df982 (Asha Rao 2026-09-11 15)         rows.append(row)

$ git blame --date=short -w -L '/^def load/,+7' scorekit/runner.py
b4b066b4 (Lab User 2026-09-09  9) def load(path):
b4b066b4 (Lab User 2026-09-09 10)     rows = []
b4b066b4 (Lab User 2026-09-09 11)     for line in open(path):
b4b066b4 (Lab User 2026-09-09 12)         row = json.loads(line)
9c8df982 (Asha Rao 2026-09-11 13)         if not row["reference"].strip():
ca7e2b7f (Asha Rao 2026-09-14 14)             raise ValueError("empty reference in row %d" % row["id"])
b4b066b4 (Lab User 2026-09-09 15)         rows.append(row)

$ git blame --date=short --ignore-rev 9c8df98 -L '/^def load/,+7' scorekit/runner.py
b4b066b4 (Lab User 2026-09-09  9) def load(path):
b4b066b4 (Lab User 2026-09-09 10)     rows = []
b4b066b4 (Lab User 2026-09-09 11)     for line in open(path):
b4b066b4 (Lab User 2026-09-09 12)         row = json.loads(line)
b4b066b4 (Lab User 2026-09-09 13)         if not row["reference"].strip():
ca7e2b7f (Asha Rao 2026-09-14 14)             raise ValueError("empty reference in row %d" % row["id"])
b4b066b4 (Lab User 2026-09-09 15)         rows.append(row)

$ cat .git-blame-ignore-revs
# Formatter run: four-space indent, double quotes
9c8df98229e009115b3044edb1181d11075ecbbb
$ git blame -s --ignore-revs-file .git-blame-ignore-revs scorekit/text.py
d926d3c5  1) import re
dd70d9e4  2) 
d926d3c5  3) _PUNCTUATION = re.compile(r"[^\w\s]")
dd70d9e4  4) 
dd70d9e4  5) 
dd70d9e4  6) def normalize(text):
d926d3c5  7)     text = _PUNCTUATION.sub("", text)
dd70d9e4  8)     return " ".join(text.split())
c0d33a57  9) 
c0d33a57 10) 
c0d33a57 11) def tokens(text):
c0d33a57 12)     return normalize(text).split()

$ git config set blame.ignoreRevsFile .git-blame-ignore-revs
$ git config set blame.markIgnoredLines true
$ git blame -s scorekit/text.py
d926d3c5  1) import re
dd70d9e4  2) 
?d926d3c  3) _PUNCTUATION = re.compile(r"[^\w\s]")
dd70d9e4  4) 
dd70d9e4  5) 
dd70d9e4  6) def normalize(text):
?d926d3c  7)     text = _PUNCTUATION.sub("", text)
?dd70d9e  8)     return " ".join(text.split())
c0d33a57  9) 
c0d33a57 10) 
c0d33a57 11) def tokens(text):
?c0d33a5 12)     return normalize(text).split()

$ cat ../short-ids.txt
9c8df98
$ git blame -s --ignore-revs-file ../short-ids.txt -L 6,8 scorekit/text.py
fatal: invalid object name: 9c8df98
[exit status: 128]

# The list is committed on main. An older commit does not have the file:
$ git switch --detach --quiet v0.2.0
$ git blame -s -L 6,8 scorekit/text.py
fatal: could not open object name list: .git-blame-ignore-revs
[exit status: 128]
$ git blame -s --ignore-revs-file "" -L 6,8 scorekit/text.py
fatal: could not open object name list: .git-blame-ignore-revs
[exit status: 128]

$ git config set blame.ignoreRevsFile ':(optional).git-blame-ignore-revs'
$ git blame -s -L 6,8 scorekit/text.py
dd70d9e4 6) def normalize(text):
9c8df982 7)     text = _PUNCTUATION.sub("", text)
9c8df982 8)     return " ".join(text.split())
$ git switch --quiet main
$ git blame -s -L 6,8 scorekit/text.py
dd70d9e4 6) def normalize(text):
?d926d3c 7)     text = _PUNCTUATION.sub("", text)
?dd70d9e 8)     return " ".join(text.split())

$ git blame -s -w scorekit/metrics.py
c0d33a57 scorekit/metrics.py  1) from scorekit.text import normalize, tokens
b2502383 scorer.py            2) 
b2502383 scorer.py            3) 
b2502383 scorer.py            4) def exact_match(prediction, reference):
b2502383 scorer.py            5)     return normalize(prediction) == normalize(reference)
bb5eb57c scorer.py            6) 
bb5eb57c scorer.py            7) 
bb5eb57c scorer.py            8) def token_f1(prediction, reference):
c0d33a57 scorekit/metrics.py  9)     pred = tokens(prediction)
c0d33a57 scorekit/metrics.py 10)     ref = tokens(reference)
ec4fad75 scorekit/metrics.py 11)     overlap = len(set(pred) & set(ref))
bb5eb57c scorer.py           12)     if overlap == 0:
bb5eb57c scorer.py           13)         return 0.0
bb5eb57c scorer.py           14)     precision = overlap / len(pred)
bb5eb57c scorer.py           15)     recall = overlap / len(ref)
bb5eb57c scorer.py           16)     return 2 * precision * recall / (precision + recall)

# dd70d9e moved normalize() from scorekit/metrics.py into the new file scorekit/text.py.
$ git blame -s dd70d9e -- scorekit/text.py
dd70d9e4 1) import string
dd70d9e4 2) 
dd70d9e4 3) _PUNCTUATION = str.maketrans('', '', string.punctuation)
dd70d9e4 4) 
dd70d9e4 5) 
dd70d9e4 6) def normalize(text):
dd70d9e4 7)   text = text.lower().translate(_PUNCTUATION)
dd70d9e4 8)   return ' '.join(text.split())
$ git blame -s -C dd70d9e -- scorekit/text.py
dd70d9e4 scorekit/text.py    1) import string
bb5eb57c scorer.py           2) 
8e2cac1a scorekit/metrics.py 3) _PUNCTUATION = str.maketrans('', '', string.punctuation)
8e2cac1a scorekit/metrics.py 4) 
bb5eb57c scorer.py           5) 
b2502383 scorer.py           6) def normalize(text):
8e2cac1a scorekit/metrics.py 7)   text = text.lower().translate(_PUNCTUATION)
8e2cac1a scorekit/metrics.py 8)   return ' '.join(text.split())

# Today only two short lines of that move are left unchanged. -C wants 40 alphanumeric characters.
$ git blame -s -C --ignore-rev 9c8df98 -L 6,8 scorekit/text.py
dd70d9e4 6) def normalize(text):
d926d3c5 7)     text = _PUNCTUATION.sub("", text)
dd70d9e4 8)     return " ".join(text.split())
$ git blame -s -C5 --ignore-rev 9c8df98 -L 6,8 scorekit/text.py
b2502383 scorer.py        6) def normalize(text):
d926d3c5 scorekit/text.py 7)     text = _PUNCTUATION.sub("", text)
dd70d9e4 scorekit/text.py 8)     return " ".join(text.split())

$ git blame -s -L 9,12 data/nightly.jsonl
c0c9a304  9) {"id": 9, "prediction": "seven", "reference": "7"}
c0c9a304 10) {"id": 10, "prediction": "oxygen", "reference": "oxygen"}
c0c9a304 11) {"id": 11, "prediction": "Canberra", "reference": "canberra"}
c0c9a304 12) {"id": 12, "prediction": "the speed of light", "reference": "speed of light"}
$ git blame -s -C -L 9,12 data/nightly.jsonl
c0c9a304  9) {"id": 9, "prediction": "seven", "reference": "7"}
c0c9a304 10) {"id": 10, "prediction": "oxygen", "reference": "oxygen"}
c0c9a304 11) {"id": 11, "prediction": "Canberra", "reference": "canberra"}
c0c9a304 12) {"id": 12, "prediction": "the speed of light", "reference": "speed of light"}
$ git blame -s -C -C -L 9,12 data/nightly.jsonl
633e3e60 data/smoke.jsonl    9) {"id": 9, "prediction": "seven", "reference": "7"}
633e3e60 data/smoke.jsonl   10) {"id": 10, "prediction": "oxygen", "reference": "oxygen"}
c0c9a304 data/nightly.jsonl 11) {"id": 11, "prediction": "Canberra", "reference": "canberra"}
c0c9a304 data/nightly.jsonl 12) {"id": 12, "prediction": "the speed of light", "reference": "speed of light"}

# A new commit moves tokens() above normalize() and changes nothing else.
$ git blame -s scorekit/text.py
d926d3c5  1) import re
dd70d9e4  2) 
9c8df982  3) _PUNCTUATION = re.compile(r"[^\w\s]")
dd70d9e4  4) 
dd70d9e4  5) 
fad07941  6) def tokens(text):
fad07941  7)     return normalize(text).split()
fad07941  8) 
fad07941  9) 
dd70d9e4 10) def normalize(text):
9c8df982 11)     text = _PUNCTUATION.sub("", text)
9c8df982 12)     return " ".join(text.split())
$ git blame -s -M scorekit/text.py
d926d3c5  1) import re
dd70d9e4  2) 
9c8df982  3) _PUNCTUATION = re.compile(r"[^\w\s]")
dd70d9e4  4) 
dd70d9e4  5) 
c0d33a57  6) def tokens(text):
9c8df982  7)     return normalize(text).split()
fad07941  8) 
fad07941  9) 
dd70d9e4 10) def normalize(text):
9c8df982 11)     text = _PUNCTUATION.sub("", text)
9c8df982 12)     return " ".join(text.split())

$ git blame -L 15,18 scorekit/rouge.py
78232321 (Lab User 2026-09-11 15:01:00 +0530 15) def rouge_l(prediction, reference):
78232321 (Lab User 2026-09-11 15:01:00 +0530 16)     pred = tokens(prediction)
78232321 (Lab User 2026-09-11 15:01:00 +0530 17)     ref = tokens(reference)
78232321 (Lab User 2026-09-11 15:01:00 +0530 18)     lcs = _lcs(pred, ref)
$ git blame -L 15,18 feat/rouge-l -- scorekit/rouge.py
80ee55fe (Asha Rao 2026-09-11 12:11:00 +0530 15) def rouge_l(prediction, reference):
80ee55fe (Asha Rao 2026-09-11 12:11:00 +0530 16)     pred = tokens(prediction)
80ee55fe (Asha Rao 2026-09-11 12:11:00 +0530 17)     ref = tokens(reference)
80ee55fe (Asha Rao 2026-09-11 12:11:00 +0530 18)     lcs = _lcs(pred, ref)

$ git log -1 --format='%h %an: %s%n%(trailers:key=Co-authored-by)' 7823232
7823232 Lab User: Add ROUGE-L metric
Co-authored-by: Asha Rao <asha@example.com>
Co-authored-by: Ravi Menon <ravi@example.com>

$ git branch --no-merged main
  feat/report
  feat/rouge-l

$ git blame -s --first-parent -L 10,12 scorekit/text.py
86572733 10) def normalize(text):
9c8df982 11)     text = _PUNCTUATION.sub("", text)
9c8df982 12)     return " ".join(text.split())

$ python3 -B -m scorekit.runner data/smoke.jsonl
rows=10 exact_match=0.400 token_f1=0.444 rouge_l=0.489
[exit status: 1]
$ git grep -n "exact_match" -- docs
docs/metrics.md:3:- exact_match: 1 when prediction and reference are equal after normalization, else 0.
$ git grep -n -W "def exact_match"
scorekit/metrics.py:4:def exact_match(prediction, reference):
scorekit/metrics.py-5-    return normalize(prediction) == normalize(reference)

$ git grep -n "def normalize"
scorekit/text.py:6:def normalize(text):
$ git blame --date=short -L :normalize scorekit/text.py
dd70d9e4 (Asha Rao 2026-09-09  6) def normalize(text):
9c8df982 (Asha Rao 2026-09-11  7)     text = _PUNCTUATION.sub("", text)
9c8df982 (Asha Rao 2026-09-11  8)     return " ".join(text.split())
c0d33a57 (Asha Rao 2026-09-09  9) 
c0d33a57 (Asha Rao 2026-09-09 10) 

$ git blame --date=short --ignore-rev 9c8df98 -L :normalize scorekit/text.py
dd70d9e4 (Asha Rao   2026-09-09  6) def normalize(text):
d926d3c5 (Ravi Menon 2026-09-10  7)     text = _PUNCTUATION.sub("", text)
dd70d9e4 (Asha Rao   2026-09-09  8)     return " ".join(text.split())
c0d33a57 (Asha Rao   2026-09-09  9) 
c0d33a57 (Asha Rao   2026-09-09 10) 

$ git show d926d3c
commit d926d3c5a77b586297eb25335c33fed4aee6f053
Author: Ravi Menon <ravi@example.com>
Date:   Thu Sep 10 13:36:00 2026 +0530

    Speed up normalize with a precompiled pattern
    
    str.translate built a table lookup per call. One compiled pattern is faster on the nightly set.

diff --git a/scorekit/text.py b/scorekit/text.py
index 48081af..3075545 100644
--- a/scorekit/text.py
+++ b/scorekit/text.py
@@ -1,10 +1,10 @@
-import string
+import re
 
-_PUNCTUATION = str.maketrans('', '', string.punctuation)
+_PUNCTUATION = re.compile(r'[^\w\s]')
 
 
 def normalize(text):
-  text = text.lower().translate(_PUNCTUATION)
+  text = _PUNCTUATION.sub('', text)
   return ' '.join(text.split())
 
 

$ git log --format='%h %ad %<(10)%an %s' --date=short -S'.lower()'
d926d3c 2026-09-10 Ravi Menon Speed up normalize with a precompiled pattern
dd70d9e 2026-09-09 Asha Rao   Move normalize into scorekit/text.py
b250238 2026-09-07 Lab User   Add exact-match scorer

# One call that separates right from wrong, at the suspect and at its parent, without touching the working tree:
$ git worktree add --detach --quiet ../before d926d3c~1
$ git worktree add --detach --quiet ../after d926d3c
$ (cd ../before && python3 -B -c 'from scorekit.metrics import exact_match; print(exact_match("Paris", "paris"))')
True
$ (cd ../after && python3 -B -c 'from scorekit.metrics import exact_match; print(exact_match("Paris", "paris"))')
False
$ git worktree remove ../before && git worktree remove ../after

$ git tag --contains d926d3c
v0.2.0
$ git branch --all --contains d926d3c
  feat/report
  feat/rouge-l
* main
$ git describe --contains d926d3c
v0.2.0~3
$ git log --oneline d926d3c..main -- scorekit/text.py
9c8df98 Reformat sources: four-space indent, double quotes

$ git rev-list --count v0.1.0..main
20
$ git bisect start
status: waiting for both 'good' and 'bad' commits
$ git bisect bad main
status: waiting for 'good' commit(s), 'bad' commit known
$ git bisect good v0.1.0
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[074d49249bcd54bc33537fe26519dd38e3c3ba85] Send warnings to stderr

$ git status
HEAD detached at 074d492
You are currently bisecting, started from branch 'main'.
  (use "git bisect reset" to get back to the original branch)

nothing to commit, working tree clean

$ python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | tail -n 1
KeyError: '--limit'
$ git bisect skip
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[d9d075d63dcc94ce04c8e404e06b5cc3173180c2] Remove the experimental BLEU scorer

$ python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | tail -n 1
rows=10 exact_match=0.400 token_f1=0.444 rouge_l=0.489
$ git bisect bad
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[72134f33df03602df3b2fa85f5b7dfb0aa08c190] Document the runner's exit status

$ python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | tail -n 1
rows=10 exact_match=0.800 token_f1=0.889
$ git bisect good
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[9c8df98229e009115b3044edb1181d11075ecbbb] Reformat sources: four-space indent, double quotes

$ python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | tail -n 1
rows=10 exact_match=0.400 token_f1=0.489
$ git bisect bad
Bisecting: 2 revisions left to test after this (roughly 1 step)
[be9ad1b4894348317c1e7ccb8276b8ad4760fb89] Fix crash when --limit is not given
$ python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | tail -n 1
rows=10 exact_match=0.800 token_f1=0.889
$ git bisect good
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[d926d3c5a77b586297eb25335c33fed4aee6f053] Speed up normalize with a precompiled pattern

$ python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | tail -n 1
rows=10 exact_match=0.400 token_f1=0.489
$ git bisect bad
d926d3c5a77b586297eb25335c33fed4aee6f053 is the first 'bad' commit
commit d926d3c5a77b586297eb25335c33fed4aee6f053
Author: Ravi Menon <ravi@example.com>
Date:   Thu Sep 10 13:36:00 2026 +0530

    Speed up normalize with a precompiled pattern
    
    str.translate built a table lookup per call. One compiled pattern is faster on the nightly set.

 scorekit/text.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

$ git for-each-ref --format="%(objectname:short) %(refname)" refs/bisect
d926d3c refs/bisect/bad
682bc71 refs/bisect/good-682bc717015f0ab43b873d1df1ac9da0ba01c393
72134f3 refs/bisect/good-72134f33df03602df3b2fa85f5b7dfb0aa08c190
be9ad1b refs/bisect/good-be9ad1b4894348317c1e7ccb8276b8ad4760fb89
074d492 refs/bisect/skip-074d49249bcd54bc33537fe26519dd38e3c3ba85
$ ls .git | grep BISECT
BISECT_ANCESTORS_OK
BISECT_EXPECTED_REV
BISECT_LOG
BISECT_NAMES
BISECT_START
BISECT_TERMS
$ cat .git/BISECT_START .git/BISECT_TERMS
main
bad
good
$ git rev-parse --abbrev-ref HEAD
HEAD

$ git bisect log
git bisect start
# status: waiting for both 'good' and 'bad' commits
# bad: [e376e5b709142415a87d49ca10af71977aaeabdc] Mention the nightly run in the README
git bisect bad e376e5b709142415a87d49ca10af71977aaeabdc
# status: waiting for 'good' commit(s), 'bad' commit known
# good: [682bc717015f0ab43b873d1df1ac9da0ba01c393] Add config module with the pass mark
git bisect good 682bc717015f0ab43b873d1df1ac9da0ba01c393
# skip: [074d49249bcd54bc33537fe26519dd38e3c3ba85] Send warnings to stderr
git bisect skip 074d49249bcd54bc33537fe26519dd38e3c3ba85
# bad: [d9d075d63dcc94ce04c8e404e06b5cc3173180c2] Remove the experimental BLEU scorer
git bisect bad d9d075d63dcc94ce04c8e404e06b5cc3173180c2
# good: [72134f33df03602df3b2fa85f5b7dfb0aa08c190] Document the runner's exit status
git bisect good 72134f33df03602df3b2fa85f5b7dfb0aa08c190
# bad: [9c8df98229e009115b3044edb1181d11075ecbbb] Reformat sources: four-space indent, double quotes
git bisect bad 9c8df98229e009115b3044edb1181d11075ecbbb
# good: [be9ad1b4894348317c1e7ccb8276b8ad4760fb89] Fix crash when --limit is not given
git bisect good be9ad1b4894348317c1e7ccb8276b8ad4760fb89
# bad: [d926d3c5a77b586297eb25335c33fed4aee6f053] Speed up normalize with a precompiled pattern
git bisect bad d926d3c5a77b586297eb25335c33fed4aee6f053
# first 'bad' commit: [d926d3c5a77b586297eb25335c33fed4aee6f053] Speed up normalize with a precompiled pattern

$ git bisect reset
Previous HEAD position was d926d3c Speed up normalize with a precompiled pattern
Switched to branch 'main'
$ git status --short --branch
## main
$ git for-each-ref refs/bisect | wc -l
       0

  v0.1.0   8e2cac1 ... 8657273  72134f3  8dc82cb  074d492  be9ad1b  d926d3c  9c8df98 ... d9d075d ... main
   good                          good(3)           skip(1)  good(5)  BAD(6)   bad(4)      bad(2)     bad
                                                                     ^ first bad commit

# An uncommitted edit in scorekit/config.py, a file that differs between the commits to visit:
$ git bisect start main v0.1.0
Bisecting: 9 revisions left to test after this (roughly 3 steps)
error: Your local changes to the following files would be overwritten by checkout:
	scorekit/config.py
Please commit your changes or stash them before you switch branches.
Aborting
[exit status: 1]
$ git bisect reset

$ cat ../check-em.sh
#!/bin/sh
# Exit 0 if this commit scores the smoke set correctly, 1 if it does not,
# 125 if the runner cannot produce a score line at all (untestable: skip).
out=$(python3 -B -m scorekit.runner data/smoke.jsonl 2>/dev/null)
case "$out" in
  *exact_match=0.800*) exit 0 ;;
  *exact_match=*)      exit 1 ;;
  *)                   exit 125 ;;
esac
$ ../check-em.sh; echo "exit status on main: $?"
exit status on main: 1

$ git bisect start main v0.1.0
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[074d49249bcd54bc33537fe26519dd38e3c3ba85] Send warnings to stderr
$ git bisect run ../check-em.sh
running '../check-em.sh'
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[d9d075d63dcc94ce04c8e404e06b5cc3173180c2] Remove the experimental BLEU scorer
running '../check-em.sh'
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[72134f33df03602df3b2fa85f5b7dfb0aa08c190] Document the runner's exit status
running '../check-em.sh'
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[9c8df98229e009115b3044edb1181d11075ecbbb] Reformat sources: four-space indent, double quotes
running '../check-em.sh'
Bisecting: 2 revisions left to test after this (roughly 1 step)
[be9ad1b4894348317c1e7ccb8276b8ad4760fb89] Fix crash when --limit is not given
running '../check-em.sh'
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[d926d3c5a77b586297eb25335c33fed4aee6f053] Speed up normalize with a precompiled pattern
running '../check-em.sh'
d926d3c5a77b586297eb25335c33fed4aee6f053 is the first 'bad' commit
commit d926d3c5a77b586297eb25335c33fed4aee6f053
Author: Ravi Menon <ravi@example.com>
Date:   Thu Sep 10 13:36:00 2026 +0530

    Speed up normalize with a precompiled pattern
    
    str.translate built a table lookup per call. One compiled pattern is faster on the nightly set.

 scorekit/text.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
bisect found first 'bad' commit

$ git bisect log
# bad: [e376e5b709142415a87d49ca10af71977aaeabdc] Mention the nightly run in the README
# good: [682bc717015f0ab43b873d1df1ac9da0ba01c393] Add config module with the pass mark
git bisect start 'main' 'v0.1.0'
# skip: [074d49249bcd54bc33537fe26519dd38e3c3ba85] Send warnings to stderr
git bisect skip 074d49249bcd54bc33537fe26519dd38e3c3ba85
# bad: [d9d075d63dcc94ce04c8e404e06b5cc3173180c2] Remove the experimental BLEU scorer
git bisect bad d9d075d63dcc94ce04c8e404e06b5cc3173180c2
# good: [72134f33df03602df3b2fa85f5b7dfb0aa08c190] Document the runner's exit status
git bisect good 72134f33df03602df3b2fa85f5b7dfb0aa08c190
# bad: [9c8df98229e009115b3044edb1181d11075ecbbb] Reformat sources: four-space indent, double quotes
git bisect bad 9c8df98229e009115b3044edb1181d11075ecbbb
# good: [be9ad1b4894348317c1e7ccb8276b8ad4760fb89] Fix crash when --limit is not given
git bisect good be9ad1b4894348317c1e7ccb8276b8ad4760fb89
# bad: [d926d3c5a77b586297eb25335c33fed4aee6f053] Speed up normalize with a precompiled pattern
git bisect bad d926d3c5a77b586297eb25335c33fed4aee6f053
# first 'bad' commit: [d926d3c5a77b586297eb25335c33fed4aee6f053] Speed up normalize with a precompiled pattern
$ git bisect reset
Previous HEAD position was d926d3c Speed up normalize with a precompiled pattern
Switched to branch 'main'

$ git bisect start main v0.1.0
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[074d49249bcd54bc33537fe26519dd38e3c3ba85] Send warnings to stderr
$ git bisect run sh -c 'exit 200'
running 'sh' '-c' 'exit 200'
error: bisect run failed: exit code 200 from 'sh' '-c' 'exit 200' is < 0 or >= 128
[exit status: 56]
$ git bisect reset
Previous HEAD position was 074d492 Send warnings to stderr
Switched to branch 'main'

$ git bisect start main v0.1.0
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[074d49249bcd54bc33537fe26519dd38e3c3ba85] Send warnings to stderr
$ git bisect run ./check-em.sh
running './check-em.sh'
'./check-em.sh': ./check-em.sh: No such file or directory
[682bc717015f0ab43b873d1df1ac9da0ba01c393] Add config module with the pass mark
running './check-em.sh'
'./check-em.sh': ./check-em.sh: No such file or directory
error: bogus exit code 127 for 'good' revision
[074d49249bcd54bc33537fe26519dd38e3c3ba85] Send warnings to stderr
[exit status: 1]
$ git bisect reset
Previous HEAD position was 074d492 Send warnings to stderr
Switched to branch 'main'

# Since which commit does the runner work again without --limit? 8dc82cb is known to crash.
$ git bisect start --term-old broken --term-new fixed
status: waiting for both 'broken' and 'fixed' commits
$ git bisect fixed main
status: waiting for 'broken' commit(s), 'fixed' commit known
$ git bisect broken 8dc82cb
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[78232321c82cb23d6fb732e4c8ee614c6f005919] Add ROUGE-L metric
$ git bisect terms
Your current terms are 'broken' for the old state
and 'fixed' for the new state.

# For bisect run, exit status 0 means the OLD state. grep exits 0 when it finds the crash.
$ git bisect run sh -c 'python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | grep -q KeyError'
running 'sh' '-c' 'python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | grep -q KeyError'
Bisecting: 2 revisions left to test after this (roughly 1 step)
[be9ad1b4894348317c1e7ccb8276b8ad4760fb89] Fix crash when --limit is not given
running 'sh' '-c' 'python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | grep -q KeyError'
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[074d49249bcd54bc33537fe26519dd38e3c3ba85] Send warnings to stderr
running 'sh' '-c' 'python3 -B -m scorekit.runner data/smoke.jsonl 2>&1 | grep -q KeyError'
be9ad1b4894348317c1e7ccb8276b8ad4760fb89 is the first 'fixed' commit
commit be9ad1b4894348317c1e7ccb8276b8ad4760fb89
Author: Lab User <you@example.com>
Date:   Thu Sep 10 12:31:00 2026 +0530

    Fix crash when --limit is not given
    
    Without the option the runner stopped with KeyError: '--limit'.

 scorekit/runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bisect found first 'fixed' commit

$ git bisect start --term-old broken --term-new fixed
status: waiting for both 'broken' and 'fixed' commits
$ git bisect bad main 2>&1 | head -n 2
error: Invalid command: you're currently in a fixed/broken bisect
fatal: unknown command: 'bad'
$ git bisect reset
Already on 'main'

$ git bisect replay ../bisect.log
status: waiting for both 'broken' and 'fixed' commits
be9ad1b4894348317c1e7ccb8276b8ad4760fb89 is the first 'fixed' commit
commit be9ad1b4894348317c1e7ccb8276b8ad4760fb89
Author: Lab User <you@example.com>
Date:   Thu Sep 10 12:31:00 2026 +0530

    Fix crash when --limit is not given
    
    Without the option the runner stopped with KeyError: '--limit'.

 scorekit/runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
$ git bisect reset
Already on 'main'

$ git rev-list --count v0.1.0..main
20
$ git rev-list --count --first-parent v0.1.0..main
18
$ git bisect start --first-parent main v0.1.0
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[be9ad1b4894348317c1e7ccb8276b8ad4760fb89] Fix crash when --limit is not given
$ git bisect reset
Previous HEAD position was be9ad1b Fix crash when --limit is not given
Switched to branch 'main'

$ git bisect start v0.1.0 main
Some 'good' revs are not ancestors of the 'bad' rev.
git bisect cannot work properly in this case.
Maybe you mistook 'good' and 'bad' revs?
[exit status: 1]
$ git bisect reset

$ git grep -n PASS_MARK
scorekit/config.py:2:PASS_MARK = 0.75
scorekit/config.py:3:NIGHTLY_PASS_MARK = 0.6
scorekit/runner.py:4:from scorekit.config import PASS_MARK
scorekit/runner.py:28:    return 0 if em >= PASS_MARK else 1
$ git grep -c tokens
docs/metrics.md:1
scorekit/metrics.py:3
scorekit/rouge.py:3
scorekit/text.py:1
$ git grep -l normalize -- scorekit
scorekit/metrics.py
scorekit/text.py

$ git grep -n -p "overlap == 0"
scorekit/metrics.py=8=def token_f1(prediction, reference):
scorekit/metrics.py:12:    if overlap == 0:
$ git grep -W "set(pred)"
scorekit/metrics.py=def token_f1(prediction, reference):
scorekit/metrics.py-    pred = tokens(prediction)
scorekit/metrics.py-    ref = tokens(reference)
scorekit/metrics.py:    overlap = len(set(pred) & set(ref))
scorekit/metrics.py-    if overlap == 0:
scorekit/metrics.py-        return 0.0
scorekit/metrics.py-    precision = overlap / len(pred)
scorekit/metrics.py-    recall = overlap / len(ref)
scorekit/metrics.py-    return 2 * precision * recall / (precision + recall)

$ git grep -n "lower()" v0.1.0
v0.1.0:scorekit/metrics.py:5:  return ' '.join(text.lower().split())
$ git grep -n "lower()" main
[exit status: 1]
$ git grep -n -i bleu v0.2.0 -- docs
v0.2.0:docs/metrics.md:6:- bleu1: unigram precision with a brevity penalty (experimental).

$ git grep -n 'PASS_MARK = ' v0.1.0 v0.2.0 main -- scorekit/config.py
v0.1.0:scorekit/config.py:2:PASS_MARK = 0.7
v0.2.0:scorekit/config.py:2:PASS_MARK = 0.75
main:scorekit/config.py:2:PASS_MARK = 0.75
main:scorekit/config.py:3:NIGHTLY_PASS_MARK = 0.6

$ git whatchanged -2
'git whatchanged' is nominated for removal.

hint: You can replace 'git whatchanged <opts>' with:
hint:	git log <opts> --raw --no-merges
hint: Or make an alias:
hint:	git config set --global alias.whatchanged 'log --raw --no-merges'

If you still use this command, here's what you can do:

- read https://git-scm.com/docs/BreakingChanges.html
- check if anyone has discussed this on the mailing
  list and if they came up with something that can
  help you: https://lore.kernel.org/git/?q=git%20whatchanged
- send an email to <git@vger.kernel.org> to let us
  know that you still use this command and were unable
  to determine a suitable replacement

fatal: refusing to run without --i-still-use-this
[exit status: 128]

$ git log --raw --no-merges --oneline -2
e376e5b Mention the nightly run in the README
:100644 100644 bfae817 08f9933 M	README.md
2652768 Add a separate pass mark for the nightly set
:100644 100644 5efef40 12c95c2 M	scorekit/config.py
$ git whatchanged --i-still-use-this --oneline -2
e376e5b Mention the nightly run in the README
:100644 100644 bfae817 08f9933 M	README.md
2652768 Add a separate pass mark for the nightly set
:100644 100644 5efef40 12c95c2 M	scorekit/config.py

# A newer plumbing command answers "which commit last touched each path". It is marked experimental.
$ git last-modified -r -- scorekit
26527689cd0ae7dc79b9627c6bea60ade2590465	scorekit/config.py
ca7e2b7fcc44f8d5a494107b68440c3d4ec23b55	scorekit/runner.py
ec4fad75a7e73526c6694c2559094bbe05e00980	scorekit/metrics.py
78232321c82cb23d6fb732e4c8ee614c6f005919	scorekit/rouge.py
9c8df98229e009115b3044edb1181d11075ecbbb	scorekit/text.py
9c8df98229e009115b3044edb1181d11075ecbbb	scorekit/__init__.py