Study format 2 cutover¶
The cutover converts every study folder of pkdb_data from study format 1 to study format 2 with pkdb migrate. Each study is converted from the entities that the format 1 parser resolved and is checked by an equivalence gate against that parser. Only studies whose data is identical, or differs only by intended changes, replace their format 1 folder. Run the cutover after the pkdb_data tooling (pre-commit, CI and the issue sync workflow) exists. The command needs Linux or macOS.
State on 2026-10-16¶
pkdb 0.12.7 on PyPI has every command of the cutover (pkdb migrate, pkdb check, pkdb issues sync, pkdb new, pkdb move, pkdb release, pkdb registry, pkdb curate, pkdb review assign), the fixes found during the cutover (#890 to #901), the curation app for the converted studies (#906 to #912, #920, #923 to #928, #943 to #953), the counts of the format 1 tables (#955), the vocabulary terms that the format 1 studies need (#969), the converter fixes for single-row array outputs and dose units with a factor (#968) and the refusal of migrations that would drop numbers that study.json does not map (unmapped_value, #983). pkdb_data pins it in requirements/pkdb.txt (pkdb_data#2644); vocabulary.lock.json is the vocabulary bundled with 0.12.6 and 0.12.7 (pkdb_data#2507). main of pkdb_data tracks its release 2.0.0 (pkdb_data#2493), the first release in study format 2. pkdb_data develop has the format 2 tooling (#2175: pkdb check in pre-commit and the required studies workflow, the issues and vocabulary workflows, AGENTS.md, the Claude skill, the pins), the format 1 fixes before the first run (#2176 to #2189) and the cutover with its follow-ups.
The first run is pkdb_data#2190, with pkdb develop at 6b157fae; v1-final tags the last state of develop before it. Of 1576 studies, 303 identical and 546 intended went to format 2 (849), and 184 folders without study.json moved to papers/; two of them, the dulaglutide baseline analysis and the chlorzoxazone assays, are no papers and moved to pkdb_models (pkdb_models#161). docs/curation_guide.md of pkdb_data is the format 2 guide since then, the format 1 guide is archived as docs/curation-guide-format-1.md, and dulaglutide/Teraruchi2014 is renamed to Terauchi2014 (pkdb_data#2191). Every format 2 study got its GitHub issue (pkdb_data#2485: 557 adopted existing issues, 292 are new); mkoenig is a reviewer (pkdb_data#2486) and a curator (pkdb_data#2488) of every format 2 study and an assignee of every study issue; every issue is labeled with the substance of its folder, every other substance that the study names, and the review status. The 349 issues that did not belong to a study were closed as not planned.
The second run on 2026-10-14 followed fixes of the format 1 studies by group, each a pull request of pkdb_data:
- The papers without curation data left
papers/(#2498: 153 folders with only a PDF or a PDF and images). - Stubs left
studies/forpapers/: 72 studies whosestudy.jsondefines no group or individual, with their digitized workbooks (#2500;acetaminophen/Chiou1974with an empty workbook was removed), and 14 studies whosestudy.jsonwas copied from another study or an unadapted template (#2505);glucose/Kerkshoffs1998is renamed toKerckhoffs1998. - Metadata of
study.json(#2501), images of the sources (#2502), the structure of study.json and workbooks (#2503) and duplicate rows and subjects (#2504). - Abstinence and fasting rows without a value get the duration from the paper as
minwith a time unit, or are commented out when the paper gives none; repeated times of a series are fixed when the paper settles them; vocabulary gaps are mapped to existing terms or use the terms of pkdb#969 (#2506). - The run itself (pkdb_data#2508, pkdb 0.12.6) wrote 367 studies (72
identical, 295intended); 19 of them were first put back because they would have dropped numbers in columns thatstudy.jsondid not map, and were converted again with the missing mappings. Their issues are adopted or created (pkdb_data#2641: 235 adopted, 132 new; seven issues with other titles were renamed first).
The same loss of unmapped numbers had happened in 68 studies of the first run and in 3 edoxaban studies of the second, which the check before its commit missed: they lacked tmax medians and ranges, CVs, geometric means or the values of a value column. pkdb_data#2642 and #2643 restored them in 68 studies (more than 500 rows or values, checked against the PDFs); lisinopril/vanSchaik1988, omeprazole/Pounder1986 and rapamycin/BasaDenes2019 need new digitizing instead. pkdb 0.12.7 refuses such migrations (unmapped_value).
Later fixes by group, each a pull request of pkdb_data that fixes the format 1 studies, runs pkdb migrate on them and gives the converted ones their issues with pkdb issues sync --adopt:
- 25
mismatchstudies with mechanical fixes (pkdb_data#2649), and 8 studies that kept the diabetes duration in the numbers of theirdiseasechoice row, which now go to the measurementdisease durationof pkdb 0.12.8 (pkdb_data#2653, pin #2650). The points that need a curator are open review items (pkdb_data#2654). - pkdb 0.12.9 (pin pkdb_data#2657) adds the vocabulary terms that more studies need (#990) and refuses to drop a row when any unmapped column holds a number (#991). Its audit found values that earlier runs had lost in 15 converted studies; pkdb_data#2655 restored them, and pkdb_data#2656 put confidence limits, quartiles and ratios that format 2 has no column for into the row comments.
- 61 studies fixed, 47 of them converted (pkdb_data#2658, issues #2677: 29 adopted, 18 new): names with commas or repeated names, schema errors of
study.json, groups of one with subgroups, unmapped columns, reversed ranges, digitizing noise below zero and scatter dimensions that the format 1 reader did not split. - 45 studies fixed, 40 of them converted (pkdb_data#2678, issues #2687: 32 adopted, 8 new): choice rows with numeric statistics, single subjects with their value outside
mean, missing units and broken formulas, and single defects of structure. Geometric means and CVs are mapped togmean,gcvandcv: format 1 readscvandgcvas fractions and the converter writes percent, so a curated percent converts 100 times too large. The audit of pkdb_data#2710 fixed the converted studies (below). Sheets thatstudy.jsonnever references (unreferenced_sheet) are not carried into format 2; they stay in the history. - 79 studies fixed, 66 of them converted (pkdb_data#2688, issues #2709: 46 adopted, 20 new; three issues with similar titles were renamed first), most of them from the PDF: repeated times of a series set to the sampling schedule of the Methods, arms, days or substances that the mapping did not tell apart, geometric means and CVs, and wrong values found on the way. The comment of pkdb_data#2688 lists what each study that stays format 1 needs.
- The CV audit (pkdb_data#2710) scanned
cvandgcvof all format 2 tables for values above 100 and for CVs 100 times off their SD, and fixed 16 converted studies against their papers: CVs divided by 100, geometric means with their CV mapped togmeanandgcvwhere the paper prints "geometric mean (%CV)" (also in 5 studies whose CV was already in percent), and SDs that the format 1 sheets derived from a CV removed. Of the format 1 studies onlyapixaban/Frost2015has a CV in percent.pvl/PVLDrugs(unpublished data) is private now, ascaffeine/IKP243already was. - 17 studies with repeated times of a series digitized again and converted (pkdb_data#2719, issues #2724: 13 adopted, 4 new; 6
identical, 11intended). Each digitized figure is kept as a WebPlotDigitizer project and each paper table as a raw table; 48 figure images were replaced by sharper crops of the PDFs. -
Scatters (pkdb 0.12.10, pin pkdb_data#2726): a scatter of study format 2 holds several points of one subject, each row of
scatters_<source>.tsvis a point, a format 1 dataset can pair its outputs by sheet row ("shared": ["row"]), andpkdb migratepairs the points as the format 1 dataset does (#1000). pkdb_data#2725 (issues #2733: 15 adopted, 5 new) converted 20 scatter studies, with their figures checked against the papers and digitized again where the curation was wrong or incomplete, and restored the scatters of 27 converted studies whose x/y points the conversion had written as unrelated outputs because format 1 had no dataset for them. Format 1 template defects of these figures had lost or garbled values:"source": "Fig2 || Fig3"split in parallel with the x || y split takes x only from the first sheet and y only from the second, and"method": "col==x_method || col==x_method"gives y the x method. A squash merge that fixed and converted studies in one commit, such as pkdb_data#2658, #2678 and #2688, holds their unfixed format 1 folder in its parent; take the folder from the parent of the head of its pull request branch. -
The remaining format 1 studies by groups, curated against their papers with the decisions of the maintainer (pkdb 0.12.11 and 0.12.12, pins pkdb_data#2735 and #2740): pkdb_data#2738 (32 fixed, 24 converted), #2739 (the 7 studies with repeated times: anonymous points of a group are outputs of the group with their time, never invented individuals, and the means stay timecourses of the group) and #2742 (the last 17). They needed:
- vocabulary terms (#1013), such as changes from baseline that may be negative,
auc_net_incremental,radioactivity concentrationwith the unitcpm,protein amount,mRNA expression, the Jaffe creatinine methods and intestinal transporters and tissues; - raw data as the source of a paper table without image (#1012), for unpublished data and data provided elsewhere (caffeine/IKP243, pvl/PVLDrugs, Gd-EOB-DTPA/Nassif2012, icg/SwineLeipzig2014, chlorzoxazone/Puris2019 TabS4);
- no PK parameters from an all-zero curve (#1007), the
unmapped_valuefalse positive fixed (#1010), and WebPlotDigitizer projects kept bypkdb migrate(#1009). The issues of the 48 studies: pkdb_data#2763 (28 adopted, 20 new). - Restore of numbers in unmapped columns (pkdb_data#2741): format 1 converts only the columns that a template maps, and
unmapped_valuerefused only rows that were dropped whole, so numbers in other columns of converted rows were dropped without notice. An audit of all converted studies at their format 1 state found about 8,900 such numbers in 300 studies (ranges of medians, CVs, digitized error bars, changes from baseline, quartiles, whole unmapped series). #2741 restored them in 265 studies against the papers and fixed the format 1 defects found on the way, among them"min": "col==max"templates, geometric means and medians stored as means, and swapped arms. pkdb#1015 extends the gate to converted rows. A squash merge that fixed and converted studies in one commit holds their unfixed format 1 folder in its parent; take the folder from the parent of the head of its pull request branch.
pkdb check of pkdb_data develop with the pinned 0.12.12: 1489 format 2 studies, 0 errors, 4259 warnings; no study is format 1 any more. The last run deleted the identifier registry studies/study_identifiers.json, because every identifier belongs to a format 2 study. The report of the second run, migration.json and migration.md, lies uncommitted in the root of the pkdb_data checkout of mkoenig; later runs wrote their reports elsewhere.
Known open items: the models of pkdb_models that read the hidden format 1 tables from a pkdb_data checkout (DATA_PATH_BASE pointing to pkdb_data/studies) no longer find them for the converted studies; the models on develop of pkdb_models read their own copies in data/. The curator roster names a GitHub login for every curator except dimitra, Ahmed-fub and lisapolzer; former curators of student groups are no collaborators of pkdb_data, so their studies' issues lack them as assignees.
Open settings of pkdb_data, for a maintainer: apply the rulesets with .github/rulesets/apply.sh (makes studies a required check), set the secret PKDB_API_KEY and the variable PKDB_ENDPOINT (until then the issues and vocabulary workflows are skipped), and allow GitHub Actions to create pull requests. vocabulary.lock.json is still the vocabulary bundled with pkdb; the vocabulary workflow refreshes it from PKDB_ENDPOINT. Dependabot cannot update pkdb_data (Dependabot::OutOfDisk: it clones the whole repository), so bump requirements/pkdb.txt by hand as docs/development.md of pkdb_data describes.
The conversion is complete: every study of pkdb_data is in study format 2, and curation continues there. The vocabulary terms that the curated studies still miss are open review items of their studies (for example pupil size, beta2-microglobulin, 99mTc-DTPA, 13C-labelled caffeine, a percent change of an AUC).
Prepare a machine¶
- Clone pkdb and pkdb_data. In pkdb, on
develop, start the local server and create the administrator as Local setup and development describes (docker compose --profile dev up --build --wait); an existing database volume is kept and migrated at startup. Sign in at http://localhost:8080 and create a personal API key. - Install pkdb: the release pinned by pkdb_data with
uv tool install --python 3.14 --force pkdb --with-requirements requirements/pkdb.txtfrom the pkdb_data checkout, or the newerdevelopfrom the pkdb checkout withuv tool install --python 3.14 --force --editable ./python. - Set the environment:
PKDB_ENDPOINT=http://localhost:18083,PKDB_USERandPKDB_API_KEYof your local account,PKDB_NO_UPDATE=1, andGH_TOKEN(for examplegh auth token) forpkdb new,pkdb moveandpkdb issues sync. - In pkdb_data, install the hooks with
uvx pre-commit installand work on branches ofdevelop.
Next steps¶
- Curate the converted studies with
pkdb curate(Local curation app), one branch and pull request per study. - Resolve the open review items of the converted studies, among them the questions of the restored numbers of pkdb_data#2741.
- When beta.pk-db.com is deployed again, set
PKDB_API_KEYandPKDB_ENDPOINTof pkdb_data (see the open settings above), so that theissuesworkflow keeps the issues in step every night, re-upload the studies to staging and compare them with production (see After the merge).
Before the run¶
Fix unreadable study.json files and the studies that are invalid in format 1 by normal pull requests. Merge or pause open curation pull requests, because they are written against format 1.
Convert¶
Work on a branch of pkdb_data. Only one pkdb migrate runs per checkout, because it locks the repository root. Add .pkdb-migrate/ to the .gitignore of pkdb_data.
The command reads the identifier registry studies/study_identifiers.json, so it needs --approver, the maintainer who approves the released studies. It writes the report migration.json and migration.md to the current folder, or to the JSON file named by --report and the Markdown file next to it. Run every command from the same folder and keep migration.json between runs: each run reads the report of the earlier runs and keeps the studies they wrote, so migration.md covers the whole cutover. A run that stops early, by Ctrl-C or a failed swap, still writes the report, and migration.md then says "Interrupted." first. The report names the vocabulary of the run, because the conversion depends on it. Like pkdb check, the command uses the vocabulary of --vocabulary, else vocabulary.lock.json of the checkout, else the one bundled with pkdb, so pkdb_data CI checks the converted studies with the vocabulary that converted them. Use the same vocabulary for every run of the cutover, as the report warns as long as it holds studies written with another one.
Read migration.md, fix the format 1 studies or report converter bugs, and run again. The tables under Manual decisions say for each study whether it was written.
Before the first run without --dry-run, reconcile the registry with the dry-run report. A study whose study.json has a PKDB identifier as sid is only converted when the registry gives that identifier to its folder; the others are not_converted with registry_sid and are listed under "Identifiers that differ from the registry". Fix these, the studies with two PKDB identifiers (double_identifier) and the "Registry paths that do not exist" in the registry or in the studies, and run the dry run again.
When the dry run looks right, run the command without --dry-run. Stage the whole result before you commit it (git add -A studies papers .gitignore): pre-commit stashes unstaged changes as one patch and cannot apply a patch of more than 1 GB again, so a commit with unstaged changes of the migration fails and leaves the checkout half restored; then reset the checkout to the last commit, remove the untracked files of the run and run the migration again. Repeat until pkdb migrate exits with 0, which means no study is mismatch, invalid_v1 or not_converted. Each run converts only the studies that pass, so reruns are safe. A dry run writes only the report.
The run after which every identifier of the registry belongs to a format 2 study deletes studies/study_identifiers.json, and its migration.md says "Registry file deleted." under Manual decisions. Commit the deletion with the converted studies.
An interrupted run leaves .pkdb-migrate/ at the repository root. The next real run finishes or undoes each interrupted swap, and a dry run refuses until that real run happened.
| Class | Result | What to do |
|---|---|---|
identical |
Replaces the format 1 folder | Nothing. |
intended |
Replaces the format 1 folder | Check the listed changes and the manual decisions. |
mismatch |
Stays format 1 | Fix the format 1 study, see below, and rerun. |
invalid_v1 |
Stays format 1 | The format 1 parser refuses the study. Fix it in format 1, see below. |
not_converted |
Stays format 1 | Fix the reason, see below, and rerun. |
The report lists the issue codes of an invalid_v1 study. pkdb validate --offline studies/<substance>/<name> shows each error with its place in study.json or the workbook; like other format 1 tools, it may rewrite the hidden .tsv files of the workbook.
A mismatch mostly means that the converted study fails format 2 validation because of format 1 data defects that format 2 refuses: records with a statistic, such as an sd, but without a central value (mean, gmean, median, min or max), timecourse rows without time, repeated rows, choice rows without numeric statistics, and a group of one that is the parent of other subjects. The report lists the format 2 error codes and messages per study. Each message names the converted table and line with the identifying cells of that row, for example validation characteristica.tsv:16 missing_value [subjects=S2 measurement=age]; find the format 1 row by these cells. Other mismatches are differences between the format 1 data and the converted data, comments included, for example an sd or se cell ABS(X - mean) whose value saved in the workbook is stale: recalculate the workbook in a spreadsheet application and save it.
A format 1 row that holds a number in a column that study.json does not map is a mismatch with the issue code unmapped_value, so that the converter never loses numbers unseen, whether it drops the row because none of its records has a value or converts the row without the number: typically the median, min and max of a tmax row in a table whose template maps only mean and sd, a cv or geomean column next to a mapped mean, confidence limits or quartiles, a baseline or a change, the digitized mean_pm of an error bar whose spread is not mapped, the other dimension of a scatter whose template maps only one, or a whole measurement that no template reads. Every column can hold a value of the row, in any case, except count, n, group_count, groupcount, individual_count, subjects, the times day, tstart, tend, t_start, t_end, traw and every column whose name starts with time (such as time_start or the digitized time_dig of a rounded time), study, group, individual, subject, name, id, label and the columns whose name contains comment or note; a column that the template of another record of the same row maps, text such as NR and anything after a # in the row are no values of the row either. Neither is a number that a record of the same row that is converted holds. It holds the cells that its template maps and every cell that held cells determine through formulas of the workbook. A cell is computed from held cells when its formula reads only held cells and cells of columns that hold no value, in any row of the sheet: the sd computed from a held se and the count, as in the C-peptide rows of glucose/LicinioPaixao1986 whose glucose was not sampled, a change computed from the mean and the mean of the baseline row, or hours computed from a clock time. A cell is an input that the formula of a held cell with a value determines when it is the only number of the formula that is neither held nor in a column without value, it is in the row of the formula and read as a single cell, not through a range, and the formula uses arithmetic, text joins and the functions ABS, SQRT, EXP, LN, LOG and LOG10 alone: =X3, =X3*2, =ABS(X3-M3) with M3 held (the mean + se column of an error bar), =X3/100*M3 (an sd computed from a cv) and a name ="S_"&X3 (an index) determine X3. An aggregate determines none of its inputs: =AVERAGE(P3:U3) none of the individual values of the row (torasemide/Lesne1982), =(Q3-P3)/3.92*SQRT(C3) neither confidence limit, =SQRT(SUM(L$3:L3)) no squared sd, =MAX(X3,M3) not X3, and =baseline+change*baseline/100 neither the baseline nor the change, so these numbers are refused too. A number is also held when it is an exact copy of a statistic of a kept record of the row, such as a value column that repeats the mapped mean, or of the end of its error bar, the central value plus or minus the sd or se, such as a mean_pm column digitized next to a typed sd; times and counts are no statistics, so a max that equals the time of the row counts. A formula that reads another sheet, a whole column or a name determines nothing, a constant is a number of its own, a cell of another row is held only by the records of that row, and the cells of a hidden table hold no formulas. An error bar end that no record of its row holds often shows a formula that reads the neighbouring row, such as ABS(N99-M100), which computes the error bar of the row from the end of the row above. Until pkdb 0.12.8 the check knew only the statistic columns value, mean, median, min, max, sd, se, cv, gmean, gsd, gcv and mean_pm, and dropped rows whose numbers were in columns such as geomean, geometric_mean, ci_low or q25, and until pkdb 0.12.12 it checked only the rows that the converter drops: an audit of the converted studies of pkdb_data found about 7,200 numbers of converted rows in 280 studies that the conversion had lost that way. Each difference of a dropped row names the table that the row would be in, the format 1 file, sheet and row, the subject and measurement, and the columns with their numbers, for example unmapped_value outputs_Tab2.tsv: Bashir2018.xlsx Tab2 row 3, subject all, cmax, substance apixaban, tissue plasma with min 147, max 219; a difference of a converted row names the format 1 file, sheet and row, the subjects and the measurements of its records, for example unmapped_value Frost2014.xlsx Fig2 row 3, subject all, concentration with value 16.87. Map the columns in study.json, or remove a number that is no value of the row, and rerun. The check reads each characteristic from its own row of the sheet: the format 1 parser located every characteristic of a group that spans several rows at the first of them.
Three format 1 gaps need no fix, because the converter writes them as intended changes: a characteristic of a measurement that needs a time, such as a concentration, gets time NR, because format 1 characteristica have no time (time_not_reported), a characteristic, output or timecourse point without any value, that is without a choice and without any statistic, is dropped (valueless_row) unless its format 1 row holds a number that study.json does not map (unmapped_value, see above), and an output or timecourse point of a group takes the count of the count column of its format 1 table when that count differs from the count of the group (table_count). No format 1 template maps count, because the format 1 server counted every output with its group, but curators kept the number of subjects of a row in that column and the formulas of their workbooks derive an sd or se from it. A count above the count of the group, such as the count of all copied into the rows of a subgroup, is not written and is listed under Manual decisions, and so is a count that is no whole number; the rows of an individual keep the count 1, and a scatter point has no count in format 2. Rows with only a count or an error type, with a spread but no central value, and rows of abstinence ... and fasting (duration), whose existence is the information, are not dropped and need a curator, and so does a repeated row with data. A timecourse that keeps only one point after the drop fails as timecourse_points (for example apixaban/Raghavan2009) and needs curation too. Text cells are written on one line, so runs of whitespace become one space and blank cells become empty (whitespace).
What to do for each not_converted reason:
| Reason | Fix |
|---|---|
missing_image |
A figure source of outputs, timecourses or scatters needs <study>_<source>.png, and a table source its image or its raw table <study>_<source>.tsv. For data that no image shows, such as unpublished data, raw data provided elsewhere or a supplement that is not available, put the raw table beside the workbook, name the sheet as a paper table such as TabS4, and remove the image of its outputs from study.json; the converted study keeps the raw table as the source of the table. A hidden format 1 table .<study>_<sheet>.tsv and a TSV file that study.json reads as a table are no raw tables. |
image_type |
The image of a source is neither PNG nor JPG. Convert it to <study>_<source>.png. |
image_unreadable |
A PNG or JPG image cannot be read or is corrupt; every image is decoded completely before it is copied. Replace it with a readable PNG or JPG. |
image_write |
An image could not be written to the work folder, such as on a full disk. The source image is fine; fix the disk and run again. |
image_conflict |
Two image files exist for one source, such as a PNG and a JPG. Keep one. |
sheet_name, image_name |
Rename to the source pattern Tab... or Fig.... |
subject_name |
Remove , ; tabs and line breaks from names, and give a group and an individual different names. |
label_name |
A timecourse label whose , ; tabs and line breaks become _ would equal another label. Rename one of them. |
scatter_* |
For example scatter_name: two scatters with one subset name in different datasets, as in the caffeine studies Wood1979 and Roberts1976. Rename the subset. |
registry_sid |
The sid of study.json is a PKDB identifier that the registry does not give to this folder. Fix the registry or the sid, see above. |
double_identifier |
Fix the identifier registry. |
no_subjects, metadata, reference |
Fix the study as named in the message. |
format |
Format 2 refuses the written folder for the reason in the message. Fix the format 1 study when the message names its data, otherwise report a converter bug. |
unreadable |
The format 1 parser refuses the folder, usually invalid_v1. |
papers_exists |
A folder without study.json moves to papers/, where a folder of that name exists. Merge the two folders by hand. |
swap |
The converted folder could not replace the format 1 folder, which is unchanged. Fix the file system problem named in the message and rerun. |
converter_error |
The converter failed or its worker process stopped. Rerun, and report a converter bug when it fails again. |
A study without reference.json is resolved on a copy by the reference resolver, which needs the network. The original format 1 folder is not written. Check the section Manual decisions in migration.md: dropped rows without any value, one line per study, table and format 1 file with the row ranges, for the studies that are written (a dropped row can be the symptom of another defect, such as || lists of different order in study.json; migration.json lists every dropped row with its comment), replaced reference snapshots (the PubMed ID of study.json wins over a reference.json of another publication), unreleased public studies written as private, studies without a creator (the approver, or pkdb without one, becomes the creator), renamed timecourse labels, dropped output labels (also of array outputs that form no timecourse series, such as correlation data of many subjects or a label with a single row, as in tirzepatide/Urva2021), converted JPG images (turned upright by their EXIF orientation, with their ICC profile kept in the PNG; an image of another format named .png is converted the same way), dropped data files (for example WebPlotDigitizer .json projects not named <study>_<source>.wpd.json of a figure), dropped WebPlotDigitizer projects <study>_<source>.wpd.json (a project is checked against the figure image as pkdb digitize import checks it and is kept as the raw extraction of the figure; one that is invalid, lies outside its image or whose figure has no image is dropped with the reason), kept WebPlotDigitizer projects whose datasets differ from the converted rows (the digitized_mismatch and unknown_dataset warnings of pkdb validate, counted with the first message), groups of one turned into individuals, and counts of the format 1 tables that were not written because they exceed the count of the group or are no whole number.
Review and merge¶
Tag v1-final on develop before the merge. In the migration pull request (pkdb_data#2190 has these changes), add studies/**/*.xlsx to .gitignore (format 2 workbooks are generated), archive the format 1 guide as docs/curation-guide-format-1.md first, then rename docs/curation-v2.md to docs/curation_guide.md, update the navigation in zensical.toml, update every link to curation-v2.md (today in AGENTS.md, CLAUDE.md, docs/development.md, docs/installation.md; find them with git grep -n curation-v2), remove the "applies after the cutover" note at the top of the guide, and revisit the install and update instructions in docs/installation.md of pkdb_data (they still say uv tool upgrade pkdb): after the cutover they must install the pinned pkdb with uv tool install --python 3.14 --force pkdb --with-requirements requirements/pkdb.txt, like the format 2 guide. Review through migration.md (summary and manual decisions) and spot checks of converted studies. Merge one pull request with green CI.
After the merge¶
Give every study its GitHub issue on a branch of pkdb_data while curation is still paused. pkdb issues sync reads the GitHub logins from the PK-DB roster, so set PKDB_ENDPOINT and PKDB_API_KEY; set GH_TOKEN to a token that may write the issues of pkdb_data, which is private, so the dry run needs it too; and name the PK-DB user of the adoption with --user or PKDB_USER.
export PKDB_ENDPOINT=https://beta.pk-db.com PKDB_API_KEY=<key> GH_TOKEN=<token>
pkdb issues sync --adopt --user <maintainer> --dry-run
The local server of Local setup and development serves the roster too: set PKDB_ENDPOINT=http://localhost:18083 (the API; the frontend runs on http://localhost:8080) and the API key of a local administrator, after the user roster with the GitHub logins was imported. Its roster must name every curator and reviewer of the studies with their GitHub login, otherwise the dry run reports User ... is not in the PK-DB roster or has no GitHub login.
Compare the number of new issues in the dry run (create N issues in the summary, the adoptions with created in --format json) with the number of studies: a large count means that the titles of the existing issues do not follow <substance>/<name>. Rename the issues named in the warnings about similar titles. Then run the command without --dry-run; it prints each adoption and change as it goes, and a rerun finishes a stopped run. A study that names the same issue as a study that cannot be read gets an error and no change: fix the unreadable study and run the command again. A warning created #N although issue #M has a similar title names an issue that the run did not adopt; close it if it duplicates the new issue. Commit the changed study.json and review.json files and merge them through a pull request.
Re-upload all studies with processing version 9 to staging and compare counts and API output with production. Delete the format 1 code once no format 1 branch is open.
Papers¶
Folders without study.json move to papers/<substance>/<name>/. pkdb new <substance>/<name> takes the PDF <name>.pdf and the images <name>_<source>.png, with a source such as Tab1, Fig2A or Text, from there into the new study. Other files, such as a study format 1 workbook <name>.xlsx, stay in the paper folder and are listed in the output; a file whose name differs from these names only in case, such as <name>.PDF, also stays and is named with the expected name. An emptied paper folder and an emptied substance folder below papers/ are removed.