Local setup and development¶

Run the backend, frontend, and PostgreSQL locally with Docker Engine and the Docker Compose plugin. No host Python, Node.js, PostgreSQL, mail server, or external authentication service is needed for this setup.

Quick start¶

Clone the repository (git clone https://github.com/matthiaskoenig/pkdb.git) and change into it (cd pkdb). From the repository root, set PKDB_ADMIN and PKDB_ADMIN_EMAIL to your chosen administrator username and email address, then start the services and create the administrator:

export PKDB_ADMIN=your-username
export PKDB_ADMIN_EMAIL=you@example.org
PKDB_BUILD_COMMIT="$(git rev-parse HEAD)" docker compose --profile dev up --build --wait
docker compose exec backend pkdb-server create-admin "$PKDB_ADMIN" --email "$PKDB_ADMIN_EMAIL"

Enter a password at the hidden prompt. Create the administrator once; subsequent starts preserve the account. There is one designated administrator per database, identified by its internal account ID. The username is your choice. The email is marked verified by this operator command, so local login and API-key creation work without SMTP.

The first build downloads dependencies. Startup applies database migrations and loads the bundled vocabulary. Defaults work without an environment file.

The frontend proxies API requests to the backend, including browser session cookies and CSRF. Use localhost consistently for browser login. Both HTTP ports bind only to the local machine.

Populate the database with our users¶

As the third setup command, load the bundled historical curator/reviewer roster and avatars:

docker compose exec backend pkdb-server import-users /app/bootstrap/curator-roster.json --avatar-root /app/frontend/public --apply

Use --dry-run instead of --apply to preview changes first, especially on an existing database. The public roster contains 68 historical entries: 66 curator/reviewer accounts are imported on a fresh database; the historical test account is excluded, and the historical administrator profile is skipped unless it matches the already designated administrator. Choosing another admin username does not rename or adopt that historical identity. The empty users.json is not the user roster. Compose mounts the bundled avatars read-only and the importer copies them into persistent backend storage.

Imported users retain their attribution, roles, and profile data, but new imported accounts have no usable password and remain disabled. Repeating the same import is safe: it preserves credentials, account state, and later profile edits. Existing role or study-assignment changes require a reviewed dry run and --update-existing on both preview and apply.

Active accounts for local testing¶

For immediate login as a curator, create a separate test account with a username absent from the historical roster:

docker compose exec backend pkdb-server create-user local-curator --role curator --email local-curator@example.org

Enter its password when prompted. Use --role reviewer to test reviewer access to public studies, or omit --role for an ordinary reader. Private-study access requires an explicit curator assignment unless the account is the administrator. Email is optional for browser login; supply it when testing personal API keys. These commands create new accounts and never overwrite imported users or reset existing passwords.

Invite the real users¶

Provide reviewed email addresses in a private JSON file outside the repository, for example /absolute/path/contacts.json:

[
  {"username": "MariiaMysh", "email": "reviewed-contact@example.org"}
]

Mount it read-only for the import. Preview, resolve any conflicts, then repeat with --apply in place of --dry-run:

docker compose run --rm --no-deps \
  --volume /absolute/path/contacts.json:/private/contacts.json:ro \
  backend pkdb-server import-users /app/bootstrap/curator-roster.json \
  --avatar-root /app/frontend/public --contacts /private/contacts.json --dry-run

Configure SMTP in .env and recreate the backend with docker compose --profile dev up -d --wait. Sign in as the administrator and use User administration → Invite. Recipients accept the invitation and choose their own password. Importing does not send mail or activate users. See account administration for contact overlays, existing account IDs, and study assignments.

Load studies and test uploads¶

The initial database contains vocabulary and any users you imported, but no studies. Follow the minimal local upload setup for the complete sequence: start Docker, create the administrator, import users, prepare attribution accounts, and validate and upload with the pkdb Python package. This uses the same running stack and does not require another backend installation.

Daily development¶

Start the live frontend at http://localhost:8080 with:

docker compose --profile dev up --build --wait

Keep this stack running while editing the GUI. Save a Vue, TypeScript, or CSS file under frontend/src/ and Vite updates the open browser through hot module replacement, preserving component state where possible. Changes to assets in frontend/public/ trigger a page reload. You do not need to rebuild the image or refresh the browser for these edits. The source directories are mounted into the Vite container. Docker development uses polling every 250 ms so changes are detected even when bind-mount filesystem events are missed. Set PKDB_DEV_USE_POLLING=false in .env to use native filesystem events when those work reliably. Rebuild after dependency or frontend configuration changes. Backend code is built into its image; after changing it, run:

PKDB_BUILD_COMMIT="$(git rev-parse HEAD)" docker compose --profile dev up --build --wait

Stop and resume without losing database or attachment data:

docker compose --profile dev stop
docker compose --profile dev up --wait

docker compose --profile dev down also preserves named volumes. Adding --volumes deletes the local database and attachments. The project is named pkdb-current; it does not reuse older deployment volumes. Inspect failures with docker compose --profile dev logs --tail=100 backend db frontend.

Configuration¶

Copy .env.example to .env if you need overrides. PKDB_HTTP_PORT changes the backend's host port (default 18083); the container frontend always reaches it on http://backend:8000. Keep PKDB_BROWSER_ORIGIN=http://localhost:8080 for the default frontend. The database password must be URL-safe because it appears in the connection URL. Changing it after database initialization does not change the stored PostgreSQL password.

Native frontend and frontend checks¶

For a host frontend with the Docker backend, start only the database and API with docker compose up --build --wait. If you use the native backend instead, keep that process running and skip this Compose startup. Stop any Docker frontend with docker compose --profile dev stop frontend. Install Node 24.21.0 and npm 12.1.0, then run:

cd frontend
npm install --global npm@12.1.0
npm ci
npm run dev

The development proxy defaults to http://127.0.0.1:18083. Set PKDB_DEV_API_TARGET=http://127.0.0.1:YOUR_PORT when starting Vite if you changed the backend host port. Keep VITE_API_BASE empty for same-origin requests. Native Vite uses filesystem events by default; set PKDB_DEV_USE_POLLING=true if your filesystem does not reliably deliver change notifications.

Run frontend checks from frontend/:

npm run test:source
npm run typecheck
npm run lint
npm run test:unit -- --run
npm run build
npm run build:curation
npx playwright install --with-deps
npm run test:e2e

Browser tests use a separate disposable Compose project. See isolated frontend browser checks. For a quick check, test one browser: npx playwright install --with-deps chromium once, then npm run test:e2e -- --project=chromium.

The unit tests of the curation app read contract fixtures in frontend/tests/fixtures/curation-contract/. python/tests/test_curation_contract.py writes them from real answers of the local server on the fixture workspace and fails when an answer changes. Regenerate them from python/ with PKDB_UPDATE_CONTRACT=1 uv run --locked pytest -q tests/test_curation_contract.py, review the diff, and run the frontend unit tests against them. Never edit them by hand.

The local curation app and the website share the views that show a study. frontend/src/study-views/ holds the documents of a study and their guards, the pure modules, the read components (the study list with its filter line, the study header, the section tabs, the Data, Problems and Review sections, the metadata and the notes of a table as text) and their global styles. A page gives them one context with provideStudyViews (src/study-views/context.ts): what its reader may see and do (edit, curation, files and jobs), the routes of their links and of their query changes, and the reads of the study. The curation app provides it from its stores and router (src/curation-app/studyViews.ts) and puts its writes around the shared components (src/curation-app/**/Curation*.vue). Nothing in src/study-views/ and nothing of the website imports src/curation-app/: npm run lint and npm run test:source refuse such an import.

The curation app has its own browser tests in Chromium against the real pkdb curate, without Docker: every spec file starts a server from the python/ environment, which uv run --project python prepares, on a fresh copy of the synthetic fixture workspace in tools/curation_testing/fixture, so the spec files run in parallel, on every CPU core in continuous integration and on half of them elsewhere. The tests of screens.spec.ts change nothing, so they also run in parallel, each worker with a server of its own. speed.spec.ts runs on a workspace of 1,000 studies that tools/curation_testing/large_workspace.py generates from caffeine/Draft2021, and fails when typing in Search studies takes more than 100 ms to paint. On a busy machine, use fewer workers, for example npm run test:curation-e2e -- --workers 4. Run them from frontend/:

npx playwright install --with-deps chromium
npm run build:curation && npm run test:curation-e2e

To run only the speed tests, use npx playwright test --config playwright.curation.config.ts --project=speed --no-deps; without --no-deps, Playwright first runs the project that the speed tests wait for, which is all other spec files. Continuous integration runs the speed tests in the job curation-speed, with one worker, and curation-e2e runs npm run test:curation-e2e -- --project=chromium, which leaves them out.

tools/curation_testing/make_fixture.py draws the figure, the table images and the WebPlotDigitizer project of that fixture from its tables; run uv run --project python python tools/curation_testing/make_fixture.py from the repository root after changing them.

Native backend server¶

Use this workflow when you need automatic backend reloads. Stop the Compose backend first so port 18083 is free; this native workflow uses a separate development database and file directory:

docker compose --profile dev stop backend frontend
docker run --detach --name pkdb-native-db \
  --publish 127.0.0.1:15438:5432 \
  --env POSTGRES_DB=pkdb_dev --env POSTGRES_USER=pkdb_dev \
  --env POSTGRES_PASSWORD=local-development-only \
  --mount source=pkdb-native-database,target=/var/lib/postgresql \
  postgres:18.6
docker exec pkdb-native-db pg_isready -U pkdb_dev -d pkdb_dev

Wait for pg_isready to report that PostgreSQL accepts connections. On subsequent starts, use docker start pkdb-native-db instead of creating it again. From the repository root:

uv sync --project backend --locked --python 3.14
export PKDB_DATABASE_URL=postgresql+psycopg://pkdb_dev:local-development-only@127.0.0.1:15438/pkdb_dev
export PKDB_FILE_ROOT="$PWD/.cache/native-files"
export PKDB_BROWSER_ORIGIN=http://localhost:8080
export PKDB_SECURE_COOKIES=false
export PKDB_ADMIN=your-username
export PKDB_ADMIN_EMAIL=you@example.org
mkdir -p "$PKDB_FILE_ROOT"
cd backend
uv run --locked alembic upgrade head
uv run --locked pkdb-server bootstrap bootstrap
uv run --locked pkdb-server create-admin "$PKDB_ADMIN" --email "$PKDB_ADMIN_EMAIL"
uv run --locked uvicorn pkdb_server.app:create_app --factory --reload --host 127.0.0.1 --port 18083

Create the administrator only once and choose its password at the prompt. In another terminal, start the native frontend; its default proxy reaches this API on port 18083. Keep the environment variables set for subsequent native API and migration commands. This local database and its accounts are independent of beta.pk-db.com.

Backend tests and checks¶

The library and backend support Python 3.14 and 3.15 (currently tested with 3.15.0rc2). Python 3.15 currently uses the Pydantic 2.14 beta for native dependency wheels and the beartype 0.23 release candidate for MCP compatibility. Python 3.14 remains the default for local development, hooks, and Docker. To build the Python 3.15 runtime, use docker build --build-arg PYTHON_VERSION=3.15 -f backend/Dockerfile ..

Install uv and Python 3.14 for the development hooks. Run from the repository root (use --python 3.15 to select Python 3.15):

uv sync --project backend --locked --python 3.14
uv run --project backend pre-commit install
docker compose -f compose.test.yaml up -d --wait
export PKDB_TEST_DATABASE_URL=postgresql+psycopg://pkdb_test:local-test-only@127.0.0.1:15439/pkdb_test
uv run --project backend pytest backend/tests -q -x -n auto --dist worksteal
uv run --project backend python -m pytest tools/backend_migration -q -x
uv run --project backend ruff check .
uv run --project backend ruff format --check .
uv run --project backend ty check --project backend

The test database keeps its data in temporary container storage and runs without durable writes (fsync, synchronous_commit and full_page_writes are off), which suits only disposable data. Each pytest process migrates a template database once, and every test gets a copy of it (CREATE DATABASE ... TEMPLATE, a few milliseconds), so -n auto (pytest-xdist) runs the tests in parallel, as CI does; the tests of the migrations migrate an empty database themselves. At the end, the run reports how long the copies took (test databases: ... copies of the template, median ... ms). --dist worksteal lets an idle worker take tests from a busy one, which was faster than keeping a test file on one worker. CI adds --durations=15 to list the slowest tests. The user of PKDB_TEST_DATABASE_URL needs the right to create databases (CREATEDB); the users of compose.test.yaml and CI are superusers. The test databases are named after the configured one (pkdb_test_<time>_<id>), and a run drops those that a killed run left more than six hours ago, unless the run that created them is still alive. Keep it separate from your upload-testing database. Stop it with docker compose -f compose.test.yaml down; after a change of compose.test.yaml, docker compose -f compose.test.yaml up -d --wait recreates it.

Image lifecycle and backup/restore tests live in backend/system_tests. They require Docker, a built image selected by PKDB_TEST_IMAGE, and the test database URL. CI runs these tests on Python 3.14; for Python 3.15 run them locally against the Python 3.15 runtime image above, with PKDB_TEST_IMAGE_PYTHON=3.15. Corpus tests require explicitly configured source data and are not part of the default suite.

PKDB_STUDY_CORPUS=<pkdb_data>/studies uv run --locked pytest -q tests/test_migration_corpus.py converts named studies of pkdb_data in a dry run on copies, and PKDB_MIGRATION_FULL=1 adds the whole corpus.

Migrations¶

Commit Alembic migrations in backend/alembic/versions/. Compose applies them before starting the API. The ASGI application itself does not mutate database schemas. For native development, set PKDB_DATABASE_URL, change into backend/, and run uv run alembic upgrade head. Check model/schema agreement with uv run alembic check.

The unified data-model baseline (p001initial) requires an empty database and fresh study uploads. Historical migration revisions are no longer supported. Follow the local upload guide for account setup and uploading, and the data-model guide for the baseline details. Migration p006studyformat upgrades an existing database to study format 2 and processing version 9; every study must be uploaded again afterwards, see upgrade to study format 2.

Documentation¶

Zensical builds independently of the backend:

uvx --python 3.14 --with-requirements docs/requirements.txt zensical build --clean
uv run --no-project --python 3.14 python scripts/llms_txt.py

Branches and releases¶

Use a topic branch and a pull request against develop. Required checks are tests, ruff, ty, and docs. The tests check covers both Python versions and a fresh Compose startup/restart smoke test. Release automation uses .bumpversion.toml to update package metadata, the lockfile, and the runtime version together. Do not edit generated changelogs manually.