mirror of
https://github.com/vacp2p/roadmap.git
synced 2026-01-09 13:48:09 -05:00
845d6b8dcd
## Summary - Introduce a standalone Python roadmap validator with a CLI entry point, modular validation pipeline, and GitHub Actions wiring so roadmap content can be linted locally and in CI. - Provide reusable validation primitives for path resolution, front-matter parsing, identity checks, task parsing, catalog enforcement, and template adherence. - Document usage, configuration, and workflow behaviour to make the validator approachable for contributors. ## Validator Details - **Core tooling** - Added the `tools/roadmap_validator/` package with `validate.py` (CLI), `validator.py` (orchestration), and helper modules (`tasks.py`, `identity.py`, `paths.py`, `constants.py`, `issues.py`). - CLI supports directory/file targets, skips default filenames, emits GitHub annotations, and integrates optional substring filtering - README explains features, environment variables, and development guidance. - **Catalog and template enforcement** - `catalog.py` verifies each allowed content unit has `index.md` and `preview.md`, confirms roadmap entries appear under the proper quarter/area, and flags stale or missing links. - `templates.py` enforces template basics: front matter completeness, `## Description` ordering/content, template placeholder cleanup, and task section detection. - **Task validation** - `tasks.py` checks required metadata (`owner`, `status`, `start-date`, `end-date`), date formats, populated descriptions/deliverables, TODO markers, tangible deliverable heuristics, and `fully-qualified-name` prefixes. - **Workflow integration** - `.github/workflows/roadmap-validator.yml` runs the validator on pushes and manual dispatch, installs dependencies, scopes validation to changed Markdown, and surfaces findings via GitHub annotations. ## Existing Roadmap Updates - Normalised 2025q4 commitments across Web, DST, QA, SC, and other units by filling in missing descriptions, deliverables, schedule notes, recurring task statuses, and maintenance tasks. - Added tasks where absent, removed remaining template placeholders, aligned fully qualified names, and ensured roadmap files conform to the new validator checks. ## Testing ```bash python tools/roadmap_validator/validate.py *2025q4* ``` CI: `Roadmap Validator` workflow runs automatically on pushes/dispatch. --------- Co-authored-by: kaiserd <1684595+kaiserd@users.noreply.github.com>
912 B
912 B
title, tags, draft, description
| title | tags | draft | description | |||
|---|---|---|---|---|---|---|
| RAG Evalutation mechanism |
|
false | Setup a evaluation process for the RAG to ensure the quality of result. |
vac:bi:rag:2025q3-rag-evaluation-pipeline
Description
Setting up a evaluation process for the RAG responses of predefined questions. This will help to determind the gain or regretion from different change in the pipelines.
Task List
Setup an evaluation process
- fully qualified name:
vac:bi:rag:2025q4-rag-evaluation-pipeline - owner: nickninov
- status: not started
- start-date: 2025/10/01
- end-date: 2025/12/31
Description
https://github.com/status-im/data-docs/issues/99
Schedule note: Dates reflect quarter bounds; update when actual timing is known.
Deliverables
- List of questions to use as evaluation process
- Pipeline to test the response
- Evaluation process: can be manual or automatic