mirror of
https://github.com/danielmiessler/Fabric.git
synced 2026-01-08 22:08:03 -05:00
refactor: restructure project to align with standard Go layout
### CHANGES - Introduce `cmd` directory for all main application binaries. - Move all Go packages into the `internal` directory. - Rename the `restapi` package to `server` for clarity. - Consolidate patterns and strategies into a new `data` directory. - Group all auxiliary scripts into a new `scripts` directory. - Move all documentation and images into a `docs` directory. - Update all Go import paths to reflect the new structure. - Adjust CI/CD workflows and build commands for new layout.
This commit is contained in:
33
docs/NOTES.md
Normal file
33
docs/NOTES.md
Normal file
@@ -0,0 +1,33 @@
|
||||
## Notes on some refactoring.
|
||||
|
||||
- The goal is to bring more encapsulation of the models management and simplified configuration management to bring increased flexibility, transparency on the overall flow, and simplicity in adding new model.
|
||||
- We need to differentiate:
|
||||
- Vendors: the producer of models (like OpenAI, Azure, Anthropic, Ollama, ..etc) and their associated APIs
|
||||
- Models: the LLM models these vendors are making public
|
||||
- Each vendor and operations allowed by the vendor needs to be encapsulated. This includes:
|
||||
- The questions needed to setup the model (like the API key, or the URL)
|
||||
- The listing of all models supported by the vendor
|
||||
- The actions performed with a given model
|
||||
|
||||
- The configuration flow works like this for an **initial** call:
|
||||
- The available vendors are called one by one, each of them being responsible for the data they collect. They return a set of environment variables under the form of a list of strings, or an empty list if the user does not want to setup this vendor. As we do not want each vendor to know which way the data they need will be collected (e.g., read from the command line, or a GUI), they will be asked for a list of questions, the configuration will inquire the user, and send back the questions with the collected answers to the Vendor. The Vendor is then either instantiating an instance (Vendor configured) and returning it, or returning `nil` if the Vendor should not be set up.
|
||||
- the `.env` file is created, using the information returned by the vendors
|
||||
- A list of patterns is downloaded from the main site
|
||||
|
||||
- When the system is configured, the configuration flows:
|
||||
- Read the `.env` file using the godotenv library
|
||||
- It configures a structure that contains the various vendors selected as well as the preferred model. This structure will be completed with some of the command line values (i.e, context, session, etc..)
|
||||
|
||||
- To get the list of all supported models:
|
||||
- Each configured model (part of the configuration structure) is asked, using a goroutine, to return the list of model
|
||||
|
||||
- Order when building message: session + context + pattern + user input (role "user)
|
||||
|
||||
|
||||
## TODO:
|
||||
- Check if we need to read the system.md for every patterns when running the ListAllPatterns
|
||||
- Context management seems more complex than the one in the original fabric. Probably needs some work (at least to make it clear how it works)
|
||||
- models on command line: give as well vendor (like `--model openai/gpt-4o`). If the vendor is not given, get it by retrieving all possible models and searching from that.
|
||||
- if user gives the ollama url on command line, we need to update/init an ollama vendor.
|
||||
- The db should host only things related to access and storage in ~/.config/fabric
|
||||
- The interaction part of the Setup function should be in the cli (and perhaps all the Setup)
|
||||
155
docs/Project-Restructured.md
Normal file
155
docs/Project-Restructured.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Project Restructuring Plan
|
||||
|
||||
Based on discussion in <https://github.com/danielmiessler/fabric/issues/1127>
|
||||
|
||||
This plan synthesizes the proposal by `ksylvan` with key clarifications and additions from `jaredmontoya`, `eugeis`, and others in the thread. The goal is to reorganize the project to align with standard Go conventions, reduce root-level clutter, and improve overall clarity for developers.
|
||||
|
||||
**Revision 2 Changes:** Added support for additional binary tools (`code_helper` and `to_pdf`) in the `cmd/` directory structure.
|
||||
|
||||
---
|
||||
|
||||
## Rationale for Restructuring
|
||||
|
||||
The current project structure mixes application code, web assets, scripts, and configuration files at the top level. This creates several challenges:
|
||||
|
||||
* **Top-Level Clutter:** A large number of files and directories in the root makes it difficult to quickly understand the project's structure and entry points.
|
||||
* **Non-Idiomatic Go Structure:** Go source code is spread across multiple top-level directories. Standard Go practice places main application code in `cmd/` and private, non-reusable package code in `internal/`.
|
||||
* **Mixed Concerns:** Application code (Go), web frontend code (Svelte), data processing scripts (Python), and infrastructure configuration (`nix`, `Dockerfile`) are intermingled, obscuring the separation of concerns.
|
||||
|
||||
The proposed restructure addresses these issues by organizing the project by function, adhering to community best practices.
|
||||
|
||||
---
|
||||
|
||||
## Proposed Final Directory Structure
|
||||
|
||||
This is the high-level view of the proposed structure. It incorporates the core plan and ensures that technically required files like `flake.nix` remain in the root directory.
|
||||
|
||||
```markdown
|
||||
.
|
||||
├── cmd
|
||||
│ ├── fabric
|
||||
│ │ └── main.go # Main application entrypoint
|
||||
│ ├── code_helper
|
||||
│ │ ├── main.go # Code analysis helper tool
|
||||
│ │ └── code.go # Supporting code for code_helper
|
||||
│ └── to_pdf
|
||||
│ └── main.go # LaTeX to PDF conversion tool (renamed from to_pdf.go)
|
||||
├── internal
|
||||
│ ├── cli # All CLI-related code
|
||||
│ ├── core # Core application logic (e.g., chatter)
|
||||
│ ├── domain # Domain types, moved from 'common'
|
||||
│ ├── patterns # Logic for loading/managing patterns
|
||||
│ ├── plugins # All plugin logic (ai, db, etc.)
|
||||
│ ├── server # The 'restapi' code, renamed for clarity
|
||||
│ ├── tools # Non-binary tool utilities (converter, jina, youtube, etc.)
|
||||
│ └── util # Specific, shared utilities (to be used sparingly)
|
||||
├── data
|
||||
│ ├── patterns/ # All pattern markdown files
|
||||
│ └── strategies/ # All strategy json files
|
||||
├── scripts
|
||||
│ ├── docker
|
||||
│ │ ├── Dockerfile
|
||||
│ │ ├── docker-compose.yml
|
||||
│ │ ├── start-docker.sh # Helper script to start docker-compose stack
|
||||
│ │ └── README.md # Docker deployment documentation
|
||||
│ ├── python_ui
|
||||
│ │ ├── streamlit.py
|
||||
│ │ └── requirements.txt
|
||||
│ ├── pattern_generation
|
||||
│ │ ├── extract_patterns.py
|
||||
│ │ └── ...
|
||||
│ └── setup_fabric.bat # Windows setup script
|
||||
├── docs
|
||||
│ ├── images/
|
||||
│ ├── NOTES.md
|
||||
│ └── Pattern_Descriptions/ # Documentation about patterns
|
||||
├── web/ # (Svelte frontend, unchanged)
|
||||
├── completions/ # (Shell completions, unchanged)
|
||||
├── nix/ # (Nix environment, unchanged)
|
||||
├── go.mod
|
||||
├── go.sum
|
||||
├── LICENSE
|
||||
├── README.md
|
||||
├── .gitignore
|
||||
├── .envrc # (Must remain in root for direnv)
|
||||
├── flake.nix # (Must remain in root for Nix)
|
||||
└── flake.lock # (Must remain in root for Nix)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Key Changes Explained
|
||||
|
||||
1. **Introduction of `cmd/` Directory**
|
||||
* **What:** All executable entry points are moved to `cmd/` subdirectories:
|
||||
* `cmd/fabric/main.go` - Main application entrypoint
|
||||
* `cmd/code_helper/` - Code analysis helper tool (moved from `plugins/tools/code_helper/`)
|
||||
* `cmd/to_pdf/` - LaTeX to PDF conversion tool (moved from `plugins/tools/to_pdf/`)
|
||||
* **Why:** This is a standard Go convention that clearly separates executable code from library code. It immediately shows new developers where all application entry points are and allows for additional binaries to be added cleanly in the future.
|
||||
|
||||
2. **Introduction of `internal/` Directory**
|
||||
* **What:** The majority of the Go packages (`cli`, `core`, `plugins`, `restapi`, `common`) are moved under `internal/`.
|
||||
* **Why:** The Go toolchain enforces that code within an `internal` directory can only be imported by code within the same repository. This makes the application's core logic private and prevents other projects from creating unintended dependencies on it, clarifying that the project is an application, not a public library.
|
||||
|
||||
3. **Reorganizing and Renaming Packages**
|
||||
* **`restapi` -> `internal/server`**: The package is renamed to describe its function (providing an HTTP server) rather than its implementation detail (REST).
|
||||
* **Dissolving `common`**: The `common` package will be broken up. Core data structures will move to a dedicated `internal/domain` package. Utility functions will be moved closer to the packages that use them, with any truly shared utilities placed in `internal/util`.
|
||||
* **`patterns` logic**: Code for loading and managing patterns will be consolidated into `internal/patterns`.
|
||||
* **`plugins/tools` -> `internal/tools`**: Non-binary tool utilities (converter, jina, youtube, etc.) are moved to `internal/tools` while binary tools move to `cmd/`.
|
||||
|
||||
4. **Consolidating Data, Scripts, and Docs**
|
||||
* **`data/`**: The `patterns/` and `strategies/` directories, which are data assets consumed by the application, are moved into a `data/` directory to distinguish them from source code.
|
||||
* **`scripts/`**: Helper scripts (Python, shell, batch files, Docker, etc.) are grouped under `scripts/` to clarify their role as auxiliary tools:
|
||||
* `scripts/docker/` - Docker deployment files and helper scripts
|
||||
* `scripts/python_ui/` - Streamlit UI and Python dependencies
|
||||
* `scripts/pattern_generation/` - Pattern extraction and generation tools
|
||||
* **`docs/`**: Miscellaneous markdown files (`NOTES.md`, etc.) and related assets like images are moved to `docs/` for better organization.
|
||||
|
||||
---
|
||||
|
||||
### Step-by-Step Migration Plan
|
||||
|
||||
1. **Create New Directories:** Create the new top-level directories: `cmd/fabric`, `cmd/code_helper`, `cmd/to_pdf`, `internal`, `data`, `scripts`, and `docs`.
|
||||
|
||||
2. **Move Binary Tools:**
|
||||
* Move `plugins/tools/code_helper/` to `cmd/code_helper/`
|
||||
* Move `plugins/tools/to_pdf/to_pdf.go` to `cmd/to_pdf/main.go` (rename file)
|
||||
* Move remaining non-binary tools from `plugins/tools/` to `internal/tools/`
|
||||
|
||||
3. **Move Go Packages:** Move the existing Go package directories (`cli`, `core`, `plugins`, `restapi`) into the new `internal/` directory.
|
||||
|
||||
4. **Refactor and Rename Go Packages:**
|
||||
* Rename `internal/restapi` to `internal/server`.
|
||||
* Break apart the `common` package, moving its contents into appropriate new locations like `internal/domain` and `internal/util`.
|
||||
|
||||
5. **Move Main Entry Point:** Move `main.go` to `cmd/fabric/main.go`.
|
||||
|
||||
6. **Update Go Imports:** This is a critical step. Use an IDE or tools like `goimports` to update all import paths in all `.go` files to reflect the new structure:
|
||||
* `.../fabric/cli` becomes `.../fabric/internal/cli`
|
||||
* `.../fabric/plugins/tools/...` becomes `.../fabric/internal/tools/...`
|
||||
* Update imports in all three binary tools (`fabric`, `code_helper`, `to_pdf`)
|
||||
|
||||
7. **Move Data Assets:** Move the `patterns/` and `strategies/` directories into the `data/` directory. Update the application code to read from these new paths.
|
||||
|
||||
8. **Move Scripts and Docs:**
|
||||
* Move Docker files (`Dockerfile`, `docker-compose.yml`) to `scripts/docker/` and create helper scripts and documentation
|
||||
* Move Python UI (`streamlit.py` and `requirements.txt`) to `scripts/python_ui/`
|
||||
* Move pattern generation scripts (`extract_patterns.py`, etc.) to `scripts/pattern_generation/`
|
||||
* Move batch files (`setup_fabric.bat`) and other helper scripts into `scripts/`
|
||||
* Move documentation files like `NOTES.md` and the `images` directory into `docs/`
|
||||
|
||||
9. **Update Build and CI/CD Processes:**
|
||||
* Review and update any build scripts, Makefiles, or CI/CD workflows that reference old paths
|
||||
* Update Dockerfile paths in `scripts/docker/Dockerfile` to reference new locations
|
||||
* Update GitHub Actions to build all three binaries: `./cmd/fabric`, `./cmd/code_helper`, `./cmd/to_pdf`
|
||||
* Update installation instructions in README.md to reflect new binary locations and Docker setup
|
||||
* Specifically, update the "Update Version File and Create Tag" GitHub Action to work with the new file structure
|
||||
|
||||
10. **Test and Validate:**
|
||||
* Run `go build ./cmd/fabric` to ensure the main application compiles correctly.
|
||||
* Run `go build ./cmd/code_helper` to ensure the code helper tool compiles correctly.
|
||||
* Run `go build ./cmd/to_pdf` to ensure the PDF tool compiles correctly.
|
||||
* Execute the full test suite with `go test ./...`.
|
||||
* Run all applications and manually test the CLI, API, pattern loading, and helper tools to confirm all functionality is intact.
|
||||
* Verify that external packaging and distribution methods, such as the Homebrew package, continue to build correctly after the reorganization.
|
||||
* Test that `go install github.com/danielmiessler/fabric/cmd/fabric@latest` works for all three tools.
|
||||
BIN
docs/images/fabric-logo-gif.gif
Normal file
BIN
docs/images/fabric-logo-gif.gif
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 42 MiB |
BIN
docs/images/fabric-summarize.png
Normal file
BIN
docs/images/fabric-summarize.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 491 KiB |
Reference in New Issue
Block a user