---
title: VacLab
tags:
  - "2024q4"
  - "dst"
  - "ift"
draft: false
description: "Scale and apply the VacLab to IFT's needs.
Anticipate untapped use cases and needs from other teams.
Achieve 25% real world time usage."
---

`vac:dst:ift:2024q4-vaclab`

## Description

The VacLab is a resource provided to Vac by Riff Labs Limited,
intended to help us perform detailed simulations
and deployments of distributed systems at scale
as well as the systems and dependencies that surround them.

With the VacLab reaching maturity where it can start being used comfortably
to advance IFT's research, development testing efforts, and quality control,
we want to ensure it is being used to its full potential
and that teams understand that the resource exists,
and if they find a useful case for their work being improved
by collaboration with the DST team through VacLab,
they should be comfortable reaching out to us,
and confident that we will be willing to try and help them,
based on our attitude and more importantly the track record
and results we will produce using these tools and our expertise.

The lab will be treated as an IaaS-style service at first,
with the raw underlying infrastructure being developed in partnership with Riff Labs
who handles the details of making that IaaS layer available to us and reliable.

As we progress through the maturity of the lab,
we will transition to supporting a more PaaS or even SaaS software model,
where as much as possible is accessible to the IFT ecosystem and teams
to use and benefit from without them needing to concern themselves with the details of the underlying infrastructure or be blocked by the need to build and manage their own.

We will move towards self service testing and deployment,
and by doing so unblock and accelerate the development, R&D and productionisation
of IFT's projects by providing a safe and reliable place to experiment and test.

It will continue to provide significant efficiency benefits in terms of cost vs output when compared to cloud providers
and even on traditional premises deployment of infrastructure,
using many independent and cheaper nodes
rather than larger more powerful vertically scaled machines,
building on second hand and used equipment,
and "patching around" the unreliability of individual hardware,
by ensuring everything is resilient and reliable even in the face of individual failures,
and in doing so continue to reduce and control the costs of testing our systems at scale.

Through the use of the VacLab
we will support the Conduit of Expertise narrative by:
* providing a unique capability to the IFT ecosystem
  that would not otherwise be available to them,
  lower the barrier to entry for teams needing research -
  or services that require infrastructure -
  by lowering the cost and removing the need for them to get it themselves
  through cloud providers that provide less flexibility and direct control,

* using our knowledge of what is possible to do with these resources,
  based on who is already using them,
  and apply that knowledge to intuit new use cases
  that will unlock better collaboration between teams and the DST,
  driving and accelerating development of IFT projects
  such as Waku, Codex, Nomos and more.

* Accelerating initiatives by providing the means, capability and encouragement
  to test every aspect of anything that can be tested in a simulation,
  across every team and use case that is interested in doing so,
  up to the limits of what the DST team can support.

We will also provide support for the Premier Research destination narrative by:
* Allowing public access to non-sensitive telemetry and metrics from non-sensitive systems such as Codex storage nodes, and potentially even probes that measure the state of networks such as The Waku Network and Status.

## Task List

### Status Page Known

* fully qualified name: `vac:dst:ift:2024q4-vaclab:status-page-known`
* owner: Wings
* status: Done
* start-date: 2024/12/01
* end-date: 2024/12/10

#### Description

A status page for the VacLab
that has wide acceptance and use
by anyone who wants to know the current status
of the VacLab and its availability.

#### Deliverables
<!--
Already done:
* ✅ Status page is created and hosted on the lab
  and made available to users.
-->
* Status page reflects reality and is accepted by the users
  as being a good fit for their needs.
* Status page sees widespread use among its users.
* Build an external probe and a fallback status page
  that can be used in case everything else is down.
 
### Better Time Slicing

* fully qualified name: `vac:dst:ift:2024q4-vaclab:better-time-slicing`
* owner: Wings
* status: 0%
* start-date: 2024/12/01
* end-date: 2024/12/14

#### Description
Do a better job of time slicing the lab.

#### Deliverables
* A report on the current state of time slicing in the lab.
* A plan for how to improve time slicing in the lab.
* A timeline for implementing the plan.
* Measurable improvements in usage of the lab
  that aims for an initial target of 25% of real world time
  being used for useful workloads and tests

Later repeats in the VacLab commitment will aim to improve this to 50%,
then 75%, then as far as possible
to the limits of the underlying infrastructure and our actual needs.

### Train Lab Staff
<!-- technically sort of external
and will be done outside of normal DST cadence
but will be managed so as not to disrupt other works
-->

* fully qualified name: `vac:dst:ift:2024q4-vaclab:train-lab-staff`
* owner: Wings
* status: 30%
* start-date: 2024/12/01
* end-date: 2024/12/31

#### Description
Fully dedicate all time outside of core DST deliverable work
to training Michaela, the VacLab (Riff Labs Perth) custodian,
in all aspects of not just managing the VacLab,
but providing support to DST's work that utilises it,
with the focus of improving both the reliability of the lab
and provide a better systems testing service.

Will - must, for practical reasons - be done in person in Perth.

Will also be used to improve the reliability and capabilities
of the VacLab as a platform for IFT's research and development needs.

**Must not impact other works outside of this task.**

#### Deliverables
- [ ] Full automation for anything we know needs doing regularly
  - [ ] Automated patching for security updates (Debian, Authentik, SeaweedFS)
  - [ ] Secure key management and rotation automation (for SSH keys)
- [ ] Michaela fully comfortable operating the lab independently
- [ ] A report on what was learned in this process
      and how we believe it improved VacLab support and operations
- [ ] Improvements to the lab that are documented, implemented and recorded.


### Automation Uplift
<!-- technically sort of external
and will be done outside of normal DST cadence
but will be managed so as not to disrupt other works
-->

* fully qualified name: `vac:dst:ift:2024q4-vaclab:automation-uplift`
* owner: Wings
* status: 0%
* start-date: 2024/12/01
* end-date: 2024/12/31

#### Description
Significantly improve the automation and management of the VacLab,
freeing up resources for Wings to focus on other work.

#### Deliverables
- [ ] Full automation for anything we know needs doing regularly
  - [ ] Automated patching for security updates (Debian, Authentik, SeaweedFS)
  - [ ] Secure key management and rotation automation (for SSH keys)
- [ ] A report on what was learned in this process
      and how we believe it improved VacLab support and operations
  - What was automated? Why? What did that change?
  - What remains manual and needs improving?
- [ ] Improvements to the lab that are documented, implemented and recorded.