AtHeartEngineer/groups
1
0
Fork 0
mirror of https://github.com/zkitter/groups.git synced 2026-01-10 07:48:12 -05:00
Code Issues Packages Projects Releases Wiki Activity

feat: check gh-contributors and dao-voters groups membership (#37)

Browse Source 
* feat: check `gh-contributors` and `dao-voters` membership  (#36)

* feat:test: get orgs with repos and voters

* feat:test: split a time range in chunks

* refactor: get gh names by space ids

Rename field in gql query directly

* feat:test: get ids of the snapshot spaces an address voted for

Query by address instead of by space ids

* refactor: rename space keys when fetching them from snapshot rest api

* refactor getSpaces and getGhOrgs

* refactor: return object from getGhNamesBySpaceIds

* refactor: rename splitArray to split. Remove splitTimestamps fn

* remove gh, daos, snapshot & scripts folders

* add test.unit nps script

* feat:test: exclude orgs from an ignore list

* remove getGhOrgs method

* remove GroupService.ts

* refactor: accept ghName to be null

Because of prisma schema definition

* refactor `getOrgsWithRepos`

Make 1 call to get spaces gh names

* fix unit and integration tests

* update prisma schema (ghName optional)

* fix: fetch first 1000 spaces in getGhNamesBySpaceIdsQuery

* fix e2e and integration tests

* refactor:test: get whitelist short return `{ daos, repos }`

* feat:test: add GET `/whitelist/{daos,repos}` endpoints

* feat:test: can check wether a user belongs to voters group

* refactor whitelist service

* add `/whitelist/{daos,repos}` to app router

* format

* feat: add `/belongs-to-*-group` endpoints

* restructure endpoints

* update openapi spec

* Update README

* format

* fix: remove parameters from /whitelist/repos openapi spec
This commit is contained in:
r1oga
2023-02-28 23:20:53 +01:00
committed by GitHub
parent 166e5fe8a0
commit d2b2d59e6c
45 changed files with 829 additions and 698 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
README.md
View File

@@ -1,18 +1,10 @@
# Zkitter Groups
Get list of GH users who contributed to the GitHub org of a given group of DAOs.
Under the hood, it:
Top snapshot organizations as per their number of followers are whitelisted and result in 2 groups:
1. Fetches a list of `maxOrgs` (default 100) spaces from snapshot.org based on their `minFollowers` (default 10_000).
2. For each space, it fetches the list of their repositories on GitHub.
3. For each of the org repository, it fetches the list of contributors between a `since` and `until`timestamps (default between now and now - 1 month).
4. Returns the list of unique contributors, removing potential bots (e.g dependencies, github action bots etc...)
- the GitHub users who contributed to one of the public repository of one of the whitelisted organization
- the ethereum addresses who participated in of the snapshot vote of one of the whitelisted organization
## API
| METHOD | PATH | DESCRIPTION | RESPONSE |
| ------ | ------------------------------------ | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| GET | `/whitelist?format=short\|long` | Get list of whitelisted organizations in `short` or `long` format | `Array<{ followers: number, followers7d?: number, snapshotId: string, snapshotName: string, ghName: string, repos: string[] }>` |
| GET | `/whitelist/refresh` | Update list of whitelisted orgs and their repos. Return updated whitelist | `Array<{ followers: number, followers7d?: number, snapshotId: string, snapshotName: string, ghName: string, repos: string[] }>` |
| GET | `/user/:username?format=short\|long` | Get user `username` in `short` (only groups info) or `long` (with repos) format | `{ belongsToGhContributorsGroup: boolean, belongsToDaoVotersGroups: boolean }` |
| GET | `/user/:username/refresh` | Update list of repos the user `username` contributed to and return updated user | `{ ghName: string, repos: string[]}` |
See [openapi.json](src/openapi.json) for the full API specification. Or try it [online](https://zkitter-groups.fly.dev/).

9
graphql/snapshot/gh-names-by-space-ids.graphql Normal file
View File

@@ -0,0 +1,9 @@
query GhNameBySpaceIds($ids: [String]){
spaces(
first: 1000
where:{id_in: $ids}
) {
snapshotId: id
ghName: github
}
}

8
graphql/snapshot/spaces.graphql
View File

@@ -1,8 +0,0 @@
query {
spaces(
where:{id_in: ["0x00pluto.eth","stgdao.eth", "ens.eth", "aave.eth", "uniswap"]}
) {
id
github
}
}

14
graphql/snapshot/voted-spaces-by-address.graphql Normal file
View File

@@ -0,0 +1,14 @@
query VotedSpacesByAddress($address: String!, $since: Int!, $until: Int!) {
votes (
where: {
created_gte: $since,
created_lte: $until,
voter_in: [$address],
vp_state: "final"
}
) {
space {
snapshotId: id
}
}
}

14
graphql/snapshot/voters.graphql
View File

@@ -1,14 +0,0 @@
query {
votes (
where: {
created_gte: 1672000000
created_lte: 1674000000
space_in: ["stgdao.eth", "ens.eth", "aave.eth", "uniswap"]
vp_state: "final"
}
orderBy: "created",
orderDirection: desc
) {
voter
}
}

9
package-scripts.yaml
View File

@@ -58,7 +58,7 @@ scripts:
script: nps db.gen-schema barrels build start.prod
description: Start in production mode
dev:
script: tsnd --cls --exit-child --ignore-watch node_modules --quiet --respawn --rs --transpile-only -r tsconfig-paths/register src
script: dotenv -e .env.dev -- tsnd --cls --exit-child --ignore-watch node_modules --quiet --respawn --rs --transpile-only -r tsconfig-paths/register src
description: Start in dev mode
prod:
script: NODE_ENV=production node -r module-alias/register dist
@@ -77,11 +77,14 @@ scripts:
once:
script: dotenv -e .env.test -- jest --config test/jest.config.ts
description: Run tests (once, coverage)
e2e:
script: dotenv -e .env.test -- jest --config test/jest.config.ts --selectProjects e2e --coverage=false
description: Run e2e tests
integration:
script: dotenv -e .env.test -- jest --config test/jest.config.ts --selectProjects integration --coverage=false
description: Run e2e tests
e2e:
script: dotenv -e .env.test -- jest --config test/jest.config.ts --selectProjects e2e --coverage=false
unit:
script: dotenv -e .env.test -- jest --config test/jest.config.ts --selectProjects unit --coverage=false
description: Run e2e tests
validate:

2
prisma/schema.prisma
View File

@@ -19,7 +19,7 @@ model User {
model Org {
id String @id @default(auto()) @map("_id") @db.ObjectId
ghName String @unique
ghName String? @unique
followers Int
followers7d Int?
repos String[]

14
scripts/get-committers-group.ts
View File

@@ -1,14 +0,0 @@
import { getCommittersGroup } from 'gh/get-committers-group'
const main = async () => {
console.log('Fetching gh group (top 100 DAOs with >= 10_000 followers)...')
const ghGroup = await getCommittersGroup()
console.log(`Fetched gh group of size: ${ghGroup.length}`, ghGroup)
}
main()
.then(() => process.exit(0))
.catch((err) => {
console.error(err)
process.exit(1)
})

38
scripts/get-spaces.ts
View File

@@ -1,38 +0,0 @@
import { hideBin } from 'yargs/helpers'
import yargs from 'yargs/yargs'
import { getSpaces } from 'snapshot/get-spaces'
const options = {
maxOrgs: {
alias: 's',
describe: 'number of org to fetch from snapshot.org. 0 means no limit',
type: 'number',
},
minFollowers: {
alias: 'm',
describe: 'DAO min number of followers on snapshot.org',
type: 'number',
},
}
// @ts-expect-error
const argv = yargs(hideBin(process.argv)).options(options).help().argv as {
minFollowers: number
maxOrgs: number
}
const main = async () => {
const spaces = await getSpaces({
maxOrgs: argv.maxOrgs,
minFollowers: argv.minFollowers,
})()
console.log({ spaces })
}
main()
.then(() => process.exit(0))
.catch((err) => {
console.error(err)
process.exit(1)
})

29
src/app.ts
View File

@@ -26,14 +26,35 @@ app.use(
'/whitelist',
Router()
.get('', whitelistController.getWhitelist.bind(whitelistController))
.get('/refresh', whitelistController.refresh.bind(whitelistController)),
.get('/refresh', whitelistController.refresh.bind(whitelistController))
.get(
'/daos',
whitelistController.getWhitelistedDaos.bind(whitelistController),
)
.get(
'/repos',
whitelistController.getWhitelistedRepos.bind(whitelistController),
),
)
app.use(
'/user',
'/gh-user',
Router()
.get('/:username', userController.getUser.bind(userController))
.get('/:username/refresh', userController.refresh.bind(userController)),
.get('/:ghUsername', userController.getUser.bind(userController))
.get('/:ghUsername/refresh', userController.refresh.bind(userController)),
)
app.use(
'/membership',
Router()
.get(
'/dao-voters/:address',
userController.belongsToVotersGroup.bind(userController),
)
.get(
'/gh-contributors/:ghUsername',
userController.belongsToGhContributorsGroup.bind(userController),
),
)
export { app }

29
src/controllers/User/index.ts
View File

@@ -8,17 +8,36 @@ export class UserController implements UserControllerInterface {
constructor(readonly userService: UserService) {}
async getUser(req: Request, res: Response) {
const { username } = req.params
const user = await this.userService.getUser(
username,
const { ghUsername } = req.params
const user = await this.userService.getGhUser(
ghUsername,
req.query.format as 'short' | 'long',
)
res.json(user)
}
async refresh(req: Request, res: Response) {
const { username } = req.params
const user = await this.userService.refresh(username)
const { ghUsername } = req.params
const user = await this.userService.refresh(ghUsername)
res.json(user)
}
async belongsToGhContributorsGroup(
req: Request,
res: Response,
): Promise<void> {
const { ghUsername } = req.params
const belongsToGhContributorsGroup = await this.userService.getGhUser(
ghUsername,
)
res.json(belongsToGhContributorsGroup)
}
async belongsToVotersGroup(req: Request, res: Response): Promise<void> {
const { address } = req.params
const belongsToVotersGroup = await this.userService.belongsToVotersGroup(
address,
)
res.json({ belongsToVotersGroup })
}
}

2
src/controllers/User/interface.ts
View File

@@ -3,4 +3,6 @@ import { Request, Response } from 'express'
export default interface UserControllerInterface {
refresh: (_: Request, res: Response) => Promise<void>
getUser: (req: Request, res: Response) => Promise<void>
belongsToVotersGroup: (req: Request, res: Response) => Promise<void>
belongsToGhContributorsGroup: (req: Request, res: Response) => Promise<void>
}

10
src/controllers/Whitelist/index.ts
View File

@@ -18,4 +18,14 @@ export class WhitelistController implements WhitelistControllerInterface {
)
res.json(orgs)
}
async getWhitelistedDaos(_: Request, res: Response): Promise<void> {
const daos = await this.whitelistService.getWhitelistedDaos()
res.json(daos)
}
async getWhitelistedRepos(_: Request, res: Response): Promise<void> {
const repos = await this.whitelistService.getWhitelistedRepos()
res.json(repos)
}
}

2
src/controllers/Whitelist/interface.ts
View File

@@ -3,4 +3,6 @@ import { Request, Response } from 'express'
export default interface WhitelistControllerInterface {
refresh: (_: Request, res: Response) => Promise<void>
getWhitelist: (_: Request, res: Response) => Promise<void>
getWhitelistedDaos: (_: Request, res: Response) => Promise<void>
getWhitelistedRepos: (_: Request, res: Response) => Promise<void>
}

46
src/daos/get-voters-group.ts
View File

@@ -1,46 +0,0 @@
import { votersQuery } from 'repositories/Snapshot/queries'
import { getSpaces } from 'snapshot/get-spaces'
import { ArraySet, getTime, minusOneMonth } from 'utils'
import { URLS } from '../constants'
export const getVotersGroup = async (
{
maxOrgs = 100,
minFollowers = 10_000,
since,
until = new Date(),
}: {
maxOrgs?: number
minFollowers?: number
since?: Date
until?: Date
} = {
maxOrgs: 100,
minFollowers: 10_000,
until: new Date(),
},
) => {
if (since === undefined) since = minusOneMonth(until)
const spacesIds = await getSpaces({ maxOrgs, minFollowers })()
const res = await fetch(URLS.SNAPSHOT_GQL, {
body: JSON.stringify({
query: votersQuery,
variables: {
created_gte: getTime(since),
created_lte: getTime(until),
space_in: spacesIds,
},
}),
headers: {
'Content-Type': 'application/json',
},
method: 'POST',
})
const { data } = await res.json()
const voters = (data.votes as Array<{ voter: string }>).map(
({ voter }) => voter,
)
return ArraySet(voters)
}

54
src/gh/get-committers-by-org.ts
View File

@@ -1,54 +0,0 @@
import { ok } from 'assert'
import { committersByOrgQuery } from 'repositories/Github/queries'
import { ArraySet, parseDate } from 'utils'
import { URLS } from '../constants'
export const getCommittersByOrg = async ({
org,
since,
until,
}: {
org: string
since: Date
until: Date
}) => {
ok(process.env.GH_PAT, 'GH_PAT is not defined')
const res = await fetch(URLS.GH_SQL, {
body: JSON.stringify({
query: committersByOrgQuery,
variables: {
login: org,
since: parseDate(since),
until: parseDate(until),
},
}),
headers: {
// eslint-disable-next-line @typescript-eslint/restrict-template-expressions
Authorization: `bearer ${process.env.GH_PAT}`,
'Content-Type': 'application/json',
},
method: 'POST',
})
const repos = (await res.json()).data?.organization?.repositories?.nodes
if (repos === undefined) return []
return ArraySet(
(repos as any[])
.reduce<string[][]>((repos, repo) => {
if (repo.defaultBranchRef !== null)
repos.push(
(repo.defaultBranchRef.target.history.nodes as any[]).reduce<
string[]
>((users, node) => {
const login: string = node.author.user?.login
if (login !== undefined) users.push(login)
return users
}, []),
)
return repos
}, [])
.flat(),
)
}

32
src/gh/get-committers-group.ts
View File

@@ -1,32 +0,0 @@
import { getGhOrgs } from 'snapshot/get-gh-orgs'
import { ArraySet, minusOneMonth, notBot } from 'utils'
import { getCommittersByOrg } from './get-committers-by-org'
export const getCommittersGroup = async (
{
maxOrgs = 100,
minFollowers = 10_000,
since,
until = new Date(),
}: {
maxOrgs?: number
minFollowers?: number
since?: Date
until?: Date
} = {
maxOrgs: 100,
minFollowers: 10_000,
until: new Date(),
},
) => {
if (since === undefined) since = minusOneMonth(until)
const orgs = await getGhOrgs({ maxOrgs, minFollowers })
const users = await Promise.all(
orgs.map(async (org) => {
// @ts-expect-error - type guarded earlier
return getCommittersByOrg({ org, since, until })
}),
)
return ArraySet(users.flat()).filter(notBot)
}

195
src/openapi.json
View File

@@ -2,11 +2,16 @@
"openapi": "3.0.3",
"info": {
"title": "Zkitter Groups - OpenAPI 3.0",
"description": "This is a sample Pet Store Server based on the OpenAPI 3.0 specification. \n\n[GitHub](https://github.com/zkitter/groups)",
"description": "Whitelisted GitHub and DAOs groups. \n\n[GitHub](https://github.com/zkitter/groups)",
"license": {
"name": "MIT",
"url": "https://github.com/zkitter/groups/blob/main/LICENSE"
},
"contact": {
"name": "r1oga",
"url": "https://github.com/r1oga",
"email": "me@r1oga.io"
},
"version": "1.0.0"
},
"servers": [
@@ -25,10 +30,14 @@
{
"name": "user",
"description": "GitHub User"
},
{
"name": "membership",
"description": "Check belonging to GitHub and/or DAO groups"
}
],
"paths": {
"/user/{username}": {
"/gh-user/{ghUsername}": {
"get": {
"tags": ["user"],
"summary": "Get user by GitHub login/username",
@@ -36,7 +45,7 @@
"operationId": "getUserByGhName",
"parameters": [
{
"name": "username",
"name": "ghUsername",
"in": "path",
"description": "The GitHub login that needs to be fetched.",
"required": true,
@@ -71,7 +80,7 @@
}
}
},
"/user/{username}/refresh": {
"/gh-user/{ghUsername}/refresh": {
"get": {
"tags": ["user"],
"summary": "Update list of repos the user contributed.",
@@ -79,7 +88,7 @@
"operationId": "refreshUserByGhName",
"parameters": [
{
"name": "username",
"name": "ghUsername",
"in": "path",
"description": "The GitHub login that needs to be refreshed.",
"required": true,
@@ -124,13 +133,70 @@
],
"responses": {
"200": {
"description": "sucessful operation",
"description": "successful operation",
"content": {
"application/json": {
"schema": {
"oneOf": [
{
"type": "array",
"items": {
"$ref": "#/components/schemas/OrgLong"
}
},
{
"$ref": "#/components/schemas/OrgShort"
}
]
}
}
}
}
}
}
},
"/whitelist/daos": {
"get": {
"tags": ["whitelist"],
"summary": "Get whitelisted DAOs.",
"description": "",
"operationId": "getWhitelistDaos",
"responses": {
"200": {
"description": "successful operation",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Org"
"description": "Id of the DAO on https://snapshot.org",
"type": "string",
"example": "aave.eth"
}
}
}
}
}
}
}
},
"/whitelist/repos": {
"get": {
"tags": ["whitelist"],
"summary": "Get whitelisted repos.",
"description": "",
"operationId": "getWhitelistRepos",
"responses": {
"200": {
"description": "successful operation",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"type": "string",
"example": "uniswap/v3-core",
"description": "Github repository name in the format of `org/repo`"
}
}
}
@@ -151,7 +217,90 @@
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/User"
"type": "array",
"items": {
"$ref": "#/components/schemas/OrgLong"
}
}
}
}
}
}
}
},
"/membership/dao-voters/{address}": {
"get": {
"tags": ["membership"],
"summary": "Check if an address belongs to the voters group.",
"description": "",
"operationId": "belongsToVotersGroup",
"parameters": [
{
"name": "address",
"in": "path",
"description": "The Ethereum address that needs to be checked.",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "successful operation",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"belongsToVotersGroup": {
"type": "boolean",
"example": true
}
},
"example": {
"belongsToVotersGroup": true
}
}
}
}
}
}
}
},
"/membership/gh-contributors/{ghUsername}": {
"get": {
"tags": ["membership"],
"summary": "Check if an address belongs to the GitHub Contributors group.",
"description": "",
"operationId": "belongsToGhContributorsGroup",
"parameters": [
{
"name": "ghUsername",
"in": "path",
"description": "The GitHub username that needs to be checked.",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "successful operation",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"belongsToGhContributorsGroup": {
"type": "boolean",
"example": true
}
},
"example": {
"belongsToGhContributorsGroup": true
}
}
}
}
@@ -182,7 +331,14 @@
}
}
},
"Org": {
"OrgLong": {
"required": [
"followers",
"ghName",
"repos",
"snapshotId",
"snapshotName"
],
"type": "object",
"properties": {
"ghName": {
@@ -203,12 +359,33 @@
},
"repos": {
"type": "array",
"description": "Array of repositories names that belong to the GitHub organization associated to this snapshot org.",
"items": {
"type": "string",
"example": ["treasure-docs", "treasure-subgraph"]
}
}
}
},
"OrgShort": {
"type": "object",
"required": ["daos", "repos"],
"properties": {
"daos": {
"type": "array",
"items": {
"type": "string",
"example": ["stgdao.eth", "aave.eth"]
}
},
"repos": {
"type": "array",
"items": {
"type": "string",
"example": ["stargate-protocol/stargate", "aave/aave-v3-core"]
}
}
}
}
}
}

83
src/repositories/Snapshot/index.ts
View File

@@ -1,8 +1,13 @@
import { Service } from 'typedi'
import { URLS } from '#'
import { ArraySet, getTime, minusOneMonth } from 'utils'
import {
Space,
SpaceGqlResponse,
SpaceRestResponse,
VoteResponse,
} from '../../types'
import SnapshotRepositoryInterface from './interface'
import { spacesQuery, votersQuery } from './queries'
import { ghNamesBySpaceIdsQuery, votedSpacesByAddress } from './queries'
@Service()
export class SnapshotRepository implements SnapshotRepositoryInterface {
@@ -27,43 +32,59 @@ export class SnapshotRepository implements SnapshotRepositoryInterface {
return res.json()
}
async getSpaces(): Promise<
Record<string, { name: string; followers?: number; followers_7d?: number }>
> {
async getSpaces(): Promise<Record<string, Space>> {
const res = await fetch(URLS.SNAPSHOT_EXPLORE)
const { spaces } = await res.json()
return spaces
}
async getGhOrgsBySpaceIds(
ids: string[],
): Promise<Array<{ ghName: unknown; snapshotId: string }>> {
const { data } = await this.gqlQuery(spacesQuery, { id_in: ids })
const spaces = data?.spaces ?? []
return (spaces as Array<{ github: string; id: string }>).map(
({ github: ghName, id: snapshotId }) => ({
ghName,
snapshotId,
}),
const { spaces }: { spaces: SpaceRestResponse[] } = await res.json()
return Object.entries(spaces).reduce<Record<string, Space>>(
(spaces, [snapshotId, spaceResponse]) => {
const space: Space = {
followers: spaceResponse.followers,
snapshotId,
snapshotName: spaceResponse.name,
}
if (spaceResponse.followers_7d !== undefined) {
space.followers7d = spaceResponse.followers_7d
}
spaces[snapshotId] = space
return spaces
},
{},
)
}
async getVoters(
async getGhNamesBySpaceIds(
ids: string[],
{ since, until = new Date(0) }: { since?: Date; until?: Date } = {
until: new Date(),
},
) {
if (since === undefined) since = minusOneMonth(until)
): Promise<Record<string, SpaceGqlResponse>> {
const { data } = await this.gqlQuery(ghNamesBySpaceIdsQuery, { ids })
return ((data?.spaces ?? []) as SpaceGqlResponse[]).reduce<
Record<string, SpaceGqlResponse>
>((spaces, { ghName, snapshotId }) => {
spaces[snapshotId] = { ghName, snapshotId }
return spaces
}, {})
}
const { data } = await this.gqlQuery(votersQuery, {
created_gte: getTime(since),
created_lte: getTime(until),
space_in: ids,
async getVotedSpacesByAddress({
address,
since,
until,
}: {
address: string
since: number
until: number
}): Promise<string[]> {
const { data } = await this.gqlQuery(votedSpacesByAddress, {
address,
since,
until,
})
return ArraySet(
(data.votes as Array<{ voter: string }>).map(({ voter }) => voter),
return (data.votes ?? ([] as VoteResponse[])).reduce(
(snapshotIds: string[], { space: { snapshotId } }: VoteResponse) => {
if (!snapshotIds.includes(snapshotId)) snapshotIds.push(snapshotId)
return snapshotIds
},
[],
)
}
}

21
src/repositories/Snapshot/interface.ts
View File

@@ -1,10 +1,17 @@
import { Space, SpaceGqlResponse } from '../../types'
export default interface SnapshotRepositoryInterface {
getSpaces: () => Promise<Record<string, any>>
getGhOrgsBySpaceIds: (
getSpaces: () => Promise<Record<string, Space>>
getGhNamesBySpaceIds: (
ids: string[],
) => Promise<Array<{ ghName: unknown; snapshotId: string }>>
getVoters: (
ids: string[],
{ since, until }: { since?: Date; until?: Date },
) => Promise<string[]>
) => Promise<Record<string, SpaceGqlResponse>>
getVotedSpacesByAddress: ({
address,
since,
until,
}: {
since: number
until: number
address: string
}) => Promise<string[]>
}

11
src/repositories/Snapshot/queries/gh-names-by-space-ids.ts Normal file
View File

@@ -0,0 +1,11 @@
export const ghNamesBySpaceIdsQuery = `
query GhNamesBySpaceIds($ids: [String]){
spaces(
first: 1000
where:{id_in: $ids}
) {
snapshotId: id
ghName: github
}
}
`

4
src/repositories/Snapshot/queries/index.ts
View File

@@ -2,5 +2,5 @@
* @file Automatically generated by barrelsby.
*/
export * from './spaces'
export * from './voters'
export * from './gh-names-by-space-ids'
export * from './voted-spaces-by-address'

10
src/repositories/Snapshot/queries/spaces.ts
View File

@@ -1,10 +0,0 @@
export const spacesQuery = `
query Spaces($id_in: [String]){
spaces(
where:{id_in: $id_in}
) {
id
github
}
}
`

16
src/repositories/Snapshot/queries/voted-spaces-by-address.ts Normal file
View File

@@ -0,0 +1,16 @@
export const votedSpacesByAddress = `
query VotedSpacesByAddress($address: String!, $since: Int!, $until: Int!) {
votes (
where: {
created_gte: $since,
created_lte: $until,
voter_in: [$address],
vp_state: "final"
}
) {
space {
snapshotId: id
}
}
}
`

16
src/repositories/Snapshot/queries/voters.ts
View File

@@ -1,16 +0,0 @@
export const votersQuery = `
query Voters($space_in: [String], $created_gte: Int, $created_lte: Int){
votes (
where: {
created_gte: $created_gte
created_lte: $created_lte
space_in: $space_in
vp_state: "final"
}
orderBy: "created",
orderDirection: desc
) {
voter
}
}
`

122
src/services/GroupService.ts
View File

@@ -1,122 +0,0 @@
import { Service } from 'typedi'
import { GithubRepository, SnapshotRepository } from 'repositories'
import { ArraySet, filterSpaces, minusOneMonth, notBot, split } from 'utils'
import { Space } from '../types'
@Service()
export class GroupService {
constructor(
readonly ghRepository: GithubRepository,
readonly snapshotRepository: SnapshotRepository,
) {}
private async getSpaceIds(
{
maxOrgs = 100,
minFollowers = 10_000,
}: { minFollowers: number; maxOrgs: number } = {
maxOrgs: 100,
minFollowers: 10_000,
},
) {
const spaces = await this.snapshotRepository.getSpaces()
// @ts-expect-error
return Object.entries(spaces as Space[])
.reduce<Array<{ id: string; followers: number }>>(
(spaces, [id, space]) => {
if (filterSpaces(minFollowers)(space)) {
spaces.push({ followers: space.followers, id })
}
return spaces
},
[],
)
.sort((a, b) => b.followers - a.followers)
.slice(0, maxOrgs)
.map(({ id }) => id)
}
private async getGhOrgsChunk(ids: string[]) {
const spaces = await this.snapshotRepository.getGhOrgsBySpaceIds(ids)
return spaces.reduce<string[]>((spaces, github) => {
// @ts-expect-error
if (github !== null) spaces.push(github)
return spaces
}, [])
}
private async getGhOrgs(
{
maxOrgs = 100,
minFollowers = 10_000,
}: {
minFollowers: number
maxOrgs: number
} = { maxOrgs: 100, minFollowers: 10_000 },
): Promise<string[]> {
const spacesIds = await this.getSpaceIds({ maxOrgs, minFollowers })
const orgs = await Promise.all(split(spacesIds).map(this.getGhOrgsChunk))
return ArraySet(orgs.flat())
}
async getCommittersGroup(
{
maxOrgs = 100,
minFollowers = 10_000,
since,
until = new Date(),
}: {
maxOrgs?: number
minFollowers?: number
since?: Date
until?: Date
} = {
maxOrgs: 100,
minFollowers: 10_000,
until: new Date(),
},
) {
if (since === undefined) since = minusOneMonth(until)
const orgs = await this.getGhOrgs({ maxOrgs, minFollowers })
const nodes = (
await Promise.all(
orgs.map(async (org) =>
// @ts-expect-error - since already type guarded
this.ghRepository.getCommittersByOrg({
org,
since,
until,
}),
),
)
).flat()
const committers = nodes
.reduce<string[][]>((repos, repo) => {
// @ts-expect-error
if (repo.defaultBranchRef !== null)
repos.push(
// @ts-expect-error
(repo.defaultBranchRef.target.history.nodes as any[]).reduce<
string[]
>((users, node) => {
const login: string = node.author.user?.login
if (login !== undefined) users.push(login)
return users
}, []),
)
return repos
}, [])
.flat()
return ArraySet(committers).filter(notBot)
}
// async getVotersGroup() {}
// async createUser() {}
// async createOrg() {}
// async findOneUser(){}
// async findOneOrg(){}
}

55
src/services/User/index.ts
View File

@@ -1,7 +1,11 @@
import { Service } from 'typedi'
import { GithubRepository, MongoRepository } from 'repositories'
import { GroupsData, UserData } from 'types'
import { intersect } from 'utils'
import {
GithubRepository,
MongoRepository,
SnapshotRepository,
} from 'repositories'
import { UserData } from 'types'
import { getTime, intersect, minusOneMonth } from 'utils'
import { WhitelistService } from '../Whitelist'
import UserServiceInterface from './interface'
@@ -10,6 +14,7 @@ export class UserService implements UserServiceInterface {
constructor(
readonly gh: GithubRepository,
readonly db: MongoRepository,
readonly snapshot: SnapshotRepository,
readonly whitelist: WhitelistService,
) {}
@@ -18,30 +23,44 @@ export class UserService implements UserServiceInterface {
return repos.map(({ name, org }) => `${org}/${name}`)
}
async getVotedOrgs({
address,
since,
until = new Date(),
}: {
address: string
since?: Date
until?: Date
}) {
if (since === undefined) since = minusOneMonth(until)
return this.snapshot.getVotedSpacesByAddress({
address,
since: getTime(since),
until: getTime(until),
})
}
async belongsToGhContributorsGroup(user: UserData | null): Promise<boolean> {
if (user !== null) {
const whitelistedRepos = await this.whitelist.getWhitelistShort()
return intersect(whitelistedRepos, user.repos)
const { repos } = await this.whitelist.getWhitelistShort()
return intersect(repos, user.repos)
}
return false
}
async belongsToVotersGroup(user: UserData | null): Promise<boolean> {
return Promise.resolve(false)
async belongsToVotersGroup(address: string): Promise<boolean> {
const votedOrgs = await this.getVotedOrgs({ address })
const { daos } = await this.whitelist.getWhitelistShort()
console.log({ address, daos, votedOrgs })
return intersect(daos, votedOrgs)
}
async getGroups(user: UserData | null): Promise<GroupsData> {
const [belongsToGhContributorsGroup] = await Promise.all([
this.belongsToGhContributorsGroup(user),
])
return { belongsToGhContributorsGroup }
}
async getUser(ghName: string, format: 'short' | 'long' = 'short') {
async getGhUser(ghName: string, format: 'short' | 'long' = 'short') {
const user = await this.db.findUser(ghName)
const groups = await this.getGroups(user)
if (format === 'short') return groups
return { ...groups, ...user }
const belongsToGhContributorsGroup =
user === null ? false : await this.belongsToGhContributorsGroup(user)
if (format === 'short') return { belongsToGhContributorsGroup }
return { ...user, belongsToGhContributorsGroup }
}
async refresh(ghName: string) {

14
src/services/User/interface.ts
View File

@@ -2,9 +2,17 @@ import { GroupsData, UserData } from '../../types'
export default interface UserServiceInterface {
getContributedRepos: (username: string) => Promise<string[]>
getVotedOrgs: ({
address,
since,
until,
}: {
address: string
since?: Date
until?: Date
}) => Promise<string[]>
belongsToGhContributorsGroup: (user: UserData | null) => Promise<boolean>
belongsToVotersGroup: (user: UserData | null) => Promise<boolean>
getGroups: (user: UserData | null) => Promise<GroupsData>
getUser: (username: string) => Promise<UserData | GroupsData | null>
belongsToVotersGroup: (address: string) => Promise<boolean>
getGhUser: (username: string) => Promise<UserData | GroupsData | null>
refresh: (username: string) => Promise<UserData>
}

3
src/services/Whitelist/ignore-list.ts Normal file
View File

@@ -0,0 +1,3 @@
export const IGNORE_SPACES = [
'treasuredao.eth', // https://snapshot.org/#/treasuredao.eth inactive space
]

93
src/services/Whitelist/index.ts
View File

@@ -26,20 +26,10 @@ export class WhitelistService implements WhitelistServiceInterface {
} = { maxOrgs: 100, minFollowers: 10_000 },
) {
const spaces = await this.snapshot.getSpaces()
return Object.entries(spaces)
.reduce<Space[]>((spaces, [snapshotId, space]) => {
return Object.values(spaces)
.reduce<Space[]>((spaces, space) => {
if (filterSpaces(minFollowers)(space)) {
const _space = {
followers: space.followers as number,
snapshotId,
snapshotName: space.name,
}
if (space.followers_7d !== undefined) {
// @ts-expect-error
_space.followers7d = space.followers_7d
}
spaces.push(_space)
spaces.push(space)
}
return spaces
}, [])
@@ -51,18 +41,6 @@ export class WhitelistService implements WhitelistServiceInterface {
}, {})
}
async getGhOrgs(snapshotNames: string[]) {
const spaces = await this.snapshot.getGhOrgsBySpaceIds(snapshotNames)
return spaces.reduce<Array<{ ghName: string; snapshotId: string }>>(
(spaces, space) => {
if (typeof space.ghName === 'string')
spaces.push({ ghName: space.ghName, snapshotId: space.snapshotId })
return spaces
},
[],
)
}
async getOrgsWithRepos(
{
maxOrgs,
@@ -73,18 +51,33 @@ export class WhitelistService implements WhitelistServiceInterface {
} = { maxOrgs: 100, minFollowers: 10_000 },
) {
const spaces = await this.getSpaces({ maxOrgs, minFollowers })
const orgs: Record<string, OrgData> = {}
const ghNames = await this.snapshot.getGhNamesBySpaceIds(
Object.keys(spaces),
)
const orgs = Object.entries(spaces).reduce<Record<string, OrgData>>(
(orgs, [snapshotId, space]) => {
orgs[snapshotId] = {
...space,
ghName: ghNames[snapshotId]?.ghName ?? null,
}
return orgs
},
{},
)
await Promise.all(
split(Object.keys(spaces)).map(async (snapshotNames) => {
const ghOrgs = await this.getGhOrgs(snapshotNames)
split<{ ghName: string; snapshotId: string }>(
Object.values(orgs).reduce<
Array<{ ghName: string; snapshotId: string }>
>((ghNames, { ghName, snapshotId }) => {
if (ghName !== null) ghNames.push({ ghName, snapshotId })
return ghNames
}, []),
).map(async (ghNames) => {
await Promise.all(
ghOrgs.map(async ({ ghName, snapshotId }) => {
ghNames.map(async ({ ghName, snapshotId }) => {
const repos = await this.gh.getReposByOrg(ghName)
if (repos.length > 0) {
orgs[snapshotId] = { ...spaces[snapshotId], ghName, repos }
}
if (repos.length > 0) orgs[snapshotId].repos = repos
}),
)
}),
@@ -94,15 +87,37 @@ export class WhitelistService implements WhitelistServiceInterface {
}
async getWhitelistShort() {
const orgs = await this.db.findAllWhitelistedOrgs()
return orgs
.map(({ ghName, repos }) => repos.map((repo) => `${ghName}/${repo}`))
.flat()
return (await this.getWhitelist('short')) as {
daos: string[]
repos: string[]
}
}
async getWhitelist(format: 'short' | 'long' = 'short') {
if (format === 'long') return this.db.findAllWhitelistedOrgs()
return this.getWhitelistShort()
const orgs = await this.db.findAllWhitelistedOrgs()
if (format === 'long') {
return orgs
} else {
return orgs.reduce<{ daos: string[]; repos: string[] }>(
(orgs, { ghName, repos, snapshotId }) => {
orgs.daos.push(snapshotId)
if (ghName !== null)
orgs.repos.push(...repos.map((repo) => `${ghName}/${repo}`))
return orgs
},
{ daos: [], repos: [] },
)
}
}
async getWhitelistedDaos(): Promise<string[]> {
const { daos } = await this.getWhitelistShort()
return daos
}
async getWhitelistedRepos(): Promise<string[]> {
const { repos } = await this.getWhitelistShort()
return repos
}
async refresh() {

14
src/services/Whitelist/interface.ts
View File

@@ -10,10 +10,6 @@ export default interface WhitelistServiceInterface {
minFollowers?: number
}) => Promise<Record<string, Space>>
getGhOrgs: (
snapshotNames: string[],
) => Promise<Array<{ ghName: string; snapshotId: string }>>
getOrgsWithRepos: ({
maxOrgs,
minFollowers,
@@ -22,7 +18,13 @@ export default interface WhitelistServiceInterface {
minFollowers?: number
}) => Promise<Record<string, OrgData>>
unWhitelist: (ghNameOrSnapshotId: string) => Promise<Org>
getWhitelistShort: () => Promise<string[]>
getWhitelist: (format: 'short' | 'long') => Promise<OrgData[] | string[]>
getWhitelist: (
format: 'short' | 'long',
) => Promise<OrgData[] | { daos: string[]; repos: string[] }>
getWhitelistShort: (
format: 'short' | 'long',
) => Promise<{ daos: string[]; repos: string[] }>
getWhitelistedDaos: () => Promise<string[]>
getWhitelistedRepos: () => Promise<string[]>
refresh: () => Promise<OrgData[]>
}

1
src/services/index.ts
View File

@@ -2,6 +2,5 @@
* @file Automatically generated by barrelsby.
*/
export * from './GroupService'
export * from './User/index'
export * from './Whitelist/index'

55
src/snapshot/get-gh-orgs.ts
View File

@@ -1,55 +0,0 @@
import { CHUNK_SIZE, URLS } from '#'
import { spacesQuery } from 'repositories/Snapshot/queries'
import { getSpaces } from './get-spaces'
const split = (arr: string[]) => {
const chunks = []
for (let i = 0; i < arr.length; i += CHUNK_SIZE) {
chunks.push(arr.slice(i, i + CHUNK_SIZE))
}
return chunks
}
const getChunk = async (ids: string[]) => {
const res = await fetch(URLS.SNAPSHOT_GQL, {
body: JSON.stringify({
query: spacesQuery,
variables: { id_in: ids },
}),
headers: {
'Content-Type': 'application/json',
},
method: 'POST',
})
const { data } = await res.json()
return (data.spaces as Array<{ github: string }>).reduce<string[]>(
(spaces, { github }) => {
if (github !== null) spaces.push(github)
return spaces
},
[],
)
}
export const getGhOrgs = async (
{
maxOrgs = 100,
minFollowers = 10_000,
}: {
minFollowers: number
maxOrgs: number
} = { maxOrgs: 100, minFollowers: 10_000 },
): Promise<string[]> => {
const spacesIds = await getSpaces({ maxOrgs, minFollowers })()
const result = new Set<string>()
await Promise.all(
split(spacesIds).map(async (ids) => {
const chunk = await getChunk(ids)
chunk.forEach((org) => result.add(org))
}),
)
return Array.from(result)
}

35
src/snapshot/get-spaces.ts
View File

@@ -1,35 +0,0 @@
import { Space } from 'types'
import { URLS } from '../constants'
import { filterSpaces } from '../utils'
export const getSpaces =
(
{
maxOrgs = 100,
minFollowers = 10_000,
}: { minFollowers: number; maxOrgs: number } = {
maxOrgs: 100,
minFollowers: 10_000,
},
) =>
async () => {
const res = await fetch(URLS.SNAPSHOT_EXPLORE)
const { spaces } = await res.json()
return Object.entries(spaces as Space[])
.reduce<Array<{ id: string; followers: number }>>(
(spaces, [id, space]) => {
if (filterSpaces(minFollowers)(space)) {
spaces.push({ followers: space.followers, id })
}
return spaces
},
[],
)
.sort((a, b) => b.followers - a.followers)
.slice(0, maxOrgs)
.map(({ id }) => id)
}
export const get100TopDaosWithMin10kFollowers = getSpaces()

23
src/types.ts
View File

@@ -3,8 +3,6 @@ export interface Space {
followers7d?: number
snapshotId: string
snapshotName: string
ghName?: string
repos?: string[]
}
export interface OrgData {
@@ -12,8 +10,8 @@ export interface OrgData {
followers7d?: number
snapshotId: string
snapshotName: string
ghName: string
repos: string[]
ghName: string | null // keep or null because optional in Prisma schema
repos?: string[]
}
export interface UserData {
@@ -24,3 +22,20 @@ export interface UserData {
export interface GroupsData {
belongsToGhContributorsGroup: boolean
}
export interface SpaceRestResponse {
followers: number
followers_7d?: number
name: string
}
export interface SpaceGqlResponse {
snapshotId: string
ghName: string | null
}
export interface VoteResponse {
space: {
snapshotId: string
}
}

18
src/utils.ts
View File

@@ -1,6 +1,7 @@
import { Request, Response } from 'express'
import { body, validationResult } from 'express-validator'
import { CHUNK_SIZE } from '#'
import { IGNORE_SPACES } from './services/Whitelist/ignore-list'
export const minusOneMonth = (date: Date) =>
new Date(new Date(date).setMonth(date.getMonth() - 1))
@@ -13,18 +14,23 @@ export const notBot = (str: string) => !str.includes('[bot]')
export const parseDate = (date: Date) => date.toISOString().split('.')[0] + 'Z'
export const split = (arr: string[]) => {
const chunks = []
for (let i = 0; i < arr.length; i += CHUNK_SIZE) {
chunks.push(arr.slice(i, i + CHUNK_SIZE))
export const split = <T>(arr: T[], chunkSize = CHUNK_SIZE): T[][] => {
const chunks: T[][] = []
for (let i = 0; i < arr.length; i += chunkSize) {
chunks.push(arr.slice(i, i + chunkSize))
}
return chunks
}
const isIncluded = (ignoreList: string[]) => (str?: string) =>
str === undefined ? true : !ignoreList.includes(str)
export const filterSpaces =
(minFollowers: number) =>
({ followers }: any) =>
followers !== undefined && followers >= minFollowers
({ followers, snapshotId }: any) =>
followers !== undefined &&
followers >= minFollowers &&
isIncluded(IGNORE_SPACES)(snapshotId)
const validations = [
body('maxOrgs').if(body('maxOrgs').exists()).isInt(),

56
test/e2e/user-routes.test.ts
View File

@@ -4,13 +4,16 @@ import { app } from 'app'
import { MongoRepository } from 'repositories'
import { UserService } from 'services'
// beforeAll(async () => {
//
// })
describe('UserController', () => {
const userService = Container.get(UserService)
const mongoRepository = Container.get(MongoRepository)
describe('GET /user/:username', () => {
describe('GET /gh-user/:gh_username', () => {
it('returns user in short format by default', async () => {
const { body } = await request(app).get('/user/r1oga').send()
const { body } = await request(app).get('/gh-user/r1oga').send()
expect(body).toMatchObject({
belongsToGhContributorsGroup: expect.any(Boolean),
@@ -18,7 +21,9 @@ describe('UserController', () => {
})
it('can return user in long format', async () => {
const { body } = await request(app).get('/user/r1oga?format=long').send()
const { body } = await request(app)
.get('/gh-user/r1oga?format=long')
.send()
expect(body).toMatchObject({
belongsToGhContributorsGroup: expect.any(Boolean),
@@ -29,12 +34,12 @@ describe('UserController', () => {
})
})
describe('GET /user/:username/refresh', () => {
describe('GET /gh-user/:gh_username/refresh', () => {
it('updates the user in the DB and returns the updated user', async () => {
jest.spyOn(userService, 'refresh')
jest.spyOn(mongoRepository, 'upsertUser')
const { body } = await request(app).get('/user/r1oga/refresh').send()
const { body } = await request(app).get('/gh-user/r1oga/refresh').send()
expect(userService.refresh).toHaveBeenCalledOnce()
expect(mongoRepository.upsertUser).toHaveBeenCalledOnce()
@@ -45,4 +50,45 @@ describe('UserController', () => {
expect(body.repos).toInclude('zkitter/groups')
})
})
describe('GET /membership/gh-contributors/:gh_username', () => {
it('returns true if user belongs to the GH contributors group', async () => {
await request(app).get('/gh-user/NoahZinsmeister/refresh').send()
const { body } = await request(app)
.get('/membership/gh-contributors/NoahZinsmeister')
.send()
expect(body).toEqual({ belongsToGhContributorsGroup: true })
})
it('returns false if user does not belong to the GH contributors group', async () => {
const { body } = await request(app)
.get('/membership/gh-contributors/r1oga')
.send()
expect(body).toEqual({ belongsToGhContributorsGroup: false })
})
})
describe('GET /membership/voters/:address', () => {
it('returns false if user does not belongs to the voters group', async () => {
const { body } = await request(app)
.get(
'/membership/dao-voters/0xF411903cbC70a74d22900a5DE66A2dda66507255',
)
.send()
expect(body).toEqual({ belongsToVotersGroup: false })
})
it('returns true if user belongs to the voters group', async () => {
const { body } = await request(app)
.get(
'/membership/dao-voters/0x329c54289Ff5D6B7b7daE13592C6B1EDA1543eD4',
) // aavechan.eth
.send()
expect(body).toEqual({ belongsToVotersGroup: true })
})
})
})

54
test/e2e/whitelist-routes.test.ts
View File

@@ -1,12 +1,14 @@
import { faker } from '@faker-js/faker'
import request from 'supertest'
import { Container } from 'typedi'
import { app } from 'app'
import { MongoRepository } from 'repositories'
import { WhitelistService } from 'services/Whitelist'
describe('refresh handler', () => {
const whitelistService = Container.get(WhitelistService)
const mongoRepository = Container.get(MongoRepository)
const mongoRepository = Container.get(MongoRepository)
const whitelistService = Container.get(WhitelistService)
describe('Whitelist Controller', () => {
jest.spyOn(whitelistService, 'getWhitelist')
jest.spyOn(mongoRepository, 'findAllWhitelistedOrgs')
@@ -16,23 +18,30 @@ describe('refresh handler', () => {
expect(whitelistService.getWhitelist).toHaveBeenCalledOnce()
expect(mongoRepository.findAllWhitelistedOrgs).toHaveBeenCalledOnce()
expect(body).toBeInstanceOf(Array)
expect(body[0]).toBeString().not.toBeEmpty()
expect(body).toMatchObject({
daos: expect.any(Array<string>),
repos: expect.any(Array<string>),
})
expect(body.daos.includes('opcollective.eth')).toBeTrue()
expect(body.repos.includes('aave/aave-js')).toBeTrue()
})
it('can return list of whitelisted orgs in long format', async () => {
const { body } = await request(app).get('/whitelist?format=long').send()
const org = body[faker.datatype.number({ max: body.length - 1, min: 0 })]
expect(whitelistService.getWhitelist).toHaveBeenCalledOnce()
expect(mongoRepository.findAllWhitelistedOrgs).toHaveBeenCalledOnce()
expect(body).toBeInstanceOf(Array)
expect(body[0]).toMatchObject({
expect(org).toMatchObject({
followers: expect.any(Number),
ghName: expect.any(String),
repos: expect.any(Array<string>),
snapshotId: expect.any(String),
snapshotName: expect.any(String),
})
expect(org.ghName).toSatisfy(
(ghName: any) => ghName === null || typeof ghName === 'string',
)
expect(org.repos).toBeArray()
})
it('GET /whitelist/refresh: should update and return the list of whitelisted orgs', async () => {
@@ -40,17 +49,40 @@ describe('refresh handler', () => {
jest.spyOn(mongoRepository, 'upsertOrgs')
const { body } = await request(app).get('/whitelist/refresh').send()
const org = body[faker.datatype.number({ max: body.length - 1, min: 0 })]
expect(whitelistService.refresh).toHaveBeenCalledOnce()
expect(mongoRepository.upsertOrgs).toHaveBeenCalledOnce()
expect(body).toBeInstanceOf(Array)
expect(body[0]).toMatchObject({
expect(org).toMatchObject({
followers: expect.any(Number),
ghName: expect.any(String),
repos: expect.any(Array<string>),
snapshotId: expect.any(String),
snapshotName: expect.any(String),
})
expect(org.ghName).toSatisfy(
(ghName: any) => ghName === null || typeof ghName === 'string',
)
expect(org.repos).toBeArray()
})
it('GET /whitelist/daos: should return the list of whitelisted DAOs', async () => {
const { body } = await request(app).get('/whitelist/daos').send()
expect(body).toBeInstanceOf(Array)
expect(body[faker.datatype.number({ max: body.length - 1, min: 0 })])
.toBeString()
.not.toBeEmpty()
expect(body.includes('opcollective.eth')).toBeTrue()
})
it('GET /whitelist/repos: should return the list of whitelisted repos', async () => {
const { body } = await request(app).get('/whitelist/repos').send()
expect(body).toBeInstanceOf(Array)
expect(body[faker.datatype.number({ max: body.length - 1, min: 0 })])
.toBeString()
.not.toBeEmpty()
expect(body.includes('aave/aave-js')).toBeTrue()
})
})
})

53
test/integration/snapshot-repository.test.ts
View File

@@ -1,7 +1,5 @@
import { SnapshotRepository } from 'repositories'
const SPACE_IDS = ['stgdao.eth', 'ens.eth', 'aave.eth', 'uniswap']
describe('SnapshotRepository', () => {
let snapshotRepository: SnapshotRepository
beforeEach(() => {
@@ -12,40 +10,45 @@ describe('SnapshotRepository', () => {
const spaces = await snapshotRepository.getSpaces()
expect(spaces).toBeObject().not.toBeEmpty()
expect(Object.values(spaces)).toBeArray().not.toBeEmpty()
expect(spaces['opcollective.eth']).toEqual(
expect.objectContaining({
followers: expect.any(Number),
followers_7d: expect.any(Number),
name: expect.any(String),
followers7d: expect.any(Number),
snapshotId: expect.any(String),
snapshotName: expect.any(String),
}),
)
})
it('gets github orgs by space ids', async () => {
const spaces = await snapshotRepository.getGhOrgsBySpaceIds(SPACE_IDS)
it('gets github org names by space ids', async () => {
const SPACE_IDS = ['stgdao.eth', 'ens.eth', 'aave.eth', 'uniswap']
const GH_NAMES = ['stargate-protocol', 'ensdomains', 'aave', 'Uniswap']
const spaces = await snapshotRepository.getGhNamesBySpaceIds(SPACE_IDS)
expect(spaces)
.toBeArray()
.not.toBeEmpty()
.toIncludeAllMembers([
{ ghName: 'stargate-protocol', snapshotId: 'stgdao.eth' },
{ ghName: 'aave', snapshotId: 'aave.eth' },
{ ghName: 'Uniswap', snapshotId: 'uniswap' },
{ ghName: 'ensdomains', snapshotId: 'ens.eth' },
])
spaces.forEach(({ ghName, snapshotId }) => {
expect(ghName).toBeString().not.toBeEmpty()
expect(snapshotId).toBeString().not.toBeEmpty()
})
expect(spaces).toMatchObject(
Object.fromEntries(
SPACE_IDS.map((snapshotId, i) => [
snapshotId,
{
ghName: GH_NAMES[i],
snapshotId,
},
]),
),
)
})
it('gets voters by space ids', async () => {
const voters = await snapshotRepository.getVoters(SPACE_IDS)
it('gets ids of the spaces an address voted to', async () => {
const spaceIds = await snapshotRepository.getVotedSpacesByAddress({
address: '0x329c54289Ff5D6B7b7daE13592C6B1EDA1543eD4',
since: 1674836517,
until: 1677514917,
})
expect(voters).toBeArray().not.toBeEmpty()
voters.forEach((voter) => {
expect(voter).toBeString().not.toBeEmpty().toStartWith('0x')
expect(spaceIds).toBeArray().not.toBeEmpty()
expect(spaceIds).toInclude('aave.eth')
spaceIds.forEach((spaceId) => {
expect(spaceId).toBeString().not.toBeEmpty()
})
})
})

2
test/jest.config.ts
View File

@@ -18,7 +18,7 @@ const jestConfig: JestConfigWithTsJest = {
'./test/jest.prettier.ts',
'./test/jest.unit.ts',
],
testTimeout: 30_000,
testTimeout: 60_000,
watchPlugins: [
'jest-watch-select-projects',
'jest-watch-typeahead/filename',

10
test/unit/filter-spaces.test.ts
View File

@@ -2,7 +2,7 @@ import { filterSpaces } from '../../src/utils'
describe('filterSpaces', () => {
const SPACE = { snapshotId: 'space.eth', snapshotName: 'space' }
it('should return true if followers is greater than min', () => {
it('should return true if followers is greater than min and not on the ignore list', () => {
const followers = 100
const min = 10
expect(filterSpaces(min)({ followers, ...SPACE })).toBe(true)
@@ -17,4 +17,12 @@ describe('filterSpaces', () => {
const min = 100
expect(filterSpaces(min)(SPACE)).toBe(false)
})
it('should return false if snapshotId in on the ignore list', () => {
const followers = 100
const min = 10
expect(
filterSpaces(min)({ followers, snapshotId: 'treasuredao.eth' }),
).toBe(false)
})
})

100
test/unit/user-service.test.ts
View File

@@ -1,6 +1,12 @@
import { TestBed } from '@automock/jest'
import { GithubRepository, MongoRepository } from 'repositories'
import { faker } from '@faker-js/faker'
import {
GithubRepository,
MongoRepository,
SnapshotRepository,
} from 'repositories'
import { UserService, WhitelistService } from 'services'
import { getTime } from '../../src/utils'
describe('UserService', () => {
const REPOS = [
@@ -14,6 +20,7 @@ describe('UserService', () => {
let userService: UserService
let db: jest.Mocked<MongoRepository>
let gh: jest.Mocked<GithubRepository>
let snapshot: jest.Mocked<SnapshotRepository>
let whitelist: jest.Mocked<WhitelistService>
beforeEach(() => {
@@ -21,10 +28,11 @@ describe('UserService', () => {
userService = unit
gh = unitRef.get(GithubRepository)
db = unitRef.get(MongoRepository)
snapshot = unitRef.get(SnapshotRepository)
whitelist = unitRef.get(WhitelistService)
})
it('getContributedRepositories: fetches from GH and returns contributed repos', async () => {
it('getContributedRepos: fetches from GH and returns repos a user has contributed to', async () => {
gh.getContributedRepos.mockResolvedValueOnce(REPOS)
await expect(userService.getContributedRepos('foo')).resolves.toEqual([
@@ -34,9 +42,30 @@ describe('UserService', () => {
expect(gh.getContributedRepos).toHaveBeenCalledOnceWith('foo')
})
it('getVotedOrgs: fetches from Snapshot and returns orgs a user has voted to', async () => {
const SPACE_IDS = ['space1', 'space2']
const address = faker.finance.ethereumAddress()
const since = faker.date.past()
const until = faker.date.future()
snapshot.getVotedSpacesByAddress.mockResolvedValueOnce(SPACE_IDS)
await expect(
userService.getVotedOrgs({ address, since, until }),
).resolves.toEqual(SPACE_IDS)
expect(snapshot.getVotedSpacesByAddress).toHaveBeenCalledOnceWith({
address,
since: getTime(since),
until: getTime(until),
})
})
describe('belongsToContributorsGroup', () => {
it('returns true if user has contributed to a whitelisted repo', async () => {
whitelist.getWhitelistShort.mockResolvedValueOnce(['A/a'])
whitelist.getWhitelistShort.mockResolvedValueOnce({
daos: [],
repos: ['A/a'],
})
await expect(
userService.belongsToGhContributorsGroup({
@@ -49,7 +78,10 @@ describe('UserService', () => {
})
it('returns false if user has not contributed to a whitelisted repo', async () => {
whitelist.getWhitelistShort.mockResolvedValueOnce(['C/a'])
whitelist.getWhitelistShort.mockResolvedValueOnce({
daos: [],
repos: ['C/a'],
})
await expect(
userService.belongsToGhContributorsGroup({
@@ -62,6 +94,40 @@ describe('UserService', () => {
})
})
describe('belongsToVotersGroup', () => {
it('returns true if user has voted to a whitelisted org', async () => {
whitelist.getWhitelistShort.mockResolvedValueOnce({
daos: ['A'],
repos: [],
})
snapshot.getVotedSpacesByAddress.mockResolvedValueOnce(['A'])
await expect(userService.belongsToVotersGroup('0x123')).resolves.toBe(
true,
)
expect(whitelist.getWhitelistShort).toHaveBeenCalledOnce()
expect(snapshot.getVotedSpacesByAddress).toHaveBeenCalledOnceWith({
address: '0x123',
since: expect.any(Number),
until: expect.any(Number),
})
})
it('returns false if user has not voted to a whitelisted org', async () => {
whitelist.getWhitelistShort.mockResolvedValueOnce({
daos: ['C'],
repos: [],
})
await expect(userService.belongsToVotersGroup('0x123')).resolves.toBe(
false,
)
expect(whitelist.getWhitelistShort).toHaveBeenCalledOnce()
})
})
describe('refresh', () => {
it('fetches from GH and stores in DB', async () => {
gh.getContributedRepos.mockResolvedValueOnce(REPOS)
@@ -74,29 +140,15 @@ describe('UserService', () => {
})
})
describe('getGroups', () => {
it('returns data about the groups the user is part of', async () => {
whitelist.getWhitelistShort.mockResolvedValueOnce(['A/a'])
await expect(
userService.getGroups({
ghName: 'foo',
repos: ['A/a', 'B/b'],
}),
).resolves.toEqual({
belongsToGhContributorsGroup: true,
})
expect(whitelist.getWhitelistShort).toHaveBeenCalledOnce()
})
})
describe('getUser', () => {
describe('getGhUser', () => {
it('can return user data in longer format', async () => {
db.findUser.mockResolvedValueOnce(USER_DATA)
whitelist.getWhitelistShort.mockResolvedValueOnce(['A/a'])
whitelist.getWhitelistShort.mockResolvedValueOnce({
daos: [],
repos: ['A/a'],
})
await expect(userService.getUser('foo', 'long')).resolves.toEqual({
await expect(userService.getGhUser('foo', 'long')).resolves.toEqual({
belongsToGhContributorsGroup: true,
...USER_DATA,
})

25
test/unit/utils.test.ts
View File

@@ -1,4 +1,11 @@
import { ArraySet, getTime, intersect, minusOneMonth, notBot } from 'utils'
import {
ArraySet,
getTime,
intersect,
minusOneMonth,
notBot,
split,
} from 'utils'
describe('utils', () => {
describe('ArraySet', () => {
@@ -39,5 +46,21 @@ describe('utils', () => {
it('should return false if the two arrays have no element in common', () => {
expect(intersect(['a', 'b', 'c'], ['d', 'e', 'f'])).toBe(false)
})
it('should return false if one of the arrays is empty', () => {
expect(intersect([], ['d', 'e', 'f'])).toBe(false)
expect(intersect(['a', 'b', 'c'], [])).toBe(false)
})
})
describe('split', () => {
it('should return a list of arrays with a given length', () => {
const arr = ['a', 'b', 'c', 'd', 'e', 'f', 'g']
const chunkSize = 3
const chunks = split(arr, chunkSize)
chunks.forEach((chunk) =>
expect(chunk.length).toBeLessThanOrEqual(chunkSize),
)
})
})
})

108
test/unit/whitelist-service.test.ts
View File

@@ -1,23 +1,46 @@
import { TestBed } from '@automock/jest'
import { GithubRepository, SnapshotRepository } from 'repositories'
import {
GithubRepository,
MongoRepository,
SnapshotRepository,
} from 'repositories'
import { WhitelistService } from 'services/Whitelist'
const ORGS = [
{
followers: 10,
ghName: 'a',
repos: ['aa', 'ab'],
snapshotId: 'a.eth',
snapshotName: 'A',
},
{
followers: 100,
ghName: 'b',
repos: ['ba', 'bb'],
snapshotId: 'b.eth',
snapshotName: 'B',
},
]
describe('WhitelistService', () => {
let whitelistService: WhitelistService
let gh: jest.Mocked<GithubRepository>
let snapshot: jest.Mocked<SnapshotRepository>
let db: jest.Mocked<MongoRepository>
beforeEach(() => {
const { unit, unitRef } = TestBed.create(WhitelistService).compile()
whitelistService = unit
gh = unitRef.get(GithubRepository)
snapshot = unitRef.get(SnapshotRepository)
db = unitRef.get(MongoRepository)
})
const SPACES = {
'a.eth': { followers: 10_000, name: 'a' },
'b.eth': { followers: 100_000, name: 'b' },
'c.eth': { followers: 100_000, name: 'c' },
'a.eth': { followers: 10_000, snapshotId: 'a.eth', snapshotName: 'a' },
'b.eth': { followers: 100_000, snapshotId: 'b.eth', snapshotName: 'b' },
'c.eth': { followers: 100_000, snapshotId: 'c.eth', snapshotName: 'c' },
}
describe('get spaces', () => {
@@ -49,7 +72,7 @@ describe('WhitelistService', () => {
it('getSpaces: returns max maxOrgs spaces with at least minFollowers', async () => {
snapshot.getSpaces.mockResolvedValueOnce({
d: { followers: 1000, name: 'd' },
d: { followers: 1000, snapshotId: 'd.eth', snapshotName: 'd' },
...SPACES,
})
@@ -71,39 +94,23 @@ describe('WhitelistService', () => {
})
})
it('getGhOrgs: return list of github orgs', async () => {
snapshot.getGhOrgsBySpaceIds.mockResolvedValueOnce([
{ ghName: 'a', snapshotId: 'a.eth' },
{
ghName: null,
snapshotId: 'b.eth',
},
])
await expect(
whitelistService.getGhOrgs(['a.eth', 'b.eth']),
).resolves.toEqual([
{
ghName: 'a',
snapshotId: 'a.eth',
},
])
})
it('getOrgs: return list of orgs', async () => {
it('getOrgsWithRepos: return list of orgs that includes repos', async () => {
snapshot.getSpaces.mockResolvedValue(SPACES)
snapshot.getGhOrgsBySpaceIds.mockResolvedValueOnce([
{ ghName: 'a', snapshotId: 'a.eth' },
{
snapshot.getGhNamesBySpaceIds.mockResolvedValueOnce({
'a.eth': { ghName: 'a', snapshotId: 'a.eth' },
'b.eth': {
ghName: 'b',
snapshotId: 'b.eth',
},
{ ghName: 'c', snapshotId: 'c.eth' },
])
gh.getReposByOrg
.mockResolvedValueOnce(['repo-aa', 'repo-ab'])
.mockResolvedValueOnce(['repo-ba', 'repo-bb'])
.mockResolvedValueOnce(['repo-ca'])
'c.eth': { ghName: 'c', snapshotId: 'c.eth' },
})
gh.getReposByOrg.mockImplementation(async (org) => {
if (org === 'a') return Promise.resolve(['repo-aa', 'repo-ab'])
if (org === 'b') return Promise.resolve(['repo-ba', 'repo-bb'])
if (org === 'c') return Promise.resolve(['repo-ca'])
return Promise.resolve([])
})
await expect(whitelistService.getOrgsWithRepos()).resolves
.toMatchInlineSnapshot(`
@@ -141,5 +148,38 @@ describe('WhitelistService', () => {
`)
})
describe('whitelist', () => {
it('getWhitelist: should return orgs in long format', async () => {
db.findAllWhitelistedOrgs.mockResolvedValueOnce(ORGS)
await expect(whitelistService.getWhitelist('long')).resolves.toEqual(ORGS)
})
it('getWhitelist: should return orgs in short format', async () => {
db.findAllWhitelistedOrgs.mockResolvedValueOnce(ORGS)
await expect(whitelistService.getWhitelist('short')).resolves.toEqual({
daos: ['a.eth', 'b.eth'],
repos: ['a/aa', 'a/ab', 'b/ba', 'b/bb'],
})
})
})
it('getWhitelistedDaos: should return list of snapshot ids of the whitelisted orgs', async () => {
db.findAllWhitelistedOrgs.mockResolvedValueOnce(ORGS)
await expect(whitelistService.getWhitelistedDaos()).resolves.toEqual([
'a.eth',
'b.eth',
])
})
it('getWhitelistedRepos: should return list of whitelisted repos', async () => {
db.findAllWhitelistedOrgs.mockResolvedValueOnce(ORGS)
await expect(whitelistService.getWhitelistedRepos()).resolves.toEqual([
'a/aa',
'a/ab',
'b/ba',
'b/bb',
])
})
it.todo('unWhitelist')
})

2
tsconfig.json
View File

@@ -1,5 +1,5 @@
{
"include": ["src", "scripts", "test"],
"include": ["src", "test"],
"exclude": ["test/coverage"],
"files": [
"node_modules/jest-chain/types/index.d.ts",