github/bower
1
1
Fork 0
mirror of https://github.com/bower/bower.git synced 2026-04-24 03:00:19 -04:00
Code Issues Packages Projects Releases Wiki Activity

Change UnitOfWork to Worker.

Browse Source
This commit is contained in:
André Cruz
2013-04-19 19:05:33 +01:00
parent f100982b04
commit b968b4a9a4
6 changed files with 200 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@@ -1,4 +1,3 @@
.DS_Store
Thumbs.db
node_modules
components
node_modules

62
README.md
View File

@@ -2,10 +2,10 @@
## Why?
Bower codebase is becoming unmanageable, especially at its core.
Bower code base is becoming unmanageable, especially at its core.
Main issues are:
- __No separation of concerns. The overall codebase has grown in a patch fashion, which has lead to a bloated and tight coupled solution.__
- __No separation of concerns. The overall code base has grown in a patch fashion, which has lead to a bloated and tight coupled solution.__
- __Monolithic Package.js that handles all package types (both local and remote `Git`, URL, local files, etc).__
- __Package.js has a big nesting level of callbacks, causing confusion and making the code hard to read.__
- Some commands, such as install and update, have incorrect behaviour ([#200](https://github.com/twitter/bower/issues/200), [#256](https://github.com/twitter/bower/issues/256))
@@ -21,7 +21,7 @@ Main issues are:
- Ease the process of gathering more contributors.
- Clear architecture and separation of concerns.
- Installation/update speedup.
- Installation/update speed-up.
- Named endpoints on the CLI install.
- Offline installation of packages, thanks to the cache.
- Ability to easily add package types (`SVN`, etc).
@@ -39,13 +39,12 @@ Main issues are:
- **Target:** `semver` range, commit hash, branch (indicates a version).
- **Endpoint:** source#target
- **Named endpoint:** name@endpoint#target
- **UoW:** Unit of Work
- **Components folder:** The folder in which components are installed (`bower_components` by default).
- **Package meta:** A data structure similar to the one found in `bower.json`, which might also contain additional information. This is usually stored in a `.bower.json` file, inside a canonical package.
### Overall strategy
![Really nicely drawn architecture diagram](http://f.cl.ly/items/44271M0R1O012H2m4234/resolve_diagram.png "Don't over think it! We already did! :P")
![Really nicely drawn architecture diagram](http://f.cl.ly/items/2z0u3B1817341P0q0H3M/bower_diagram2.jpg "Don't over think it! We already did! :P")
Bower is composed of the following components:
@@ -59,11 +58,9 @@ Bower is composed of the following components:
- Requesting the `PackageRepository` to fail-fast, in case it realises there is no resolution for the current dependency tree.
- `PackageRepository`: Abstraction to the underlying complexity of heterogeneous source types. Responsible for:
- Storing new entries in `ResolveCache`.
- Queueing resolvers into the `UoW`, if no suitable entry is found in the `ResolveCache`.
- Queueing resolvers into the `Worker`, if no suitable entry is found in the `ResolveCache`.
- `ResolveCache`: Keeps a cache of previously resolved endpoints. Lookup can be done using an endpoint.
- `UnitOfWork`: Work coordinator, responsible for:
- Keeping track of which resolvers are being resolved.
- Limiting amount of parallel resolutions.
- `Worker`: A service responsible for limiting amount of parallel executions of tasks of the same type.
- `ResolverFactory`: Parses an endpoint and returns a `Resolver` capable of resolving the source type.
- `Resolver`: Base resolver, which can be extended by concrete resolvers, like `UrlResolver`, `GitRemoteResolver`, etc.
@@ -96,7 +93,7 @@ Here's an overview of the dependency resolve process:
6. **CACHE HIT VALIDATION** - At this stage, and only for the cache hits, the `PackageRepository` will question the `Resolver` if there is any version higher than the one fetched from cache that also complies with the endpoint target. Some considerations:
- This step is ignored in case a flag like `offline` is passed.
- How the `Resolver` checks this, depends on the `Resolver` type. (e.g. `GitRemoteResolver` would fetch the git refs with `git ls-remote --tags --heads`, and check if there is a higher version that complies with the target).
- How the `Resolver` checks this, depends on the `Resolver` type. (e.g. `GitRemoteResolver` would fetch the git refs with `git ls-remote --tags --heads`, and check if there is a higher version that complies with the target).
- This check should be as quick as possible. If the process of checking a new version is too slow, it's preferable to just assume there is a new version.
- If there is no way to check if there is a higher version, assume that there is.
- If the `Resolver` indicates that the cached version is outdated, then it is treated as a cache miss.
@@ -148,7 +145,7 @@ Simple function that takes a *named endpoint*/endpoint with options and creates
function createResolver(endpoint, options) -> Promise
```
This function will perform transformations/normalisations to the endpoint, like expanding shorthand andendpoints.
This function will perform transformations/normalisations to the endpoint, like expanding shorthand endpoints.
The function is async to allow querying the Bower registry, etc.
@@ -243,7 +240,7 @@ Creates a temporary dir.
Reads `bower.json`/`component.json`, possibly by using a dedicated `read-json` node module that will be available in the Bower organisation.
This method also normalises the `package meta`, filling in any missing information, inferring when possible.
This method also generates the `package meta` based on the `json`, filling in any missing information, inferring when possible.
`Resolver#_decoratePkgMeta(meta)`: Promise
@@ -283,37 +280,48 @@ The `ResolverFactory` knows these types, and is able to fabricate suitable resol
This architecture makes it very easy for the community to create others package types, for instance, a `MercurialFsResolver`, `MercurialResolver`, `SvnResolver`, etc.
#### Unit of work
#### Worker
The work coordinator, responsible for keeping track of which resolvers are being resolved.
The number of parallel resolutions may be limited and configured.
A worker responsible for limiting execution of parallel tasks.
The number of parallel tasks may be limited and configured per type.
This component will be a service that can be accessed to perform tasks.
------------
#### Constructor
`UnitOfWork(options)`
`Worker(defaultConcurrency, types)`
Options:
The `defaultConcurrency` is the default maximum concurrent functions being run.
The `types` allows you to specify different concurrencies for different types.
Use `-1` to specify no limits.
- `maxConcurrent`: maximum number of concurrent resolvers running (defaults to 5)
Example:
```js
var worker = new Worker(15, {
'network_io': 10,
'disk_io': 50
});
```
------------
#### Public methods.
`UnitOfWork#enqueue(resolver)`: Promise
`Worker#enqueue(func, type)`: Promise
Enqueues a resolver to be ran.
The promise is fulfilled when the resolver successfully resolved or is rejected if it failed to resolve.
Enqueues a function to be ran. The function is expected to return a promise.
The returned promise is resolved when the function promise is also resolved.
`UnitOfWork#has(resolver)`: Boolean
The `type` argument is optional and can be a `string` or an array of `strings`.
Use it to specify the type(s) associated with the function.
If multiple types are specified, the function will only be ran when a free slot on every type list is found.
Checks if a resolver is already in the unit of work.
`Worker#abort()`: Promise
`UnitOfWork#abort()`: Promise
Aborts all current work being done.
Returns a promise that is resolved when the current running functions finish to execute.
Any function that was in the queue waiting to be ran is removed immediately.
Aborts the current work being done, by removing any resolvers waiting to resolve.
Note that resolvers running can't be aborted.
Returns a promise that is fulfilled when the current running resolvers finish the resolve process. Any `Resolver` that was in queue to be ran is aborted immediately.

144
lib/resolve/UnitOfWork.js
View File

@@ -1,144 +0,0 @@
var Q = require('q');
var util = require('util');
var mout = require('mout');
var events = require('events');
var UnitOfWork = function (options) {
// Ensure options defaults
this._options = mout.object.mixIn({
maxConcurrent: 5
}, options);
// Parse some of the options
this._options.maxConcurrent = this._options.maxConcurrent > 0 ? this._options.maxConcurrent : 0;
// Initialize some needed properties
this._queue = [];
this._beingResolved = [];
};
util.inherits(UnitOfWork, events.EventEmitter);
// -----------------
UnitOfWork.prototype.enqueue = function (resolver) {
var deferred;
if (this.has(resolver)) {
throw new Error('Attempting to enqueue an already enqueued resolver');
}
deferred = Q.defer();
// Add to the queue
this._queue.push({
resolver: resolver,
deferred: deferred
});
// Process the queue shortly later so that handlers can be attached to the returned promise
Q.fcall(this.doWork.bind(this));
return deferred.promise;
};
UnitOfWork.prototype.has = function (resolver) {
var index;
// Check in the queue
index = this._indexOf(this._queue, resolver);
if (index !== -1) {
return true;
}
// Check in the being resolved list
index = this._indexOf(this._beingResolved, resolver);
if (index !== -1) {
return true;
}
return false;
};
UnitOfWork.prototype.abort = function () {
var promises,
emptyFunc = function () {};
// Empty queue
this._queue = [];
// Wait for pending resolvers to resolve
promises = this._beingResolved.map(function (entry) {
// Please note that the promise resolution/fail is silenced
return entry.deferred.promise.then(emptyFunc, emptyFunc);
});
return Q.all(promises);
};
// -----------------
UnitOfWork.prototype._doWork = function () {
// Check if the number of allowed packages being resolved reached the maximum
if (this._options.maxConcurrent && this._beingResolved.length >= this._options.maxConcurrent) {
return;
}
// Find candidates for the free slots
var freeSlots = this._options.maxConcurrent ? this._options.maxConcurrent - this._beingResolved.length : -1,
entry,
resolver,
x;
for (x = 0; x < this._queue.length && freeSlots; ++x) {
entry = this._queue[x];
resolver = entry.resolver;
// Remove from the queue and
this._queue.splice(x--, 1);
// Put it in the being resolved list
this._beingResolved.push(entry);
freeSlots--;
// Resolve it, waiting for it to be done
this.emit('pre_resolve', resolver);
resolver.resolve().then(this._onResolveSuccess.bind(this, entry), this._onResolveFailure.bind(this, entry));
}
};
UnitOfWork.prototype._onResolveSuccess = function (entry, result) {
var index;
// Remove the package from the being resolved list
index = this._beingResolved.indexOf(entry);
this._beingResolved.splice(index, 1);
entry.deferred.resolve(result);
this.emit('post_resolve', entry.resolver, result);
// A free spot became available, so let's do some more work
this._doWork();
};
UnitOfWork.prototype._onResolveFailure = function (entry, err) {
var index;
// Remove the package from the being resolved list
index = this._beingResolved.indexOf(entry);
this._beingResolved.splice(index, 1);
entry.deferred.reject(err);
this.emit('fail', entry.resolver, err);
// A free spot became available, so let's do some more work
this._doWork();
};
UnitOfWork.prototype._indexOf = function (arr, pkg) {
return mout.array.findIndex(arr, function (item) {
return item.pkg === pkg;
});
};
module.exports = UnitOfWork;

150
lib/resolve/Worker.js Normal file
View File

@@ -0,0 +1,150 @@
var Q = require('q');
var util = require('util');
var events = require('events');
var mout = require('mout');
var Worker = function (defaultConcurrency, types) {
this._defaultConcurrency = defaultConcurrency != null ? defaultConcurrency : 10;
// Initialize some needed properties
this._queue = {};
this._slots = types || {};
this._executing = [];
};
util.inherits(Worker, events.EventEmitter);
// -----------------
Worker.prototype.enqueue = function (func, type) {
var deferred = Q.defer(),
types,
entry;
type = type || '';
types = Array.isArray(type) ? type : [type];
entry = {
func: func,
types: types,
deferred: deferred
};
// Add the entry to all the types queues
types.forEach(function (type) {
this._queue[type] = this._queue[type] || [];
this._queue.push(entry);
}, this);
// Process the entry shortly later so that handlers can be attached to the returned promise
Q.fcall(this._processEntry.bind(this, entry));
return deferred.promise;
};
Worker.prototype.abort = function (type) {
var promises = [],
types;
type = type || '';
types = Array.isArray(type) ? type : [type];
// Empty the queue of each specified types
types.forEach(function (type) {
this._queue[type] = [];
}, this);
// Wait for all pending functions of the specified type to finish
promises = this._executing.map(function (entry) {
return entry.deferred.promise;
});
return Q.allResolved(promises)
.then(function () {}); // Resolve with no value
};
// -----------------
Worker.prottoype._processQueue = function (type) {
var queue = this._queue[type],
length = queue ? queue.length : 0,
x;
for (x = 0; x < length; x += 1) {
if (!this._processEntry(this._queue[x])) {
break;
}
}
};
Worker.prototype._processEntry = function (entry) {
var allFree = entry.types.every(this._hasSlot, this);
// If there is a free slot for every tag
if (allFree) {
// Foreach type
entry.types.forEach(function (type) {
// Remove entry from the queue
mout.utils.remove(this._queue[type], entry);
// Take slot
this._takeSlot(type);
}, this);
// Execute the function
this._executing.push(entry);
entry.func().then(
this._onResolve.bind(this, entry, true),
this._onResolve.bind(this, entry, false)
);
}
return allFree;
};
Worker.prototype._onResolve = function (entry, ok, result) {
// Resolve/reject the deferred based on sucess/error of the promise
if (ok) {
entry.deferred.resolve(result);
} else {
entry.deferred.reject(result);
}
// Remove it from the executing list
mout.array.remove(this._executing, entry);
// Free up slots for every type
entry.types.forEach(this._freeSlot, this);
// Find candidates for the free slots of each type
entry.types.forEach(this._processQueue, this);
};
Worker.prototype._hasSlot = function (type) {
var freeSlots = this._slots[type];
if (freeSlots == null) {
freeSlots = this._defaultConcurrency;
}
return freeSlots > 0;
};
Worker.prototype._takeSlot = function (type) {
if (this._slots[type] == null) {
this._slots[type] = this._defaultConcurrency;
} else if (!this._slots[type]) {
throw new Error('No free slots');
}
// Decrement the free slots
--this._slots[type];
};
Worker.prototype._freeSlot = function (type) {
if (this._slots[type] != null) {
++this._slots[type];
}
};
module.exports = Worker;

5
package.json
View File

@@ -2,6 +2,7 @@
"name": "bower",
"version": "0.0.0",
"description": "The browser package manager.",
"author": "Twitter",
"licenses": [
{
"type": "MIT",
@@ -9,7 +10,7 @@
}
],
"main": "lib",
"homepage": "http://twitter.github.com/bower",
"homepage": "http://bower.io",
"engines": {
"node": ">=0.8.0"
},
@@ -30,8 +31,6 @@
"scripts": {
"test": "mocha -R spec"
},
"author": "Twitter",
"license": "MIT",
"bin": {
"bower": "bin/bower"
},

82
resolve_diagram.graphml
View File

@@ -95,22 +95,6 @@
</data>
</node>
<node id="n5">
<data key="d6">
<y:GenericNode configuration="ShinyPlateNode">
<y:Geometry height="53.0" width="71.0" x="518.4999999999998" y="333.0"/>
<y:Fill color="#FF9900" transparent="false"/>
<y:BorderStyle hasColor="false" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="18.1328125" modelName="custom" textColor="#000000" visible="true" width="29.951171875" x="20.5244140625" y="17.43359375">UoW<y:LabelModel>
<y:SmartNodeLabelModel distance="4.0"/>
</y:LabelModel>
<y:ModelParameter>
<y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/>
</y:ModelParameter>
</y:NodeLabel>
</y:GenericNode>
</data>
</node>
<node id="n6">
<data key="d6">
<y:GenericNode configuration="ShinyPlateNode">
<y:Geometry height="53.0" width="108.0" x="499.9999999999998" y="-148.42089843749997"/>
@@ -126,7 +110,7 @@
</y:GenericNode>
</data>
</node>
<node id="n7">
<node id="n6">
<data key="d6">
<y:GenericNode configuration="ShinyPlateNode">
<y:Geometry height="53.0" width="133.0" x="-39.49999999999952" y="25.50000000000037"/>
@@ -142,7 +126,7 @@
</y:GenericNode>
</data>
</node>
<node id="n8">
<node id="n7">
<data key="d6">
<y:GenericNode configuration="ShinyPlateNode">
<y:Geometry height="53.0" width="133.0" x="-39.49999999999952" y="102.5000000000002"/>
@@ -158,7 +142,7 @@
</y:GenericNode>
</data>
</node>
<node id="n9">
<node id="n8">
<data key="d6">
<y:GenericNode configuration="ShinyPlateNode">
<y:Geometry height="53.0" width="133.0" x="-39.49999999999952" y="179.50000000000003"/>
@@ -174,7 +158,7 @@
</y:GenericNode>
</data>
</node>
<node id="n10">
<node id="n9">
<data key="d6">
<y:GenericNode configuration="ShinyPlateNode">
<y:Geometry height="53.0" width="133.0" x="-39.49999999999952" y="256.0000000000002"/>
@@ -190,7 +174,7 @@
</y:GenericNode>
</data>
</node>
<node id="n11">
<node id="n10">
<data key="d6">
<y:GenericNode configuration="ShinyPlateNode">
<y:Geometry height="53.0" width="133.0" x="-39.49999999999952" y="332.5000000000004"/>
@@ -206,7 +190,7 @@
</y:GenericNode>
</data>
</node>
<edge id="e0" source="n6" target="n1">
<edge id="e0" source="n5" target="n1">
<data key="d10">
<y:ArcEdge>
<y:Path sx="53.96875" sy="0.0" tx="0.0" ty="0.0">
@@ -365,49 +349,7 @@ endpoint<y:LabelModel>
</y:ArcEdge>
</data>
</edge>
<edge id="e8" source="n2" target="n5">
<data key="d10">
<y:ArcEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
<y:Point x="520.375" y="282.75"/>
</y:Path>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="standard"/>
<y:EdgeLabel alignment="center" backgroundColor="#FFFFFF" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasLineColor="false" height="32.265625" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" visible="true" width="53.76953125" x="-41.79681396484375" y="34.1219482421875">Enqueue
Resolver<y:LabelModel>
<y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/>
</y:LabelModel>
<y:ModelParameter>
<y:SmartEdgeLabelModelParameter angle="0.0" distance="30.0" distanceToCenter="true" position="center" ratio="1.0" segment="0"/>
</y:ModelParameter>
<y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/>
</y:EdgeLabel>
<y:Arc height="-33.625" ratio="-0.8762214779853821" type="fixedRatio"/>
</y:ArcEdge>
</data>
</edge>
<edge id="e9" source="n5" target="n2">
<data key="d10">
<y:ArcEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
<y:Point x="590.625" y="282.75"/>
</y:Path>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="standard"/>
<y:EdgeLabel alignment="center" backgroundColor="#FFFFFF" configuration="AutoFlippingLabel" distance="2.0" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasLineColor="false" height="32.265625" modelName="custom" preferredPlacement="anywhere" ratio="0.5" textColor="#000000" visible="true" width="55.6796875" x="-11.59735107421875" y="-66.3875732421875">Resolved
Resolver<y:LabelModel>
<y:SmartEdgeLabelModel autoRotationEnabled="false" defaultAngle="0.0" defaultDistance="10.0"/>
</y:LabelModel>
<y:ModelParameter>
<y:SmartEdgeLabelModelParameter angle="0.0" distance="30.0" distanceToCenter="true" position="center" ratio="0.0" segment="-1"/>
</y:ModelParameter>
<y:PreferredPlacementDescriptor angle="0.0" angleOffsetOnRightSide="0" angleReference="absolute" angleRotationOnRightSide="co" distance="-1.0" frozen="true" placement="anywhere" side="anywhere" sideReference="relative_to_edge_flow"/>
</y:EdgeLabel>
<y:Arc height="-36.625" ratio="-0.9543973803520203" type="fixedRatio"/>
</y:ArcEdge>
</data>
</edge>
<edge id="e10" source="n2" target="n4">
<edge id="e8" source="n2" target="n4">
<data key="d10">
<y:ArcEdge>
<y:Path sx="66.52301487651081" sy="10.136170353159002" tx="-66.526611328125" ty="0.0">
@@ -428,7 +370,7 @@ package<y:LabelModel>
</y:ArcEdge>
</data>
</edge>
<edge id="e11" source="n3" target="n7">
<edge id="e9" source="n3" target="n6">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
@@ -441,7 +383,7 @@ package<y:LabelModel>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e12" source="n3" target="n9">
<edge id="e10" source="n3" target="n8">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
@@ -451,7 +393,7 @@ package<y:LabelModel>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e13" source="n3" target="n10">
<edge id="e11" source="n3" target="n9">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
@@ -464,7 +406,7 @@ package<y:LabelModel>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e14" source="n3" target="n11">
<edge id="e12" source="n3" target="n10">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
@@ -477,7 +419,7 @@ package<y:LabelModel>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e15" source="n3" target="n8">
<edge id="e13" source="n3" target="n7">
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">