From 50081716815d1c9db1501306ee44bfac93a0c922 Mon Sep 17 00:00:00 2001 From: Jan Dvorak Date: Thu, 27 Oct 2022 20:58:42 +0200 Subject: [PATCH 1/5] Outline scaling article --- guide/_config.yml | 1 + guide/source/scaling.md | 128 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 guide/source/scaling.md diff --git a/guide/_config.yml b/guide/_config.yml index 7a99eb70f2..4e2d6d1e07 100644 --- a/guide/_config.yml +++ b/guide/_config.yml @@ -66,6 +66,7 @@ sidebar_categories: Production: - security - deployment + - scaling Meta: - CONTRIBUTING - CHANGELOG diff --git a/guide/source/scaling.md b/guide/source/scaling.md new file mode 100644 index 0000000000..6ac7e34024 --- /dev/null +++ b/guide/source/scaling.md @@ -0,0 +1,128 @@ +--- +title: Scaling +description: How to optimize your Meteor application for higher performance when you start growing. +--- + +This guide focuses on providing you tips and common practices on how to scale your Meteor app. +It is important to note that at the end of the day Meteor is a Node.js app tied closely to MongoDB, +so a lot of the problems you are going to encounter are common to other Node.js and MongoDB apps. +Also do note that every app is different so there are unique challenges to each when scaling, so +practices describe in this guide should be used as a guiding posts rather than absolutes. + +This guide has been heavily inspired by [Marcin Szuster's Vazco article](https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects) +on this issue and talk by Paulo Mogollón's talk at Impact 2022 titled "First steps on scaling Meteor realtime data". + +TODO video from Impact 2022 https://impact.meteor.com/meetings/virtual/uo2Er8YPqx2vuRcne + +

Performance monitoring

+ +Before any optimization can take place we need to know what is our problem. This is where APM (Application Performance Monitor) comes in. +If you are hosting on Galaxy then this is automatically included in the [Professional plan](https://www.meteor.com/cloud#pricing-section) +and you can learn more about in its [own dedicated guide article](https://cloud-guide.meteor.com/apm-getting-started.html). +For those hosting outside of Galaxy the most popular solution is to go with [Monti APM](https://montiapm.com/) which shares +all the main functionality with Galaxy APM. You can also choose other APM for Node.js, but they will not show you Meteor +specific data that Galaxy APM and Monti APM specialize in. For this guide we will focus on showing how to work with Galaxy APM, +which is the same as with Monti APM, for simplicity. + +Once you setup either of those APMs you will need to add a package to your Meteor app to start sending them data. + +#### Galaxy APM [package](https://atmospherejs.com/mdg/meteor-apm-agent) +```sh +meteor add mdg:meteor-apm-agent +``` + +#### Monti APM [package](https://atmospherejs.com/montiapm/agent) +```sh +meteor add montiapm:agent +``` + +

Finding issues in APM

+APM will start with providing you with an overview of how your app is performing. You can then dive deep into details of +publications, methods, errors happening (both on client and server) and more. You will spend a lot of time in the detailed +tabs looking for methods and publications to improve and analyzing the impact of your actions. The process, for example for +optimizing methods, will look like this: + +1. Go to the detailed view under the Methods tab. +2. Sort the Methods Breakdown by Response Time. +3. Click on a method name in the Methods Breakdown. Assess the impact if you improve the selected method. +4. Look at the response time graph and find a trace. +5. Improve your method if you feel it is the right moment to do so. + +Not every long-performing method has to be improved. Take a look at the following example: +* methodX - mean response time 1 515 ms, throughput 100,05/min +* methodY - mean response time 34 000 ms, throughput 0,03/min + +At first glance, the 34 seconds response time can catch your attention, and it may seem that the methodY +is more relevant to improvement. But don’t ignore the fact that this method is being used only once in +a few hours by the system administrators or scheduled cron action. + +And now, let’s take a look at the methodX. Its response time is evidently lower BUT compared to the frequency +of use, it is still high, and without any doubt should be optimized first. + +It’s also absolutely vital to remember that you shouldn't optimize everything as it goes. +The key is to think strategically and match the most critical issues with your product priorities. + +

Publications

+

Low observer reuse

+ +https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects + +

Redis Oplog

+14:26 +* reduces load on server +* channels +* only publish the changes you need + +

Methods

+ +

Heavy actions

+6:49 + +

Reoccurring jobs

+ +

Rate limiting

+ +

MongoDB

+* always limit access to your cluster (IP whitelisting) + +

Indexes

+10:00 +* compound indexes +* ESR (equity, sort, range) +* only the ones needed +* n + 1 +* read from secondaries +* do not use regex +* too many indexes actually slow things down + +

Find strategies

+17:46 +* all queries should have an index +* Fields that filter the most should be first + +

Beware of collection hooks

+ +

Caching

+* query caching +* use aggregation & save them + +

Methods over publications

+also consider GraphQL or REST +8:00 + +

Scaling

+ +

Vertical and horizontal scaling

+There are mainly two different ways of scaling: the vertical and horizontal one. + +* **Vertical scaling** boils down to adding more resources (CPU/RAM/disk) to your server, while horizontal scaling refers to adding more machines or containers to your pool of resources. +* **Horizontal scaling** for Meteor projects typically includes running multiple instances of your app on a single server with multiple cores, or running multiple instances on multiple servers. + +

Autoscaling

+ +

Packages

+ +During development it is very tempting to add packages to solve issue or support some features. +This should be done carefully and each package should be wetted carefully if it is a good fit for the application. +Besides security and maintenance issues you also want to know which dependencies given package introduces and +as a whole what will be the impact on performance. From e6b673d684c8da376a0d7fc0b7863030d8cc3120 Mon Sep 17 00:00:00 2001 From: Jan Dvorak Date: Thu, 24 Nov 2022 17:20:26 +0100 Subject: [PATCH 2/5] Container autoscaling section --- guide/source/scaling.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/guide/source/scaling.md b/guide/source/scaling.md index 6ac7e34024..b21776b50a 100644 --- a/guide/source/scaling.md +++ b/guide/source/scaling.md @@ -10,9 +10,7 @@ Also do note that every app is different so there are unique challenges to each practices describe in this guide should be used as a guiding posts rather than absolutes. This guide has been heavily inspired by [Marcin Szuster's Vazco article](https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects) -on this issue and talk by Paulo Mogollón's talk at Impact 2022 titled "First steps on scaling Meteor realtime data". - -TODO video from Impact 2022 https://impact.meteor.com/meetings/virtual/uo2Er8YPqx2vuRcne +on this issue and talk by Paulo Mogollón's talk at Impact 2022 titled ["First steps on scaling Meteor realtime data"](https://impact.meteor.com/meetings/virtual/uo2Er8YPqx2vuRcne).

Performance monitoring

@@ -118,7 +116,18 @@ There are mainly two different ways of scaling: the vertical and horizontal one. * **Vertical scaling** boils down to adding more resources (CPU/RAM/disk) to your server, while horizontal scaling refers to adding more machines or containers to your pool of resources. * **Horizontal scaling** for Meteor projects typically includes running multiple instances of your app on a single server with multiple cores, or running multiple instances on multiple servers. -

Autoscaling

+

Container autoscaling

+ +It is important to be ready for a sudden spikes of traffic. +While all the other measures mentioned here will help, but a certain point it becomes impossible to support more users on one container and additional containers need to be added to support these users. +Today most hosting solutions offer scaling triggers that you can set to automatically scale up (and down) the number of containers for your app based on things like number of connection, CPU and RAM usage. +Galaxy has these as well. Learn more about [setting triggers for scaling on Galaxy](https://galaxy-guide.meteor.com/triggers.html). + +Setting these is vital, so that your application can keep on running when you have extra people come and then saves you money by scaling down when the containers are not in use. +When initially setting these pay a close attention to the performance of your app. you need to learn when is the right time to scale your app so it has enough time to spin up new containers before the existing one get overwhelmed by traffic and so on. +There are other points to pay attention to as well. For example if your app is used by corporation you might want to setup that on weekdays the minimum number of containers is going to increase just before the start of working hours and the then decrease the minimum to 1 for after hours and on weekends. + +Usually when you are working on performance issues you will have higher numbers of containers as you optimize your app. It is therefore vital to revisit your scaling setting after each rounds of improvements to ensure that scaling triggers are properly optimized.

Packages

From fc2c40d04813ef8ff47f64ee9e275ebb2d341f82 Mon Sep 17 00:00:00 2001 From: Jan Dvorak Date: Mon, 28 Nov 2022 18:24:00 +0100 Subject: [PATCH 3/5] Wording improvement --- guide/_config.yml | 2 +- ...{scaling.md => performance-improvement.md} | 49 +++++++++++++------ 2 files changed, 35 insertions(+), 16 deletions(-) rename guide/source/{scaling.md => performance-improvement.md} (75%) diff --git a/guide/_config.yml b/guide/_config.yml index 4e2d6d1e07..b709b58c8d 100644 --- a/guide/_config.yml +++ b/guide/_config.yml @@ -66,7 +66,7 @@ sidebar_categories: Production: - security - deployment - - scaling + - performance-improvement Meta: - CONTRIBUTING - CHANGELOG diff --git a/guide/source/scaling.md b/guide/source/performance-improvement.md similarity index 75% rename from guide/source/scaling.md rename to guide/source/performance-improvement.md index b21776b50a..ee5ffd33cf 100644 --- a/guide/source/scaling.md +++ b/guide/source/performance-improvement.md @@ -1,16 +1,16 @@ --- -title: Scaling +title: Performance improvements description: How to optimize your Meteor application for higher performance when you start growing. --- -This guide focuses on providing you tips and common practices on how to scale your Meteor app. +This guide focuses on providing you tips and common practices on how to improve performance of your Meteor app (sometimes also called scaling). It is important to note that at the end of the day Meteor is a Node.js app tied closely to MongoDB, so a lot of the problems you are going to encounter are common to other Node.js and MongoDB apps. -Also do note that every app is different so there are unique challenges to each when scaling, so +Also do note that every app is different so there are unique challenges to each, therefore practices describe in this guide should be used as a guiding posts rather than absolutes. -This guide has been heavily inspired by [Marcin Szuster's Vazco article](https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects) -on this issue and talk by Paulo Mogollón's talk at Impact 2022 titled ["First steps on scaling Meteor realtime data"](https://impact.meteor.com/meetings/virtual/uo2Er8YPqx2vuRcne). +This guide has been heavily inspired by [Marcin Szuster's Vazco article](https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects), the official [Meteor Galaxy guide](https://galaxy-guide.meteor.com/), +and talk by Paulo Mogollón's talk at Impact 2022 titled ["First steps on scaling Meteor realtime data"](https://impact.meteor.com/meetings/virtual/uo2Er8YPqx2vuRcne).

Performance monitoring

@@ -24,6 +24,8 @@ which is the same as with Monti APM, for simplicity. Once you setup either of those APMs you will need to add a package to your Meteor app to start sending them data. +For working with Galaxy APM and optimizing your app through the data there, don't forget to visit the [Meteor APM guide](https://galaxy-guide.meteor.com/apm-getting-started.html). + #### Galaxy APM [package](https://atmospherejs.com/mdg/meteor-apm-agent) ```sh meteor add mdg:meteor-apm-agent @@ -60,9 +62,23 @@ of use, it is still high, and without any doubt should be optimized first. It’s also absolutely vital to remember that you shouldn't optimize everything as it goes. The key is to think strategically and match the most critical issues with your product priorities. -

Publications

-

Low observer reuse

+For more information about all the things you can find in Galaxy APM take a look at the Meteor APM section in [Galaxy Guide](https://galaxy-guide.meteor.com/apm-getting-started.html). +

Publications

+Publications allow for the most prominent aspect of Meteor, live data. +At the same this is the most resource intensive part of a Meteor application. + +

Proper use of publications

+ +

Methods over publications

+8:00 + +

Publication replacements

+consider GraphQL or REST + +

Low observer reuse

+https://galaxy-guide.meteor.com/apm-know-your-observers.html +https://galaxy-guide.meteor.com/apm-make-your-app-faster.html https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects

Redis Oplog

@@ -81,17 +97,24 @@ https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects

Rate limiting

MongoDB

-* always limit access to your cluster (IP whitelisting) +The following section offers some guidance on optimizing performance of your Meteor application when it comes to the database. +You can find these and more information in other places that deal with MongoDB performance optimization, like on the [official MongoDB website](https://www.mongodb.com/basics/best-practices). +These are all applicable and you should spend some time researching into them as well. The guide here offers some initial and most common patterns. + +

IP whitelisting

+If your MongoDB hosting provider allows it, you should make sure that only whitelisted + +https://galaxy-guide.meteor.com/container-environment.html#network-outgoing

Indexes

10:00 * compound indexes * ESR (equity, sort, range) * only the ones needed +* too many indexes actually slow things down * n + 1 * read from secondaries * do not use regex -* too many indexes actually slow things down

Find strategies

17:46 @@ -104,17 +127,13 @@ https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects * query caching * use aggregation & save them -

Methods over publications

-also consider GraphQL or REST -8:00 -

Scaling

Vertical and horizontal scaling

There are mainly two different ways of scaling: the vertical and horizontal one. -* **Vertical scaling** boils down to adding more resources (CPU/RAM/disk) to your server, while horizontal scaling refers to adding more machines or containers to your pool of resources. -* **Horizontal scaling** for Meteor projects typically includes running multiple instances of your app on a single server with multiple cores, or running multiple instances on multiple servers. +* **Vertical scaling** boils down to adding more resources (CPU/RAM/disk) to your containers, while horizontal scaling refers to adding more machines or containers to your pool of resources. +* **Horizontal scaling** for Meteor projects typically includes running multiple instances of your app on a single container with multiple cores, or running multiple instances on multiple containers.

Container autoscaling

From f711a662850eed709f147ea848ad4c22cace3172 Mon Sep 17 00:00:00 2001 From: Jan Dvorak Date: Sun, 1 Oct 2023 12:23:59 +0200 Subject: [PATCH 4/5] Finish initial guide on performance --- guide/package-lock.json | 38 +++---- guide/source/performance-improvement.md | 143 +++++++++++++++++++----- 2 files changed, 133 insertions(+), 48 deletions(-) diff --git a/guide/package-lock.json b/guide/package-lock.json index 919850c84d..187638a3da 100644 --- a/guide/package-lock.json +++ b/guide/package-lock.json @@ -1830,13 +1830,13 @@ "ansi-regex": { "version": "2.1.1", "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz", - "integrity": "sha512-TIGnTpdo+E3+pCyAluZvtED5p5wCqLdezCyhPZzKPcxvFplEt4i+W7OONCKgeZFT3+y5NZZfOOS/Bdcanm1MYA==", + "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=", "dev": true }, "ansi-styles": { "version": "2.2.1", "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-2.2.1.tgz", - "integrity": "sha512-kmCevFghRiWM7HB5zTPULl4r9bVFSWjz62MhqizDGUrq2NWuNMQyuv4tHHoKJHs69M/MF64lEcHdYIocrdWQYA==", + "integrity": "sha1-tDLdM1i2NM914eRmQ2gkBTPB3b4=", "dev": true }, "anymatch": { @@ -1905,7 +1905,7 @@ "babel-code-frame": { "version": "6.26.0", "resolved": "https://registry.npmjs.org/babel-code-frame/-/babel-code-frame-6.26.0.tgz", - "integrity": "sha512-XqYMR2dfdGMW+hd0IUZ2PwK+fGeFkOxZJ0wY+JaQAHzt1Zx8LcvpiZD2NiGkEG8qx0CfkAOr5xt76d1e8vG90g==", + "integrity": "sha1-Y/1D99weO7fONZR9uP42mj9Yx0s=", "dev": true, "requires": { "chalk": "^1.1.3", @@ -1916,7 +1916,7 @@ "babel-messages": { "version": "6.23.0", "resolved": "https://registry.npmjs.org/babel-messages/-/babel-messages-6.23.0.tgz", - "integrity": "sha512-Bl3ZiA+LjqaMtNYopA9TYE9HP1tQ+E5dLxE0XrAzcIJeK2UqF0/EaqXwBn9esd4UmTfEab+P+UYQ1GnioFIb/w==", + "integrity": "sha1-8830cDhYA1sqKVHG7F7fbGLyYw4=", "dev": true, "requires": { "babel-runtime": "^6.22.0" @@ -1925,7 +1925,7 @@ "babel-plugin-syntax-decorators": { "version": "6.13.0", "resolved": "https://registry.npmjs.org/babel-plugin-syntax-decorators/-/babel-plugin-syntax-decorators-6.13.0.tgz", - "integrity": "sha512-AWj19x2aDm8qFQ5O2JcD6pwJDW1YdcnO+1b81t7gxrGjz5VHiUqeYWAR4h7zueWMalRelrQDXprv2FrY1dbpbw==", + "integrity": "sha1-MSVjtNvePMgGzuPkFszurd0RrAs=", "dev": true }, "babel-plugin-transform-decorators-legacy": { @@ -1942,7 +1942,7 @@ "babel-polyfill": { "version": "6.26.0", "resolved": "https://registry.npmjs.org/babel-polyfill/-/babel-polyfill-6.26.0.tgz", - "integrity": "sha512-F2rZGQnAdaHWQ8YAoeRbukc7HS9QgdgeyJ0rQDd485v9opwuPvjpPFcOOT/WmkKTdgy9ESgSPXDcTNpzrGr6iQ==", + "integrity": "sha1-N5k3q8Z9eJWXCtxiHyhM2WbPIVM=", "dev": true, "requires": { "babel-runtime": "^6.26.0", @@ -1953,7 +1953,7 @@ "regenerator-runtime": { "version": "0.10.5", "resolved": "https://registry.npmjs.org/regenerator-runtime/-/regenerator-runtime-0.10.5.tgz", - "integrity": "sha512-02YopEIhAgiBHWeoTiA8aitHDt8z6w+rQqNuIftlM+ZtvSl/brTouaU7DW6GO/cHtvxJvS4Hwv2ibKdxIRi24w==", + "integrity": "sha1-M2w+/BIgrc7dosn6tntaeVWjNlg=", "dev": true } } @@ -1961,7 +1961,7 @@ "babel-runtime": { "version": "6.26.0", "resolved": "https://registry.npmjs.org/babel-runtime/-/babel-runtime-6.26.0.tgz", - "integrity": "sha512-ITKNuq2wKlW1fJg9sSW52eepoYgZBggvOAHC0u/CYu/qxQ9EVzThCgR69BnSXLHjy2f7SY5zaQ4yt7H9ZVxY2g==", + "integrity": "sha1-llxwWGaOgrVde/4E/yM3vItWR/4=", "dev": true, "requires": { "core-js": "^2.4.0", @@ -1971,7 +1971,7 @@ "babel-template": { "version": "6.26.0", "resolved": "https://registry.npmjs.org/babel-template/-/babel-template-6.26.0.tgz", - "integrity": "sha512-PCOcLFW7/eazGUKIoqH97sO9A2UYMahsn/yRQ7uOk37iutwjq7ODtcTNF+iFDSHNfkctqsLRjLP7URnOx0T1fg==", + "integrity": "sha1-3gPi0WOWsGn0bdn/+FIfsaDjXgI=", "dev": true, "requires": { "babel-runtime": "^6.26.0", @@ -1984,7 +1984,7 @@ "babel-traverse": { "version": "6.26.0", "resolved": "https://registry.npmjs.org/babel-traverse/-/babel-traverse-6.26.0.tgz", - "integrity": "sha512-iSxeXx7apsjCHe9c7n8VtRXGzI2Bk1rBSOJgCCjfyXb6v1aCqE1KSEpq/8SXuVN8Ka/Rh1WDTF0MDzkvTA4MIA==", + "integrity": "sha1-RqnL1+3MYsjlwGTi0tjQ9ANXZu4=", "dev": true, "requires": { "babel-code-frame": "^6.26.0", @@ -2001,7 +2001,7 @@ "babel-types": { "version": "6.26.0", "resolved": "https://registry.npmjs.org/babel-types/-/babel-types-6.26.0.tgz", - "integrity": "sha512-zhe3V/26rCWsEZK8kZN+HaQj5yQ1CilTObixFzKW1UWjqG7618Twz6YEsCnjfg5gBcJh02DrpCkS9h98ZqDY+g==", + "integrity": "sha1-o7Bz+Uq0nrb6Vc1lInozQ4BjJJc=", "dev": true, "requires": { "babel-runtime": "^6.26.0", @@ -2113,7 +2113,7 @@ "chalk": { "version": "1.1.3", "resolved": "https://registry.npmjs.org/chalk/-/chalk-1.1.3.tgz", - "integrity": "sha512-U3lRVLMSlsCfjqYPbLyVv11M9CPW4I728d6TCKMAOJueEeB9/8o+eSsMnxPJD+Q+K909sdESg7C+tIkoH6on1A==", + "integrity": "sha1-qBFcVeSnAv5NFQq9OHKCKn4J/Jg=", "dev": true, "requires": { "ansi-styles": "^2.2.1", @@ -2248,7 +2248,7 @@ "core-decorators": { "version": "0.11.2", "resolved": "https://registry.npmjs.org/core-decorators/-/core-decorators-0.11.2.tgz", - "integrity": "sha512-47n1NWwwc+qPmOMtY9zUKCM1cYfoxLvBRxKzirFrqhE61yqK+yZP/BOA3gjaBUVb9P46J1RyJjasrtqYoWCbvA==", + "integrity": "sha1-GyQzFZQa598a8938tX3+6mPIsXE=", "dev": true }, "core-js": { @@ -2547,7 +2547,7 @@ "has-ansi": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/has-ansi/-/has-ansi-2.0.0.tgz", - "integrity": "sha512-C8vBJ8DwUCx19vhm7urhTuUsr4/IyP6l4VzNQDv+ryHQObW3TTTp9yB68WpYgRe2bbaGuZ/se74IqFeVnMnLZg==", + "integrity": "sha1-NPUEnOHs3ysGSa8+8k5F7TVBbZE=", "dev": true, "requires": { "ansi-regex": "^2.0.0" @@ -2600,7 +2600,7 @@ "hexo-inject": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/hexo-inject/-/hexo-inject-1.0.0.tgz", - "integrity": "sha512-Ly0k7FO3G5+XNvFNE7yjSENSWy8QTnzl8cNFWYuMXRYMogbHd/Q0Ane8WCKYb5QD/A+WXC3rHb32wIGb0YAfVw==", + "integrity": "sha1-pTVXVgUdWrJ5yCtzfacrEzx+Ju0=", "dev": true, "requires": { "babel-plugin-transform-decorators-legacy": "^1.3.4", @@ -2740,7 +2740,7 @@ "js-tokens": { "version": "3.0.2", "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-3.0.2.tgz", - "integrity": "sha512-RjTcuD4xjtthQkaWH7dFlH85L+QaVtSoOyGdZ3g6HFhS9dFNDfLyqgm2NFe2X6cQpeFmt0452FJjFG5UameExg==", + "integrity": "sha1-mGbfOVECEw449/mWvOtlRDIJwls=", "dev": true }, "kind-of": { @@ -3338,7 +3338,7 @@ "strip-ansi": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz", - "integrity": "sha512-VhumSSbBqDTP8p2ZLKj40UjBCV4+v8bUSEpUb4KjRgWk9pbqGF4REFj6KEagidb2f/M6AzC0EmFyDNGaw9OCzg==", + "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=", "dev": true, "requires": { "ansi-regex": "^2.0.0" @@ -3347,13 +3347,13 @@ "supports-color": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-2.0.0.tgz", - "integrity": "sha512-KKNVtd6pCYgPIKU4cp2733HWYCpplQhddZLBUryaAHou723x+FRzQ5Df824Fj+IyyuiQTRoub4SnIFfIcrp70g==", + "integrity": "sha1-U10EXOa2Nj+kARcIRimZXp3zJMc=", "dev": true }, "to-fast-properties": { "version": "1.0.3", "resolved": "https://registry.npmjs.org/to-fast-properties/-/to-fast-properties-1.0.3.tgz", - "integrity": "sha512-lxrWP8ejsq+7E3nNjwYmUBMAgjMTZoTI+sdBOpvNyijeDLa29LUn9QaoXAHv4+Z578hbmHHJKZknzxVtvo77og==", + "integrity": "sha1-uDVx+k2MJbguIxsG46MFXeTKGkc=", "dev": true }, "to-object-path": { diff --git a/guide/source/performance-improvement.md b/guide/source/performance-improvement.md index ee5ffd33cf..a6c5a4d394 100644 --- a/guide/source/performance-improvement.md +++ b/guide/source/performance-improvement.md @@ -10,7 +10,7 @@ Also do note that every app is different so there are unique challenges to each, practices describe in this guide should be used as a guiding posts rather than absolutes. This guide has been heavily inspired by [Marcin Szuster's Vazco article](https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects), the official [Meteor Galaxy guide](https://galaxy-guide.meteor.com/), -and talk by Paulo Mogollón's talk at Impact 2022 titled ["First steps on scaling Meteor realtime data"](https://impact.meteor.com/meetings/virtual/uo2Er8YPqx2vuRcne). +and talk by Paulo Mogollón's talk at Impact 2022 titled ["First steps on scaling Meteor realtime data"](https://www.youtube.com/watch?v=aOqhExZn_5A).

Performance monitoring

@@ -68,64 +68,149 @@ For more information about all the things you can find in Galaxy APM take a look Publications allow for the most prominent aspect of Meteor, live data. At the same this is the most resource intensive part of a Meteor application. +Under the hood WebSockets are being used with additional abilities provided by DDP. +

Proper use of publications

+Since publications can get resource intensive they should be reserved for usage that requires up to date, live data or +that are changing frequently and you need the users to see that. +You will need to evaluate your app to figure out which situations these are. As a rule of thumb any data that are not +required to be live or are not changing frequently can be fetched once via other means and re-fetched as needed, +in most cases the re-fetching shouldn't be necessary. + +But even before you proceed any further there are a few improvements that you can make here. +First make sure that you only get the fields you need, limit the number of documents you send to the client to what you need +(aka always set the `limit` option) and ensure that you have set all your indexes.

Methods over publications

-8:00 +The first easiest replacement is to use Meteor methods instead of publications. In this case you can use the existing publication +and instead of returning a cursor you will call `.fetchAsync()` and return the actual data. The same performance improvements +to get the method work faster apply here, but once called it sends the data and you don't have the overhead of a publication. + +What is crucial here is to ensure that your choice of a front-end framework doesn't call the method every time, but only once +to load the data or when specifically needed (for example when the data gets updated due to user action or when the user requests it).

Publication replacements

-consider GraphQL or REST +Using methods has its limitations and there are other tools that you might want to evaluate as a potential replacement. + +[Grapher](https://github.com/cult-of-coders/grapher) is a favorite answer and allows you to easily blend with another +replacement which is [GraphQL](https://graphql.org/) and in particular [Apollo GraphQL](https://www.apollographql.com/), +which also has an integration [package](https://atmospherejs.com/meteor/apollo) with Meteor. Finally, you can also go back to using REST as well. + +Do note, that you can mix all of these based on your needs.

Low observer reuse

-https://galaxy-guide.meteor.com/apm-know-your-observers.html -https://galaxy-guide.meteor.com/apm-make-your-app-faster.html -https://www.vazco.eu/blog/how-to-optimize-and-scale-meteor-projects +Observers are among the key components of Meteor. They take care of observing documents on MongoDB and they notify changes. +Creating them is an expensive operations, so you want to make sure that Meteor reuses them as much as possible. + +> [Learn more about observers](https://galaxy-guide.meteor.com/apm-know-your-observers.html) + +The key for observer reuse is to make sure that the queries requested are identical. This means that user given values +should be standardised and so should any dynamic input like time. Publications for users should check if user is signed in +first before returning publication and if user is not signed in, then it should instead call `this.ready();`. + +> [Learn more on improving observer reuse](https://galaxy-guide.meteor.com/apm-improve-cpu-and-network-usage)

Redis Oplog

-14:26 -* reduces load on server -* channels -* only publish the changes you need + +[Redis Oplog](https://atmospherejs.com/cultofcoders/redis-oplog) is a popular solution to Meteor's Oplog tailing +(which ensures the reactivity, but has some severe limitations that especially impact performance). Redis Oplog as name +suggests uses [redis](https://redis.io/) to track changes to data that you only need and cache them. This reduces load on +the server and database, allows you to track only the data that you want and only publish the changes you need.

Methods

+While methods are listed as one of the possible replacements for publications, they themselves can be made more performant, +after all it really depends on what you put inside them and APM will provide you with the necessary insight on which +methods are the problem. +

Heavy actions

-6:49 + +In general heavy tasks that take a lot of resources or take long and block the server for that time should be taken out +and instead be run in its own server that focuses just on running those heavy tasks. This can be another Meteor server +or even better something specifically optimized for that given task.

Reoccurring jobs

+Reoccurring jobs are another prime candidate to be taken out into its own application. What this means is that you will have +an independent server that is going to be tasked with running the reoccurring jobs and the main application will only add to +the list and be recipient of the results, most likely via database results. +

Rate limiting

+Rate limit your methods to reduce effectiveness of DDOS attack and spare your server. This is also a good practice to +ensure that you don't accidentally DDOS your self. For example a user who clicks multiple time on a button that triggers +an expensive function. In this example you should also in general ensure that any button that triggers a server event +should be disabled until there is a response from the server that the event has finished. + +You can and should rate limit both methods and collections. + +> [Learn more about rate limiting](https://docs.meteor.com/api/methods.html#ddpratelimiter) +

MongoDB

+ The following section offers some guidance on optimizing performance of your Meteor application when it comes to the database. You can find these and more information in other places that deal with MongoDB performance optimization, like on the [official MongoDB website](https://www.mongodb.com/basics/best-practices). -These are all applicable and you should spend some time researching into them as well. The guide here offers some initial and most common patterns. +These are all applicable, and you should spend some time researching into them as well. The guide here offers some initial and most common patterns.

IP whitelisting

-If your MongoDB hosting provider allows it, you should make sure that only whitelisted -https://galaxy-guide.meteor.com/container-environment.html#network-outgoing +If your MongoDB hosting provider allows it, you should make sure that you whitelist the IPs of your application servers. +If you don't then your database servers are likely to come under attack from hackers trying to brute force their way in. +Besides the security risk this also impacts performance as authentication is not a cheap operation and it will impact performance. + +See [Galaxy guide](https://galaxy-guide.meteor.com/container-environment.html#network-outgoing) on IP whitelisting to get IPs for your Galaxy servers.

Indexes

-10:00 -* compound indexes -* ESR (equity, sort, range) -* only the ones needed -* too many indexes actually slow things down -* n + 1 -* read from secondaries -* do not use regex + +While single indexes on one field are helpful on simple query calls, you will most likely have more advance queries with +multiple variables. To cover those you will need to create compound indexes. For example: + +```javascript +Statistics.createIndexAsync( + { + pageId: 1, + language: 1, + date: 1 + }, + { unique: true } +) +``` +When creating indexes you should sort the variables in ESR (equity, sort, range) style. +Meaning, first you put variables that will be equal to something specific. Second you put variables that sort things, +and third variables that provide range for that query. +Further you should order these variables in a way that the fields that filter the most should be first. + +Make sure that all the indexes are used and remove unused indexes as leaving unused indexes will have negative impact +on performance as the database will have to still keep track on all the indexed variables.

Find strategies

-17:46 -* all queries should have an index -* Fields that filter the most should be first + +To optimize finds ensure that all queries have are indexed. Meaning that any `.find()` variables should be indexed as described above. + +All your finds should have a limit on the return so that the database stops going through the data once it has reached +the limit, and you only return the limited number of results instead of the whole database. + +Beware of queries with `n + 1` issue. For example in a database that has cars and car owners. You don't want to get cars, +and then call the database for each car owner, instead you want to use only two queries. One where you get the all the cars and +second where you get all the owners and then match the data on the front-end. + +Check all queries that run longer than 100ms as there might be issues. + +Do not use RegEx for your queries as these queries have to go through all the data to do that match. + +If you still have issues make sure that you read data from secondaries.

Beware of collection hooks

+While collection hooks can help in many cases beware of them and make sure that you understand how they work as they might +create additional queries that you might not know about. Make sure to review packages that use them so that they won't +create additional queries. +

Caching

-* query caching -* use aggregation & save them + +Once your user base increases you want to invest into query caching like using Redis, Redis Oplog and other. +For more complex queries or when you are retrieving data from multiple collections, then you want to use [aggregation](https://www.mongodb.com/docs/manual/aggregation/) +and save their results.

Scaling

@@ -142,7 +227,7 @@ While all the other measures mentioned here will help, but a certain point it be Today most hosting solutions offer scaling triggers that you can set to automatically scale up (and down) the number of containers for your app based on things like number of connection, CPU and RAM usage. Galaxy has these as well. Learn more about [setting triggers for scaling on Galaxy](https://galaxy-guide.meteor.com/triggers.html). -Setting these is vital, so that your application can keep on running when you have extra people come and then saves you money by scaling down when the containers are not in use. +Setting this is vital, so that your application can keep on running when you have extra people come and then saves you money by scaling down when the containers are not in use. When initially setting these pay a close attention to the performance of your app. you need to learn when is the right time to scale your app so it has enough time to spin up new containers before the existing one get overwhelmed by traffic and so on. There are other points to pay attention to as well. For example if your app is used by corporation you might want to setup that on weekdays the minimum number of containers is going to increase just before the start of working hours and the then decrease the minimum to 1 for after hours and on weekends. @@ -150,7 +235,7 @@ Usually when you are working on performance issues you will have higher numbers

Packages

-During development it is very tempting to add packages to solve issue or support some features. +During development, it is very tempting to add packages to solve issue or support some features. This should be done carefully and each package should be wetted carefully if it is a good fit for the application. Besides security and maintenance issues you also want to know which dependencies given package introduces and as a whole what will be the impact on performance. From da19a61b23873c78693545c53ba0d1dc780fc794 Mon Sep 17 00:00:00 2001 From: Jan Dvorak Date: Tue, 3 Oct 2023 18:12:48 +0200 Subject: [PATCH 5/5] Update guide/source/performance-improvement.md link Co-authored-by: Frederico Maia --- guide/source/performance-improvement.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guide/source/performance-improvement.md b/guide/source/performance-improvement.md index a6c5a4d394..f6721550f7 100644 --- a/guide/source/performance-improvement.md +++ b/guide/source/performance-improvement.md @@ -15,7 +15,7 @@ and talk by Paulo Mogollón's talk at Impact 2022 titled ["First steps on scalin

Performance monitoring

Before any optimization can take place we need to know what is our problem. This is where APM (Application Performance Monitor) comes in. -If you are hosting on Galaxy then this is automatically included in the [Professional plan](https://www.meteor.com/cloud#pricing-section) +If you are hosting on Galaxy then this is automatically included in the [Professional plan](https://www.meteor.com/cloud/pricing) and you can learn more about in its [own dedicated guide article](https://cloud-guide.meteor.com/apm-getting-started.html). For those hosting outside of Galaxy the most popular solution is to go with [Monti APM](https://montiapm.com/) which shares all the main functionality with Galaxy APM. You can also choose other APM for Node.js, but they will not show you Meteor