Discussions » Greasy Fork Feedback

High update check counts from Chrome users making the site slower

§
Posted: 2016-04-08
Edited: 2016-04-08

High update check counts from Chrome users making the site slower

On April 4, Chrome users checking for updates represented 10% of traffic. Today, it's 90%. Overall, traffic to the site is 10 times normal. So the server is really busy, so don't be surprised by slowness or 503 pages.

I'm guessing it has to do with a recent Tampermonkey upgrade which "Fix a problem that caused the script update check to run only on startup".

§
Posted: 2016-04-09

The access logs filled up the entire disk for a while there, crashing the site.

If anyone has a direct line to the Tampermonkey people...

§
Posted: 2016-04-09

@JasonBarnabe I'm so sorry. It seems that under certain conditions (which in theory never should occur) the update check runs into an endless loop. Affected people should be able to work-around the issue by setting the "Script Update" -> "Check Interval" option to another value than "Never".

I'm going to prepare a new stable release now. Sorry again for any inconvenience. :(

See also: https://github.com/OpenUserJs/OpenUserJS.org/issues/944#issuecomment-207719752

§
Posted: 2016-04-09

http://tampermonkey.net/changelog.php?version=4.0.25&show=dhdg

Waiting for the Chrome store to publish it...

§
Posted: 2016-04-09

OK, it's published. Chrome should auto-update Tampermonkey soon, but you can also drag and drop the new release[1] to Chrome's extension page.

Sorry again for any inconvenience.

[1] https://tampermonkey.net/test/versions/stable/4.0.25/fa9adf10f06/dhdgffkkebhmkfjojejmpbldmpobfkfo_main.crx

§
Posted: 2016-04-09

I haven't seen any improvement yet, still crossing my fingers that it's to come.

§
Posted: 2016-04-09

It always takes a some time for the updates to be broadcasted and additionally after Chrome downloaded the update a browser restart is required to apply the update. I hope you'll see some improvement at the end of your day.

§
Posted: 2016-04-10
Edited: 2016-04-10

It's at about 70% of the maximum I've seen over the past few days right now. Not sure if it's the day or time of day, or if things are getting better. Still about 8X normal.

§
Posted: 2016-04-10

The problem at this level of traffic doesn't seem to be directly related to CPU, memory, or disk I/O. The site runs with 8 Passenger processes and the problems seem to occur when some of them get stuck. The when it goes down to 6 active processes, they can't seem to keep up with the requests, and so 503s start happening.

I've upgraded Passenger and nginx to see if the stuck processes issue gets fixed. If not, I will file a bug report with them.

§
Posted: 2016-04-10

It's hard to say how many users already updated. :( But I hope the "missing" 30% percent are a result of some users using a fixed TM version.
I also released new Beta and Safari versions and I hope the Opera extension will be reviewed soon.

§
Posted: 2016-04-10

I found the cause of the hung Passenger processes (the VM didn't allow enough processes, which made the Ruby processes hang, or something). I fixed it and now things look much better. I'm assuming that this misconfiguration multiplied the problem caused by TM users, possibly by making them retry their request.

Right now the traffic looks like down to it's 2X normal.

§
Posted: 2016-04-10

Things are a bit worse now, but the site is giving timeouts instead of 503 pages.

I hope this is still just the remnants of the people on the old version. Some poor guy has tried to download "Neverwinter gateway - Profession Automation" 600000 times today.

§
Posted: 2016-04-10

Hmm, the nasty thing is that only a few people with the issue and a potent internet connection can cause a lot of traffic. Is something like https://github.com/kickstarter/rack-attack an option? The description looks like it's capable to stop the 600000-times guy by throttling the requests to script URLs.
Sorry again. :(

§
Posted: 2016-04-11

The thing is that the requests on the server side are actually really quick - about 12ms for the server to handle them. The problem is the number of requests coming in. With rack-attack, all the requests are still happening.

§
Posted: 2016-04-11

In fact, I'm looking at what the top requesters are doing... They seem to make a request then cancel it about 15 times a second. Not sure if that behaviour matches the problem you fixed.

§
Posted: 2016-04-11
Edited: 2016-04-11

Unfortunately yes. It's a endless loop that starts new update checks. Many requests are canceled due to their amount, but some start downloads.

According to my research this happens to TM users that set the script update interval to "Never" before February 2013, never modified the setting again and never uninstalled or reset Tampermonkey and then updated to 4.0.10. Backups that are made under this conditions are also affected.

§
Posted: 2016-04-11

did you read this::

Today, we’re announcing the end of Chrome’s support for Windows XP, as well as Windows Vista, and Mac OS X 10.6, 10.7, and 10.8, since these platforms are no longer actively supported by Microsoft and Apple. Starting April 2016, Chrome will continue to function on these platforms but will no longer receive updates and security fixes.

It seems they additionally stopped addon upgrading.

§
Posted: 2016-04-12

@JasonBarnabe Is there any news on this topic?

§
Posted: 2016-04-13

I banned a few IPs a couple days ago which had a large immediate effect, and the traffic has been slowly subsiding since then. It's at 2-3X normal now, and the max over the last 24 hours has been 6X normal.

§
Posted: 2016-04-13

And I guess a higher amount of traffic may be expected as that's what the original change was designed to do.

§
Posted: 2016-04-13

Glad to hear that things start to return to normal. Of course, a little bit more traffic is normal, because the update check now runs at long-living sessions more than once. The default interval is 24h what should quite often match the interval user's do open their Chrome.
Thanks for your patience.

§
Posted: 2016-04-16

Pretty much back to normal now.

Post reply

Sign in to post a reply.