Greasy Fork

Seems fanfiction.net has implemented some sort of Cloudflare protection. I know that ficsave/fanficfare (for downloading fics) are currently broken because of it (https://github.com/waylaidwanderer/FicSave/issues/245). It also seems to affect this script. Based on what I'm seeing in the network tab of dev tools, it seems if you try and parse 25+ pages from a source, anything past 25 gets blocked with 429 error (too many requests). It may not actually be Cloudflare, but something at the hosting level. Well, I don't know all the specifics.
(EDIT: Apparently it IS Cloudflare. I finally got it on the actual site today on a normal page, rather than a script, and it was a Cloudflare captcha.)

I imagine it's probably not something that will be easy to work around, but I wonder if it would be possible to adjust the script to populate the pages it already has processed (for example, 1-24). Instead the progress bar up top will show it's loaded those pages, but it won't add them to the current page because it's hung on the later ones. If that makes any sense? Perhaps a timeout or something and then it just dumps what it has. Even if you can't load all pages, it would probably help to see what it's already loaded.

I love the addon and have been using it religiously for years. Even if you don't update it, I'll continue to use it. I just wanted to add what I could based on what I've noticed since these changes came about a few weeks ago. I'm technically still using version 0.0.1.20171117190949 anyways, since I didn't want to migrate away from ViolentMonkey. Although maybe they've added that new GM API by now? (Ah, it seems they did!)

Thanks for notifying me, and for the detailed report :)
I don't use this script that often right now and I'm a bit busy.

The blue bar doesn't show *which* chapters are downloaded, just the *count*. They are still displayed right after green which does erroneusly imply that they could be added. It's just many times easier to code it as a simple bar that indicates the count instead of showing blue dots at appropriate locations to indicate which chapters have been downloaded.

The reason they are not added to the page (and contribute to the green instead of the blue bar) is because chapter 2 or something is missing. It could be that you have 1 and 3-26 but not 2, or 1 and 76-100 or 1 and some random 24 chapters within all the chapters. Basically it loads all chapters at once in parallel but only appends them in order and waits for missing chapters before continuing. As it is, if the green bar isn't progressing then the white part at the end is actually "before" the blue part (at least some of it) in which chapters it represents.

I'm considering rewriting the progress bar. Before when it never got stuck (bakc when ff.net was working) it didn't really matter since it completed within seconds for a hundred chapters so no one would notice the difference. And if I rate limit it then it should "just work" so it doesn't really matter then either. Maybe just moving the blue bar to the end (after white part) would communicate less badly what it represents? But that seems like it would also be confusing so maybe not.

If cloudflare is dropping requests because they happen "to fast" (a 100 chapter book is like a single image in size so it's kind of silly to stop quick tiny requests like these) then just rate limiting the requests sent might be enough to solve the problems. Then maybe it's not a biggie to fix it.

I don't remember if I got the settings saving functionality working with the new API or not.

New update:
v0.0.1.20201228094824 41 seconds ago - Rate limit chapter downloading down to 1 per second (faster for first few chapters) to not hit cloudflare rate limits

Now chapters are requested at a slower rate and in order when rate limited. The first 10 chapters are downloaded in a burst like before but after that a small delay that increases up to 1 second is used. It's still instant for shorter fics but it handles https://www.fanfiction.net/s/5782108/1/Harry-Potter-and-the-Methods-of-Rationality (which has 122 chapters) fine for me now, though slower than before, which is unavoidable.

There's still no checks to see if requests time out and retry them later if so but as long as the cloudflare rate limit isn't hit it shouldn't matter.

I'm considering rewriting the progress bar.

I did the thing. It has been uploaded. Now if there's a chapter missing it's clearly visible on the progress bar that there's a gap.

I don't remember if I got the settings saving functionality working with the new API or not.

I checked, and it's working :)

v0.0.1.20201228094824 41 seconds ago - Rate limit chapter downloading down to 1 per second (faster for first few chapters) to not hit cloudflare rate limits

Now chapters are requested at a slower rate and in order when rate limited. The first 10 chapters are downloaded in a burst like before but after that a small delay that increases up to 1 second is used. It's still instant for shorter fics but it handles https://www.fanfiction.net/s/5782108/1/Harry-Potter-and-the-Methods-of-Rationality (which has 122 chapters) fine for me now, though slower than before, which is unavoidable.

There's still no checks to see if requests time out and retry them later if so but as long as the cloudflare rate limit isn't hit it shouldn't matter.

Awesome, the update works great! Thanks so much for updating this so fast! Especially as you mentioned you aren't using it that often anymore. I still use this religiously, all day every day, and it had been kind of tricky doing things the way I wanted with the changes to FFNet. Even my fallback of just DLing the fic using a scrapper to see the entire thing at once wasn't really an option, since most scripts are still scrambling to get around this. It's great how your rate limiting is such a simple yet efficient solution. Eventually the whole fic does get downloaded.

Once again, thanks as always for prompt responses and updates! It's hard for me to imagine FFNet without this script anymore.

I did notice that functionality no longer seems to work on Review pages. When clicking the Qomplete button it does flip over to the Qomplete page, but doesn't seem to load any additional pages. Just the current one you're on.

Not a huge deal, and I'm hugely grateful that you fixed the main functionality on story pages. But I just thought I'd bring it to your attention, especially since you said you don't use it much yourself at the moment, and might not have noticed.

Thanks again for such a great script!

I did notice that functionality no longer seems to work on Review pages.

Is it a feature that you use? I initially added it because someone requested it for a one-time usecase. I've been considering removing it because I suspect that it isn't actually used, and if you read reviews you might be fine with the standard interface anyways? Is it worth the maintenance?

I do use it. In fact that might have actually been what brought me to this script in the first place, I can't remember. Before I used this, I used to grab all the review pages with wget and slice all the containers together with a Notepad++ macro or something like that, since I don't know Javascript too well, lol.

It's really useful for stories that have hundreds or thousands of reviews, especially since FFNet Story Parser never implemented the auto-load next page feature for reviews. But it's not hugely important. I imagine with rate limiting it could take quite some time to load pages for stories with like 300+ review pages, anyways.

Well, anyways, if it's simple to keep the functionality since it looks like it works out of the same scripting as the story appender, I definitely use it and will continue to. But if it's going to add extra work to fix and maintain it, don't worry about it. Although it is what brought me to this script, I definitely use the story appender more now. I love building out the entire story instantly (or near instantly, now), with the streamlined look and customized formatting that Qomplete offers. Makes the whole reading experience much better. I even have Stylus modify the Qomplete page slightly as well to make it read more like a PDF, and I love it. The review pages are probably only a minor thing for me now.

I will consider fixing the review page Qompletions then.

> I even have Stylus modify the Qomplete page slightly as well to make it read more like a PDF, and I love it.

What Stylus style are you using?

It’s just a simple one I made. Just tweaks the font size, style, line spacing, and page width. A lot of it is already built into FFNet and Qomplete but I couldn’t get it just as I wanted so I just had Stylus overwrite it all. Also to keep it consistent between a Qomplete and non Qomplete FFNet page. It more or less makes the text the same as when I read fics as PDFs in Foxit full page view.

And thanks if you do fix the reviews. If not, it’s no biggie. As I said while it was what first brought me to this script, I mostly use it for story pages now. Right now I’m just really pleased with how quickly and efficiently your rate limiting fixed this script. Especially as many of the scrapper/download scripts are still working on fixes 😄

I tried to get review Qompletion working. But for some reason it breaks the actual fanfiction story chapter Qompletions.

since it looks like it works out of the same scripting as the story appender

Some parts of the code is good, the appending and stuff that works for story pages work fine. The part of the code that detects which chapter/page you are on and which website you are on and what rules to follow for different pages is horrible. It's super confusing trying to fix the code on one page without breaking it on another. That part needs to be rewritten to make these adjustments easy. I can't do it now. I have a browser addon WIP that is a spiritual replacement for this script, that addon has much better support for working with any webpage. But the addon is not ready for release and this script is too messy to use code from the addon. And that addon, while being a different and bigger project, kind of makes it pointless to improve this script since the addon already has the improvements I would add to this.

This script was never intended to work with anything except story pages on fanfiction.net so that's what I will make sure works reliably for now. Might try again later but I've tried some minutes or hours several different days now and it's just pain and failure. :(

I have a browser addon WIP that is a spiritual replacement for this script, that addon has much better support for working with any webpage. But the addon is not ready for release and this script is too messy to use code from the addon. And that addon, while being a different and bigger project, kind of makes it pointless to improve this script since the addon already has the improvements I would add to this.

This script was never intended to work with anything except story pages on fanfiction.net so that's what I will make sure works reliably for now. Might try again later but I've tried some minutes or hours several different days now and it's just pain and failure. :(

It's no big deal if you don't get reviews working. You asked if I used it, which I answered honestly, but it's not as important to me anymore as story pages. Again I appreciate that you've fixed that as quickly and effectively as you have. In the weeks since you updated the script and added the dynamic chapter loading and rate-limiting, and I've had more time to use it, I've come to appreciate it even more.

Your addon sounds nice. Let us know when to keep an eye out for it when it's done.
Thank again!