Keep track of the HITs you've done (and more!). Cross browser compatible.
< Feedback on MTurk HIT Database Mk.II
@romanv Thanks for the screenshot! That made things a lot easier. You'll notice that it says QuotaExceededError which only occurs when it's trying to save some transient data into localStorage, but localStorage cannot hold any more information. It's also worth noting that localStorage is distinct from the database.
What you'll want to do is clean up your localStorage as it has a default cap of ~10MB per domain (depending on browser/implementation. There are two options, either purge everything by clearing cache and cookies (not recommended) or manually go through and delete unnecessary and unused clutter (recommended).
Use this snippet I wrote to get a list of localStorage keys and their sizes. You can remove a key and its associated data with localStorage.removeItem('key'), obviously replacing key with the actual key listed under the index column of the returned table.
Use this snippet I wrote to get a list of
localStoragekeys and their sizes. You can remove a key and its associated data withlocalStorage.removeItem('key'), obviously replacingkeywith the actual key listed under theindexcolumn of the returned table.
I should've mentioned, that i ran script like that (not that exact one, but another similar, that i found somewhere in discussions here), and my local storage is way smaller, than 10MB, if i did interpret results correctly. If TOTAL is indeed total size of all records, then it is just 152KB, as you can see.
@romanv Okay, there is a second possibility, though I initially dismissed it since the script will spit out a different error immediately and not allow you to even begin the update process. But just to confirm, please verify:
Never remember history, or if you're using custom settings for history, that Always use private browsing mode is not checked.These are really the only two ways you can get a QuotaExceededError--either local storage is full, or you are running in private browsing mode.
No, i do not run my browser in private mode and it saving history just fine. There is two weird details with this bug:
- i do not always get QuotaExceededError, sometimes it is just a few thousands of AbortError.
- i have about 55k submitted hits and update database every morning, and yet everything was fine until today. So initially, script was able to parse way more, that 42k records. However now i cannot import my backup file with ~48k records into empty database. I had tried that with both *.bak and csv backups without any success.
Do you get any different errors while importing vs updating?
Okay, i did more tests with empty database, and here is what i found:
- importing Nov 7 backup (33k records) - everything is fine
- attempting to update it (+20k new records) - QuotaExceededError + AbortError (145 of aborts)
- importing Nov 22 backup (42.8k records) - fine
- updating (11k new) - QuotaExceededError + AbortError (690)
- importing Dec 20 backup (52k hits) - fine
- updating (1500 new) - AbortError (250) without QuotaExceededError
- updating just last 45 days with empty db (23k hits) - ok, total information is correct, but only last 15 days have actual hit info in them (see screenshot)
This is interesting because from your tests, it seems the QuotaExceededError was not coming from local storage as I had assumed, but rather indexedDB. What this suggests is that you have very little free disk space available. Firefox puts a hard limit on indexedDB at 20% per origin of the global limit. The global limit is defined as 50% of free disk space.
So for example, if you have 10GB free, the global limit is 5GB and the maximum space available for any single database is 1GB.
Now, in the above example, even 1GB is quite large. I had a test db for performance testing with 500k entries and that only took ~100MB. You seem to be running out of space at ~53.2k entries. Can you confirm that the error is being caused by insufficient disk space?
At the moment, i have 1.5 GB of free space on system partition. If i understand correctly, this should limit my db size to something around 150 MB? If {numbers}HBIDT.sqlite in firefox profile folder is db file, its size is only 5 MB. Is there any possibility, that, while db file itself is small, during the process of importing some temp files can take all available free space and crash the script?
Ok, i did clean additional 6 GB of free space, and it fixed the problem. It seems 1.5 GB was, for some reason, not enough for database update.
Error when writing to database/restoring from backup
I've been using Firefox verdion of hitdb for about six months now, and everything was smooth, until today, when it started to hang up every time when updating database. I have a few backups of my database, so after a few attempts of regular update, i did try to completely delete my current database, import one of the backups and update it again. What happens, however, is that i can't import backups with 42000+ hits, only the ones with less, than that. And when i import backup with, let's say, 40000 of hits and try to do regular update, script hangs up again every time somewhere in 42000 hits region with the same error, as on attached screenshot.