Greasy Fork

State of Denial :-(

Sorry this is so long. Try to give you as much data as possible to maybe allow you to help me.

I had a >5000 HIT DB in Firefox that recently got corrupted I think while I was running both HitDB versions (while in Beta).

I'm not blaming running both at the same time ... the old version I think has a history of such corruption happening for unclear reasons. And maybe the new one will too if the .sqlite stuff is the real culprit.

I have engaged the SQLite Manger extension in Firefox, but not actually changed anything yet. But I'm prepared to try that if suggested.

I was hoping to remove everything and start over with just the latest version, and maybe import .csv files which I have stored religiously along the way from the old version as I had heard tales of corruption ...

At the moment, the new version starts up with

UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code.

I tried renaming idb to idbO under ~/.mozilla/firefox/jschkouc.default/storage/default/https+++www.mturk.com/
(I'm running this particular one on openSUSE 13.2)
and chrome to chromeO in ~/.mozilla/firefox/jschkouc.default/storage/permanent>

I also moved .metadata to .metadataO

Are in fact both of these directories used by DB II under FIrefox?

What is the purpose of the chrome directory (if it's for DB II) in a Firefox directory?

Is there anything else to do ... is there another file or cookie or whatever .... that is being used here that leads to the error message that also needs to be removed or edited?

For the new version, do you anticipate possibly writing another script that was wipe everything clean so you could start over? (along with the same 33 warnings that you are asking to remove everything and don't blame me! :-) )

As for the import of .csv files, are there any limit on how many HITs (or days?) can be in any file you import?

I did try the import function (after the error message listed above) which I hoped might just create a new DB, but it complained about wrong number of arguments at line 2248 ...
turned out the Requester comment was multiple lines long. LibreCalc reads the file past this line (to completion), but it just puts the first line into the "comment" field for that line. Note that new line characters s are within quote marks, so if your scanner was a little different (not to say it's easy to do) you might get around that. Once I shortened those comments it worked as the next case:

I also tried a 5 line test .csv file in this failed start state, and while it didn't complain about any line numbers, It didn't work either, and I got the purple undulating line to infinity ... and either removing the tab or reloading it still started with the same error message.


As I recall, there's a script that will delete the DB for the old version. Is there any way to go back to the old version, delete it, and then move to the new? Or am I likely to get the same message once done with the old and I re-enable the new version because of some magic cookie or magic something else?

Can you now (or maybe in the future?) import multiple files that say contain just a month or a year of data and have them just add them on instead of replacing the previous version? (which I am assuming that is what happens on import) If it fact that's what happens on Import, maybe you could have it "wipe everything" before it starts the import?

I am concerned that at some point the file to import might get "too big" .... or maybe that will be my final straw that gets me to get a real job! I understand that big files lead to really slow searches unless I use the start and end dates that are a really nice addition.

I was having trouble with the import function on the old version, but I never tried it with the edited .csv file with the multiline comments being shortened.

Thanks. Any guidance would be appreciated. I have working versions on Fedora 22 and in Chrome on both OSes, so this isn't an emergency, but none of them have my full HIT History in their databases like this one did up to 10-11-2015.

Oooooookay. You weren't kidding about this being long! First of all, the database is, and always has been, an indexed database, hence the name of the API indexedDB. It is not a relational database like SQL; it uses a key:value system to store data in containers called object stores, which is not directly translatable into a relational database format.

 What is the purpose of the chrome directory
It stores settings related to UI appearance like toolbars and application windows. It also contains stuff related to extensions and themes.

 Are in fact both of these directories used by DB II under FIrefox?
I actually have no idea; I've never looked into directory stuff.

 Is there anything else that leads to the error message that also needs to be removed or edited?
I've never seen that error before. I searched around and the most common cause of the error is due to trying to access the database out of a file, i.e. file:///* instead of http(s)://*. I don't think that applies here. What exactly did you do before getting the UnknownError?

 Are there any limit on how many HITs (or days?) can be in any file you import?
Not as far as I can tell. I've successfully imported files containing records of over 500,000 HITs.

 ...turned out the Requester comment was multiple lines long
Thanks for letting me know. I'll update to take care of multi-line feedback.

 Can you now (or maybe in the future?) import multiple files ... and have them just add them on instead of replacing the previous version? ... maybe you could have it "wipe everything" before it starts the import?
I can add support for multi-file imports in the future. As I mentioned earlier, the database is a key:value store. Each key needs to be unique and HITs are stored by their HIT id. This means only one entry of a HIT can exist. Any future entry containing the same HIT id will overwrite the previous. This ensures there are no duplicates in the database.

Import allows users to add HITs that are missing from the current database for whatever reason, whether that may be a previous deletion, migration to a new computer/browser, etc. Deleting everything before importing defeats the purpose here.

Hard to say "what I did" ... I was flailing a bit. I know just enough to be dangerous as they say (I'm a computer programmer used to Linux systems) My guess is that the first non-Beta run of DBII tried to "merge" the "old style" DB which was corrupted in some way (that is if I tried to do Update DB, it would say it was starting, but never do anything, and never complete, and if I tried to search it would never produce a .csv file as I guess something key was locked) and got very confused, which of course is complete understandable. Guess I should have asked first if there was a way to either start completely fresh or whether there could be some flag to ignore the DB-old-style. I'm guessing if it seems to be there, you try to use it.

Perhaps DBII barely started, sensed some lock that prevents anything from happening because the old DB was being "updated," and things on the "new DBII" side got left in a bad state.

Of course I don't understand what you did if I had been using the Beta version before (which I had, and in fact concurrently with DBI which was okay as I understood it) ... did you ignore whatever the Beta had created and started fresh with the last DBI version to create a new DBII DB?

So what is the base name of the indexDB file(s) I should be looking for?
Of failing that, the file extension?

I searched for filenames with *ndex* in them under the "FF profile" directory on the working Fedoroa system (no messing around there) but didn't find anything.

Is the idb directory not an abbreviation/acronym for Indexed DataBase?
Here's what's there on the Fedora system that's working (in FF)

[wtb@wtbathlon https+++www.mturk.com]$ ll
total 16
drwxr-xr-x. 3 wtb wtb 4096 Jun 19 13:49 .
drwxr-xr-x. 6 wtb wtb 4096 Oct 19 15:37 ..
drwxr-xr-x. 4 wtb wtb 4096 Oct 19 18:09 idb
-rw-rw-r--. 1 wtb wtb 47 Oct 19 18:12 .metadata
[wtb@wtbathlon https+++www.mturk.com]$ cd idb
[wtb@wtbathlon idb]$ ll
total 5540
drwxr-xr-x. 4 wtb wtb 4096 Oct 19 18:09 .
drwxr-xr-x. 3 wtb wtb 4096 Jun 19 13:49 ..
drwxr-xr-x. 2 wtb wtb 4096 Jun 22 09:53 2632765872QBuDal.files
-rw-r--r--. 1 wtb wtb 524288 Oct 19 16:12 2632765872QBuDal.sqlite
drwxr-xr-x. 2 wtb wtb 4096 Jun 19 13:49 4000284130HBIDT.files
-rw-r--r--. 1 wtb wtb 5128192 Oct 19 18:08 4000284130HBIDT.sqlite

I have been using Chrome exclusively in the last few days on Fedora, so the 19th dates are not significant. But I did use it back on Monday to make sure it was working in FF on Fedora and Chrome, and that the old DB script was disabled in all of them.

I thought I saw a warning that you shouldn't have both DBI and DBII scripts active at the same time. Is it really the execution order that's the issue? If not, and you depend on both not being active, then I would say that some % of people will get in trouble ...

The beta used a separate database named HITDB_TESTING. Coming out of beta, it updated the schema of the original database, HITDB, merged the beta database into the original, then deleted the beta database. No HIT data from the original database would have been altered or deleted during this process^*.

The .sqlite files are indeed how Firefox stores the database, but like I said they aren't directly translatable. If you look at the files, you'll be able to see the keys, but the data will all be unreadable blobs. Making and saving changes may or may not corrupt the database; I haven't tried.

So, what are you trying to do? Tell me your goal and I'll help you from there.

^*If there were overlapping HITs, the entries in the original would be updated to whatever was in the beta

I would like to start on this Firefox instance with a blank slate, import all my HITs through 10-11-2015 from the "corrected" .csv file I have, and then update the resulting DB to be up to date.

If it's any easier, (and I doubt it is) start with a blank slate, update to get the last 45 days, and then use import to merge all the HITS from the .csv file that has data through 10-11-2015. I gather than would just overwrite the duplicated data for those records that were also in the last 45 days, but theoretically that shouldn't be a problem, right?

I suppose I could achieve this by starting a new Firefox Profile, but then all my cookies and passwords and bookmarks and God knows what else would have to be copied over by hand, which of course is also prone to "user error" ... Truthfully most of my passwords are stored in Chrome and bookmarks in Diigo, so this isn't a huge deal, just an inconvenience

I believe for the old style there was a way to delete your old DB and start over ... maybe a one time script you ran to achieve that, then you disabled it, and re-enabled DBI and off you went. Perhaps creating something similar would be as easy as trying to figure out how to undo this weird state I'm in that prevents me from doing any updates or searches (and I haven't even looked at what it's doing to the interaction with Hit Scraper)

I dare say though that I won't be the last person than somehow ends up with a corrupted DB ... stuff happens, disks get flakey, power spikes or interruptions happen ... whatever. So having a way to wipe it and start over will probably be needed again by other folks.

I've been hesitant to try using the .csv import feature on any of the 3 instances that got converted from Beta just fine ( I think the error messages from the lines with newlines in the Approve comment before I figured the issue out spooked me from trying it since we're gone "official.")

I really want a DB II with all my HIT data in it somewhere, but preferable everywhere (both OSes, both browsers) . Right now I mainly use Fedora and Chrome for HITS, so there first.

Thanks again.

1. make sure you have nothing else accessing the db (HIT Scraper tab, another dashboard tab)
2. disable the script
3. refresh the dashboard
4. open the console (ctrl+shift+I)
5. run the command, indexedDB.deleteDatabase('HITDB')
6. run the command, localStorage.removeItem('hitdb_fetchData')
7. reenable the script and refresh the dashboard again

Then you can run an update and import.

no joy.

First command: (via cut and paste)
indexedDB.deleteDatabase('HITDB')

First command returned:
IDBOpenDBRequest { onblocked: null, onupgradeneeded: null, source: null, transaction: null, readyState: "pending", onsuccess: null, onerror: null }

Second command:
localStorage.removeItem('hitdb_fetchData')

Second command returned:
undefined

Possible spelling error? Is undefined the expected result?

hold on ... I didn't have an idb directory ... I will try empty one first, and then try restoring the idb the way it was previously ....

Okay, I restored the idb directory, but got the same results.

This readyState being pending on the first command ... any issue with that?

No, those are expected results. The two commands don't return anything useful. It's the same on both failure and success.

Okay ... so there's no change ... I still get the error message.

Opening up the Console with DBII enabled the msgs are:

Ready Turkmaster_(Mturk).user.js:144:2
DOMError { name: "UnknownError", message: "The operation failed for reasons un…" } MTurk_HIT_Database_MkII.user.js:557:17
Utils.errorHandler() MTurk_HIT_Database_MkII.user.js:557
dbh.onerror() MTurk_HIT_Database_MkII.user.js:1168

This site makes use of a SHA-1 Certificate; it's recommended you use certificates with signature algorithms that use hash functions stronger than SHA-1.[Learn More] i.imgur.com
Use of getPreventDefault() is deprecated. Use defaultPrevented instead.

Try opening a test database manually. Input this into the console:
indexedDB.open('testdb')
Does that give you an error?

Ready Turkmaster_(Mturk).user.js:144:2
This site makes use of a SHA-1 Certificate; it's recommended you use certificates with signature algorithms that use hash functions stronger than SHA-1.[Learn More] i.imgur.com
Use of getPreventDefault() is deprecated. Use defaultPrevented instead. jquery-min.js:3:0
indexedDB.open('testdb')
IDBOpenDBRequest { onblocked: null, onupgradeneeded: null, source: null, transaction: null, readyState: "pending", onsuccess: null, onerror: null }
UnknownError



afterwards:

wtb@linux:~/.mozilla/firefox/jschkouc.default/storage> find . -type f -print | xargs ls -l
-rw-r--r-- 1 wtb users 1105920 Oct 11 17:00 ./default/https+++drive.google.com/idb/830257170ceeglalroo_ts.sqlite
-rw-r--r-- 1 wtb users 51 Oct 11 18:26 ./default/https+++drive.google.com/.metadata
-rw-r--r-- 1 wtb users 524288 Oct 3 07:00 ./default/https+++twitter.com/idb/437107801ddma_ethyape.sqlite
-rw-r--r-- 1 wtb users 47 Jun 10 13:41 ./default/https+++twitter.com/.metadata
-rw-r--r-- 1 wtb users 59 Mar 25 2015 ./default/https+++www.guidedtrack.com/.metadata
-rw-r--r-- 1 wtb users 1179648 Oct 22 23:22 ./default/https+++www.mturk.com/idb/2632765872QBuDal.sqlite
-rw-r--r-- 1 wtb users 8069120 Oct 22 23:22 ./default/https+++www.mturk.com/idb/4000284130HBIDT_20151021204310.sqlite
-rw-r--r-- 1 wtb users 1179648 Oct 7 10:57 ./default/https+++www.mturk.com/idb-Back/2632765872QBuDal.sqlite
-rw-r--r-- 1 wtb users 8069120 Oct 21 20:43 ./default/https+++www.mturk.com/idb-Back/4000284130HBIDT_20151021204310.sqlite
-rw-r--r-- 1 wtb users 8069120 Oct 11 16:02 ./default/https+++www.mturk.com/idb-Back/4000284130HBIDT.sqlite
-rw-r--r-- 1 wtb users 32768 Oct 21 20:43 ./default/https+++www.mturk.com/idb-Back/4000284130HBIDT.sqlite-shm
-rw-r--r-- 1 wtb users 0 Oct 21 20:43 ./default/https+++www.mturk.com/idb-Back/4000284130HBIDT.sqlite-wal
-rw-r--r-- 1 wtb users 47 Oct 22 06:57 ./default/https+++www.mturk.com/.metadata
-rw-r--r-- 1 wtb users 47 Oct 11 18:21 ./default/https+++www.mturk.com/.metadataO
-rw-r--r-- 1 wtb users 17351680 Oct 22 23:15 ./default/https+++www.mturk.com/out3.cpio
-rw-r--r-- 1 wtb users 524288 Oct 3 07:00 ./default/http+++www.autoevolution.com/idb/1560848701eBcD_dIenxde.sqlite
-rw-r--r-- 1 wtb users 62 Mar 1 2015 ./default/http+++www.autoevolution.com/.metadata
-rw-r--r-- 1 wtb users 884736 Oct 3 07:00 ./default/TURK-BACKUP/2015-06-03/idb/2632765872QBuDal.sqlite
-rw-r--r-- 1 wtb users 5996544 Oct 3 07:00 ./default/TURK-BACKUP/2015-06-03/idb/4000284130HBIDT.sqlite
-rw-r--r-- 1 wtb users 47 Jun 1 14:21 ./default/TURK-BACKUP/2015-06-03/.metadata
-rw-r--r-- 1 wtb users 1179648 Oct 18 16:17 ./default/TURK-BACKUP/2015-10-11/idb/2632765872QBuDal.sqlite
-rw-r--r-- 1 wtb users 8069120 Oct 18 16:17 ./default/TURK-BACKUP/2015-10-11/idb/4000284130HBIDT.sqlite
-rw-r--r-- 1 wtb users 40960 Oct 22 22:18 ./permanent/chrome/idb/2918063365piupsah.sqlite
-rw-r--r-- 1 wtb users 29 Oct 22 22:17 ./permanent/chrome/.metadata
-rw-r--r-- 1 wtb users 29 Aug 18 18:22 ./permanent/chromeO/.metadata
-rw-r--r-- 1 wtb users 40960 Oct 23 20:28 ./permanent/indexeddb+++fx-devtools/idb/478967115deegvatroootlss--cans.sqlite
-rw-r--r-- 1 wtb users 63 Oct 22 22:40 ./permanent/indexeddb+++fx-devtools/.metadata
-rw-r--r-- 1 wtb users 270 Jun 10 12:16 ./temporary/https+++www.facebook.com/asmjs/metadata
-rw-r--r-- 1 wtb users 1106405 Jun 8 18:12 ./temporary/https+++www.facebook.com/asmjs/module13
-rw-r--r-- 1 wtb users 1106405 Jun 8 18:12 ./temporary/https+++www.facebook.com/asmjs/module14
-rw-r--r-- 1 wtb users 1106405 Jun 8 18:12 ./temporary/https+++www.facebook.com/asmjs/module15
-rw-r--r-- 1 wtb users 53 Jun 8 18:12 ./temporary/https+++www.facebook.com/.metadata
wtb@linux:~/.mozilla/firefox/jschkouc.default/storage>

I think the Beta DB would have had maybe 45-60 days worth of data in it before the
switchover. I never tried to import all the records from my backup file on this openSUSE/Firefox instance.

So you got an UnknownError with that command? If the open request fails with the same error on an empty test database, it suggests the problem lies with your configuration. I'm honestly not sure what you did to break it or how to fix it.

1. make sure you have nothing else accessing the db (HIT Scraper tab, another dashboard tab)
2. disable the script
3. refresh the dashboard
4. open the console (ctrl+shift+I)
5. run the command, indexedDB.deleteDatabase('HITDB')
6. run the command, localStorage.removeItem('hitdb_fetchData')
7. reenable the script and refresh the dashboard again

Then you can run an update and import.

THANK YOU for this! I had upgraded my Waterfox, realized that once again Mozilla changed things that shouldn't be changed, downgraded back to the version I had, and had the OP's issue with my DB.

I followed the quoted steps and all is well. Whew!