Greasy Fork

bug: HIT was counted twice: shown in Mk.II report twice, but with different AA time

(Update: I guess it's not a bug, I think the HIT doubled because the requester setup an automatic bonus or something. Because after one of my database searches one of those doubled records showed 0,00 for Bonus column. Although the weird thing is that when I searched for the requester/HIT title again, and checked Pending again, that note was no longer shown.)

1. I updated the Mk.II database, and it showed 2 records for the same HIT on the same date, one goes with AA n/a, another with AA 2 days.
Both have "Pending Approval" status.
2. I manually checked the list of the HITs completed on that day, and the day before and after, and that HIT was done by me only once. Moreover, it was a survey, so I wouldn't even be able to do it twice.
3. I've updated the database again, and the problem is still here.

The same thing is happening to me. One record shows the AA days and the other shows AA n/a. The difference with me is that both records are showing the value of the hit. 2 $4 HITs, 2 $.50 HITs, etc. so it's inflating my total pending $$. This is for pretty much every entry across the board.

Same thing here. Glad this was reported, I've been scanning the web for this. Double results for the same HIT, one is n/a the other's showing the exact number of AA days (2, 3, etc).

Same. It's showing double on the pending HITs but when you go to daily review, it shows the true amount of actual pending HITs. Please fix this ASAP.

Thanks for letting me know. This has been fixed in the new version. It does not retroactively remove the duplicated entries, however. To do that please do the following:

• Open the console with F12 (navigate to the console tab if it's not selected by default)
• Paste in the following code snippet new Promise(resolve => { let d = 0; HITStorage.db .transaction('HIT', 'readwrite') .objectStore('HIT') .index('date') .openCursor(IDBKeyRange.lowerBound('2016-09-09')) .onsuccess = e => { const c = e.target.result; if (!c) return resolve(d); if (/[&%+]/.test(c.value.hitId) || c.value.hitId.length != 30) d++ && c.delete(); c.continue(); } }).then(console.log.bind(console, 'deleted:'))

Doesn't work. It's still showing HITs and whatnot I already got paid for.

All it's doing is deleting data from the 9th of september. An entire week before this was even an issue.

Pending earnings at the bottom doesn't even display the correct amount; it says I have 24 dollars pending when it's really 13.44. The pending earnings at the top and weekly/daily projection is accurate. The pending overview is messed up.

I've uninstalled and reinstalled the new script about a dozen times and tried it in 3 other Chromium based browsers to no avail.

It deletes duplicates starting from the 9th to the most recent update. It doesn't only look at the 9th. When it finishes running, it will have printed out a line telling you how many entries it deleted.

If it's not deleting the duplicates, please let me know what the output of the above code is and give me an example of the HIT ID of your duplicated HITs.

Alright. I've just updated my database again. I currently have 137 HITs pending from a requester that AAs in 24 hours. Never earlier, never later. These are the only HITs I have pending in a 30 day or even the 45 day period.

When I click on "Pending Overview" I get that request labeled as 250 HITs and another requester that has a 1 hr AA as 353 HITs when the last HIT from them approved 5 hours ago.

Running the tool the first time provided 3K+ deleted HITs, none of which solved my problem. Running the tool now provides "deleted: 1" as the outcome result. Now with the duplicates gone, I still have "pending" HITs from the 15th that approved and cleared on the 15th. Am I supposed to download the CSV and manually edit all these by hand and pray the script doesn't mess up again if Amazon changes something yet again?


This is what comes up after updating the database in the console:

Metrics for DATABASE_UPDATE
VM1255:1866 Process completed in 197.131 seconds
VM1255:1869 [PRE]09172016p16 occurred after 11.066 seconds, resolving in 6.459 seconds
VM1255:1869 [PRE]09162016p15 occurred after 11.184 seconds, resolving in 6.407 seconds
VM1255:1869 [PRE]09172016p21 occurred after 20.742 seconds, resolving in 6.437 seconds
VM1255:1869 [PRE]09162016p20 occurred after 20.945 seconds, resolving in 6.473 seconds
VM1255:1869 [PRE]09172016p25 occurred after 29.731 seconds, resolving in 9.614 seconds
VM1255:1869 [PRE]09162016p28 occurred after 33.017 seconds, resolving in 6.438 seconds
VM1255:1869 [PRE]09172016p30 occurred after 42.616 seconds, resolving in 6.443 seconds
VM1255:1869 [PRE]09162016p33 occurred after 42.897 seconds, resolving in 6.439 seconds
VM1255:1869 [PRE]09172016p34 occurred after 51.651 seconds, resolving in 9.642 seconds
VM1255:1869 [PRE]09162016p40 occurred after 54.322 seconds, resolving in 6.435 seconds
VM1255:1869 [PRE]09172016p39 occurred after 64.748 seconds, resolving in 3.438 seconds
VM1255:1869 [PRE]09162016p47 occurred after 65.692 seconds, resolving in 3.412 seconds
VM1255:1869 [PRE]09162016p48 occurred after 69.6 seconds, resolving in 3.42 seconds
VM1255:1869 [PRE]09172016p42 occurred after 70.125 seconds, resolving in 3.365 seconds
VM1255:1869 [PRE]09172016p44 occurred after 74.722 seconds, resolving in 6.499 seconds
VM1255:1869 [PRE]09162016p51 occurred after 75.003 seconds, resolving in 6.45 seconds
VM1255:1869 [PRE]09162016p56 occurred after 84.864 seconds, resolving in 3.44 seconds
VM1255:1869 [PRE]09172016p50 occurred after 85.399 seconds, resolving in 3.397 seconds
VM1255:1869 [PRE]09162016p59 occurred after 90.44 seconds, resolving in 6.505 seconds
VM1255:1869 [PRE]09172016p53 occurred after 90.801 seconds, resolving in 6.47 seconds
VM1255:1869 [PRE]09162016p63 occurred after 99.69 seconds, resolving in 6.563 seconds
VM1255:1869 [PRE]09172016p57 occurred after 100.001 seconds, resolving in 6.495 seconds
VM1255:1869 [PRE]09162016p68 occurred after 109.752 seconds, resolving in 6.493 seconds
VM1255:1869 [PRE]09162016p96 occurred after 137.694 seconds, resolving in 6.553 seconds
VM1255:1869 [PRE]09162016p130 occurred after 171.704 seconds, resolving in 6.567 seconds


And this is what happens when I request the "Pending Overview"

Metrics for DATABASE_RECALL::PENDING
VM1255:1866 Process completed in 0.143 seconds
VM1255:1869 data retrieval occurred after 0 seconds, resolving in 0.107 seconds
VM1255:1869 HTML construction occurred after 0.107 seconds, resolving in 0.033 seconds


It says 353 Pending by Giphy from 9/15 - which isn't the case because those were approved within and hour and paid out soon after. It also says 250 by Opendoor from 9/15 and 9/17. These turn to approved in 24 hours. Only the ones from 9/17 would be actually pending now, and that's 137 of them.

Now, when I look at the DAILY OVERVIEW, it only says 137 pending. However, when I do a search through the database, it says I have 353 pending on Giphy from the 15th, but that's impossible. The same goes for Opendoor. It's only these two requesters that semed to be affected by whatever Amazon did at around 5:40 PM PST on the 15ths. When everyone began having troubles.

Should I just go ahead and manually edit those by exporting the CSV, opening them up in a text editor and then find and replacing it and doing it that way? Because it also shows over 1200 approved payment instead of paid for the 15th.

Err, I went ahead and did it. I'll report back in a week if there's any more issues. Curious about one thing; the difference between Requester Overview Totals and Daily Overview Totals is that daily takes into account any bonuses, and request does not, correct?

Please AT, I've lost track here. Could you be kind enough and just write us 3 lines (like 1. Do this, 2. Do that, 3. Delete this) to say what exactly did you do and in what order? I've lost my morning playing with the *.csv delimiters with no luck.

Also did you understand why this happened? Did Amazon change its code?

And finally, feihtality, thank you very much for your time and effort and support in this script. We appreciate it!

Okay, reporting back. I really rely on Mk. II so instead of waiting or manually trying to fiddle with delimeters and error reports, I took a shot and copied-pasted the author's snippet from above. OK. No more duplicate results, no more AA:n/a descriptions. I'm going back to work and see how it goes.

Also, feihtality, you said "this is fixed in the new version". Where is it??? :-) Is there an update on the road?

Thanks!

Okay, reporting back. I really rely on Mk. II so instead of waiting or manually trying to fiddle with delimeters and error reports, I took a shot and copied-pasted the author's snippet from above. OK. No more duplicate results, no more AA:n/a descriptions. I'm going back to work and see how it goes.

I did the same, but I clearly don't know how to run it. I haven't been able to remove the duplicates just pasting in the code. Help would be so appreciated.
ETA: This error: VM374:3 Uncaught (in promise) ReferenceError: HITStorage is not defined(…

AT, the code above only deletes duplicates, as long as the total number of HITs in the database on any given day matches the total number of HITs you submitted, it did its job. If the HITs are pending when they shouldn't be, then yes, you will need to manually edit them via export and import.

ToucHDowN, the new version is already released.

housekat, what version of the script are you running? That error suggests you are running an extremely old version.

Problem still here after all of the above. Again, double hits, one instance as AA: 3days the other as AA: n/a. Deleted manually and see how it goes.

Question: Is it normal to see AA:n/a in our pending overview? If yes what does it mean? How can it be that an AA time is not available? Why some HITs in Pending Overview have explicit AA time and others n/a?

The n/a in your case is caused by duplicate entries. If you are still getting duplicates, please verify that you're using the latest version of the script.

i am also still getting duplicates after updating

If you're still getting duplicates, I need more information. Please provide a sample of the duplicated HITs, specifically their HIT ID.

The n/a in your case is caused by duplicate entries. If you are still getting duplicates, please verify that you're using the latest version of the script.

1) I use v1.1.029. Checked in the Tampermonkey's dashboard.
2) I attach examples: first image is the report by vanilla MTurk for the requester Fred G. Morstatter, 09/15/2016. Second image is the Pending overview report of the Mk.II. As you can see, there are 5 Pending Approval HITs but the AA = n/a.

I attach here also the *.csv of the above Mk.II report (5 hits) for your convenience.

EDIT: Can't upload *.csv as it seems, so here you have a select all -> copy-paste from the csv.

hitId date requesterName requesterId title pay bonus status autoAppTime feedback
38RHULDV9YFAGIAX81DVEAC6MVMWI3 2016-09-15 Fred G. Morstatter A1SBRVC3EW7UCL Word Intrusion 0.20 Pending Approval
3KTZHH2ONIFLM046VDVHSNG1LX3M8U 2016-09-15 Fred G. Morstatter A1SBRVC3EW7UCL Word Intrusion 0.20 Pending Approval
3NC6WP7WJHKSHDU2XIHAF1LE16KWWA 2016-09-15 Fred G. Morstatter A1SBRVC3EW7UCL Word Intrusion 0.20 Pending Approval
3PGQRAZX02K3Z18TSEPX393961NYSV 2016-09-15 Fred G. Morstatter A1SBRVC3EW7UCL Word Intrusion 0.20 Pending Approval
3TX9T2ZCB91WB8Z7851HE5FWI8OWZ0 2016-09-15 Fred G. Morstatter A1SBRVC3EW7UCL Word Intrusion 0.20 Pending Approval

You said it showed up on one and was blank on the other. I had assumed the AA would be attached to the primary entry instead of the duplicate, but it seems the opposite happened which should not have been possible. Perhaps they were overwritten by an import?

Regardless, the root of the problem, entry duplication, should hopefully be fixed, so I don't expect this to be an issue in the future. If you absolutely must have them, you can add them in manually. Since you already have the csv, just input the AA in the proper place and import it back in. The AA is recorded in seconds, so for 3 days, you'll need the value 259200.

You said it showed up on one and was blank on the other. I had assumed the AA would be attached to the primary entry instead of the duplicate, (...)?

(...)

Indeed, it was showing up on one and was black on the other but this was another example. I fixed that previous example by manually editing. What I attached here though with this Fred requester guy is not a duplication problem but an erroneous AA:n/a report.

Of course and I will try to manually add the AA time but I thought I'd let you know.

Hi,
When I attempted to use your fix that you upload up above, I got the following error.

"VM2404:3 Uncaught (in promise) ReferenceError: HITStorage is not defined
at resolve (:3:3)
at :1:1"

I don't know much of anything about scrips except for how to install them, update them, and use them, so I have no idea what any of that means.

Thankfully I usually keep track of my projected earnings using a different script, so this doesn't bother me too much, I'd just like to clean it up if I could.

Thanks for fixing it so quickly though!

jdzane, what version of the script are you running?

v. 1.1.029

Are you running the code snippet in the console originating from the dashboard? It must be used on a page in which the script is actually running.

Are you running the code snippet in the console originating from the dashboard? It must be used on a page in which the script is actually running.

Ok I figured it out. After I pressed F12 to bring up the console, I left clicked on the actual database section and the chose to "inspect element", then I clicked on the console tab again, pasted in the code, and it fixed it! Thanks for the fix!

I realized that you still might not have been given why the HIT still keeps reappearing even after deleting, because it's still there. After I make manual changes, it still won't remove or let me permanently change anything.
The duplicate HIT Ids look like this:
3BC9H1KCYTB4ARMN0TPO31XOIA7YW0 2016-09-15 Dilney Albornoz Goncalves AQ2A2VMMVGEP9 Take a few short academic surveys - WOMEN ONLY (about 10 minutes) 0.75 Paid

3BC9H1KCYTB4ARMN0TPO31XOIA7YW0&hitDescription=&hitTitle=Take+a+few+short+academic+surveys+-+WOMEN+ONLY+%28about+10+minutes%29 2016-09-15 Dilney Albornoz Goncalves AQ2A2VMMVGEP9 Take a few short academic surveys - WOMEN ONLY (about 10 minutes) 0.75 Pending Approval

So if I have duplicates from 9.5 to 9.17 no matter what I do they still come back
The import won't allow me to delete anything.
-It now won't accept that HIT ID back in.
-If I edit the HIT ID to anything else, it then makes another entry on top of the errored HIT ID.
-If I change the HIT ID to just match the other one, it all imports fine and looks good, but once I update the database it just reverts it back to what it was. So my changes are just lost with that maneuver.

Hopefully that is info that helps and I'm going to try to paste in the code again after looking at the additional info from jdzane

ETA: Nope I am too stupid now to figure it out, and my current error msg is:

Promise {[[PromiseStatus]]: "rejected", [[PromiseValue]]: ReferenceError: HITStorage is not defined
at resolve (:3:3)
at :1:1}
VM530:3 Uncaught (in promise) ReferenceError: HITStorage is not defined(…)resolve @ VM530:3(anonymous function) @ VM530:1

Are you running the code snippet in the console originating from the dashboard? It must be used on a page in which the script is actually running.

Ok I figured it out. After I pressed F12 to bring up the console, I left clicked on the actual database section and the chose to "inspect element", then I clicked on the console tab again, pasted in the code, and it fixed it! Thanks for the fix!

You did that in the HITDB script, right? Any particular line #? I am not able to figure it out.

@housekat No, just go to the mturk dashboard and hit F12 to open the console, paste in the provided code, and hit enter. I've not instructed modifying the actual script in any way.

@housekat No, just go to the mturk dashboard and hit F12 to open the console, paste in the provided code, and hit enter. I've not instructed modifying the actual script in any way.

Thank you. The snippet worked perfect, and I am very embarrassed.
Can I just change it to 9/5 to get the earlier duplicates removed?

I have approximately 30000 HITs submitted. When I used to update my database, all of these HITs were accounted for. After the most recent update to HIT DB, when I update my database, only 5000 of those HITs are accounted for. Any help regarding this issue would be awesome.

Edit: I do remember clearing cookies a few days ago.

@feihtality, I believe I have found a secondary problem where some entries have not been deleted, or possibly the duplicates had different statuses and the Paid HIT was removed.
For example, I still have a PredictWallStreet.com hit in my Pending Overview from 9-15, however it would have been paid on the 19th. And for another requester, PTI, the Requester Overview shows 20 HITs and the Pending Overview still shows 12 HITs, even though all have been Paid(AA is 4 days on these).

I'm getting another n/a hit issue that isn't going away with the console script nor is editing a CSV file and deleting it working.