Hit_scraper with hit export script added CUZ IT'S MORE CONVENIENT! Here's a few guides, one on mturkforum.com and one on mturkgrind.com
v1.5 update!
This one isn't as big, but I felt it warranted a minor release because there was some updated functionality.
Features added:
- Added a new column, "M?", which shows if hits are Masters or not (more useful for those of us with Masters, but useful for both)
- Added qual listing, mouseover the "M?" column to see the quals for that hit.
- Added TO listing, mouseover TO link to see all the ratings
- Minor tweaking, some verbiage fixes, stuff you probably won't see/notice
v1.5.1: Fixed "notqualified" link processing
v1.5.2: Hopefully fixed the firefox issue, changed the way values are stored/recalled. This will have the unfortunate effect of clearing everyone's blocklist, but hopefully this will not change in the future.
v1.5.3: Fixed storage to properly handle requesters with commas in their names, I didn't realize.
v1.5.4: Updated to fix it not scraping when you're logged out, because apparently that's a thing people do.
v1.5.5: Updated to fix 1.5.4 again, because Amazon changed the way links work. Now, if you're not qualified (IE logged out), clicking the title (and/or exporting) should give you a link to the requester page, instead of a non-functioning link.
v1.5.6: Fixed 1.5.5 again, hopefully now it'll work when you're logged out again. Made it so "qualified" is not default when not logged in, hopefully fixed "duplicate" issue (where a hit will sometimes show up twice)
TODO:
- Add in functionality to remove hits that match certain TO criteria
- Add in sort by TO functionality
- Add in "max reward" functionality
These are going to be rough, as they involve editing the list after it's been populated, and I have trouble with that. Also, expect bug fixes/subreleases in the near future with more enhancements/fixes.
v1.4 update!
Big update here, I added a lot of cosmetic and other functionality to it. Let me know if there are any issues, new functions will probably be buggy until taken care of.
Features added:
- Ability to block by title and requester (so you can block individual hits you've done)
- Ability to view only certain requesters with Include list (Must add requesters to list individually for the moment, if there's a desire I'll add in a button like the blocklist)
- Ability to make scraper make a "ding" noise when it finds new work.
- Tied in with HitDB so clicking the R/T at the end will show you the work you've done for that requester (only for green items, might not work on firefox)
- Added A-Z sort
- Added inverse sort
- Added checkbox for "Correct For Skips" (mouseover the checkbox to see what it does, or try it out! On by default, will change to off by default if necessary).
Cosmetics:
- Re-organized a bit of the header section with some | characters to separate things
- Added some helpful "status" messages to explain some things a bit (IE why it's scraping more than the pages you told it to)
- Moved the status messages to below the header
Subtle:
- Made it pull the blocklist every time you run it so you can have multiple instances and they'll work together properly.
TODO:
- Save everything to localstorage so you won't have to set it up individually each time
- Add capability for multiple export templates (so you can have one scraper for a bunch of sites)
- Make it easier to theme (Add a table with colors you can edit and such)
v1.4.1 changes:
- Initial themeing support. Put all the color values up at the top of the code, with descriptors, so they can be changed easily
v1.4.2 changes:
Nothing really. Just a bit for some of our...friends...You shouldn't see anything different really.
v1.4.3 changes:
Reverted v1.4.2
v1.4.4 changes:
Added another descriptive status message.
Older update logs:
Updated to fix an issue with the export not getting the proper quals for the proper hit.
Updated so it wouldn't clobber the normal hit export script
Updated to fix a bug, and now the requester list is case insensitive.
Added description as mouseover text for title link. Hold the mouse over the title to see it.
1.3.0.10: Added ability to block requesters dynamically, and revert to the blocklist set in the code. Default blocklist contains:
"oscar smith", "Diamond Tip Research LLC", "jonathon weber", "jerry torres", "Crowdsource", "we-pay-you-fast", "turk experiment", "jon brelig"
To clear any of those from the default, just remove them from the code (line 18, remove the " marks and comma as well). To add a requester to the block list, click the "BLOCK" button next to their name. To reset to default, click "Reset blocklist" at the top.
1.3.0.11: Added a line (line 24) to change the hit export to text symbol to whatever you'd like.
1.3.0.12: Changed such that the "reset blocklist" is now a confirm dialog in case you misclick.
1.3.0.13: Updated an error with no TO hits.
1.3.0.14: Initial method of editing the existing blocklist to add/remove requesters manually. I'd like a better way of doing it, expect that to be coming.
1.3.0.15: Added "hits available" to default template per request.
1.3.1.0: Major release because of all the changes so far. This one has logical updating of the block list. What's that mean? It means when you click "Edit Blocklist" you'll get a textarea you type in. Remove requesters, add requesters, whatever you'd like. Then just click save and it saves.
1.3.1.1: Updated with Miku's new API link.
1.3.1.2: Fixed correct for skips to accurately reflect the pages you select.
