Greasy Fork

Hi,

is there a way to use a webhook with a private GitLab repository? Everything that I tried so far has failed, namely:

1. Use a "sync identifier" that contains an access token for GreasyFork to be able to access the script, which looks something like that:
https://gitlab.com/api/v4/projects/00000/repository/files/script.user.js/raw?ref=main&private_token=glpat-XXXX

This will allow the periodic auto sync of the script, but does not work with webhooks.

When GreasyFork tries to match the generated sync_identifiers for every changed file it uses the following method, which generates totally different identifiers to the one I have configured:
https://github.com/JasonBarnabe/greasyfork/blob/ac683598b7b24aec97b7c826cec2a273083998b9/lib/gitlab.rb#L36-L41

The generated urls (https://gitlab.com/user/repository/raw/main/script.user.js) will not match the sync url that I have configured, so when the webhook is called, I get a 200 response, but with the following payload and the script is not updated:

{"updated_scripts":[],"updated_failed":[],"message":"No scripts found."}


2. If I put the "sync_identifier" as expected by GreasyFork (https://gitlab.com/user/repository/raw/main/script.user.js), this leads to a 500 error when the webhook is called with the following error:

"Response body is empty"

I assume the reason for that is that GreasyFork cannot really download the script file, as it is located in a private repository and the access token is missing from the url. Note that I have manually uploaded an initial version of the script, which contains an @updateURL and @downloadURL, which are set to the url, which allows downloading it (https://gitlab.com/api/v4/projects/00000/repository/files/script.user.js/raw?ref=main&private_token=glpat-XXXX)

Is there a way to make it working?

Thanks!

When GreasyFork tries to match the generated sync_identifiers for every changed file it uses the following method, which generates totally different identifiers to the one I have configured:
https://github.com/JasonBarnabe/greasyfork/blob/ac683598b7b24aec97b7c826cec2a273083998b9/lib/gitlab.rb#L36-L41

Can you provide more detail on this? It may be possible to modify the logic.

When GreasyFork tries to match the generated sync_identifiers for every changed file it uses the following method, which generates totally different identifiers to the one I have configured:

I havent run it locally, I only browsed the code on GitHub and I dont have any Ruby experience, but my understanding of the GitLab (specifically) webhook process is the following:

1. Based on the webhook payload, for all "modified" files the following dict/map is created:
https://github.com/JasonBarnabe/greasyfork/blob/8dd7285243509a2732acbd8f76911494d58e32dc/lib/gitlab.rb#L15-L20

For every changed file the commit hash, the commit message and most importantly a couple of urls where the file might be located will be created.

2. As there is only webhook per user (and not one per script), GreasyFork needs to figure out to which script those changed files belong to, which is done here:
https://github.com/JasonBarnabe/greasyfork/blob/8dd7285243509a2732acbd8f76911494d58e32dc/app/controllers/concerns/webhooks.rb#L116-L121

Every file in the changed file map from step 1 will be augmented and additional information about the related script and script attributes.
This is done using a "sync_identifier" query, which uses the urls generated in step 1 to find the correct script. I think this is where the mismatch happens, which results in the "No scripts found." error:

The sync_identifier of my script looks like that: https://gitlab.com/api/v4/projects/00000/repository/files/script.user.js/raw?ref=main&private_token=glpat-XXXX
While the generated urls for my script look like: https://gitlab.com/user/repository/raw/main/script.user.js

As a result the "script" and "script_attributes" query most likely return nil.

3. Later both "script" and "script_attributes" properties are checked, and as they are nil, the following check fails and the update does not work:
https://github.com/JasonBarnabe/greasyfork/blob/55a76674b656d8c4a97039ec2e8549a5361540c4/app/controllers/concerns/webhooks.rb#L131-L137


If you can extend the url generation to include those private urls to also include the url

https://gitlab.com/api/v4/projects/{payload['project_id']}/repository/files/{filename}/raw?ref={ref.split('/').last}

and drop the "private_token" at least when doing the identifier query in step 2, I think this should help.

Let me know if you need more info.

Thanks for the details. One more thing - can you send me a sample payload that GitLab is sending to Greasy Fork as the webhook? (With the secret token changed of course.) Then I write a test for this scenario.

The webhook request does not contain the secret token, as least not the same token which is used in the sync url.

Here is the sample payload:


{
"object_kind": "push",
"event_name": "push",
"before": "c07b16267070314ee1da3d77f24c727378614125",
"after": "ea6c525a17d2c6f23ebd36b2acfbfb1eb7b36cd1",
"ref": "refs/heads/main",
"ref_protected": true,
"checkout_sha": "ea6c525a17d2c6f23ebd36b2acfbfb1eb7b36cd1",
"message": null,
"user_id": 123456,
"user_name": "MY NAME",
"user_username": "430i",
"user_email": "",
"user_avatar": "https://secure.gravatar.com/avatar/d1a4df204cd9d900bd92207459885b9d?s=80&d=identicon",
"project_id": 456789,
"project": {
"id": 456789,
"name": "repository",
"description": "",
"web_url": "https://gitlab.com/username-or-group/repository",
"avatar_url": null,
"git_ssh_url": "git@gitlab.com:username-or-group/repository.git",
"git_http_url": "https://gitlab.com/username-or-group/repository.git",
"namespace": "namespace",
"visibility_level": 0,
"path_with_namespace": "username-or-group/repository",
"default_branch": "main",
"ci_config_path": "",
"homepage": "https://gitlab.com/username-or-group/repository",
"url": "git@gitlab.com:username-or-group/repository.git",
"ssh_url": "git@gitlab.com:username-or-group/repository.git",
"http_url": "https://gitlab.com/username-or-group/repository.git"
},
"commits": [
{
"id": "ea6c525a17d2c6f23ebd36b2acfbfb1eb7b36cd1",
"message": "debug: update\n",
"title": "debug: update",
"timestamp": "2024-01-06T00:43:53+01:00",
"url": "https://gitlab.com/username-or-group/repository/-/commit/ea6c525a17d2c6f23ebd36b2acfbfb1eb7b36cd1",
"author": {
"name": "MY NAME",
"email": "[REDACTED]"
},
"added": [

],
"modified": [
"script.user.js"
],
"removed": [

]
}
],
"total_commits_count": 1,
"push_options": {
},
"repository": {
"name": "repository",
"url": "git@gitlab.com:username-or-group/repository.git",
"description": "",
"homepage": "https://gitlab.com/username-or-group/repository",
"git_http_url": "https://gitlab.com/username-or-group/repository.git",
"git_ssh_url": "git@gitlab.com:username-or-group/repository.git",
"visibility_level": 0
}
}


And here are the headers:


Content-Type: application/json
User-Agent: GitLab/16.8.0-pre
X-Gitlab-Event: Push Hook
X-Gitlab-Webhook-UUID: bb7d6f9c-4d2a-4435-96a2-21a443c8a9a5
X-Gitlab-Instance: https://gitlab.com
X-Gitlab-Token: [REDACTED]
X-Gitlab-Event-UUID: c1f7a6bb-cf26-41d3-a0c5-be8d9bddcd53


The token has been redacted by GitLab, so I dont know which token is that, but most likely its the GreasyFork secret.

Hope this helps!

I've made some changes, let me know how it works now.

Thanks for the quick changes! I have configured the sync url to the private url with the access token and the webhook now fails with 500 http code and "Response body is empty".

Looking at your commit I think the culprit might be trying to fetch the script with the "public" url and not with the private url, which was configured as sync identifier, but that's just a speculation.
https://github.com/JasonBarnabe/greasyfork/commit/3591808dd59436b4b33fc36667576d176ff02852#diff-454783472cee6ec78936f91fe3a3ed21f3a0566a437d1c6ed2c9bf20f2754fd4R253

Hmm yeah it expects to be able to get the file through git. I will have to add an exception in this case so that it gets it from the sync URL.

I've adjusted it to pull over HTTP instead git if it thinks it's a private repo.

Hi, finally had the chance to try it out and its working perfectly. Thanks a bunch for the quick fix!