Wikidata talk:REST API feedback round

Feedback on the JSON format

Hi everyone,

We are getting closer to a first proper version of the Wikibase REST API and there is one more thing I'd love to have your feedback on. One of the important benefits of the REST API should be that it is much better tailored to the Wikibase data model than the existing ActionAPI. We've now finished the JSON format for the REST API and written down the differences from the Action API. You can find the detail at https://doc.wikimedia.org/Wikibase/master/php/rest_data_format_differences.html. I would appreciate your feedback on this. Does this make sense to you? Anything we overlooked? Anything that's odd?

Cheers Lydia Pintscher (WMDE) (talk) 16:51, 5 December 2022 (UTC)

LGTM though I would've liked to have seen examples of no-value or unknown value BrokenSegue (talk) 22:25, 5 December 2022 (UTC)

Wow, that's some significant changes but the old format was pretty verbose so this looks a lot better. The one thing that stood out to me as possibly a problem was removing the grouping of qualifiers by property; I'm not sure if that's a big deal, the list of qualifiers on a single statement shouldn't be very long. But I think that lookup by qualifier instead of having to search through the list of qualifiers is helpful in some cases at least. For "statements" you have property as a key with value an array of the claims for that property, so I think it might make sense to do the same for qualifiers. But probably not a serious issue to leave as is. ArthurPSmith (talk) 22:29, 5 December 2022 (UTC)

I'm glad the "snak" term is gone, that's probably my favorite change here. Everything is a lot simpler with the new API, it all makes sense to me. Nicereddy (talk) 03:32, 6 December 2022 (UTC)

For Statement data could we adjust it so that the data-type doesn't have to be repeated for every statement and can just be a field of the statement object?

{

"statements": {

"P31": {

"data-type": "string",

"statements": [

{

"id": "Q11$4A2F60EA-C779-42D5-8516-A8C26E3ED571",

"rank": "normal",

"qualifiers": [],

"references": [],

"value": {

"type": "value",

"content": "something"

}

}

]

}

}

} Lectrician1 (talk) 20:22, 12 December 2022 (UTC)

@Lectrician1: I think it’s good to have a uniform structure used both for statements and for qualifiers and references. That being said, I can imagine omitting the data types – but then omit them entirely, and provide this information through another endpoint. Which would make sense, as most API users probably don’t need this information in each and every request; they can cache or even hardcode the data types, saving bandwidth, as they change very, very rarely, and even if they do change, it’s mostly directly after being created. (P.S. could you please edit your above comment to use a <syntaxhighlight> block instead of unformatted text? It would make it much easier to read.) —Tacsipacsi (talk) 09:02, 15 December 2022 (UTC)

@Lydia Pintscher (WMDE): I have two comments about the data type change:

• Why the datatype → data-type renaming? As far as I see, it’s the only key in the new structure that’s not a valid identifier in JavaScript (and many other languages). If it’s called datatype, I can access like an object field:

  const datatype = response.statements.P31.property.datatype
  

  but with it being called data-type, I can access it only like an array key:

  const datatype = response.statements.P31.property['data-type']
  

  and I’m not sure if it’s possible to create TypeScript interfaces describing it or object destructuring:

  const { id, datatype } = response.statements.P31.property // `data-type` would be invalid here
  

• The old structure contained two types: mainsnak.datatype and mainsnak.datavalue.type. While in the example, both are string, they mean different things: the former is the data type of the property, the latter is the value type. I think both can be equally important, so please either remove both (as I suggested above), or retain both.

—Tacsipacsi (talk) 09:02, 15 December 2022 (UTC)

Thank you! I'll discuss that with the team. I'd also love to hear hwat others here think. Lydia Pintscher (WMDE) (talk) 17:31, 18 December 2022 (UTC)

@Lydia Pintscher (WMDE) I definitely would want to keep the data-type (with whatever name we want to call it) in the statements, because it's important context. The alternative would be to either pull down every property from the Wikibase instance at runtime to cache it, or we'd need to make queries to a Property endpoint whenever we hit a new property we'd not seen before (or both!). I think having that data directly in the response is *a lot* simpler. Nicereddy (talk) 23:54, 27 December 2022 (UTC)

One thing I've noticed after going through and updating the Ruby gem I've built for the new REST API is that there are some types that are a bit weird. Namely, the "url" type, which used to map to "string" when we had the "datatype" and "value.type" distinction. I don't think it's worth including both types anyway, I can easily just update the code to map URL to the String type in my gem, but I figured I'd note it as a potential weird quirk. Also, do we have a full list of possible data-types that the API can expose? Nicereddy (talk) 23:58, 27 December 2022 (UTC)

We do have a human-readable list of possible data types (with corresponding value types): Special:ListDatatypes. I didn’t find a machine-readable equivalent for it, though. —Tacsipacsi (talk) 10:59, 29 December 2022 (UTC)

Thanks! That's helpful. Unfortunately, I think that doesn't quite match what's used by the REST API, e.g. `wikibase-entityid` is unused by the REST API at this point, and has been replaced by `wikibase-item`, and `globecoordinate` in this list is called `globe-coordinate` in the REST API. Nicereddy (talk) 17:14, 29 December 2022 (UTC)

Fields param

I know this isn't really related but everytime I see a "fields" param on a REST API to control what is being returned it just makes me wonder why they aren't using GraphQL instead. RPI2026F1 (talk) 12:13, 21 December 2022 (UTC)

GraphQL is on the table for later as well. Lydia Pintscher (WMDE) (talk) 13:38, 6 January 2023 (UTC)

Feedback from Nicereddy (round 2)

Hi! I'm updating my Wikidatum Ruby gem to support the new format of the REST API. One thing that I think should probably be fixed prior to release is the type "commonsMedia". It should be "commons-media", to match "globe-coordinate", "wikibase-item", etc. Nicereddy (talk) 22:28, 27 December 2022 (UTC)

These are the possible "data-type" values I know of:

• globe-coordinate
• monolingualtext
• quantity
• string
• time
• wikibase-item
• commonsMedia
• url

"monolingualtext" and "commonsMedia" are inconsistent with the rest of the options, and should probably be updated to use a hyphen like globe-coordinate and wikibase-item. Nicereddy (talk) 00:02, 28 December 2022 (UTC)

Thanks for noticing this. I'll file a ticket for it. Lydia Pintscher (WMDE) (talk) 13:39, 6 January 2023 (UTC)

Another thing I've run into: If you use the bot flag as an IP user or as a user without bot permissions, you get a generic 403 "rest-write-denied" error. This should 1) probably be documented in the error possibilities in the OpenAPI spec, and 2) should probably have a more specific error case? I only figured this out by looking through the Wikibase repo to find the logic that triggered this error. Nicereddy (talk) 16:49, 28 December 2022 (UTC)

I'll file a ticket for this as well. Thanks. Lydia Pintscher (WMDE) (talk) 13:40, 6 January 2023 (UTC)

Another question: Will there be (or is there already?) documentation about how authentication works with the REST API? I haven't implemented auth for my gem yet, as I've been waiting for documentation on how it works/how to implement it for a library using the REST API. That'd be quite nice to have, otherwise I'll be pretty stuck reading the PHP code and looking through the tests in the repo trying to figure out how it's supposed to work 😅 Nicereddy (talk) 17:43, 28 December 2022 (UTC)

I've pinged about it again in phab:T314874. Lydia Pintscher (WMDE) (talk) 13:40, 6 January 2023 (UTC)

I think I've got my Ruby gem mostly updated to support the latest REST API JSON format now! You can see the gem here: https://github.com/connorshea/wikidatum

I've only implemented the basic GET endpoints and the DELETE and POST endpoints for statements, though. I'll wait to implement the PATCH/PUT endpoints for later, after things have fully stabilized. I've tested that these endpoints all work, although I've been unable to test the GlobeCoordinates or Time types due to https://phabricator.wikimedia.org/T324680. Overall, the changes that have been made were absolutely for the better, and make the response body *a lot* simpler to follow. I still need to test further with creating references and qualifiers, and I haven't implemented authentication. It also doesn't support the commonsMedia or URL data types yet. But generally, it's working pretty well for what I've implemented thus far. Thanks to everyone on the WMDE team that worked on the new REST API over the last year! :) Nicereddy (talk) 17:50, 28 December 2022 (UTC)

\o/

That's great to hear. Thanks for updating the Ruby gem. I agree it makes sense to wait with the PATCH route. Hopefully that is resolved within the next few weeks when the security review is finished.

Fixing the URL issue on beta is next on my list. Lydia Pintscher (WMDE) (talk) 13:42, 6 January 2023 (UTC)

The Wikibase REST API is now available for testing on Test.Wikidata (January 2023)

The new REST API is available for testing on test.wikidata.org now. You can find out more in the announcement.

Feedback from BrokenSegue (talk)

What have you tried doing with the new Wikibase REST API?

Just messed around with it in my browser.

What worked well?

Definitely less verbose than the existing API.

What didn't work well?

I assume it's intentional that the `/entities/items/{item_id}/labels` endpoint doesn't work? I'm a little confused by the example that does https://wikidata.beta.wmflabs.org/w/rest.php/wikibase/v0/entities/items/Q598520?_fields=labels since that seems to replace the need for this endpoint? Also why is there an underscore in the param name? This isn't mentioned in the API specification page. It's cleaner the original way.

Is there a reason to support two formats for patching/putting/deleting statements? If I have the statement ID why would I bother with the the form that uses the item ID? Seems redundant.

What are you missing the most?

Adding a reference/qualifier to a claim is a common operation. Maybe that could get its own endpoint? I imagine I could make a JSON patch to do this but I'm not familiar with that format though I could learn.

One thing I would like (though I imagine this isn't easy or in scope) would be to get constraint violations on a statement when I query a statement.

Other

It's a little weird that the URL is /w/rest.php/wikibase/v0/entities/items/Q598520/statements Wouldn't it be cleaner if it were just /w/wikibase/v0/... Really minor point though.

BrokenSegue (talk) 17:49, 19 January 2023 (UTC)

Feedback from Pintoch

I somehow missed this page and wrote feedback there: Wikidata_talk:REST_API#Datavalue_types_and_consistency_with_the_JSON_dump. I hope it's okay! − Pintoch (talk) 20:43, 22 January 2023 (UTC)

The Wikibase REST API is now available on Wikidata (January 2023)

The new REST API is available on Wikidata now. We would love to hear your feedback and input for how to build it out further.

Feedback from XXX

What have you tried doing with the new Wikibase REST API?

Can you briefly describe what you have been doing with the new Wikibase REST API

What worked well?

Is there anything that you especially like?

What didn't work well?

Did you run into any bugs? Something you expected to work differently?

What are you missing the most?

The new Wikibase REST API is not complete. What is currently missing still that would make the API much more useful for you? That will help us prioritize the next steps.

Other

Notes, questions, other piece of feedback or anything else you'd like to bring up?

Feedback from JakobVoss

What have you tried doing with the new Wikibase REST API?

• I'm trying to write an interactive script to change subclass-of statements into instance-of statements.

What worked well?

• Read access via documented endpoints.

What didn't work well?

• The base URL documented at https://doc.wikimedia.org/Wikibase/master/js/rest-api/ does not work (always 404 errors), so I used https://wikidata.beta.wmflabs.org/w/rest.php/wikibase/v0 instead.

What are you missing the most?

• The JSON format could still be simplified.
  • At least statement field property should directly hold the property id
  • property.data-type should be moved to type or datatype (please avoid non-letters in field names to not require escaping in several programming languages!)
  • value should directly contain value.content and always be string, object or null
  • special value types novalue and somevalue should be encoded in type (nobody is interested in the property data type of a novalue/somevalue) and value should be set to null in this case.
• The statement identifier always contains the entity identifier (e.g. Q42 in Q42$F078E5B3-F9A8-480E-B7AC-D97778CBBEF9), so an endpoint /entities/statements/{statement_id} would make more sense. Otherwise client applications either need to pass around two ids or parse entity id from statement id. I wrote a helper function to map statement_id to /entities/items/{item_id}/statements/{statement_id} but why add this step to every client?
• I did not try editing because I did not find any documentation on authentification.

Other

Thanks! -- JakobVoss (talk) 20:36, 25 January 2023 (UTC)

Comments

Thank you! The URL in the docs has been improved and will be further improved in phab:T329080. We have also added documentation for authentication at Wikidata:REST API/Authentication. For the remaining points I'll have a closer look with the team. --Lydia Pintscher (WMDE) (talk) 17:22, 7 February 2023 (UTC)

Feedback from Bean49

Editing

Could someone help me, why do I get HTTP 403 Forbidden when I try to add a statement?

I tried POST /entities/items/{item_id}/statements on https://www.wikidata.org/w/rest.php/wikibase/v0/ with Jersey-client basic authentication. --Bean49 (talk) 17:20, 27 January 2023 (UTC)

did you follow Wikidata:REST API/Authentication? BrokenSegue (talk) 18:11, 27 January 2023 (UTC)

No. I didn't know that page. --Bean49 (talk) 18:31, 27 January 2023 (UTC)

 Resolved I succeeded adding a statement. --Bean49 (talk) 21:21, 27 January 2023 (UTC)

Other

I think it would be useful to add a

GET /entities/items/{item_id}/statements/{property_id} or

GET /entities/items/{item_id}/statements?property={property_id}

to retrieve only statements with certain property, just like Lua function mw.wikibase.getAllStatements or action API wbgetclaims.

What is the purpose of GET /entities/items/{item_id}/statements/{statement_id}? Who and when knows the statement_id before retrieving the whole bunch? Why would he retrieve it again? --Bean49 (talk) 11:51, 28 January 2023 (UTC)

The property ID thing seems reasonable. E.g. from the "real" wikidata, https://www.wikidata.org/wiki/Q42 Douglas Adams the author has a date of birth https://www.wikidata.org/wiki/Property:P569 . So anytime you would look for a bunch of people, if you would want to get their date of birth but nothing else, you would just retrieve P569 every time. It's like "SELECT name(P1559), date_of_birth(P569) FROM my_people". 2.205.76.198 13:00, 28 January 2023 (UTC)

I'm novice in REST, but to conform with other parts of the API, I read that we could add filters to the results, something like:

GET /entities/items/Q42/statements?property=P569

--Bean49 (talk) 09:26, 1 February 2023 (UTC)

Comments

Thank you! We now have phab:T329092 for getting data based on the Property ID of the statement. --Lydia Pintscher (WMDE) (talk) 17:25, 7 February 2023 (UTC)

Feedback from Nicereddy

What have you tried doing with the new Wikibase REST API?

Writing a basic script for scraping data off of associated English Wikipedia articles, given the Wikidata item.

What worked well?

I really like how much easier it is to read and parse the responses :)

What didn't work well?

One issue I ran into is that the sitelinks (e.g. on https://www.wikidata.org/w/rest.php/wikibase/v0/entities/items/Q2377) will have spaces and such in them, which need to be URL-encoded in the specific way that MediaWiki likes to do these things. This is somewhat annoying, and it'd be nice if it were easier to get the actual URLs for the sitelinks.

What are you missing the most?

It'd be nice if the URL-encoded titles (or even full URLs) were provided as part of the sitelinks response. At the least, it'd be good to have something documented for the specifics on converting the titles returned by the API into usable URLs. This is especially important for special characters in article titles.

Nicereddy (talk) 01:14, 6 February 2023 (UTC)

Comments

Thank you! I've filed phab:T329093 to look into the sitelink issues more closely. --Lydia Pintscher (WMDE) (talk) 17:30, 7 February 2023 (UTC)

Feedback from Wolfgang Fahl

What have you tried doing with the new Wikibase REST API?

Trying to find the specification / interface description, examples and a working demo

What worked well?

I could leave my feedback here.

What didn't work well?

• I didn't find the specification/interface description or working demo - the links are missing in these discussions

What are you missing the most?

1. specification of the REST API
2. description of the REST API
3. examples for using the REST API
4. a working demo
5. a python library for using the REST API

Comments

WolfgangFahl (talk) 09:55, 13 February 2023 (UTC)

Hi Wolfgang,

The documentation and specification we have are all listed on Wikidata:REST API. Lydia Pintscher (WMDE) (talk) 11:17, 13 February 2023 (UTC)

Feedback from Erick

I haven't tried the API unfortunately. The API looks much cleaner than my pywikipedia script. Great.

I used to try to import a database with the help from Wikidata:Dataset Imports. Wikidata:Dataset Imports was not flexible compared to REST API. If I have the API, it should be much straightforward to get an import. I also used to do research with Wikidata. For that, I use Wikidata dump and Spark. There are a lot of items (in reality, mostly papers).

I would recommend a note to people working on dataset imports. And the RESTful API looks also useful for librarains. It could be a good engagement. Then the harder question. What to build? You have the choices for sure. I think people doing batch imports would love it. It can be a question whether to support it in the server side or in the client side, i.e. pywikipedia. Fantasticfears (talk) 18:06, 7 March 2023 (UTC)

Can I use REST API without providing an email address?

I have started using the REST API, but I can't use REST API on my bot account because email address must be confirmed before creating OAuth applications. I don't want to receive any emails from my bot account. Midleading (talk) 11:41, 20 April 2023 (UTC)

In Special:Preferences you have the option to disable email from other users, as well as the option to disable email notifications in the "Notifications" section of Preferences. With those settings you should not get any emails (other than ones that are unavoidable, which should be extremely few in number). Harej (talk) 18:35, 23 July 2023 (UTC)

Batching requests

I notice that the API is structured to submit individual statements at a time. If I am adding a large number of statements, including multiple statements to a single entity. This could potentially slow things down if I am limited to one edit per request. Being able to batch requests together into a single request would be a useful feature. Harej (talk) 18:30, 23 July 2023 (UTC)

We will look into this as part of editing on the level of whole Items. For now it's unfortunately not possible. Lydia Pintscher (WMDE) (talk) 15:44, 4 September 2023 (UTC)