Wikidata talk:Tools/Archive 2

Use of wb_terms

Hello everyone,

Wikidata's wb_terms database table is replicated on toolforge, and people can build scripts, tools, etc using these data.

As we're considering phasing out this database table, we want to understand what and how data is used, so we can offer some reasonable replacement(s) for the users.

If you are the author of a tool using wb_terms replica, or use replicas in any other way, please provide some basic information on what are you using, and how, preferably by adding a comment to this Phabricator task.

Example:

• Usage: I have created a tool that finds Wikidata items that have English label, but are missing a label in my native language
• What data I use: I query label data from wb_terms, and then process results in my tool to find the gaps.

We don't set any hard deadline for this survey, but it would be great to have some overview of existing usage of the replica by the end of June 2018. If you have any other questions that you think are related, do not hesitate to ask on Phabricator.

Cheers, Lea Lacroix (WMDE) (talk) 10:12, 19 June 2018 (UTC)

Wikibase’s maxlag now takes dispatch lag in account

This change impacts people running bots and semi-automated tools to edit Wikidata.

Hello all,

Based on the previous discussions that happened around the limitation set up to fix the important dispatch lag on clients, we came with a new solution to try.

The database behind Wikidata is replicated to several other database servers. At each edit, the changes are replicated to these other servers. There is always a short lag, which is usually less than a second. If this lag is too high, the other databases can’t synchronize correctly, which can cause problems for reading and editing Wikidata, or reusing data on other projects.

If the lag is too high on too many servers, the master database stops accepting new edits. When the lag is close to the limit, the system is prioritizing “humans” edits and ignore the edits from bots, sending back an error. This limit is set up by the maxlag option in the API.

People writing bots can set up a number as maxlag for their bot. The default value is 5. This number is used to evaluate two things: the replication lag between master database and replicas, and the size of the job queue.

On Tuesday, July 3rd, maxlag will also evaluate the dispatch lag between Wikidata and clients (eg Wikipedias).

The dispatch lag is the latency between an edit on Wikidata and the moment when it’s shown on clients. Its median value is around 2 minutes.

If you’re running a bot and using a standard configuration (maxlag=5), when the median of dispatch lag is more than 300 seconds, your bot edits won’t be saved and will return an error.

If this change is impacting your work too much, please let us know by letting a comment in this ticket. This is also where you can ask any question. You can also change your configuration in order to increase the maxlag limit.

More information: Wikidata dispatch Grafana board

Thanks for your constructive feedback, Lea Lacroix (WMDE) (talk) 12:30, 28 June 2018 (UTC)

[Breaking change] Important for Wikidata tools maintainers: wb_terms table to be dropped at the end of May

Hello all,

This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by querying directly Labs database replicas.

In May-June 2019, the Wikidata development team will drop the wb_terms table from the database in favor of a new optimized schema. Over years, this table has become too big, causing various issues.

This change requires the tools using wb_terms to be updated. Developers and maintainers will need to adapt their code to the new schema before the migration starts and switch to the new code when the migration starts.

The migration will start on May 29th. On May 15th, a test system will be available for you to test your code.

The table being used by plenty of external tools, we are setting up a process to make sure that the change can be done together with the developers and maintainers, without causing issues and broken tools. Most of the documentation and updates will take place on Phabricator:

• In this Phabricator task, you can find a description of the changes and the process, and you can ask for more details or for help in the comments. This is also where updates will be announced if necessary.
• On the Tool Builders Migration board you will find all the details about the migration, how to update your tool, and you can add your own tasks.
• If you need to discuss with the Wikidata developers or get more specific help, we set up two dedicated IRC meetings and a session at the Wikimedia hackathon. More information in this task.

We are aware that this change will ask you to make some important changes in your code, and we are willing to help you as much as our resources allow us to. We hope that you will understand that this change is made to avoid bigger issues in the near future.

Note that this change is not impacting Wikibase instances outside of Wikidata. A dedicated migration plan and announcement will follow.

We strongly encourage you to not wait until last minute to make the changes in your code. If you have any question or issue, we will be happy to help. In order to keep the discussions in one place, please ask questions or raise issues directly in the Phabricator task and board.

Thanks for your understanding, Lea Lacroix (WMDE) (talk) 14:26, 24 April 2019 (UTC)

Scholia gadget link

I am wondering if it would be possible to get a Scholia link gadget approved? Specifically I am thinking a version Daniel Mietchen modified from the (Magnus Manske's?) Reasonator gadget. Mietchen's version is available here: [1] appropriately renamed to "Gadget-Scholia.js". — Finn Årup Nielsen (fnielsen) (talk) 12:07, 17 May 2019 (UTC)

Change the property of a statement

Is there a tool to change the property of a statement? For example, Eva Bacharach (Q1379109) claims that Jicchak Hakohen Katz (Q21875216) was her child (P40). This seems unlikely as Jicchak was 30 years older than Eva; he was in fact her father (P22). I know I can type in the new property, copy all the references, qualifiers, etc. from the old one and delete the old one but that would be a very time-consuming and error-prone process for multiple cases and I'm sure you have developed a better way. I've found tools to do similar things such as moving the statement to a different item but nothing that does quite what I need. I expect that behind the scenes such a tool would actually insert and delete; that's exactly what I need as long as the end result is equivalent to changing the property name.

Although an experienced enwiki editor, I'm still finding my way around Wikidata so please feel free to move this discussion to a more appropriate venue. Thanks, Certes (talk) 14:50, 22 October 2019 (UTC)

I found WT:Tools/Archive 1#Property changing which suggests using Daty, but that seems to require libhandy which requires Ubuntu 19.04 and I'm not ready to upgrade that far yet. Certes (talk) 16:41, 22 October 2019 (UTC)

I, too, have been looking for such a tool. It would be useful to do quick error correction without having to retype it in under the new Property. Senator2029 21:28, 26 February 2020 (UTC)

Looking for a tool: Class tree viewer

I'm looking for a tool that I haven't seen in a while, that doesn't seem to be listed in any of these lists. It generates a tree of classes and subclasses, with different colored arrows for subclass and instance-of relations. Anyone remember what it was called, or where it can be found? --Yair rand (talk) 19:38, 1 January 2020 (UTC)

Ah, it was the "Wikidata Taxonomy Browser", which was mentioned in this post on the mailing list by Markus Kroetzsch, but the link is now broken. Pinging @Markus Krötzsch: Do you know if that tool is still available anywhere? --Yair rand (talk) 22:11, 2 January 2020 (UTC)

Quick query at coordinates

Similarly to that "..." tool

, it would be nice have same link with coordinate location (P625) ==> it would help find other duplicates

. Bouzinac (talk) 10:30, 30 June 2020 (UTC)

Finding duplicates based on coordinates is not as easy as based on item statements, as coordinates are rarely exact duplicates, but rather often the accuracy is different and/or the coordinates themselves differ by up to a few arch seconds. I’m not sure how to handle this, but adding a search without taking this into account would be pointless. —Tacsipacsi (talk) 11:29, 30 June 2020 (UTC)

A query somewhat like that one https://w.wiki/Vg9 would be indeed useful. Bouzinac (talk) 13:35, 30 June 2020 (UTC)