Wikibase’s maxlag now takes dispatch lag in accountEdit
This change impacts people running bots and semi-automated tools to edit Wikidata.
Based on the previous discussions that happened around the limitation set up to fix the important dispatch lag on clients, we came with a new solution to try.
The database behind Wikidata is replicated to several other database servers. At each edit, the changes are replicated to these other servers. There is always a short lag, which is usually less than a second. If this lag is too high, the other databases can’t synchronize correctly, which can cause problems for reading and editing Wikidata, or reusing data on other projects.
If the lag is too high on too many servers, the master database stops accepting new edits. When the lag is close to the limit, the system is prioritizing “humans” edits and ignore the edits from bots, sending back an error. This limit is set up by the maxlag option in the API.
People writing bots can set up a number as maxlag for their bot. The default value is 5. This number is used to evaluate two things: the replication lag between master database and replicas, and the size of the job queue.
On Tuesday, July 3rd, maxlag will also evaluate the dispatch lag between Wikidata and clients (eg Wikipedias).
The dispatch lag is the latency between an edit on Wikidata and the moment when it’s shown on clients. Its median value is around 2 minutes.
If you’re running a bot and using a standard configuration (maxlag=5), when the median of dispatch lag is more than 300 seconds, your bot edits won’t be saved and will return an error.
If this change is impacting your work too much, please let us know by letting a comment in this ticket. This is also where you can ask any question. You can also change your configuration in order to increase the maxlag limit.
More information: Wikidata dispatch Grafana board
[Breaking change] Important for Wikidata tools maintainers: wb_terms table to be dropped at the end of MayEdit
This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by querying directly Labs database replicas.
In May-June 2019, the Wikidata development team will drop the
wb_terms table from the database in favor of a new optimized schema. Over years, this table has become too big, causing various issues.
This change requires the tools using
wb_terms to be updated. Developers and maintainers will need to adapt their code to the new schema before the migration starts and switch to the new code when the migration starts.
The migration will start on May 29th. On May 15th, a test system will be available for you to test your code.
The table being used by plenty of external tools, we are setting up a process to make sure that the change can be done together with the developers and maintainers, without causing issues and broken tools. Most of the documentation and updates will take place on Phabricator:
- In this Phabricator task, you can find a description of the changes and the process, and you can ask for more details or for help in the comments. This is also where updates will be announced if necessary.
- On the Tool Builders Migration board you will find all the details about the migration, how to update your tool, and you can add your own tasks.
- If you need to discuss with the Wikidata developers or get more specific help, we set up two dedicated IRC meetings and a session at the Wikimedia hackathon. More information in this task.
We are aware that this change will ask you to make some important changes in your code, and we are willing to help you as much as our resources allow us to. We hope that you will understand that this change is made to avoid bigger issues in the near future.
Note that this change is not impacting Wikibase instances outside of Wikidata. A dedicated migration plan and announcement will follow.
We strongly encourage you to not wait until last minute to make the changes in your code. If you have any question or issue, we will be happy to help. In order to keep the discussions in one place, please ask questions or raise issues directly in the Phabricator task and board.
I am wondering if it would be possible to get a Scholia link gadget approved? Specifically I am thinking a version Daniel Mietchen modified from the (Magnus Manske's?) Reasonator gadget. Mietchen's version is available here:  appropriately renamed to "Gadget-Scholia.js". — Finn Årup Nielsen (fnielsen) (talk) 12:07, 17 May 2019 (UTC)