Wikidata:SPARQL query service/WDQS backend update/August 2021 scaling update
Wikidata community members,
Thank you for all of your work helping Wikidata grow and improve over the years. In the spirit of better communication, we would like to take this opportunity to share some of the current challenges Wikidata Query Service (WDQS) is facing, and some strategies we have for dealing with them.
WDQS currently risks failing to provide acceptable service quality due to the following reasons:
1. Blazegraph scaling
- a. Graph size. WDQS uses Blazegraph as our graph backend. While Blazegraph can theoretically support 50 billion edges, in reality Wikidata is the largest graph we know of running on Blazegraph (~13 billion triples), and there is a risk that we will reach a size limit of what it can realistically support. Once Blazegraph is maxed out, WDQS can no longer be updated. This will also break Wikidata tools that rely on WDQS.
- b. Software support. Blazegraph is end of life software, which is no longer actively maintained, making it an unsustainable backend to continue moving forward with long term.
Blazegraph maxing out in size poses the greatest risk for catastrophic failure, as it would effectively prevent WDQS from being updated further, and inevitably fall out of date. Our long term strategy to address this is to move to a new graph backend that best meets our WDQS needs and is actively maintained, and begin the migration off of Blazegraph as soon as a viable alternative is identified.
In the interim period, we are exploring disaster mitigation options for reducing Wikidata’s graph size in the case that we hit this upper graph size limit: (i) identify and delete lower priority data (e.g. labels, descriptions, aliases, non-normalized values, etc); (ii) separate out certain subgraphs (such as Lexemes and/or scholarly articles). This would be a last resort scenario to keep Wikidata and WDQS running with reduced functionality while we are able to deploy a more long-term solution.
2. Update and access scaling
- a. Throughput. WDQS is currently trying to provide fast updates, and fast unlimited queries for all users. As the number of SPARQL queries grows over time alongside graph updates, WDQS is struggling to sufficiently keep up in each dimension of service quality without compromising anywhere. For users, this often leads to timed out queries.
- b. Equitable service. We are currently unable to adjust system behavior per user/agent. As such, it is not possible to provide equitable service to users: for example, a heavy user could swamp WDQS enough to hinder usability by community users.
In addition to being a querying service for Wikidata, WDQS is also part of the edit pipeline of Wikidata (every edit on Wikidata is pushed to WDQS to update the data there). While deploying the new Flink-based Streaming Updater will help with increasing throughput of Wikidata updates, there is a substantial risk that WDQS will be unable to keep up with the combination of increased querying and updating, resulting in more tradeoffs between update lag and querying latency/timeouts.
In the near term, we would like to work more closely with you to determine what acceptable trade-offs would be for preserving WDQS functionality while we scale up Wikidata querying. In the long term, we will be conducting more user research to better understand your needs so we can (i) optimize querying via SPARQL and/or other methods, (ii) explore better user management that will allow us to prevent heavy use of WDQS that does not align with the goals of our movement and projects, and (iii) make it easier for users to set up and run their own query services.
Though this information about the current state of WDQS may not be a total surprise to many of you, we want to be as transparent with you as possible to ensure that there are as few surprises as possible in the case of any potential service disruptions/catastrophic failures, and that we can accommodate your work as best as we can in the future evolution of WDQS. We plan on doing a session on WDQS scaling challenges during WikidataCon this year at the end of October 2021.
Thanks for your understanding with these scaling challenges, and for any feedback you have already been providing. If you have new concerns, comments and questions, you can best reach us on this page's talk page. Additionally, if you have not had a chance to fill out our survey yet, please tell us how you use the Wikidata Query Service (see privacy statement)! Whether you are an occasional user or create tools, your feedback is needed to decide our future development.
Best,
WMF Search + WMDE