Wikidata:SPARQL query service/WDQS backend update/April 2022 scaling update

Wikidata Query Service scaling update, April 2022 edit

  • The Search team kicked off Q4 (April - June 2022) with a primary goal for WDQS scaling to create a testing framework for the shortlisted graph backend candidates. We also hope that this testing framework can be more widely published as a standard benchmark for the industry. Due to team resources, it is not within the scope of work this quarter to evaluate any of the candidate backends on our short list, and consequently, we will not be making a final decision at this time regarding which option to move forward with.
  • Andrea has been looking into which Blazegraph features and capabilities can/should be ported to a new backend, so that user impact is minimized. More details here (work in progress, subject to change)
  • Andrea has also started creating a design and suggestions for Wikidata SPARQL query testing, so that we can evaluate shortlisted candidate backends for how they will perform on Wikidata. More details here (work in progress, subject to change)
  • Aisha is continuing to productionize WDQS subgraph and query analysis code, so that we will be able to replicate her previous analyses in the future
  • The WDQS update lag SLO (service level objective) has been reduced from all updates being reflected in WDQS in under 10 minutes 99% of the time to 95% of the time. See more details about this change here.
  • jvmquake was deployed and activated. This will help Blazegraph servers to recover when they struggle with high memory pressure
  • One blazegraph server went down for 4+ days during a long week-end due to a known instability.