Wikidata

Wikidata:New convenience functions for Lua

This input page is made for everyone who is using Lua on the Wikimedia projects.

The Wikidata development team would like to provide more Lua functions for Wikibase, in order to improve the experience of people who write Lua scripts to reuse Wikidata's data on the Wikimedia projects. Our goal is to help harmonizing the existing modules across the Wikimedia projects, to make coding in Lua easier for the communities, and to improve the performance of the language.

We would like to know more about your habits, your needs, and what could help you.

To answer, please copy the template section below, paste it as a new section below the template, rename the copy's title to indicate who is answering, and feel free to answer the questions that are relevant for you. If you don't feel comfortable with writing in English, you can also write in your preferred language. Thanks a lot for your help!

Template edit

About you edit

  • Username:
  • Main Wikimedia projects where you're active:
  • How would you qualify your level of experience with Lua?

What you're already doing edit

What are the Lua modules you're using? edit

Give us some examples of the modules you're often using, and the modules you've been working on. Please provide a short description of the main purpose of the module, and a link to the documentation or to the code.

What is currently hard to do? edit

Is there anything that you wanted to do, but couldn't, or had to fight hard to find how to do?

Did you run into performance problems? edit

If so, feel free to provide links and examples to help us understand what happened.

What you would like to have edit

Data access edit

What kind of data access do you need? What pieces of data do you need?

What would you like to do (more efficiently)? edit

Feel free to provide concrete examples to help us understand your needs.

What kind of functions do you need? edit

What kind of convenience functions could you imagine us to provide? What new functions would you use?

Other edit

Notes, questions, other piece of feedback

Answer from Yuri Astrakhan edit

About you edit

  • Username: yurik
  • Main Wikimedia projects where you're active: *
  • How would you qualify your level of experience with Lua? advanced/expert

What you're already doing edit

OSM Wikibase installation

What is currently hard to do? edit

  • Module unit testing

What you would like to have edit

Ability to simulate fake entity creation and lookup/injection into a Lua module.

Data access edit

Ideally - data querying, as that would allow Lua-based table generations without relying on bots. --Yurik (talk) 01:02, 18 December 2018 (UTC)[reply]

Answer from the Sanskrit community edit

What kind of functions do you need? edit

What kind of convenience functions could you imagine us to provide? What new functions would you use?

Sanskrit community wants counting with Devanagari Digits in temples like - Death, Age, Born and other calculative works. We need parser function Devanagari digit compatible.

(Hindi - संस्कृत विकिपीडिया में कोई भी सदस्य ऐसा नहीं जो प्रोद्योगिकी के सन्दर्भ में कुछ जानता हो। अतः लुआ के द्वारा उत्पन्न दोषों को संस्कृत विकिपीडिया से हटाना है। बहुत से पृष्ठ हैं, जिन में लुआ त्रुटि आ जाती है, परन्तु हमें ये सब पता ही नहीं। यदि कोई लुआ जानकार व्यक्ति संस्कृत विकिपीडिया की लुआ प्रणालि के दोषों को दूर करने में सहायता करें, तो बहुत अच्छा होगा। इसके साथ साथ ये जो देवनागरी अङ्कों की बात में कर रहा हूं, वो भी आवश्यक है। देवनागरी यदि गणना वाले टेम्पलेट में डालते हैं, तो वो गणना में लुआ त्रुटि ही बाताती है।)

 – The preceding unsigned comment was added by NehalDaveND (talk • contribs).

Answers from Jérémy-Günther-Heinz Jähnick edit

About you edit

  • Username: Jérémy-Günther-Heinz Jähnick
  • Main Wikimedia projects where you're active: Wikimedia Commons (just to upload photos, not involved in the community), Wikidata (now my home Wiki), FR Wiki, other Wikis (generally to add pictures)
  • How would you qualify your level of experience with Lua? Good to explain a result I wish, hard to produce something, except if it is easy.

What you're already doing edit

What are the Lua modules you're using? edit

By a collaborative work, since 2015, we have developped the Module:Cycling race, an algorithm able to work in more than 20 languages. Different users bring their translations, their needs to adaptations, and their skills. On FR Wiki, I use fr:Modèle:Infobox Localité, fr:Modèle:Infobox Biographie2, fr:Modèle:Infobox Film...

What is currently hard to do? edit

To make the code more clever, it suppose to add more and more lines. Some people are able to do that, me not, but it is not a so big problem.

Did you run into performance problems? edit

Yes, the famous limit of 500 functions IF, for example with cycling race with numerous stages as Tirreno-Adriatico 2018. So it is a loss of time because some Wikipedias must write classifications as by the past. By the past, I ask to raise this limit, but it was not done.

What you would like to have edit

Data access edit

For towns, I need the populations according years. For the rest, Wikidata has already more and more datas.

What would you like to do (more efficiently)? edit

The display of datas entered on Wikidata in Wikipedias articles.

What kind of functions do you need? edit

Function more complex that are able to make a better display of datas.

Other edit

My wish is to see functions developped and translated for all Wikipedias. For example, when I use the Infobox Localité for Somain, a former canton is displayed. If I encounter this problem on FR Wiki, other users will have the problem, so in this case, the solution is global. Share datas as we do for photos will lead at more datas. So I like this initiative. We are better when we work together.

On another hand, I discover today on Wikimedia Commons the template Wikidata Infobox and I adore it. When something is well, it is good to say it.

My dream is to have super modules translated in all the languages we use on Wikipedias (and they are numerous). This way, everybody could benefit of datas from Wikidata and work on Wikidata in their language by the addition of translations. It is the key to raise the development of this database and then from all Wikipedias. In parallele, we must have solid documentations, and understandable by everybody. For me, if X from XX Wikipedia can't reuse datas and must enter information by hand in its articles, we loose something.

Sorry for my English. Jérémy-Günther-Heinz Jähnick (talk) 12:16, 27 March 2018 (UTC)[reply]

PS : Another wish, it is a template that is able to give the nationality of a person at a precise moment, with the correct flag (it sometimes change, see flag of Mauritania (Q102977)). Not only we could display changes in infobox for everybody, but we also could use it in competitions. I also read with interest propositions for other Wikimedians and I agree with them. Jérémy-Günther-Heinz Jähnick (talk) 09:53, 28 March 2018 (UTC)[reply]

PPS : I think we should send at all local communities for translation

  • first the list of around 5000 properties that currently exist on Wikidata
  • second a list of around 5000 items the most used or the most important

This way, it will be more easy to permit everybody make their contributions in their own language. For smallest Wikipedias or languages with very few speakers, another navigation will be possible with Wikimedia Commons and the brand new template Wikidata Infobox. Why not an edit-a-thon. Well explained, especially with the link with the new infobox avaible on Commons, this can be a success and bring enormous changes and a better access to information. (I see this as a first phase, a second phase should be to give them infoboxes and tables, the idea is always to work together) Jérémy-Günther-Heinz Jähnick (talk) 09:07, 8 April 2018 (UTC)[reply]

PPPS : it will be interesting to invent new templates able to display in every languages populations of an area or frequentation of places, with graphes. Jérémy-Günther-Heinz Jähnick (talk) 07:58, 10 April 2018 (UTC)[reply]

Hello @Jérémy-Günther-Heinz Jähnick:,
Thanks for your detailed answer and many suggestions. I'd like to know if you have more concrete examples (or even links to things you tried to to already) regarding these two points:
  • What would you like to do (more efficiently)? --> "The display of datas entered on Wikidata in Wikipedias articles."
  • What kind of functions do you need? --> "Function more complex that are able to make a better display of datas."
(Si tu le souhaites, tu peux répondre en français, je me chargerai de la traduction pour mes collègues)
Thanks, Lea Lacroix (WMDE) (talk) 08:53, 12 April 2018 (UTC)[reply]

Answers from Jarek Tuszynski edit

About you edit

  • Username: jarekt
  • Main Wikimedia projects where you're active: Commons, Wikidata
  • How would you qualify your level of experience with Lua?
    • Advanced. I am always learning, but at the moment I wrote and maintain Lua codes used on majority of pages on Commons.

What you're already doing edit

What are the Lua modules you're using? edit

Give us some examples of the modules you're often using, and the modules you've been working on. Please provide a short description of the main purpose of the module, and a link to the documentation or to the code.

What is currently hard to do? edit

Is there anything that you wanted to do, but couldn't, or had to fight hard to find how to do?

  • Need documentation on how to use Lua debug console
  • functions in mw:Extension:Wikibase Client/Lua could use examples. Several of my attempts to add information about Lua function behavior on MediaWiki were reverted as obvious or unnecesary, but we need some space to have per function discussion, expanded documentation, links to related phabricator tickets, etc.

Most difficulties are not due to Lua limitations, but due to MediaWiki, or Wikidata limitations. For example:

  • fetching data from Wikidata that traverses multiple item, like
    • city of birth for people, since it requires SELECT DISTINCT ?city { ?city ^(wdt:P20/wdt:P131*) wd:Q7315; wdt:P31/wdt:P279* wd:Q515 . } data query.
    • genus, family or order of a given organism
There are Wikipedias that write Lua codes to do something equivalent, but it is difficult. See phabricator:T167521

Did you run into performance problems? edit

If so, feel free to provide links and examples to help us understand what happened.

  • fetching data from Wikidata that traverses multiple item discussed above would lead to performance problems.
  • date formatting can be done 2 different ways:
    • datestr = mw.language.new(lang):formatDate( dFormat, timeStamp)
    • datestr = mw.getCurrentFrame():callParserFunction( "#time", { dFormat, timeStamp, lang } )
Both do the same. The first approach seems like a better idea since it is custom made for the purpose. Unfortunately it produces "Lua error: too many language codes requested." on some pages while the second approach does not.
Probably not related to Wikidata --Jarekt (talk) 17:53, 28 March 2018 (UTC)[reply]

What you would like to have edit

Data access edit

What kind of data access do you need? What pieces of data do you need?

  • As discussed above if I am at some commons page and that page is linked from Wikidata through one of the properties than I would like to be able to detect item number or numbers (can place a limit if may items link to the same page)
  • mw.wikibase.getEntityIdForTitle at the moment works only for page title in the local wiki. More useful would be to look it us from any wiki like mw.wikibase.sitelink function does.
  • mw.wikibase.getEntity( id ) returns entity based on item ID, and if redirects to other item returns that entity. I need a way to check if an item is a redirect.

What would you like to do (more efficiently)? edit

Feel free to provide concrete examples to help us understand your needs.

  • mw.wikibase.getEntity( id ) should not be an expensive call if called from page connected to the item through properties like P373 instead of sitelinks
  • c:Module:Wikidata_label _getLabel function fetches label based on item ID and desired language. It tries to call wikibase.getLabelWithLang( id ) if the desired language is not the one returned by that function than much more expensive mw.wikibase.entity:getLabel has to be called. It would be easier to just fetch all labels in all languages or implement wikibase.getLabel that allows you to specify language.

What kind of functions do you need? edit

What kind of convenience functions could you imagine us to provide? What new functions would you use? Many of my modules start with a few very basic functions, which seems too simple to place in specialized module:

  • LangSwitch, as in c:Module:Size
  • common parts of Module:Wikidata should become part of mw.wikibase.entity. Any localization should be for provided language which defaults to language of the uses if not provided or language of project if IP. For example I could use w:en:Module:Wikidata's "dump" function for debugging
  • function for normalizing template inputs:
    • merge inputs from frame.args and frame:getParent().args
    • ignore case of arguments and allow "_" to stand in for " "
    • fetch user language in parameter "lang" if not provided
    • treat empty named parameters and missing parameters the same way
  • operations on lists stored in tables:
    • Something like matlab "find" function to find if entry is in the list and return it's index or indeces
    • Intersection, union or diff of 2 lists
    • table sort that returns sort index
Many of those are not related to Wikidata. If you are looking for Wikidata functions only than please ignore --Jarekt (talk) 17:53, 28 March 2018 (UTC)[reply]

Other edit

Hello @Jarekt:,

Thanks a lot for your answer. I'd like to ask more details about some topics.

  • "I need a way to check if an item is a redirect." Do you have concrete examples of situations where you need this? Feel free to give more details, links, etc.
  • "It would be easier to just fetch all labels in all languages or implement wikibase.getLabel that allows you to specify language." Can you give us one or several use cases where you would need this?

Thanks a lot, Lea Lacroix (WMDE) (talk) 08:56, 12 April 2018 (UTC)[reply]

Hello again @Jarekt:,
For your information, look up for an entity ID for any wiki is already possible, the wiki can be passed as second parameter, like: mw.wikibase.getEntityIdForTitle( 'Берлин', 'ruwiki' ).
About operations on lists stored in tables, is there anything Wikidata-related that you'd like to mention? Lea Lacroix (WMDE) (talk) 12:41, 8 May 2018 (UTC)[reply]


Lea Lacroix (WMDE), sorry I missed your initial ping.
  • About redirects. I figured it out. Code entity = mw.wikibase.getEntity(itemID); if entity.id~=itemID then (we detected redirect) end can detect redirects. Problem solved.
  • About wikibase.getLabel. Currently on Commons the c:template:Label takes 2 parameters: the item ID and a language. To implement it in Lua c:Module:Wikidata label (used on 5M pages) we calls getLabelWithLang function (which pulls only a single label without the need to pull entire entity) if desired language matches what getLabelWithLang would return or we load entire entity and find the language we need otherwise. We could minimize the need to pull entire entity if we allow to specify the language in wikibase.getLabel or allow to pull labels in all languages. So the use case would be a rewrite of c:Module:Wikidata label's p._getLabel function so there is no need for mw.wikibase.getEntity call.
  • Thanks for heads up about getEntityIdForTitle. I will look into it.
  • About operations on lists stored in tables. Please disregard. I misunderstood the request I was thinking "New_convenience_functions_for_Lua" not Wikidata related "New_convenience_functions_for_Lua"
--Jarekt (talk) 13:45, 8 May 2018 (UTC)[reply]

Answers from Thayts edit

About you edit

  • Username: Thayts
  • Main Wikimedia projects where you're active: enwiki
  • How would you qualify your level of experience with Lua? Starting to get experienced :)

What you're already doing edit

What are the Lua modules you're using? edit

I've created en:Module:Wd which is used to include data from Wikidata in all shapes and forms on Wikipedia.

What is currently hard to do? edit

I haven't found a better documentation about the Wikidata datamodel than the one at Help:Wikidata datamodel, but that one needs to be updated. Dumping tables is the way I figured out much of the structure.

Did you run into performance problems? edit

The module I wrote has a lot of functions calling a lot of other functions, and there is some recursion in there. One time I hit the recursion depth limit, but fortunately I didn't really need that extra function call.

What you would like to have edit

Data access edit

There's nothing I'm in need of right now.

What would you like to do (more efficiently)? edit

Nothing that comes to mind.

What kind of functions do you need? edit

Recently I needed a way to get the global site ID (like "enwiki"), but found there is no library function available. I used the Scribunto site library to write a hacky solution as can be seen on line 162 of en:Module:Wd. I've found some comments on the interwebs saying that the function I need could become part of that same site library, but apparently it didn't make it yet.

Other edit

Simply: thanks! Thayts (talk) 21:22, 27 March 2018 (UTC)[reply]

Answers from Jean-Jacques Georges edit

About you edit

  • Username: Jean-Jacques Georges
  • Main Wikimedia projects where you're active: fr, commons, to a lesser extent en + some other projects where I may add pictures
  • How would you qualify your level of experience with Lua? : limited, and unpleasant. Mainly struggling with infoboxes.

What you're already doing edit

What are the Lua modules you're using? edit

None of them, really. I find most Lua modules unpleasant to work with. Most of the time, when I stumble upon a Lua infobox, I tend to replace it with a module not using Wikidata. The worst offender, to me, is fr:Modèle:Infobox Biographie2 which I find very unsatisfactory. IMHO, they need much improvement.

What is currently hard to do? edit

Quite simply : changing what is inside an infobox withouth going to Wikidata. I may start enjoying Wikidata when I'm not forced to go there.

Did you run into performance problems? edit

All the time. I find the infoboxes using wikidata to be very unstable. If they have a graphical charter, it may change whenever you change a single element in wikidata.

Also, lua models seem unable to manage redirections. I.e., when a data actually points to a redirection on Wikipedia fr, the model will be unable to point to the article, and will instead display a "(d)" link pointing to Wikidata, which is completely useless. It is also confusing to the reader who comes to Wikipedia to have useful info, clicks on this kind of link and finds himself on Wikidata, which is of no use and no interest to him.

What you would like to have edit

To be clear, I will find fr:Modèle:Infobox Biographie2 satisfactory when it will be possible to do with it everything that I can do with fr:Modèle:Infobox Biographie. The biographical models should look more like fr:Modèle:Infobox Artiste, which is, I think, a good start. I need to see the code so I can see what is where, and change the model's content at will, adding things, removing some, etc, without having to tamper with Wikidata's content.

What kind of functions do you need? edit

As I said above, I think the models should look like fr:Modèle:Infobox Artiste. What is needed is a wikicode displayed inside the model, so the data can be modified directly in wikipedia, without having to go to Wikidata. I also need to be able to add whatever additional data I want inside the code. I'd also like to have functions that would make it easier to stop the display of unwanted data extracted from Wikidata by the models.

Also, the little crayons inside the models pointing directly to Wikidata are completely useless, both to the readers and to the users. First, they are visually intrusive and should be replaced by something less conspicuous, like a different color shade. Second, it would be much better to have an editing function which would allow me to edit the model itself, and not wikidata.

Generally, I could not care less about Wikidata : what I want is Wikidata to help me complete Wikipedia, not the other way around with Wikipedia forcing me to go to Wikidata when I want to change something.

Other edit

The (d) signs pointing directly to wikidata inside the models are useless and irritating : they should disappear entirely.

Answer from Matěj Suchánek edit

About you edit

  • Username: Matěj Suchánek
  • Main Wikimedia projects where you're active: Czech Wikipedia, Wikidata
  • How would you qualify your level of experience with Lua? not bad

What you're already doing edit

What are the Lua modules you're using? edit

I've been maintaining a module system at Czech Wikipedia: w:cs:Modul:Wikidata. It has a rich subpages system with the ability to somehow customize the behavior (like custom formatting or sorting of values). Its most recent versions has been exported to at least two other wikis (eg. elwiki).

What is currently hard to do? edit

Efficient exploring of entity trees is a field I haven't been successful with.

Did you run into performance problems? edit

Interestingly, only once there was a problem that a particular infobox had problem with the amount of data loaded. It was a very rare case that could be quickly fixed.

What you would like to have edit

Data access edit

You know, the most significant missing feature is the ability to execute queries (which I think won't be easy to implement, if at all). Perhaps a new tool has yet to be invented for this.

What would you like to do (more efficiently)? edit

Some Lua functions don't resolve redirects, see phab:T157868. Detecting and resolving them on the client would influence performance.

What kind of functions do you need? edit

I can imagine a single function that does everything in one call, with just a few settings as a bit-mask: 

mw.wikibase.formatStatements( 'Q123', 'P456', mw.wikibase.BEST_RANK + mw.wikibase.ASCENDING + ... )

Otherwise, I don't miss anything.

Other edit

Just a big thanks! Matěj Suchánek (talk) 16:07, 28 March 2018 (UTC)[reply]

Hey @Matěj Suchánek:,
Thanks for your feedback. I'd need a bit more details on some of your information.
  • w:cs:Special:PrefixIndex/Modul:Wikidata/Sorters -> can you explain a bit what is the need behind this? What kind of sorting are you doing?
  • Efficient exploring of entity trees: did you have a look at this ticket? Would that solve your need or would you need something else?
  • Ability to execute queries: can you provide some use cases? What kind of queries would you like to run?
Thanks a lot, Lea Lacroix (WMDE) (talk) 09:18, 12 April 2018 (UTC)[reply]
  • This is a module which wraps around several methods to put statements into some order (because the order of statements on Wikidata is arbitrary). For example, w:cs:Albert Einstein has an infobox with information about schools, spouses, employers, awards etc. sorted by qualifiers. Or w:cs:Třebíč#Externí odkazy has a navbox ("Části města Třebíč") with the neighborhoods in alphabetical order. Or w:cs:Maxime Vachier-Lagrave has an infobox which shows the current (preferred) and the best ("Nejvyšší") ELO. To get the best ELO, I need to sort the statements and choose the first one.
  • Yes, I'm aware of that ticket. The ideas there look very good.
  • It isn't directly about me, I just recalled some demands for ability to query for backlinks, eg. list of films a person starred in / directed. (We can use ListeriaBot for this, though.)
Matěj Suchánek (talk) 18:05, 12 April 2018 (UTC)[reply]

Answers from Simon Villeneuve edit

About you edit

  • Username:Simon Villeneuve
  • Main Wikimedia projects where you're active: wikidata, frwiki, wikibooks, commons
  • How would you qualify your level of experience with Lua? limited. I copy-past code from one module to another and adapt it the best I can.

What you're already doing edit

What are the Lua modules you're using? edit

Give us some examples of the modules you're often using, and the modules you've been working on. Please provide a short description of the main purpose of the module, and a link to the documentation or to the code.

What is currently hard to do? edit

Is there anything that you wanted to do, but couldn't, or had to fight hard to find how to do?

  • Yes, almost everything.

Did you run into performance problems? edit

If so, feel free to provide links and examples to help us understand what happened.

  • Not really.

What you would like to have edit

Data access edit

What kind of data access do you need? What pieces of data do you need?

  • I don't know enough about Lua to answer these questions. Maybe I can talk about the Wikidata:WikiProject Weather observations/en, where we plan to use a lot of data from the Canadian federal government meteo service. For now, following the « New York City way », I think that Commons will be a better place to upload them than Wikidata, but maybe you can prove me wrong. Eventually, we'll need Lua modules to use these data, but there's a lot to do before that.

What would you like to do (more efficiently)? edit

Feel free to provide concrete examples to help us understand your needs.

  • I would like to do better in my development of Lua modules. For that, I think there's two main way :
    • Develop and translate in French help pages with a lot of examples. For now, the only place I can see/understand informations is on fr:Module:Wikidata.
    • Find a way to have an active Scribunto project on frwiki. I have asked a lot of questions about Lua modules on the Scribunto French project and almost everytime, I didn't have any answer.
      For now, there's 1 or 2 users on frwiki who can/want to help Lua newbies to solve their problems. They are really nice people, but they cannot do everything alone.

What kind of functions do you need? edit

What kind of convenience functions could you imagine us to provide? What new functions would you use?

  • Examples of things I want to learn :
    • fr:Module:Infobox/Film
      • How to put the title of the infobox in italic
      • How to class, in the "distribution" field, the actors by their "importance". Regularly, they are classed by alphabetical order.
        Generally, well known actors have big Wikidata element and/or are linked to a lot of pages. So I suppose that a function who can class the elements by the number of declarations they have or by the number of linked pages they have will do the job.
    • In the Lua infobox, a function who can show the name of a locality at a specific time. By example, for the P569 of Q3262609, the new function can "see" the qualifiers P582 of P1448 of Q576889 and select "Retteg" because it was the name of the place at the time the painter is born.

Other edit

Notes, questions, other piece of feedback

  • For me, the actual main problem with Lua functions is not a technical one, but an educational one. If we want that the Wikimedians switch from a "template PoV" to a "module PoV", we need to explain to them how to do the basic, beginning by how you can reproduce old templates things with Lua modules. I think that the energy must be putted on the help pages and the reactivity of the Scribunto projects talk pages before creating new functions, or that the new functions created help to improve these 2 points.

Answers from Jberkel edit

About you edit

  • Username: User:Jberkel
  • Main Wikimedia projects where you're active: wiktionary (en/fr), wikisource (en/fr), wikidata
  • How would you qualify your level of experience with Lua? 2-3 years of experience, mostly in Wikimedia projects. Have contributed patches to Scribunto.

What you're already doing edit

What are the Lua modules you're using? edit

More recently I used mw.wikibase.sitelink to generate links to Wikipedia (from Wiktionary, for languages).

What is currently hard to do? edit

Have only used Wikidata for simple things to far.

Did you run into performance problems? edit

Not directly Wikidata related but enwiktionary has Lua memory problems. See wikt:Wiktionary:Grease pit/2018/February#light and Lua error: not enough memory.

What you would like to have edit

Data access edit

Access to interwiki links via Lua. See phab:T163734. In the future: easy access to lexical data. Easy linking of items in different locations (Wiktionary-Wikidata).

What would you like to do (more efficiently)? edit

Memory handling. The existing memory restrictions are quite tight, and there's no good way to profile memory usage. See phab:T188492. It's also often not clear what memory/performance implication functions interfacing with Wikidata have, this should be documented in the API docs. Caching is only mentioned in passing in the documentation.

What kind of functions do you need? edit

  • On Wiktionary there's a need for good multi-language support. For example getting labels of items in multiple languages with one single API call.
  • Testing: how can Modules which depend on an external service (Wikidata) be tested efficiently?
  • Usage statistics / tracing of elements coming from Wikidata.

Other edit

  • A interactive "playground" to interact with Wikidata would be great, without going through slow page save/reload cycles.
  • Ways to input / insert data into Wikidata, from Wiktionary. Maybe forms could get automatically generated for specific data types.
Hello @Jberkel:,
Thanks for your answers.
  • "getting labels of items in multiple languages with one single API call" -> can you give me some examples of use cases? In what case would you like to display labels in a different language than the language of the Wiktionary you're writing on?
  • "tracing of elements coming from Wikidata" -> there is a thing that we call data usage. It's enabled on English Wiktionary, you can see it in the page info (example), at the bottom of the table, section "Wikidata entities used in this page". Is that what you meant, or would you need something else?
Thanks, Lea Lacroix (WMDE) (talk) 09:31, 12 April 2018 (UTC)[reply]
Lea Lacroix (WMDE) multiple labels: prefilling translation tables could be a use case, or for building a multilingual picture dictionary displaying labels in multiple languages (probably a user's target languages). Didn't know about data usage, that's quite useful. Can this data be queried? (e.g. get all wiktionary pages using QXXXX). btw, merci pour les 🧦 – Jberkel (talk) 17:36, 12 April 2018 (UTC)[reply]
I think that's possible using Quarry. Unfortunately I don't know how to use this tool so I can't help more. Lea Lacroix (WMDE) (talk) 12:37, 8 May 2018 (UTC)[reply]

Answers from HastaLaVi2 edit

About you edit

  • Username: HastaLaVi2
  • Main Wikimedia projects where you're active: Turkish Wiktionary, Turkish Wikipedia, Wikidata
  • How would you qualify your level of experience with Lua? Well enough to write complicated modules for Turkish Wiktionary.

What you're already doing edit

What are the Lua modules you're using? edit

I'm mostly dealing with modules which deal with Turkish language characters and terms to give automatic outcome of every word's declension. Lately I was working on the module wikt:tr:Modül:tr-ad

What is currently hard to do? edit

Tracked in Phabricator
Task T194153

When I try to implement Turkish terms in the functions starts with "mw.ustring...", it gives me an error like "the string is not UTF-8" because we have some different letters which are not included in UTF-8.

Did you run into performance problems? edit

Not so much, I think the performance is very well at this point, at least in the areas I encountered.

What you would like to have edit

Data access edit

What would you like to do (more efficiently)? edit

Well as I mentioned above, isn't there any way to include other characters which are not in the UTF-8 to the wikimedia functions? If so, it would be really a life saver.

What kind of functions do you need? edit

Other edit

I started learning Lua language about a year ago when I first joined Turkish Wiktionary, since then I think I improved myself and learned how to use lots of wikimedia functions. I don't know if this already can be done, but wouldn't it be nice to use the datas stored in Lua modules in other areas of a wiki, for example in javascript coding pages? Because at some points I create javascript pages which needs datas from Lua modules, since I cannot find a way to reach them yet I started creating other array pages for javascript coding.

Hello @HastaLaVi2:,
Thank you for your answers!
About the problem you encounter with UTF-8, I created a ticket for it. Would you have a more precise example, maybe even a link pointing to the module you try to write where the error happens?
Thanks, Lea Lacroix (WMDE) (talk) 12:47, 8 May 2018 (UTC)[reply]
Hi!
In the entry wikt:tr:mühür, the module removes the last "ü" letter to create the declension of the word. But before I wrote a new function in the module "terscevir", wikt:tr:Modül:tr-ad line "terscevir( sonAd )" kept giving error. More specifically [https://tr.wiktionary.org/w/index.php?title=Modül:tr-ad&oldid=2945196 this version has an error on the pages like wikt:tr:akıl, wikt:tr:kayıp. If the lost vowel is one of these: "ı, ü". HastaLaVi2 (talk) 14:51, 8 May 2018 (UTC)[reply]

Answers from RexxS edit

About you edit

  • Username: RexxS
  • Main Wikimedia projects where you're active: English Wikipedia, Commons
  • How would you qualify your level of experience with Lua? Reasonably competent

What you're already doing edit

What are the Lua modules you're using? edit

Principally modules used to import Wikidata into enwiki:

Documentation is in place.

What is currently hard to do? edit

Handling different datatypes is hard work. At present there is no generic 'renderSnak' for mainsnaks. I write specific handlers for common datatypes in order to implement requested features (e.g. linking wikibase-items and dealing with missing sitelinks and labels), but as new datatypes are introduced, there is no "catch-all" to render them generically.

It is hard to write code that avoids using the entity object, which we need to do if we don't want false positives in enwiki watchlists when a property we're not using is changed on Wikidata.

I need to be able to access the collection of sitelinks without loading the entity object (for similar reasons as given above). I'm not able to look for possible interlanguage links, for example, without loading the entity object.

Did you run into performance problems? edit

Not that I'm aware of.

What you would like to have edit

Data access edit

Better documentation and many more examples. mw:Extension:Wikibase Client/Lua is not too bad, but mw:Extension:Scribunto/Lua reference manual is appallingly poor.

What would you like to do (more efficiently)? edit

Make use of all of the entity: methods as functions that don't call getEntity. That would allow use of getBestStatements/getAllStatements instead, so that only those properties actually used on enwiki show in enwiki watchlists.

What kind of functions do you need? edit

  • renderMainSnak as described above
  • getSitelinks collection as described above

Other edit

Full, detailed documentation and copious examples are a pre-requisite for more editors to make use of the Lua-Wikidata interface. Every single new feature needs to be documented and well-explained.

@RexxS: Hi, looking through your comment, I have some potential suggestions/ questions. First of all you mention that "At present there is no generic 'renderSnak'". Have you looked into mw.wikibase.formatValue for that?

Also you mention that you have troubles working without loading full entities. For many use cases this should no longer be needed, as we have functions like mw.wikibase.getBestStatements now. Nevertheless, we don't have such a function for getting all sitelinks at once yet, would you need that? - Hoo man (talk) 03:39, 20 July 2018 (UTC)[reply]

@Hoo man: Thanks for your comments.
I looked at mw.wikibase.formatValue some time ago. It only works for qualifiers and references. If you try to use it on a mainsnak, it gives an error "Failed to deserialize data." So, although this works: 
local props = mw.wikibase.getBestStatements( 'Q42', 'P26' ); return mw.wikibase.formatValue(props[1]['qualifiers']['P580'][1])
you'll find that this doesn't: 
local props = mw.wikibase.getBestStatements( 'Q42', 'P26' ); return mw.wikibase.formatValue(props[1])
That's why I was asking for a generic function to render mainsnaks as well.
I rewrote en:Module:WikidataIB to eliminate getEntity and replace it with getBestStatements/getAllStatements, so I'm aware of the improved functionality. However, some methods documented in mw:Extension:Wikibase Client/Lua such as mw.wikibase.entity:formatPropertyValues are only available as methods of an entity object, and not as a corresponding wikibase function. That means that when you convert code to remove getEntity and replace it with getBestStatements/getAllStatements, you no longer have an entity object to call the method on. That wouldn't matter if we had a generic renderSnak, but for now it means that I write handlers for each datatype as you can see in en:Module:WikidataIB.
Let's say I'm working on a multi-lingual wiki like Commons. When I come across an property value that is a wikibase-entity, I need to render it in the best form possible for the reader, so I'll try to use a sitelink or label for that entity. Labels and sitelinks are monolingual text. If the reader's set language (or one of its fallbacks) has a sitelink or label set in that language, then there's no problem: I can just return that. However, what am I to return to represent that entity if the labels and sitelinks do not include the reader's language (or its fallbacks)? Do I return nothing? or the entity-id (Q-number)? or a link to the Wikidata page for the entity? On the English Wikipedia, none of those are acceptable; other wikis may encounter similar objections. I would prefer, therefore, to be able to get an array of all sitelinks (and the same for labels, if needed) available for the entity. I could then devise an algorithm to return a plain-text sitelink or label that would at least indicate to the reader that the value of the property exists in some language. That might act as a spur for editors to add to the Wikidata entity a label in their own language at least.
I hope that clarifies the extra functionality that I think would be useful in the future. --RexxS (talk) 11:23, 20 July 2018 (UTC)[reply]

See also edit

Retrieved from "https://www.wikidata.org/w/index.php?title=Wikidata:New_convenience_functions_for_Lua&oldid=814018525"