Wikidata

Wikidata:Requests for permissions/Bot/Pi bot 19

< Wikidata:Requests for permissions | Bot
The following discussion is closed. Please do not modify it. Subsequent comments should be made in a new section. A summary of the conclusions reached follows.
 Approved --Lymantria (talk) 16:51, 9 January 2021 (UTC)[reply]

Pi bot 19

edit

Pi bot (talk • contribs • new items • new lexemes • SUL • Block log • User rights log • User rights • xtools)
Operator: Mike Peel (talk • contribs • logs)

Task/s: Create new items from Wikipedias

Code: on Bitbucket

Function details: The code looks for articles and categories without Wikidata items (via e.g., en:Special:UnconnectedPages, with a double-check that the Wikidata item doesn't exist) that are not redirects. It checks that they haven't been edited in the last day (extended to 7 days if a search of Wikidata finds matches) and hasn't been created in the last 14 days, to give time for editors to match new items to existing items. If it meets those criteria, then it will create a new Wikidata item with the sitelink to the article or category, and if it's a category then it adds instance of (P31)=Wikimedia category (Q4167836). Example items (with the checks on last edit/creation times turned off for the tests) are 12th Macau International Movie Festival (Q104641925) and Category:1905 suicides (Q104642008). The code would be run daily. It currently only runs on enwp, but the code is general enough that I can include other wikis on request.

I was hoping that this could be done by Wikidata:Requests for permissions/Bot/RegularBot 2, but the opposition there makes it unlikely that will pass, hence this proposal. Creating items is an important first step before more information can be imported, which Pi bot already does by importing descriptions from enwp, Commons category links and hopefully soon, coordinates. Identifying instance of (P31) values is tricky, which is why the script only includes them for category items (the adopted Emijrpbot task already sorts out most biography items, which are simpler to handle than any random article). I am hoping that I can code more tasks to import more info from Wikipedias in the future, but it's simplest if they can import to an existing item rather than also having to create them.

Thanks. Mike Peel (talk) 19:26, 3 January 2021 (UTC)[reply]

 Support though I'll note your code's checking for colons in titles isn't fully robust since there can be a colon in a any title BrokenSegue (talk) 19:35, 3 January 2021 (UTC)[reply]
@BrokenSegue: It seemed the safest option, as I don't know how to check which namespace MediaWiki thinks the page is in... THanks. Mike Peel (talk) 19:42, 3 January 2021 (UTC)[reply]
@BrokenSegue: Improved code that checks namespaces is now included. Thanks. Mike Peel (talk) 19:46, 5 January 2021 (UTC)[reply]
  •  Comment "articles and categories" would mean only these two namespaces. I suppose it excludes articles about people as they are already handled by another bot task. What is it doing with disambiguation pages (they are or used to be handled by PlBot)? It would be good if we could come up with more cases where it adds P31 directly, but I think @Multichill:'s bot takes care of some. --- Jura 19:48, 3 January 2021 (UTC)[reply]
    • @Jura1: Your assumptions were already covered above. It doesn't currently look to see if the page is a disambiguation page or not. If you can identify cases that we can match with P31 values, then they could be added, but that might be better as a separate task. Thanks. Mike Peel (talk) 19:57, 3 January 2021 (UTC)[reply]
      • So this will replace item creation for people? Disambiguations are simple to handle, as page properties at Wikipedia include the magic word. Once the templates listed below are taken in account, would you run a more extensive test lot of 500 items (ideally excluding people, as we already know these work reasonably well)? --- Jura 17:11, 4 January 2021 (UTC)[reply]
        • @Jura1: Item creation for people will continue as before, the scripts that do that should create those items before this script will do so, except in the case that there are Wikidata search results that the script can't be matched to the page (in which case, this new script would create empty items after the 14/7 day periods). Adding P31 values for disambig pages is now included. Happy to run test sets as needed - I was planning on a staggered start anyway with some small batch runs before catching up with the backlog, help checking for issues would be appreciated. Thanks. Mike Peel (talk) 19:46, 5 January 2021 (UTC)[reply]
          • sounds good. In general, I'm comfortable with the way you operate your bot(s). Maybe you want to announce new wikis in the project chat of the relevant language. --- Jura 14:52, 9 January 2021 (UTC)[reply]
Some notes:

--GZWDer (talk) 20:44, 3 January 2021 (UTC)[reply]

    • @GZWDer: Thanks! 'Wikis to skip' isn't relevant, since this is enwp-only to start with, and I'll only add other Wikipedias if asked to do so. Templates to skip are now included. For categories, I'm now excluding hidden and/or empty categories, I'm not sure how to exclude the maintenance categories beyond that though. Thanks. Mike Peel (talk) 19:46, 5 January 2021 (UTC)[reply]
  • @Charles Matthews: Do you have any advices about potential duplicates?--GZWDer (talk) 20:47, 3 January 2021 (UTC)[reply]
  • Did you base your code on mw:Manual:Pywikibot/newitem.py? How is it different? Multichill (talk) 23:24, 3 January 2021 (UTC)[reply]
    • @Multichill: No, I started the code from scratch. I wasn't aware of the newitem.py script, but looking at it now, it seems like I have a different coding style (I write scripts rather than using classes), and as such, starting from scratch makes it easier for me to maintain the code anyway. I have adopted a few features of that code now - particularly the code for checking for uses of templates (nice approach using the QIDs for that!), and some pywikibot checking functions I wasn't aware of, as well as the trick of purging the page after creating the item. I think my code is now doing everything that newitem.py does, plus extra things like checking for search results, and adding P31 values. The updated version is on bitbucket, link above, if you want to check. Thanks. Mike Peel (talk) 19:46, 5 January 2021 (UTC)[reply]
  •  Support The delay timetable seems reasonable to me, as well as the example items done here. It would be nice to add other instance of (P31) cases where possible (such as the disambiguation case mentioned) but even as it is this seems useful to me. ArthurPSmith (talk) 19:32, 4 January 2021 (UTC)[reply]
The above discussion is preserved as an archive. Please do not modify it. Subsequent comments should be made in a new section.

New section: suggestions for improvement

edit
Use disambiguator
  • For enwiki, maybe if the article titles includes a disambiguator, it could use that and remove it from the label it generates, e.g.
    • "(film)" or "(upcoming film)" → P31=Q11424, [1][2][3][4][5]
    • "(<yyyy> film)" → P31=Q11424, P577=yyyy, description="yyyy film" [6][7][8][9][10][11][12][13]
    • "(politician)" → P31=Q5,P106=Q82955 [14][15][16][17]
    • "(surname)"→ P1705=mul:<article name without disambiguator>", P282=Q8229, P31=Q101352, description="family name" [18][19][20]
    • also "(EP)" [21], "(album)" [22], "(song)" [23], "(TV series)" [24][https:/www.wikidata.org/wiki/Q104829948], "(<yyyy> book)", "(company)" [25], maybe "(footballer)" if soccer can be detected.
    • Other disambiguators useful for statements: "(priest)" [26][27], "(band)" [28], "(historian)" [29], "(actor)" [30], "(baseball coach)" []
All, assuming you filter disambiguations ahead. If you log the values in parentheses, maybe we could come up with more.
Use correct label
  • If the bot already determines an occupation covered by the disambiguator, the label should probably be cleaned-up directly [31].
Use naming conventions for statements
  • Also for enwiki, some titles could be decode, e.g. "List of " [32][33], maybe "<yyyy> in <place or topic>".
General points
  • All statements added on creation makes things easier for others.
  • Ideally, the edit summary of the first edit includes a link to the enwiki article (instead of "Created a new Item: Creating item from enwiki", using "Created a new Item: from en:bla".
  • Also, maybe items without any statements could be logged somewhere separately, per week or per month. We could use this to further improve it or to add statements with other means.
  • Maybe the number of empty items/non empty items per wiki could be calculated.

Just for future reference, above a couple of suggestions discussed with Mike Peel on my userpage. When creating potential duplicates, it's preferable that they are non-empty and well formed. --- Jura 14:25, 19 January 2021 (UTC)[reply]

Use correct label
Other wikis
Retrieved from "https://www.wikidata.org/w/index.php?title=Wikidata:Requests_for_permissions/Bot/Pi_bot_19&oldid=1448486483"