Wikidata:WikiProject Source Reliability

WikiProject source reliability, aka the Credibility Ratings + Assessments Project, is an effort to identify and aggregate online sources of assessments of the reliability and credibility of sources. These assessments include estimates of bias, verifiability of claims, level of editorial oversight or peer review, and expertise in specific topics. Assessments may be of individual documents, or aggregate assessments of authors or domains.

Types of assessments include:

Compilations of self-assessments. Examples include the TRANSPOSE project for collating self-reported peer-review practices of journals).
Evaluations by groups created specifically to evaluate source reliability. Examples include Media Bias Fact Check, other fact checking sites, and sites like Ad Fontes Media that produce visuals referenced by others.
Evaluations by communities of practice, as a by-product of their work reviewing or sourcing information to others. Examples include Perennial Sources lists on various language Wikipedias, topical Reliable Sources lists from individual WikiProjects, and newsrooms that publish their internal measures of source reliability.
Compilations of secondary assessments, including the above. Examples include Iffy.news.

Goals

Short-term

Add source credibility ratings on Wikidata (including topical focus on news outlets and vaccines)
- the catalog of sources,
- for each catalog entry, a timestamped snapshots of its reliability assessments,
- adding assessments to a WD entity for the source [create it if it doesn't exist yet]
Curate master Wikipedian-source reliability lists (as versioned tables w/ explicit schema)
- Federating WD (specialist sets of tabular data)
- citeable (as archival soure w/ identifier)
- explicit schemas (+ maps between them)
- versioned (schemas + datasets)
- provenanced (chains of derivatives for composite data)

Mid-term

Model source credibility as a schema using new or existing properties.
Catalog a range of sources for such ratings
Set up versioned, archival backups of the snapshots

Long-term

Integration into future shared citations / wikicite efforts

Tasks

Propose properties (or identify existing properties) for: assessment of factual-reporting, assessment of credibility, assessment of political bias
assessed source reliability - could this be a single property w/ qualifiers for "type of reliability" and also for "according to whom"?
Propose properties for Permanent IDs for these sources
Compare: MBFC ID, AllSides ID, PolitiFact ID. Get sources to create identifiers for entities if they don't have one.
Add these properties to entities in the datasets below (CU, Iffy, &c)
Create entities for news associations that recognize other orgs (State + national press associations)
Relate publications to the press association to which they belong (instance of: member?).
Define a shared schema : data model
Figure out how to add "hijacked journal" as "sourcing circumstances" option, or otherwise make it an accessible qualifier. (Example)
...

Interested participants

Data model

Derived from "perennial sources" data model:

url → official website (P856)
title → title (P1476)
type → instance of (P31): relevant class item
- fake news website (Q27881073)) : e.g. Disclose.tv (Q115099512) – an instance of (P31) this
- evaluation (Q1379672) : evaluation (of merit, scope, worth, significance)
- peer review (Q215028)
- assessment (Q851176) : evaluation (of a fact, an object or a person)
- media criticism (Q1753773)
- source critic (Q56417579) : scientist? evaluating the quality of sources
- Intelligence source and information reliability (Q6043974) : field-specific
- systematic review (Q1504425) : including of reliability (ref'ed by WikiFactMine)
Commons quality assessment (P6731)
sourcing circumstances (P1480) : qualification of the truth or accuracy of a source: circa (Q5727902), near (Q21818619), presumably (Q18122778), etc. clustered with aliases: accuracy, reliability, confidence, precision, certainty, validity, qualitative valuation
assessment outcome (P9259)

Property proposals

assessment → proposed properties for "assessed source reliability" and "assessed source bias"
Reference:
- discussionForum → published in (P1433): e.g. Wikipedia:Reliable sources/Noticeboard (Q11425125)
- discussionLastDate → point in time (P585)
- discussionLink → reference URL (P854)
verifiability of property
Peer review process for journals (to revisit, now that there's a PID)

see also: Proposed schema for reusable source data

Indexes

Media Bias Fact Check [MBFC] (mainly news)
- Measures: Accuracy, right-left bias (US scale)
Iffy.news (incorporates MBFC)
- Measure: Iffyness
- Thoughts on incorporating into WD: notes
All Sides
Ad Fontes rankings by source (mainly news; slow site)
- Measures: Reliability + news value, Right-Left bias (US scale),
NewsQ[1] (mainly news)
- Measures: identification, journalistic content, journalistic behavior, diversity, external assessment, external affiliation, traffic + engagement
Beall's List (journals)
- Measures: tags: predatory journals, hijacked journals

Source lists

General sources lists

Perennial sources

wide range of source sites, including journals + popular websites

EN
FA
ZH
RU
EL
FR
LT
PT
NE
TR
VI
SV

Topical sources lists

Current compilations of reliability indicators

w:User talk:Headbomb/unreliable.js (Headbomb) –
CiteUnseen (SuperHamster) – categorized domains (json)
w:Wikipedia:New page patrol source guide (Novem Linguae, Rosguill, &c.)
w:User:Novem Linguae/Scripts/CiteHighlighter/AllSourcesExceptNPPSG – (NL)

External sources:

NewsQ - had an online API; soon a public dataset
Media Bias/Fact Check - 2.4k entries