[Tf-curation] License/Availability was WG: Re: LicenseAvailabilityMap.xml in vlo/trunk/vlo-commons/src/main/resources – CLARIN Trac

[Krister Lindén]

Dear all,

One thing we discussed afterwards in CLIC, was the best approach to 
utilize the legal metadata in a query. We have no illusions that the 
legal metadata would be the primary criterion people use to select their 
data. Other criteria such as name of the resource, languages covered, 
data type and usage purpose are probably more crucial, but if users have 
a choice, they probably look for resources that have a clearly defined 
legal status and have as few restrictions as possible.

"As few restrictions as possible" implies a sorting order. However, a 
user may dislike some restrictions for practical purposes. Therefore, it 
would make sense to let the user check the tags he would like to filter 
out and if none are checked, none are filtered, i.e. all resources 
conforming to the primary criteria are shown. In addition, the user 
should be able to filter out resources whose legal status is undefined, 
because the user would not know how to legally use the resource, even if 
it exists. If displayed within a sorting order of "as few restrictions 
as possible", resources with undefined legal status should be at the end 
of the list.

Regards,
Krister

On 27.11.2015 16:08, Durco, Matej wrote:
> Dear all,
>
> I only very late found out that there was a follow-up on the License
> issue right after the conference (see email below).
>
> Penny were you able to proceed on that?
>
> Meanwhile we did experimented quite a bit and compiled information, so
> here is our current take on this for our (TF Curation / ACDH-OEAW) side:
>
> We put down an overview (and would like to collect there more findings
> and decisions as we go along) in clarin-trac [1]
>
> Main points:
>
> 1.Some of the concepts are linked to both facets (not necessarily bad,
> but a hint that we don’t have a clear distinction
>
> 2.There is a normalisation file employed, which is however incomplete
> (new unmapped values exist, some of which are however obviously in the
> completely wrong place (like size in kB) )
>
> 3.With current concept-mapping we cover only some 60.000 out of 800.000
> records !!!
>
> Regarding 2: the Normalization
>
> The current normalization uses the 3-4 values distinction: Free; Free
> for academic use; Restricted; Upon request (in line with PUB/ACA/RES –
> laundry tags)
>
> This sounds easy, but as far as I could gather, it is problematic (in
> many ways).
>
> In Wroclaw, we discussed with Krister an alternative approach:
>
> We could try to map to the license categories as they are defined [2] by
> the Legal Issues Committee and available also in the License Category
> Calculator [3]. By that we would avoid the problematic reduction, still
> keeping the “laundry-tag” approach. And we would be in sync with the
> Legal committee recommendations. Also each of these atomic tags is well
> defined and most of them broadly used in the webs.
>
> We could employ here the decomposition approach, in line with what we
> try to adopt for resourceType and other facets, that means, we wouldn’t
> have facet values: [ “PUB”, “PUB+BY”, “PUB+BY+SA”] but rather [“PUB”,
> “BY”, “SA”].
>
> Allowing multiple possible values for the facet in each record in
> combination with the (already implemented) multi-select feature in VLO
> this should cover for all use cases and be more ergonomic (e.g. if I am
> interested only in the Non-Commercial clause, I need to select only one
> facet value and don’t have to search for all the combination that
> contain NC.)
>
> There is already a ​normalisation map used in production
> <https://github.com/clarin-eric/VLO/blob/master/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml>
> [4](committed 2015-04-23). But there are new values that are not mapped
> yet. ​Normalisation map as gsheet
> <https://drive.google.com/open?id=1Pf8Jk_P7RaA-7-dj8fcLOKNH5DjprraFEWXgQ3FvVtQ>
> [5] with already existing mappings (see normalisation map above) + new
> values encountered not yet normalized; Values come from elements
> annotated with concepts linked to one of the two facets
> License/Availability.
>
> If we agree on the decomposition approach, this list would need to be
> reviewed completely, but it’s just around 240 entries.
>
> And ad 3. Missing values
>
> Here we have 3 possible situations:
>
> 1.Profile does not have any information about licensing/availability
> (worst case)
>
> 2.Profile has information about L/A, but is not linked to a concept, or
> the concept is not in the facet mapping
>
> 3.Profile is well defined, with linking to one of the concepts in the
> facet mapping, but the information is simply not filled in the record.
>
> We prepared a list ​profile/facet coverage
> <https://docs.google.com/spreadsheets/d/1eeOr0ShOWxdY8BLzp62LDyfGgHo0gZ95Myw0qauzLxU/edit#gid=0&vpid=A1>
> [6] with special considerations of availability and licensing facet.
> Especially also the individual concepts contributing to the facet are
> plotted (see the c-* columns).
>
> If you want to further investigate this issue, I strongly recommend our
> experimental instance of the VLO on Minerva
> <https://minerva.arz.oeaw.ac.at/vlo/> [7].
>
> It features normalized and unnormalized facets, explicit [missing
> values], profileID and name as facets, data provider facet showing the
> actual data provider, multi-value selection and also special facets for
> the concepts contributing to facet availability (i.e. every concept is
> plotted as a separate facet; these are marked with prefix c-)
>
> With all this you can only to easily see that the biggest contributor to
> missing values in availability facet is Meertens [8] (Playing the blame
> game ;)  And you can equally easily see what are the respective profiles
> (just open the profile Name facet).
>
> So much to our findings until now. We would love to hear from you, what
> do you think, perhaps we c/should arrange a telco to discuss how to go
> on about this.
>
> Best,
>
> Matej
>
> [1]
> https://trac.clarin.eu/wiki/Taskforces/Curation/ValueNormalization/License
>
> [2] https://www.clarin.eu/content/license-categories
>
> [3] https://www.clarin.eu/content/clarin-license-category-calculator
>
> [4]
> https://github.com/clarin-eric/VLO/blob/master/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml
>
> [5]
> https://drive.google.com/open?id=1Pf8Jk_P7RaA-7-dj8fcLOKNH5DjprraFEWXgQ3FvVtQ
>
> [6]
> https://docs.google.com/spreadsheets/d/1eeOr0ShOWxdY8BLzp62LDyfGgHo0gZ95Myw0qauzLxU/edit#gid=0&vpid=A1
>
> [7] https://minerva.arz.oeaw.ac.at/vlo/
>
> [8]
> http://minerva.arz.oeaw.ac.at/vlo/search?fq=dataProvider:Meertens_Institute_Metadata_Repository&fq=availability:%5Bmissing+value%5D
>
>
>
> -------- Forwarded Message --------
>
> *Subject: *
>
> 	
>
> Re: LicenseAvailabilityMap.xml in
> vlo/trunk/vlo-commons/src/main/resources – CLARIN Trac
>
> *Date: *
>
> 	
>
> Sat, 17 Oct 2015 10:34:04 +0200
>
> *From: *
>
> 	
>
> Twan Goosen <twan.goosen at mpi.nl> <mailto:twan.goosen at mpi.nl>
>
> *To: *
>
> 	
>
> Penny Labropoulou <penny at ilsp.gr> <mailto:penny at ilsp.gr>
>
> *CC: *
>
> 	
>
> Thomas Eckart <teckart at informatik.uni-leipzig.de>
> <mailto:teckart at informatik.uni-leipzig.de>, Matej Durco <xnrn at gmx.net>
> <mailto:xnrn at gmx.net>
>
>
>
> That would be great. To get more information on the mapping from the
> values in resourceInfo records to VLO facets, you can enter the profile
> id 'clarin.eu:cr1:p_1361876010571' in the input box of the "check
> profile" form at
> <https://lux17.mpi.nl/isocat/clarin/vlo/mapping/index.html>
> <https://lux17.mpi.nl/isocat/clarin/vlo/mapping/index.html>.
>
> This will give you quite a lot of information but the relevant sections
> would be
>
>     Facet: availability
>          Matched CMD Element ConceptLink:
>     http://hdl.handle.net/11459/CCR_C-2457_45bbaa1a-7002-2ecd-ab9d-57a189f694a6
>     /c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:licence/text()
>     xpath accepted
>          Matched CMD Element ConceptLink:
>     http://hdl.handle.net/11459/CCR_C-2453_1f0c3ea5-7966-ae11-d3c6-448424d4e6e8
>     /c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:restrictionsOfUse/text()
>     xpath accepted
>
> and
>
>     Facet: license
>          Matched CMD Element ConceptLink:
>     http://hdl.handle.net/11459/CCR_C-2457_45bbaa1a-7002-2ecd-ab9d-57a189f694a6
>     /c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:licence/text()
>     xpath accepted
>          Matched CMD Element ConceptLink:
>     http://hdl.handle.net/11459/CCR_C-2453_1f0c3ea5-7966-ae11-d3c6-448424d4e6e8
>     /c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:restrictionsOfUse/text()
>     xpath accepted
>
>
> So the two fields 'licence' and 'restrictionsOfUse' are mapped to both
> facets (via the concepts 'availability' and 'license'). By looking at
> the mapping file we can able to see why this results in the three
> different availability levels we are now getting in the VLO (at least in
> the case of
> <http://catalog-clarin.esc.rzg.mpg.de/vlo/search?q=perso&fq=country:Finland>
> <http://catalog-clarin.esc.rzg.mpg.de/vlo/search?q=perso&fq=country:Finland>):
> - license 'CLARIN_ACA-NC' maps to 'Free for academic use'
> - restriction 'attribution' maps to 'Free'
> - restriction 'noRedistribution' maps to 'Restricted'
>
> The next step is to decide what would be the desired mapping (logic).
>
> Best,
> Twan
>
> On 16/10/15 22:14, Penny Labropoulou wrote:
>
>     No problem! Glad to do it - it was more or less on our agenda for
>     CLIC, so I'll have a look and let you know of the outcomes.
>
>     Best,
>
>     Penny
>
>     On 16 October 2015 at 16:06, Twan Goosen <twan.goosen at mpi.nl
>     <mailto:twan.goosen at mpi.nl>> wrote:
>
>         Thanks for your offer to look through this mapping!
>         I will also send you a link to Menzo's mapping tool.
>
>         https://trac.clarin.eu/browser/vlo/trunk/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml
>
>