[Tf-curation] License/Availability was WG: Re: LicenseAvailabilityMap.xml in vlo/trunk/vlo-commons/src/main/resources – CLARIN Trac

[Krister Lindén]

Quick answer to Penny's question about PUB/ACA/RES is that they should 
also be facets that you can restrict, i.e. choose to get data that is 
neither ACA nor RES leaving only open or public data with varying licenses.

Despite our good efforts to collect data, people will be happy to find a 
resource at all. I do not really think that they have the luxury of 
choosing e.g. whether they want a treebank for a particular language 
with a CC-BY and not an MIT license, or vice versa, but they may wish to 
say that they want a treebank with an open or public license, if available.

It is this final "if available", that has gotten me thinking that we 
should probably also let legal metadata provide a sorting order, because 
other criteria will be more important, i.e. if a favorite license is not 
on offer, I may settle for a slightly-more-difficult-to-manage license, 
as the legal status of the resource is more like a price tag, i.e. it 
will cost me more effort to deal with a restricted resource than an open 
one, but if I need an English speech data set, I will not settle for 
some Russian text data simply because the license is more interesting.

This said, if it is not too much of an effort, we could of course 
provide the option to also write the name (or part of a name) of a 
license as a search criterion. After all, that can be implemented as 
rather straightforward string matching in the license name field.

--
Krister

On 30.11.2015 13:11, Penny Labropoulou wrote:
> Hi Matej, Krister and all
>
> Some thoughts on the topics raised:
> - license & availability are indeed too close semantically and that's where the confusion comes; moreover, for the normalization, the values are taken from different attributes which brings about the contradicting outcomes we noticed in Wroclaw.
> - Now, if I understand correctly the new approach, both facets will be replaced by the License Categories, is that it? If yes, I think this would improve the situation and we need to check the new mappings. In this case, my only question to Krister and the CLIC, is whether the PUB/ACA/RES should be treated at the same level as the other tags.
> - in this scenario, I agree in principle with Krister's email about the sorting of the resources when shown to the user - but I don't know what other sortings (apart from alphabetical ordering on the resource name) you have also implemented on the new VLO; it would also be nice to somewhere state that this is the ordering of the resources or allow the user decide on the sorting, perhaps?
> - the only problem I have with keeping only license categories in the facets, is that we "lose" information of resources that are licensed with a standard license, e.g. CC, GNU etc. Could we have a second facet for these? Problems with this: (a) confusion between the two facets (will the user understand that "attribution" will give him more results than CC-BY?) and (b) mapping of resources without a standard license value... I'm just putting up the question without having a definite answer.
>
> And to Matej's question: sorry, I haven't done anything yet :-(. Do you have a deadline for the normalization issues? I can look at it closer in the next couple of weeks, taking into account the current discussion outcomes. And, if possible, it would be nice to know the attributes these concepts come from and the combinations thereof (i.e. if the same resource has two or more licensing-related attributes and/or values, get the combinations thereof).
>
> Best,
> Penny
>
>
> -----Original Message-----
> From: Krister Lindén [mailto:krister.linden at helsinki.fi]
> Sent: Friday, November 27, 2015 5:55 PM
> To: Durco, Matej <Matej.Durco at oeaw.ac.at>; penny at ilsp.gr; Menzo Windhouwer2 <menzo.windhouwer at meertens.knaw.nl>; Twan Goosen <twan.goosen at mpi.nl>; Thomas Eckart <teckart at informatik.uni-leipzig.de>; Ostojic, Davor <Davor.Ostojic at oeaw.ac.at>
> Cc: tf-curation at lists.clarin.eu
> Subject: Re: License/Availability was WG: Re: LicenseAvailabilityMap.xml in vlo/trunk/vlo-commons/src/main/resources – CLARIN Trac
>
> Dear all,
>
> One thing we discussed afterwards in CLIC, was the best approach to utilize the legal metadata in a query. We have no illusions that the legal metadata would be the primary criterion people use to select their data. Other criteria such as name of the resource, languages covered, data type and usage purpose are probably more crucial, but if users have a choice, they probably look for resources that have a clearly defined legal status and have as few restrictions as possible.
>
> "As few restrictions as possible" implies a sorting order. However, a user may dislike some restrictions for practical purposes. Therefore, it would make sense to let the user check the tags he would like to filter out and if none are checked, none are filtered, i.e. all resources conforming to the primary criteria are shown. In addition, the user should be able to filter out resources whose legal status is undefined, because the user would not know how to legally use the resource, even if it exists. If displayed within a sorting order of "as few restrictions as possible", resources with undefined legal status should be at the end of the list.
>
> Regards,
> Krister
>
> On 27.11.2015 16:08, Durco, Matej wrote:
>> Dear all,
>>
>> I only very late found out that there was a follow-up on the License
>> issue right after the conference (see email below).
>>
>> Penny were you able to proceed on that?
>>
>> Meanwhile we did experimented quite a bit and compiled information, so
>> here is our current take on this for our (TF Curation / ACDH-OEAW) side:
>>
>> We put down an overview (and would like to collect there more findings
>> and decisions as we go along) in clarin-trac [1]
>>
>> Main points:
>>
>> 1.Some of the concepts are linked to both facets (not necessarily bad,
>> but a hint that we don’t have a clear distinction
>>
>> 2.There is a normalisation file employed, which is however incomplete
>> (new unmapped values exist, some of which are however obviously in the
>> completely wrong place (like size in kB) )
>>
>> 3.With current concept-mapping we cover only some 60.000 out of 800.000
>> records !!!
>>
>> Regarding 2: the Normalization
>>
>> The current normalization uses the 3-4 values distinction: Free; Free
>> for academic use; Restricted; Upon request (in line with PUB/ACA/RES –
>> laundry tags)
>>
>> This sounds easy, but as far as I could gather, it is problematic (in
>> many ways).
>>
>> In Wroclaw, we discussed with Krister an alternative approach:
>>
>> We could try to map to the license categories as they are defined [2] by
>> the Legal Issues Committee and available also in the License Category
>> Calculator [3]. By that we would avoid the problematic reduction, still
>> keeping the “laundry-tag” approach. And we would be in sync with the
>> Legal committee recommendations. Also each of these atomic tags is well
>> defined and most of them broadly used in the webs.
>>
>> We could employ here the decomposition approach, in line with what we
>> try to adopt for resourceType and other facets, that means, we wouldn’t
>> have facet values: [ “PUB”, “PUB+BY”, “PUB+BY+SA”] but rather [“PUB”,
>> “BY”, “SA”].
>>
>> Allowing multiple possible values for the facet in each record in
>> combination with the (already implemented) multi-select feature in VLO
>> this should cover for all use cases and be more ergonomic (e.g. if I am
>> interested only in the Non-Commercial clause, I need to select only one
>> facet value and don’t have to search for all the combination that
>> contain NC.)
>>
>> There is already a ​normalisation map used in production
>> <https://github.com/clarin-eric/VLO/blob/master/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml>
>> [4](committed 2015-04-23). But there are new values that are not mapped
>> yet. ​Normalisation map as gsheet
>> <https://drive.google.com/open?id=1Pf8Jk_P7RaA-7-dj8fcLOKNH5DjprraFEWXgQ3FvVtQ>
>> [5] with already existing mappings (see normalisation map above) + new
>> values encountered not yet normalized; Values come from elements
>> annotated with concepts linked to one of the two facets
>> License/Availability.
>>
>> If we agree on the decomposition approach, this list would need to be
>> reviewed completely, but it’s just around 240 entries.
>>
>> And ad 3. Missing values
>>
>> Here we have 3 possible situations:
>>
>> 1.Profile does not have any information about licensing/availability
>> (worst case)
>>
>> 2.Profile has information about L/A, but is not linked to a concept, or
>> the concept is not in the facet mapping
>>
>> 3.Profile is well defined, with linking to one of the concepts in the
>> facet mapping, but the information is simply not filled in the record.
>>
>> We prepared a list ​profile/facet coverage
>> <https://docs.google.com/spreadsheets/d/1eeOr0ShOWxdY8BLzp62LDyfGgHo0gZ95Myw0qauzLxU/edit#gid=0&vpid=A1>
>> [6] with special considerations of availability and licensing facet.
>> Especially also the individual concepts contributing to the facet are
>> plotted (see the c-* columns).
>>
>> If you want to further investigate this issue, I strongly recommend our
>> experimental instance of the VLO on Minerva
>> <https://minerva.arz.oeaw.ac.at/vlo/> [7].
>>
>> It features normalized and unnormalized facets, explicit [missing
>> values], profileID and name as facets, data provider facet showing the
>> actual data provider, multi-value selection and also special facets for
>> the concepts contributing to facet availability (i.e. every concept is
>> plotted as a separate facet; these are marked with prefix c-)
>>
>> With all this you can only to easily see that the biggest contributor to
>> missing values in availability facet is Meertens [8] (Playing the blame
>> game ;)  And you can equally easily see what are the respective profiles
>> (just open the profile Name facet).
>>
>> So much to our findings until now. We would love to hear from you, what
>> do you think, perhaps we c/should arrange a telco to discuss how to go
>> on about this.
>>
>> Best,
>>
>> Matej
>>
>> [1]
>> https://trac.clarin.eu/wiki/Taskforces/Curation/ValueNormalization/License
>>
>> [2] https://www.clarin.eu/content/license-categories
>>
>> [3] https://www.clarin.eu/content/clarin-license-category-calculator
>>
>> [4]
>> https://github.com/clarin-eric/VLO/blob/master/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml
>>
>> [5]
>> https://drive.google.com/open?id=1Pf8Jk_P7RaA-7-dj8fcLOKNH5DjprraFEWXgQ3FvVtQ
>>
>> [6]
>> https://docs.google.com/spreadsheets/d/1eeOr0ShOWxdY8BLzp62LDyfGgHo0gZ95Myw0qauzLxU/edit#gid=0&vpid=A1
>>
>> [7] https://minerva.arz.oeaw.ac.at/vlo/
>>
>> [8]
>> http://minerva.arz.oeaw.ac.at/vlo/search?fq=dataProvider:Meertens_Institute_Metadata_Repository&fq=availability:%5Bmissing+value%5D
>>
>>
>>
>> -------- Forwarded Message --------
>>
>> *Subject: *
>>
>> 	
>>
>> Re: LicenseAvailabilityMap.xml in
>> vlo/trunk/vlo-commons/src/main/resources – CLARIN Trac
>>
>> *Date: *
>>
>> 	
>>
>> Sat, 17 Oct 2015 10:34:04 +0200
>>
>> *From: *
>>
>> 	
>>
>> Twan Goosen <twan.goosen at mpi.nl> <mailto:twan.goosen at mpi.nl>
>>
>> *To: *
>>
>> 	
>>
>> Penny Labropoulou <penny at ilsp.gr> <mailto:penny at ilsp.gr>
>>
>> *CC: *
>>
>> 	
>>
>> Thomas Eckart <teckart at informatik.uni-leipzig.de>
>> <mailto:teckart at informatik.uni-leipzig.de>, Matej Durco <xnrn at gmx.net>
>> <mailto:xnrn at gmx.net>
>>
>>
>>
>> That would be great. To get more information on the mapping from the
>> values in resourceInfo records to VLO facets, you can enter the profile
>> id 'clarin.eu:cr1:p_1361876010571' in the input box of the "check
>> profile" form at
>> <https://lux17.mpi.nl/isocat/clarin/vlo/mapping/index.html>
>> <https://lux17.mpi.nl/isocat/clarin/vlo/mapping/index.html>.
>>
>> This will give you quite a lot of information but the relevant sections
>> would be
>>
>>      Facet: availability
>>           Matched CMD Element ConceptLink:
>>      http://hdl.handle.net/11459/CCR_C-2457_45bbaa1a-7002-2ecd-ab9d-57a189f694a6
>>      /c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:licence/text()
>>      xpath accepted
>>           Matched CMD Element ConceptLink:
>>      http://hdl.handle.net/11459/CCR_C-2453_1f0c3ea5-7966-ae11-d3c6-448424d4e6e8
>>      /c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:restrictionsOfUse/text()
>>      xpath accepted
>>
>> and
>>
>>      Facet: license
>>           Matched CMD Element ConceptLink:
>>      http://hdl.handle.net/11459/CCR_C-2457_45bbaa1a-7002-2ecd-ab9d-57a189f694a6
>>      /c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:licence/text()
>>      xpath accepted
>>           Matched CMD Element ConceptLink:
>>      http://hdl.handle.net/11459/CCR_C-2453_1f0c3ea5-7966-ae11-d3c6-448424d4e6e8
>>      /c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:restrictionsOfUse/text()
>>      xpath accepted
>>
>>
>> So the two fields 'licence' and 'restrictionsOfUse' are mapped to both
>> facets (via the concepts 'availability' and 'license'). By looking at
>> the mapping file we can able to see why this results in the three
>> different availability levels we are now getting in the VLO (at least in
>> the case of
>> <http://catalog-clarin.esc.rzg.mpg.de/vlo/search?q=perso&fq=country:Finland>
>> <http://catalog-clarin.esc.rzg.mpg.de/vlo/search?q=perso&fq=country:Finland>):
>> - license 'CLARIN_ACA-NC' maps to 'Free for academic use'
>> - restriction 'attribution' maps to 'Free'
>> - restriction 'noRedistribution' maps to 'Restricted'
>>
>> The next step is to decide what would be the desired mapping (logic).
>>
>> Best,
>> Twan
>>
>> On 16/10/15 22:14, Penny Labropoulou wrote:
>>
>>      No problem! Glad to do it - it was more or less on our agenda for
>>      CLIC, so I'll have a look and let you know of the outcomes.
>>
>>      Best,
>>
>>      Penny
>>
>>      On 16 October 2015 at 16:06, Twan Goosen <twan.goosen at mpi.nl
>>      <mailto:twan.goosen at mpi.nl>> wrote:
>>
>>          Thanks for your offer to look through this mapping!
>>          I will also send you a link to Menzo's mapping tool.
>>
>>          https://trac.clarin.eu/browser/vlo/trunk/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml
>>
>>
>
>