[Tf-curation] License/Availability was WG: Re: LicenseAvailabilityMap.xml in vlo/trunk/vlo-commons/src/main/resources – CLARIN Trac

[Durco, Matej]

Dear all,

I only very late found out that there was a follow-up on the License issue right after the conference (see email below).
Penny were you able to proceed on that?
Meanwhile we did experimented quite a bit and compiled information, so here is our current take on this for our (TF Curation / ACDH-OEAW) side:

We put down an overview (and would like to collect there more findings and decisions as we go along) in clarin-trac [1]

Main points:

1.     Some of the concepts are linked to both facets (not necessarily bad, but a hint that we don’t have a clear distinction

2.     There is a normalisation file employed, which is however incomplete (new unmapped values exist, some of which are however obviously in the completely wrong place (like size in kB) )

3.     With current concept-mapping we cover only some 60.000 out of 800.000 records !!!

Regarding 2: the Normalization

The current normalization uses the 3-4 values distinction: Free; Free for academic use; Restricted; Upon request (in line with PUB/ACA/RES – laundry tags)
This sounds easy, but as far as I could gather, it is problematic (in many ways).

In Wroclaw, we discussed with Krister an alternative approach:
We could try to map to the license categories as they are defined [2] by the Legal Issues Committee and available also in the License Category Calculator [3]. By that we would avoid the problematic reduction, still keeping the “laundry-tag” approach. And we would be in sync with the Legal committee recommendations. Also each of these atomic tags is well defined and most of them broadly used in the webs.

We could employ here the decomposition approach, in line with what we try to adopt for resourceType and other facets, that means, we wouldn’t have facet values: [ “PUB”, “PUB+BY”, “PUB+BY+SA”] but rather [“PUB”, “BY”, “SA”].
Allowing multiple possible values for the facet in each record in combination with the (already implemented) multi-select feature in VLO this should cover for all use cases and be more ergonomic (e.g. if I am interested only in the Non-Commercial clause, I need to select only one facet value and don’t have to search for all the combination that contain NC.)

There is already a ​normalisation map used in production<https://github.com/clarin-eric/VLO/blob/master/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml> [4](committed 2015-04-23). But there are new values that are not mapped yet. ​Normalisation map as gsheet<https://drive.google.com/open?id=1Pf8Jk_P7RaA-7-dj8fcLOKNH5DjprraFEWXgQ3FvVtQ> [5] with already existing mappings (see normalisation map above) + new values encountered not yet normalized; Values come from elements annotated with concepts linked to one of the two facets License/Availability.
If we agree on the decomposition approach, this list would need to be reviewed completely, but it’s just around 240 entries.


And ad 3. Missing values

Here we have 3 possible situations:

1.     Profile does not have any information about licensing/availability (worst case)

2.     Profile has information about L/A, but is not linked to a concept, or the concept is not in the facet mapping

3.     Profile is well defined, with linking to one of the concepts in the facet mapping, but the information is simply not filled in the record.

We prepared a list ​profile/facet coverage<https://docs.google.com/spreadsheets/d/1eeOr0ShOWxdY8BLzp62LDyfGgHo0gZ95Myw0qauzLxU/edit#gid=0&vpid=A1> [6] with special considerations of availability and licensing facet. Especially also the individual concepts contributing to the facet are plotted (see the c-* columns).

If you want to further investigate this issue, I strongly recommend our experimental instance of the VLO on Minerva<https://minerva.arz.oeaw.ac.at/vlo/> [7].
It features normalized and unnormalized facets, explicit [missing values], profileID and name as facets, data provider facet showing the actual data provider, multi-value selection and also special facets for the concepts contributing to facet availability (i.e. every concept is plotted as a separate facet; these are marked with prefix c-)
With all this you can only to easily see that the biggest contributor to missing values in availability facet is Meertens [8] (Playing the blame game ;)  And you can equally easily see what are the respective profiles (just open the profile Name facet).

So much to our findings until now. We would love to hear from you, what do you think, perhaps we c/should arrange a telco to discuss how to go on about this.

Best,
Matej


[1] https://trac.clarin.eu/wiki/Taskforces/Curation/ValueNormalization/License
[2] https://www.clarin.eu/content/license-categories
[3] https://www.clarin.eu/content/clarin-license-category-calculator
[4] https://github.com/clarin-eric/VLO/blob/master/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml
[5] https://drive.google.com/open?id=1Pf8Jk_P7RaA-7-dj8fcLOKNH5DjprraFEWXgQ3FvVtQ
[6] https://docs.google.com/spreadsheets/d/1eeOr0ShOWxdY8BLzp62LDyfGgHo0gZ95Myw0qauzLxU/edit#gid=0&vpid=A1
[7] https://minerva.arz.oeaw.ac.at/vlo/
[8] http://minerva.arz.oeaw.ac.at/vlo/search?fq=dataProvider:Meertens_Institute_Metadata_Repository&fq=availability:%5Bmissing+value%5D


-------- Forwarded Message --------
Subject:

Re: LicenseAvailabilityMap.xml in vlo/trunk/vlo-commons/src/main/resources – CLARIN Trac

Date:

Sat, 17 Oct 2015 10:34:04 +0200

From:

Twan Goosen <twan.goosen at mpi.nl><mailto:twan.goosen at mpi.nl>

To:

Penny Labropoulou <penny at ilsp.gr><mailto:penny at ilsp.gr>

CC:

Thomas Eckart <teckart at informatik.uni-leipzig.de><mailto:teckart at informatik.uni-leipzig.de>, Matej Durco <xnrn at gmx.net><mailto:xnrn at gmx.net>



That would be great. To get more information on the mapping from the values in resourceInfo records to VLO facets, you can enter the profile id 'clarin.eu:cr1:p_1361876010571' in the input box of the "check profile" form at <https://lux17.mpi.nl/isocat/clarin/vlo/mapping/index.html><https://lux17.mpi.nl/isocat/clarin/vlo/mapping/index.html>.

This will give you quite a lot of information but the relevant sections would be

Facet: availability
    Matched CMD Element ConceptLink: http://hdl.handle.net/11459/CCR_C-2457_45bbaa1a-7002-2ecd-ab9d-57a189f694a6
/c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:licence/text()
xpath accepted
    Matched CMD Element ConceptLink: http://hdl.handle.net/11459/CCR_C-2453_1f0c3ea5-7966-ae11-d3c6-448424d4e6e8
/c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:restrictionsOfUse/text()
xpath accepted
and

Facet: license
    Matched CMD Element ConceptLink: http://hdl.handle.net/11459/CCR_C-2457_45bbaa1a-7002-2ecd-ab9d-57a189f694a6
/c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:licence/text()
xpath accepted
    Matched CMD Element ConceptLink: http://hdl.handle.net/11459/CCR_C-2453_1f0c3ea5-7966-ae11-d3c6-448424d4e6e8
/c:CMD/c:Components/c:resourceInfo/c:distributionInfo/c:licenceInfo/c:restrictionsOfUse/text()
xpath accepted

So the two fields 'licence' and 'restrictionsOfUse' are mapped to both facets (via the concepts 'availability' and 'license'). By looking at the mapping file we can able to see why this results in the three different availability levels we are now getting in the VLO (at least in the case of <http://catalog-clarin.esc.rzg.mpg.de/vlo/search?q=perso&fq=country:Finland><http://catalog-clarin.esc.rzg.mpg.de/vlo/search?q=perso&fq=country:Finland>):
- license 'CLARIN_ACA-NC' maps to 'Free for academic use'
- restriction 'attribution' maps to 'Free'
- restriction 'noRedistribution' maps to 'Restricted'

The next step is to decide what would be the desired mapping (logic).

Best,
Twan
On 16/10/15 22:14, Penny Labropoulou wrote:
No problem! Glad to do it - it was more or less on our agenda for CLIC, so I'll have a look and let you know of the outcomes.
Best,
Penny

On 16 October 2015 at 16:06, Twan Goosen <twan.goosen at mpi.nl<mailto:twan.goosen at mpi.nl>> wrote:
Thanks for your offer to look through this mapping!
I will also send you a link to Menzo's mapping tool.

https://trac.clarin.eu/browser/vlo/trunk/vlo-commons/src/main/resources/LicenseAvailabilityMap.xml



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clarin.eu/pipermail/tf-curation/attachments/20151127/f7b17fd5/attachment-0001.html>