[Tf-aai] Filtering suspicious IdPs in CLARIN SPF

Thu Jun 8 17:35:51 CEST 2017

Hi everybody,

I have set up a page on the CLARIN trac describing the process of requesting changes to the AAI SPF blacklist. This page also keeps trac of the currently blacklisted IdPs, as well as ongoing blacklist requests (tickets). The page can be found here:

https://trac.clarin.eu/wiki/ServiceProviderFederation/IdpBlacklist

Please take a look when you have the opportunity. At this point any input is very welcome!

Regards,
----
André Moreira
CLARIN ERIC
https://www.clarin.eu

> On 29 May 2017, at 11:27, André Moreira <andre at clarin.eu> wrote:
> 
> Hi Jozef,
> 
>> Anyone?
>> Seems trac is not working as fast as (I would have) expected.
> 
> My apologies but I really did not have the opportunity to look into this before. I am finishing some external assignments this month so it is difficult for me to handle all the SPF issues as quickly as I would like to. The good news is that starting next month, I will be working full time for CLARIN ERIC so I will definitely be able to handle trac issues in a more reasonable time frame.
> 
> Anyway, I will try to look into this today or at most tomorrow!
> 
>> This has been implemented, write me if you want to receive these notifications.
> 
> 
> Yes, can you please add my address to receive them?
> 
> 
> Kind regards,
> ----
> André Moreira
> CLARIN ERIC
> https://www.clarin.eu
> 
> 
> 
>> On 25 May 2017, at 09:57, Jozef Misutka <misutka at ufal.mff.cuni.cz> wrote:
>> 
>> 
>> 
>> On 12 May 2017 at 10:24, Jozef Misutka <misutka at ufal.mff.cuni.cz> wrote:
>> And we have a case study from couple of hours ago,
>> https://trac.clarin.eu/ticket/1008#ticket
>> 
>> Anyone?
>> Seems trac is not working as fast as (I would have) expected.
>> 
>> Jozef
>> 
>> 
>> 
>> 
>> Steps 1. and 2. are done,
>> for 3. we do not have the requirements yet but anyway, please comment in the trac ticket.
>> 
>> 2Andre: (I am not trac admin) but I suggest you
>> a) create specific component in trac (AAI IdP Review?) and update the tickets;
>> b) create the parallel trac page (I am not sure I know what you want there) and include listing of all bugs that are open with specific component on that page.
>> 
>> Best,
>> Jozef
>> 
>> 
>> On 10 May 2017 at 17:53, André Moreira <andre at clarin.eu> wrote:
>> Hi Jozef,
>> 
>>> And yes, I was talking only about the process flow for reviewing an IdP and possibly blacklisting it.
>> 
>> Clear.
>> 
>>> Because you are? responsible for clarin's pyff we have to work on this together or rather you have to agree to it :).
>> 
>> Fair enough!
>> 
>>> 1. someone (here, the term someone is ok) finds suspicious IdP
>>> 2. creates ticket in TRAC
>>> 3. someone (here, the term someone should be better defined - tf-aai?) reviews the requirements and comments whether any violations have been found.
>>> 4. Andre closes the ticket, if there is a violation he will do it after clarin's pyff is updated.
>> 
>> 
>> I still need the advantages of keeping a parallel trac page with the overall status of the blacklisting activities, especially since there is the potential for multiple people to handle step 3, this page can come handy. It is also a nice place to write the workflow we are discussing here in a wiki style.
>> Thus I would modify 3 and 4:
>> 3. someone (here, the term someone should be better defined - tf-aai?) reviews the requirements and comments whether any violations have been found updating the IdP blacklist status page on trac accordingly.
>> 4. André closes the ticket and updates the blacklist status page accordingly, if there is a violation he will do it after clarin's pyff is updated.
>> 
>>> One comment though, I do not think the xpath selection in this format is sustainable in the long run. But it is up to you Andre to decide what to do about it if anything.
>> 
>> This has also been one of my remarks when I was introduced to the system, so I fully agree!
>> My current idea is to create a small script that will convert a text file with all blacklisted IdPs (one per line) into the necessary XPaths and insert them into the job_c.fd code. I would like to hear if someone as an alternative idea.
>> 
>> If more people are interested we can arrange a tf-aai meeting during the CLARIN centre meeting on Wednesday the 17th after lunch, since the afternoon is supposed to be dedicated to taskforce meetings anyway. Also feel free to grab me for an informal chat about this at anytime during the centre meeting.
>> 
>> 
>> Regards,
>> ----
>> André Moreira
>> CLARIN ERIC
>> https://www.clarin.eu
>> 
>> 
>> 
>>> On 2 May 2017, at 19:42, Jozef Misutka <misutka at ufal.mff.cuni.cz> wrote:
>>> 
>>> Andre, you answered everything, thank you.
>>> 
>>> And yes, I was talking only about the process flow for reviewing an IdP and possibly blacklisting it.
>>> Because you are? responsible for clarin's pyff we have to work on this together or rather you have to agree to it :).
>>> 
>>> The VC or possibly the meeting at the dev conf. can help us defining the baseline requirements. Besides that, we have this:
>>> 1. someone (here, the term someone is ok) finds suspicious IdP
>>> 2. creates ticket in TRAC
>>> 3. someone (here, the term someone should be better defined - tf-aai?) reviews the requirements and comments whether any violations have been found.
>>> 4. Andre closes the ticket, if there is a violation he will do it after clarin's pyff is updated.
>>> 
>>> One comment though, I do not think the xpath selection in this format is sustainable in the long run. But it is up to you Andre to decide what to do about it if anything.
>>> 
>>> Best,
>>> Jozef
>>> 
>>> 
>>> 
>>> 
>>> On 1 May 2017 at 17:20, André Moreira <andre at clarin.eu> wrote:
>>> Hi Jozef,
>>> 
>>> My apologies for the late answer.
>>> I must admit I am a bit out of context here; is your idea to streamline a workflow for the detection and blacklisting of suspicious IdPs, or just the later?
>>> 
>>>>> 2. In 1a) or 1b) someone finds a suspicious IdP and CLARIN SPF should be notified about it
>>>>>   QUESTION: what is the best/preferred way to notify CLARIN SPF - trac? email?
>>> 
>>> 
>>> My preference is via a trac ticket, so we could have both: an immediate notification sent by email, plus the status and history of the actions taken.
>>> Furthermore, I think it is also important to keep an inventory of the all the blacklisted IdPs and the respective reasoning for their blacklisting. So I suggest we create a trac page for the 'IdPs blacklist status’, but that the initial reports of misbehaving IdPs are done with the creation of trac tickets. Then the operator handling one of these tickets, would be responsible for updating the 'IdPs blacklist status’ page.
>>> 
>>> 
>>>>> 4. SPF will add this IdP into the filters
>>>>>  QUESTION: is filtering (blacklisting) of IdPs in CLARIN SPF happening before they are published to https://infra.clarin.eu/aai/prod_md_about_spf_idps.xml and if so then where?
>>>>>  ACTION POINT: make filtered IdPs public (will be decided what is the best way [1])
>>> 
>>> Yes, filtering happens before publishing. You can see the PyFF code we are using here: https://github.com/clarin-eric/pyFF_config/blob/master/job_c.fd (see ’select’ XPath expressions).
>>> 
>>> 
>>> But again, I feel like I am not fully contextualised here, so I might not be giving you the answers you were looking for. In that case, if it suits you, we could schedule a skype call so we can have a more agile discussion?
>>> 
>>> 
>>> Anyhow I hope this helps,
>>> Kind regards,
>>> ----
>>> André Moreira
>>> CLARIN ERIC
>>> https://www.clarin.eu
>>> 
>>> 
>>> 
>>>> On 18 Apr 2017, at 12:29, Jozef Misutka <misutka at ufal.mff.cuni.cz> wrote:
>>>> 
>>>> Hi Andre,
>>>> 
>>>> (cc-ing tf-aai@ too for comments)
>>>> 
>>>> we should have a more formal and public way how to blacklist IdPs from CLARIN SPF. We have the tools, so we really need only minor changes and better documentation. In reality, this will be very rare but should be documented nevertheless.
>>>> 
>>>> The flow is like this:
>>>> 
>>>> 1. User authenticates to one of CLARIN SPF SPs
>>>>   a) we have information about the IdP in https://lindat.mff.cuni.cz/services/aaggreg/ so we can check if it looks suspicious or not;
>>>>   b) different web applications inform about new users (e.g., clarin-dspace sends an email) so we can even check if the attributes seem ok.
>>>>   ACTION POINT (optional): email new IdPs from attribute aggregator
>>>> 
>>>> 2. In 1a) or 1b) someone finds a suspicious IdP and CLARIN SPF should be notified about it
>>>>   QUESTION: what is the best/preferred way to notify CLARIN SPF - trac? email?
>>>> 
>>>> 3. CLARIN evaluates if that IdP is really problematic or not
>>>>   ACTION POINT: define a problematic/non problematic IdP e.g., problematic IdP is the one that does not validate its users more that by sending emails
>>>>   ACTION POINT: make this definition public (will be decided what is the best way [1])
>>>> 
>>>> 4. SPF will add this IdP into the filters
>>>>  QUESTION: is filtering (blacklisting) of IdPs in CLARIN SPF happening before they are published to https://infra.clarin.eu/aai/prod_md_about_spf_idps.xml and if so then where?
>>>>  ACTION POINT: make filtered IdPs public (will be decided what is the best way [1])
>>>> 
>>>> 
>>>> To all, please feel free to comment, share practices etc. At LINDAT, we have the following page
>>>> https://lindat.mff.cuni.cz/en/how-do-i-sign-up (see section Supported Organisations)
>>>> 
>>>> Best,
>>>> Jozef
>>>> 
>>>> 
>>>> [1] In my todo list is still to prepare a short CLARIN AAI requirements document where we could add it and publish that document.
>>> 
>>> 
>> 
>> 
>> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP
URL: <https://lists.clarin.eu/cgi-bin/mailman/private/tf-aai/attachments/20170608/5d9cddb5/attachment.sig>