[Clarin-Be] [LLMs4SSH] Subgroup Evaluation/benchmarking of LLMs

[Vincent Vandeghinste]

Hi Grzegorz,

I think we need two mailing lists: 1 for the K-centre in total and 1 for 
the subgroup of Benchmarking LLMs.

We can ask Dieter to set these up at https://lists.clarin.eu/ and have 
e.g. llms4ssh at lists.clarin.eu and llms4ssh-benchmarking at lists.clarin.eu

We can then invite peopel to subscribe to these mailing lists, so they 
can manage their own subscriptions, so we don't need google accounts.

What do you think?

v.

On 2025-06-20 10:27, Grzegorz Chodak wrote:

> Dear Vincent,
> 
> thank you for your e-mail. Mailing list it is a good idea. I attach 
> list of the e-mail we used to inform about K-Centre LLM4SSH last 
> meeting. This is in Word format. I also copied this list into Google 
> Docs and shared it with You.
> 
> I have created a Google Drive folder for LLM4SSH. Unfortunately, 
> sharing the folder is only possible with email addresses from the 
> Google domain or those linked to a Google account. This poses a 
> problem, as most of the email addresses on this mailing list are not 
> associated with Google.
> 
> Best Regards,
> 
> Grzegorz Chodak
> 
> W dniu 19.06.2025 o 16:37, Vincent Vandeghinste pisze:
> 
> Thank you Nikola,
> 
> I know Spela from a long time ago in a Translation context, can't 
> remember where.
> 
> I'll send a mail to the group with Spela included, so anyone replying 
> on the thread includes her as well.
> 
> @Maciej/Grzegorz: maybe we should create a mailing list where people 
> can subscribe to this subgroup. We could ask Dieter to set one up 
> centrally, if you want. We need to work on the workshop programme some 
> more.
> 
> cheers,
> 
> v.
> 
> On 2025-06-19 16:28, Nikola Ljubešić wrote:
> 
> Dear Vincent,
> 
> Can you, please, add Špela Vintar (cc) to the follow-up communication 
> on the LLM benchmarking workgroup within the LLMs4SSH K-Centre?
> 
> Špela is a seasoned researcher with great interest in LLM benchmarking. 
> Špela, please have a look at the current status of the document giving 
> an overview on the current benchmarking practices. You might even want 
> to expand our section with your recently published pragmatic benchmark.
> 
> Špela, I guess that Vincent is still searching for the volunteer on the 
> following point (Vincent, do let us know if this is not the case 
> anymore):
> 
> * Can someone of the people in this subgroup that is involved in 
> hands-on setting up evaluation / benchmarking volunteer to take the 
> lead on the practical side of this subgroup, such as coming up with a 
> proposal on how you would go about setting up an approach in which an 
> LLM can be evaluated with your language specific benchmarks, without 
> opening up the benchmark? This would provide us with a starting point 
> for discussion. Feel free to start a new tab in the google doc to 
> describe what you propose so it is easily sharable with the others in 
> this subgroup.
> 
> Vincent, both Taja and me will be in Vienna for the LLMs4SSH workshop. 
> We will be happy to present. Špela, we can talk if you would be 
> interested in going there as well (Tomaž has just opened up the 
> discussion on the CLARIN.SI [1] quota).
> 
> Greetings to both,
> 
> Nikola
> 
> On Tue, Apr 22, 2025 at 10:58 AM Vincent Vandeghinste 
> <vincent at ccl.kuleuven.be> wrote:
> 
> Dear all,
> 
> As you are part of the workgroup within the LLMs4SSH K-Centre, we are 
> contacting you for a first exploratory information round to tell us 
> what are the current practices of LLM evaluation for your languages.
> 
> You can put this information in this document: 
> https://docs.google.com/document/d/17th32j9W0h42AqS8sr0Q-HdaW9N4PuDQf8bzcrRF80A/edit?usp=sharing
> 
> We are interested in
> 
> * methods
> * test sets
> * evaluation platforms
> * ...
> 
> This can then serve as a preparation for our first online meeting. We 
> are waiting for the meeting about the post-conference CLARIN workshop 
> about LLMs before setting up a meeting for this subgroup, in order not 
> to confuse things.
> 
> So, if you could provide such information by the 9th of May 2025, that 
> would be greatly appreciated.
> 
> kind regards,
> 
> Vincent (also on behalf of Henk)


Links:
------
[1] http://CLARIN.SI
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clarin.eu/pipermail/clarin-be/attachments/20250620/77dce964/attachment.htm>