Events:
<p>
<dl>
<a name="1720103113.0.19"></a>
<dt> 2024-07-04 at 16:21

 [dardel]
<dd> New job starts resumed.



A subset of jobs started earlier today did experience access issues underneath /cfs/klemming/projects/supr/, this should now have been remedied.



<!-- lama -->

<a name="1720098726.0.19"></a>
<dt> 2024-07-04 at 15:00

 [dardel]
<dd> a temporary block of new job starts is in place while investigating a glitch that has affected a subset of all job starts.



<!-- lama -->

<a name="1719564987.0.19"></a>
<dt> 2024-06-28 at 10:51

 [dardel]
<dd> The login nodes login1 and login4 (Thinlinc) will be restarted next Wednesday, 2024-07-03, at 13:00 CEST (11:00 UTC). You will be logged out from the login nodes. The operation is expected to take less than 30 minutes.

Running jobs will not be affected.



<!-- ragge -->

<a name="1718362188.0.19"></a>
<dt> 2024-06-14 at 12:46

 [dardel]
<dd> Most commands/file-accesses et cetera got stuck roughly half an hour ago, and were in that state for roughly 20 minutes. Actual root cause needs to be investigated.



<!-- lama -->

<a name="1718120389.0.19"></a>
<dt> 2024-06-11 at 17:35

 [dardel]
<dd> The Dardel GPU partition is now back in operation.

<P> There might still be some issues though, please report problems to support@pdc.kth.se.



<!-- ragge -->

<a name="1717001019.0.19"></a>
<dt> 2024-05-29 at 18:35

 [dardel]
<dd> Due to a configuration mistake jobs may have failed with an error

reported by slurmstepd indicating a missing task prolog at the

start of a job step, e.g. srun invocation:



<P> error: run_command: slurm task_prolog can not be executed

(/etc/slurm/omnivector-task_prolog.sh) No such file or directory



<P> Multi-step jobs may have failed partially completed, please

inspect the job output before resubmitting. Failed jobs have to

be resubmitted to the queue, we apologize for this inconvenience.

<!-- noname -->

<a name="1716901658.0.19"></a>
<dt> 2024-05-28 at 15:06

 [dardel]
<dd> Dardel (CPU) is now back in operation, login and jobs are enabled. We are experiencing some problems with the GPU partition, so Dardel GPU is not yet back in production. We are working to resolve the issues as quickly as possible, a new flashnews message will be sent out as soon as Dardel GPU is operational.



<P>Please note that, as announced earlier, the upgraded system will have an upgraded software stack with e.g. a newer version of the Cray programming environment (CPE 23/12) so applications may need to be recompiled or reconfigured.

<!-- foo -->

<a name="1716801047.0.19"></a>
<dt> 2024-05-27 at 11:00

 [dardel]
<dd> The Dardel upgrade is taking a little bit longer than expected.

We are now testing the system and provided everything goes well

we should be able to open it to all users sometime tomorrow

(Tuesday May 28.)



<!-- sysadm -->

<a name="1715794503.0.19"></a>
<dt> 2024-05-15 at 19:31

 [dardel]
<dd> the entity in charge of the slurm (batch system) master daemon got issues a couple of hours ago and has been restarted. Operation should be back to normal.

<!-- lama -->

<a name="1715593504.0.19"></a>
<dt> 2024-05-13 at 11:44

 [dardel]
<dd> Move to performance cluster manager software system HPCM, update of Cray Operating System COS on all nodes, and update of Cray Programming Environment software CPE is expected to start 2024-05-20, Monday May 20, around 08 in the morning.



<P>Expect the system to be unavailable throughout the week.



(corrected: name of month above)



<!-- sysadm -->

<a name="1713630286.1.19"></a>
<dt> 2024-04-20 at 18:21

 [dardel]
<dd> Singularity and other container based jobs can now be used again on the compute nodes.



<P>We aim at also restart login nodes coming Tuesday 23:rd of April around 09:00. After the restart, containers will work on those too.



<P>All new jobs started will run on a compute where the lustre file-system client has been updated.



<!-- RS+LM -->

<a name="1712864028.0.19"></a>
<dt> 2024-04-11 at 21:29

 [dardel]
<dd> Earlier today, around 15:15, and over a couple of hours, some jobs may have been affected by getting a void default PDC module. Jobs specifying explicit PDC/version number should not have been affected.



<!-- sysadm -->

<a name="1712331654.0.19"></a>
<dt> 2024-04-05 at 17:39

 [dardel]
<dd> Containers using user namespaces are disabled until further notice.

This means that for example Singularity will likely not work, also other programs such as Firefox.



<!-- ragge -->

<a name="1710787072.0.19"></a>
<dt> 2024-03-18 at 19:29

 [dardel]
<dd> The dardel login node is having issues and is being rebooted again.



<P>Until resolved/an update is available we ask any user to resist other actions than submitting/checking jobs, edit plain files.



<P>Please avoid spawning a new ssh session every other second, to initiate a massive file transfer, or start I/O intensive multi-cpu/multi-task heavy pre/post-processing analysis of very large data sets, &c



<!-- sysadm -->

<a name="1710768954.0.19"></a>
<dt> 2024-03-18 at 14:34

 [dardel]
<dd> The dardel login node is having issues and is being rebooted.



<!-- sysadm -->

<a name="1710430175.0.14"></a>
<dt> 2024-03-14 at 16:30

 [klemming]
<dd> The server side upgrade of the Klemming and Scania file systems are now done. Job start has been resumed. Please report anything out of the ordinary.

<!-- slas -->

<a name="1710340844.0.14"></a>
<dt> 2024-03-13 at 15:24

 [klemming]
<dd> Tomorrow, March 14th, starting at 10AM CET, we will upgrade the Klemming and Scania file systems to a version that should fix the server side bug. The file systems should stay available during the procedure, but a number of shorter freezes will occur. Already running jobs will be allowed to continue, but new job starts will be delayed during the procedure to minimize the risk of jobs being disturbed.

<!-- slas -->

<a name="1710152894.0.19"></a>
<dt> 2024-03-11 at 11:25

 [dardel]
<dd> The lustre server side of /cfs/klemming/ was restarted shortly after 08:00 this morning.

<P> Any job running making use of /cfs/klemming/ beetween roughly 2024-03-10/20:00 and the restart this morning likely affected. Potentially completely stuck.<BR>

<P> As many compute nodes got flagged being in poor shape, and do not run any jobs, we will take this opportunity and re-start them with a bug fix (CAST-35315) aimed at the lustre _client_ kernel bug.<BR>

<P>The lustre server side bug remain.

<P>New jobs started will run on a compute where lustre client kernel bug is fixed.

<!-- gerttest -->

<a name="1710105064.0.19"></a>
<dt> 2024-03-10 at 22:07

 [dardel]
<dd> Login nodes and compute nodes did loose contact with important parts of the server side of /cfs/klemming/ roughly an hour ago.<BR>

<BR>

 Possibility to login and to access files seriously affected as is likely any running job needing access to klemming as well.



<!-- gerttest -->

<a name="1709042898.0.19"></a>
<dt> 2024-02-27 at 15:05

 [dardel]
<dd> The file system issue has been identified, awaiting fix from the vendor. Jobs are slowly	being started but please be aware that there is a risk of further outages until the fix has been delivered and applied. We are really sorry for the inconvenience.

<!-- slas -->

<a name="1708801644.0.19"></a>
<dt> 2024-02-24 at 20:07

 [dardel]
<dd> Serious file system problems, job starts have been disabled again, investigation is ongoing.



<!-- sysadm -->

<a name="1708794736.0.19"></a>
<dt> 2024-02-24 at 18:12

 [dardel]
<dd> System maintenance done, Dardel is running jobs since a few hours.



<!-- sysadm -->

<a name="1707930342.0.19"></a>
<dt> 2024-02-14 at 18:00

 [dardel]
<dd> Issues related to flapping network connectivity between file-servers and compute clients addressed. Job starts resumed since half an hour.



<P> Please be aware of that the forthcoming extensive update 2024-02-19, and that the internal bug in the lustre file-system both remain. 



<P> Important info can be found at <a href="https://www.pdc.kth.se/about/pdc-news/dardel-s-lustre-system-will-be-updated-during-the-week-of-february-19th-1.1315070"> issues/update </a>.



<!-- gerttest -->

<a name="1707767872.0.19"></a>
<dt> 2024-02-12 at 20:55

 [dardel]
<dd> As issues continue (also involving flapping connectivity between file-servers and clients) no jobs will be allowed to enter running state, should they reside under /cfs/klemming.



<P> Please find more pieces of info on this, and info on forthcoming update starting 2024-02-19 at 

<a href="https://www.pdc.kth.se/about/pdc-news/dardel-s-lustre-system-will-be-updated-during-the-week-of-february-19th-1.1315070"> issues/update </a>.



<!-- gerttest -->

<a name="1707759614.0.19"></a>
<dt> 2024-02-12 at 18:30

 [dardel]
<dd> Status of the ongoing serious issues regarding the lustre client (/cfs/klemming) and of forthcoming extensive upgrade, starting 2024-02-19, can be found at <a href="https://www.pdc.kth.se/about/pdc-news/dardel-s-lustre-system-will-be-updated-during-the-week-of-february-19th-1.1315070"> issues/update </a>.



<!-- gerttest -->

<a name="1707141660.0.19"></a>
<dt> 2024-02-05 at 14:58

 [dardel]
<dd> After the updates last week (starting Wednesday 2024-01-31) many applications have hit what seem to be an internal bug in the lustre file-system client.



<P> Typically this manifests itself through jobs not terminating/finishing properly. Nodes get stuck 'completing' after job finish for longer periods of time. Other jobs fail to start up properly on all nodes.



<P> Several applications seem to be hit by the bug. However, 'vasp' applications seem more unfortunate.



<P> Work to apply a work-around is on-going.



<!-- gerttest -->

<a name="1706791693.1.19"></a>
<dt> 2024-02-01 at 13:47

 [dardel]
<dd> System software update finished, jobs running since a while.



<P> Please find <a href="https://www.pdc.kth.se/about/pdc-news/dardel-being-updated-starting-on-31-january-1.1313016"> description of updates </a>.



<P> Be equipped with some patience on new login sessions as

it takes time to populate user private module cache.



<!-- gerttest -->

<a name="1706788800.1.19"></a>
<dt> 2024-01-24 at 16:16

 [dardel]
<dd> System will be unavailable due to system software upgrade starting Wednesday 2024-01-31 at 10:00. The work is estimated to be finished within two days and the system available again Friday 2024-02-02. More information will follow in the beginning of next week.



<P> Please find <a href="https://www.pdc.kth.se/about/pdc-news/dardel-being-updated-starting-on-31-january-1.1313016"> Dardel being updated starting on 31 January </a>



<!-- gerttest -->

<a name="1705403735.0.19"></a>
<dt> 2024-01-16 at 12:13

 [dardel]
<dd> POD and workload manager restart/restore complete.



<!-- gerttest -->

<a name="1705401213.0.19"></a>
<dt> 2024-01-16 at 11:27

 [dardel]
<dd> the POD hosting the slurm workload manager master-daemon got a failure roughly half an hour ago. Restart/restore in progress.



<!-- gerttest -->

</dl>
All flash news for 
<a href="/cgi-bin/flash/flash.py?year=2024">2024</a>, 
<a href="/cgi-bin/flash/flash.py?year=2023">2023</a>, 
<a href="/cgi-bin/flash/flash.py?year=2022">2022</a>, 
<a href="/cgi-bin/flash/flash.py?year=2021">2021</a>, 
<a href="/cgi-bin/flash/flash.py?year=2020">2020</a>, 
<a href="/cgi-bin/flash/flash.py?year=2019">2019</a>, 
<a href="/cgi-bin/flash/flash.py?year=2018">2018</a>, 
<a href="/cgi-bin/flash/flash.py?year=2017">2017</a>, 
<a href="/cgi-bin/flash/flash.py?year=2016">2016</a>, 
<a href="/cgi-bin/flash/flash.py?year=2015">2015</a>, 
<a href="/cgi-bin/flash/flash.py?year=2014">2014</a>, 
<a href="/cgi-bin/flash/flash.py?year=2013">2013</a>, 
<a href="/cgi-bin/flash/flash.py?year=2012">2012</a>, 
<a href="/cgi-bin/flash/flash.py?year=2011">2011</a>, 
<a href="/cgi-bin/flash/flash.py?year=2010">2010</a>, 
<a href="/cgi-bin/flash/flash.py?year=2009">2009</a>, 
<a href="/cgi-bin/flash/flash.py?year=2008">2008</a>, 
<a href="/cgi-bin/flash/flash.py?year=2007">2007</a>, 
<a href="/cgi-bin/flash/flash.py?year=2006">2006</a>, 
<a href="/cgi-bin/flash/flash.py?year=2005">2005</a>, 
<a href="/cgi-bin/flash/flash.py?year=2004">2004</a>, 
<a href="/cgi-bin/flash/flash.py?year=2003">2003</a>, 
<a href="/cgi-bin/flash/flash.py?year=2002">2002</a>, 
<a href="/cgi-bin/flash/flash.py?year=2001">2001</a>, 
<a href="/cgi-bin/flash/flash.py?year=2000">2000</a>, 
<a href="/cgi-bin/flash/flash.py?year=1999">1999</a>, 
<a href="/cgi-bin/flash/flash.py?year=1998">1998</a>, 
<a href="/cgi-bin/flash/flash.py?year=1997">1997</a>, 
<a href="/cgi-bin/flash/flash.py?year=1996">1996</a>, 
<a href="/cgi-bin/flash/flash.py?year=1995">1995</a>
<br><br><a href="https://www.pdc.kth.se/">Back to PDC</a>
<br><a href="https://www.pdc.kth.se/cgi-bin/flash/rss.py">Subscribe to rss</a>