Events:
<p>
<dl>
<a name="1651672887.0.19"></a>
<dt> 2022-05-04 at 16:01

 [dardel]
<dd> Required operations made, investigation finished, login re-enabled, and jobs starts enabled again.



The root cause (likely certificate related) remain, but the intention is that it can be addressed without interfering with access/running jobs.



<!-- lama -->

<a name="1651598794.0.19"></a>
<dt> 2022-05-03 at 19:25

 [dardel]
<dd> A fortnight ago, 2022-04-21, certificates in use by internal kubernetes services were updated. These are services required to operate the system. Past days basic operations and services have been performing less than optimal, some not at all. One assumption is that more certificates are in need to be extended. 



No logins will be allowed while this is investigated. Running jobs will continue until finished.



<!-- lama -->

<a name="1651146710.0.19"></a>
<dt> 2022-04-28 at 13:34

 [dardel]
<dd> Late last evening, 2022-04-27, a internal server within dardel went gradually un-responsive. 



An overlooked impact was that jobs needing licenses from external license servers could not reach these servers during its absence, and license checkouts did fail.



The server in question has been restarted over the past couple of hours. License checkouts should now work as usual.



<!-- lama -->

<a name="1650642447.1.19"></a>
<dt> 2022-04-22 at 17:42

 [dardel]
<dd> Updates of certificates in use by internal kubernetes services have been completed, and jobs are running again. The system is expected to perform as prior the updates, please let us know if you experience something else.



<!-- lama -->

<a name="1650562831.1.19"></a>
<dt> 2022-04-21 at 19:30

 [dardel]
<dd> The certificate update work finished for today, and will continue tomorrow 2022-04-22 starting at 09:30.



Certain operations today have had impact on some of the running jobs.



To avoid uncertainties a scheduler block is placed tomorrow at 09:30. Only jobs that can finish before that time will be allowed to get running.



<!-- lama -->

<a name="1650529650.1.19"></a>
<dt> 2022-04-21 at 10:18

 [dardel]
<dd> During initial updates of certificates the internal name-services went out of order. Job starts of jobs in queue have been blocked. Running jobs likely affected in many cases.



<!-- lama -->

<a name="1650529649.1.19"></a>
<dt> 2022-04-14 at 12:31

 [dardel]
<dd> Updates of certificates in use by internal kubernetes services will start in the morning of Thursday 2022-04-21, and continue throughout Friday 2022-04-22.



During activation many services will be restarted. I.e. the pod running slurmctld (scheduler master daemon) will temporarily be non-responsive.

<!-- lama -->

<a name="1648464141.0.19"></a>
<dt> 2022-03-28 at 12:38

 [dardel]
<dd> around 11:47 today a network reconfiguration caused several jobs to experience mpi related timeouts/hangs/failures. The issue should have been rectified roughly five minutes later.



<!-- lama -->

<a name="1647894515.1.0"></a>
<dt> 2022-03-21 at 21:22

<dd> Due to external circumstances out of our reach the planned test of safety and backup power tomorrow 2022-03-22 is cancelled. Scheduler blocks over test removed.



<!-- lama -->

<a name="1647846000.1.0"></a>
<dt> 2022-03-17 at 14:35

<dd> On Tuesday, 2022-03-22 at 10:00, we will simulate a power outage to test our safety and backup power machinery, this by shutting down the power feed to our building. Dardel and beskow schedulers have reservations made so that systems have no jobs in a running state over test.



<!-- lama -->

<a name="1647472893.0.19"></a>
<dt> 2022-03-17 at 00:20

 [dardel]
<dd> As was reported (yesterday, 2022-03-16, around ~18:54) access to dardel was not possible anymore. This should now have been remedied.



Network equipment between dardel and the outside of it has been replaced.



Any user job requiring access to external services have been affected.



<!-- lama -->

<a name="1647459104.0.19"></a>
<dt> 2022-03-16 at 20:11

 [dardel]
<dd> Dardel not reachable from network. Investigating.



<a name="1647258751.0.19"></a>
<dt> 2022-03-14 at 12:50

 [dardel]
<dd> 

Yesterday, 2022-03-13, a HSN restart was performed, and jobs have been let running on gradually larger sections of dardel over night. User logins are now re-enabled. 

<p>

From the perspective of a user/job, no changes have intentionally been made. Execution environment should be the same as before.

<p>

Should the HSN fail on us again there are several forward paths to choose among, ranging from repeat a plain restart, to request software upgrades to be made requiring down-time in the order of days.



<!-- lama -->

<a name="1647110627.0.19"></a>
<dt> 2022-03-12 at 19:39

 [dardel]
<dd> It seems like the HSN network and/or lustre file-system have issues. Among symptoms are frozen file-system accesses.



All new logins blocked and no new jobs allowed to start, until further notice.



<!-- lama -->

<a name="1647099945.1.19"></a>
<dt> 2022-03-12 at 16:45

 [dardel]
<dd> Setup/reconfiguration of the HSN fabric manager finished. All HSN switches restarted. Jobs running since a while. Login about to be re-enabled. From the perspective of a user job, no changes have intentionally been made. Execution environment should be the same as before.

<!-- lama -->

<a name="1647027900.1.19"></a>
<dt> 2022-03-11 at 20:44

 [dardel]
<dd> The setup/reconfiguration of the HSN fabric manager is still ongoing. The HSN, high speed network, needs to be reliably up and running for a well functioning system. Work is still ongoing. Next progress update will be sent Monday, or earlier.



<!-- lama -->

<a name="1647027802.1.19"></a>
<dt> 2022-03-10 at 20:56

 [dardel]
<dd> The setup/reconfiguration for new nodes/future nodes is taking slightly more time than three days. We think there is a good chance having the system on-line running jobs some time over the day tomorrow.



<!-- lama -->

<a name="1647027801.0.0"></a>
<dt> 2022-03-02 at 17:01

<dd> A network switch between Dardel and i.e. external license servers got intermittent failures, starting yesterday 2022-03-01, and eventually gave up earlier this afternoon. An alternative switch should now be in operation. License check-outs should be operational again.



<!-- lama -->

<a name="1647027800.1.19"></a>
<dt> 2022-02-25 at 13:00

 [dardel]
<dd> dardel: On Tuesday March 8 in the morning Dardel will be shut down to add more nodes and to prepare for adding additional nodes later this spring. HPE estimates the downtime to be one to three days. SLURM queues will be drained in advance.



<!-- gert -->

<a name="1645732614.0.19"></a>
<dt> 2022-02-24 at 20:57

 [dardel]
<dd> dardel: four HSN (slingshot) switches were reset and traffic between nodes and all file-servers have resumed. Likely jobs accessing files over the drop-out were affected.

<!-- lama -->

<a name="1645726537.0.19"></a>
<dt> 2022-02-24 at 19:04

 [dardel]
<dd> dardel: since roughly an hour there are issues related to reaching file-system servers from compute/login nodes. Accesses freeze. One typical effect is that login fail, as home-directory access freeze.



<!-- lama -->

<a name="1645190485.1.17"></a>
<dt> 2022-02-18 at 14:15

 [beskow]
<dd> Remote access to beskow.pdc.kth.se (beskow-login2.pdc.kth.se) has been disabled for most users in active allocations, and disabled for users of retired allocations.



<!-- lama -->

<a name="1641737301.0.17"></a>
<dt> 2022-01-09 at 14:57

 [beskow]
<dd> A file-system self-test for errors in /cfs/klemming did indicate failure. This is not (anymore) the case, and jobs have resumed start running since a couple of hours.



<!-- lama -->

<a name="1641680123.0.17"></a>
<dt> 2022-01-08 at 23:03

 [beskow]
<dd> Since a couple of hours compute nodes does not pass self test after job termination. Such nodes are automatically set aside, and cannot serve further jobs.



Fewer and fewer batch nodes will be available for jobs until further notice.



<!-- lama -->

</dl>
All flash news for 
<a href="/cgi-bin/flash/flash.py?year=2022">2022</a>, 
<a href="/cgi-bin/flash/flash.py?year=2021">2021</a>, 
<a href="/cgi-bin/flash/flash.py?year=2020">2020</a>, 
<a href="/cgi-bin/flash/flash.py?year=2019">2019</a>, 
<a href="/cgi-bin/flash/flash.py?year=2018">2018</a>, 
<a href="/cgi-bin/flash/flash.py?year=2017">2017</a>, 
<a href="/cgi-bin/flash/flash.py?year=2016">2016</a>, 
<a href="/cgi-bin/flash/flash.py?year=2015">2015</a>, 
<a href="/cgi-bin/flash/flash.py?year=2014">2014</a>, 
<a href="/cgi-bin/flash/flash.py?year=2013">2013</a>, 
<a href="/cgi-bin/flash/flash.py?year=2012">2012</a>, 
<a href="/cgi-bin/flash/flash.py?year=2011">2011</a>, 
<a href="/cgi-bin/flash/flash.py?year=2010">2010</a>, 
<a href="/cgi-bin/flash/flash.py?year=2009">2009</a>, 
<a href="/cgi-bin/flash/flash.py?year=2008">2008</a>, 
<a href="/cgi-bin/flash/flash.py?year=2007">2007</a>, 
<a href="/cgi-bin/flash/flash.py?year=2006">2006</a>, 
<a href="/cgi-bin/flash/flash.py?year=2005">2005</a>, 
<a href="/cgi-bin/flash/flash.py?year=2004">2004</a>, 
<a href="/cgi-bin/flash/flash.py?year=2003">2003</a>, 
<a href="/cgi-bin/flash/flash.py?year=2002">2002</a>, 
<a href="/cgi-bin/flash/flash.py?year=2001">2001</a>, 
<a href="/cgi-bin/flash/flash.py?year=2000">2000</a>, 
<a href="/cgi-bin/flash/flash.py?year=1999">1999</a>, 
<a href="/cgi-bin/flash/flash.py?year=1998">1998</a>, 
<a href="/cgi-bin/flash/flash.py?year=1997">1997</a>, 
<a href="/cgi-bin/flash/flash.py?year=1996">1996</a>, 
<a href="/cgi-bin/flash/flash.py?year=1995">1995</a>
<br><br><a href="https://www.pdc.kth.se/">Back to PDC</a>
<br><a href="https://www.pdc.kth.se/cgi-bin/flash/rss.py">Subscribe to rss</a>