<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>When investigating something else, I noticed that when starting
ultrascan applications the following output was present:<br>
</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">libGL
error: glx: failed to create dri3 screen</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">libGL
error: failed to load driver: nouveau</p>
<p>I never noticed this before. When running <font face="monospace">journalctl
| grep -iE "nouveau|libGL error"</font> I noticed that those
occure since the last restart of demeler9. When I checked
demeler2, I observed the same output, but the journal entries go
further back.<br>
</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">$
journalctl | grep -iE "nouveau|libGL error"</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Mar
30 16:18:02 nrch.umt.edu dracut[56122]: -rw-r--r-- 2 root
root 0 Sep 4 2024
etc/modprobe.d/nvidia-installer-disable-nouveau.conf</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Mar
30 16:18:07 nrch.umt.edu dracut[56122]: -rw-r--r-- 2 root
root 76 Sep 4 2024
usr/lib/modprobe.d/nvidia-installer-disable-nouveau.conf</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Mar
30 16:18:07 nrch.umt.edu dracut[56122]: drwxr-xr-x 2 root
root 0 Dec 17 21:52
usr/lib/modules/5.15.0-318.199.3.2.el8uek.x86_64/kernel/drivers/gpu/drm/nouveau</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Mar
30 16:18:07 nrch.umt.edu dracut[56122]: -rw-r--r-- 1 root
root 853360 Dec 17 21:52
usr/lib/modules/5.15.0-318.199.3.2.el8uek.x86_64/kernel/drivers/gpu/drm/nouveau/nouveau.ko.xz</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Mar
30 16:21:12 nrch.umt.edu org.gnome.Shell.desktop[70407]: libGL
error: glx: failed to create dri3 screen</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Mar
30 16:21:12 nrch.umt.edu org.gnome.Shell.desktop[70407]: libGL
error: failed to load driver: nouveau</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Mar
30 16:21:12 nrch.umt.edu org.gnome.Shell.desktop[70407]: libGL
error: glx: failed to create dri3 screen</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Mar
30 16:21:12 nrch.umt.edu org.gnome.Shell.desktop[70407]: libGL
error: failed to load driver: nouveau</p>
<p _ngcontent-ng-c1443916501=""
class="query-text-line ng-star-inserted"
style="margin: 0px; color: rgb(227, 227, 227); font-family: "Google Sans Flex", "Google Sans", "Helvetica Neue", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: none; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(40, 42, 44); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><br>
</p>
<p>So basically as soon as the driver changes were done, the errors
started within 3 minutes. With this change, there is no hardware
acceleration anymore for rendering the screen and instead the CPU
is doing it. This would especially affect the VNC in addition to
the network latency. It also matches the observation that killing
the mpi jobs helped to reduce it, because they heavily use CPU and
memory. When looking at the running processes, I also noticed that
things like gnome-shell would spike, especially if dragging
something. Or more notably, if you close a dialog box (for example
run details in us_edit) the cpu load spikes after closing, due to
the render updates around the desktop manager.</p>
<p><img src="cid:part1.NGAN6P5v.UKwMDAEg@uni-konstanz.de" alt=""></p>
<p>btop runs per ssh to not contribute to the load on gnome-shell.
What you see is me opening firefox and going to youtube. Where I
also can't watch a video currently, youtube says that the browser
can't play a video.</p>
<p>To verify:</p>
<p>- Current NVIDIA driver 580.95.05 is installed (<font
face="monospace">grep "NVIDIA GLX Module" /var/log/Xorg.0.log</font>
or <font face="monospace">nvidia-smi -q</font>)</p>
<p>- open-source nouveau driver is blacklisted at kernel level (<font
face="monospace">grep -r "nouveau" /etc/modprobe.d/
/usr/lib/modprobe.d/</font>)</p>
<p>- The gpus are not configured to contribute to the display output
(<font face="monospace">nvidia-smi -q | grep -A 2 "Display"</font> gives
"Display Active: Disabled" for all gpus)</p>
<p>- This forces the the VNC session to use software rendering (<font
face="monospace">glxinfo -B | grep "OpenGL renderer"</font>
returnts llvmpip (software renderer))</p>
<p>- All gnome-shell sessions combined have the thread count of the
cpu llvmpipe threads (for the subthreads of gnome-shell <font
face="monospace">ps -T -C gnome-shell | awk '{print $5}' | sort
| uniq -c | sort -nr</font>)</p>
<p>According to AI the chain of effect is:<br>
<br>
Why it worked with Nouveau:<br>
Nouveau is deeply integrated into the standard Linux kernel and
the open-source Mesa graphics stack. It fully supports Kernel Mode
Setting (KMS). Because of this deep integration, standard display
servers (like Xorg) can automatically detect and initialize
nouveau to provide basic 2D and 3D hardware acceleration via
standard generic interfaces, even without physical monitors
attached or an xorg.conf file present.<br>
<br>
Why it fails with the Proprietary Driver:<br>
The proprietary NVIDIA driver is closed-source and operates
outside the standard Linux KMS framework. It strictly relies on
its own proprietary modules (glxserver_nvidia).<br>
<br>
By default, the proprietary driver expects a physical monitor to
be connected to initialize a rendering screen. Because your Tesla
V100s are headless compute cards, the proprietary driver sees zero
monitors. Without a physical monitor, and without an explicit
xorg.conf file instructing it to create a "Virtual" off-screen
buffer, the NVIDIA driver simply refuses to initialize the display
engine.<br>
<br>
Consequently, Xorg crashes out of the hardware acceleration
attempt and falls back to CPU software rendering. </p>
<p><br>
</p>
<p>From my understanding, the vnc always used software rendering,
but with the default driver the defaults around the rendering and
especially opengl seem to have prevent this from happening. To
verify this, I tested the current main on Konstanz and ASTFVM
which had both no gpu related changes, and wasn't able to observe
the same issues there.</p>
<p><br>
</p>
<p>Have a nice day</p>
<p><br>
</p>
<p>Lukas</p>
<p><br>
</p>
<div class="moz-signature"><b>Lukas Dobler</b>, M.Sc.<br>
Ph.D. student<br>
Universität Konstanz<br>
AG Prof. Cölfen<br>
Fachbereich Chemie<br>
Universitätsstraße 10, Box 714<br>
78464 Konstanz<br>
<br>
Raum L 1050<br>
Tel. +49 (0)7531 88 2019<br>
<br>
<br>
</div>
<div class="moz-cite-prefix">On 02.04.2026 00:30, Saeed Mortezazadeh
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAMSFKXGDtg3H1ydUrNU-a=VWEHm=fLpzQyFXVm_0EhaFqm1e3g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">us_mpi_analysis is updated!
<div>-Saeed</div>
</div>
<br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">On Wed, Apr 1, 2026 at 3:26 PM
Borries Demeler via Demelerlab <<a
href="mailto:demelerlab@biophysics.uleth.ca"
moz-do-not-send="true" class="moz-txt-link-freetext">demelerlab@biophysics.uleth.ca</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">I believe I found the culprit of the slowness.
Killing all mpi jobs doing AUC analysis restored the regular
speed. So us_mpi_analysis appears to be a problem.
<div>I did not have to reboot. We should perhaps recompile
the mpi libraries and make sure it is all updated. Saeed,
can you please take care of that? Let us all know when it
is ready to go again, and perhaps Haben, Sophia, Sigang
and Reece and retry to do their jobs then to see if it
happens again?</div>
<div>Thanks, and sorry for the inconveniences.</div>
<div>-Borries</div>
</div>
_______________________________________________<br>
Demelerlab mailing list<br>
<a href="mailto:Demelerlab@biophysics.uleth.ca"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">Demelerlab@biophysics.uleth.ca</a><br>
<a
href="https://biophysics.uleth.ca/mailman/listinfo/demelerlab"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://biophysics.uleth.ca/mailman/listinfo/demelerlab</a><br>
</blockquote>
</div>
</blockquote>
</body>
</html>