From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Wed, 3 Apr 2019 19:09:35 +0300 From: Michael Shigorin To: devel@lists.altlinux.org Message-ID: <20190403160935.GE1878@imap.altlinux.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="VS++wcV0S1rZb1Fb" Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.5.23.88.hg577987ca2d02 (2014-03-12) Subject: [devel] =?koi8-r?b?zc7Px8/Qz9TP3s7B0SDTws/Sy8E=?= X-BeenThere: devel@lists.altlinux.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: ALT Linux Team development discussions List-Id: ALT Linux Team development discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Apr 2019 16:09:36 -0000 Archived-At: List-Archive: List-Post: --VS++wcV0S1rZb1Fb Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: 8bit Здравствуйте. Коллеги, вот мы тут за сборки, пересборки, коммиты пакетов... а есть ещё одна порой простая штука, которая важна: сколько ядер/процессоров задействуется при сборке пакета. Вот верхний десяток строк из разбора архива заданий для sisyphus_e2k на "Эльбрус-8С" (список "более 300 секунд в один поток" прилагаю, как и пару наколенных скриптов, ожидающих в /tasks соответствующий каталог сборочницы или его частичное зеркало): python-module-lxml: 26005 99% wesnoth1.12: 24803 99% samba-DC: 16774 94% samba: 14355 94% jfreechart: 10732 88% Mesa: 10110 96% python-module-wx: 7734 74% python-module-numpy: 7346 98% bouncycastle: 6833 99% chicken: 6529 97% В случае с lxml сделать что-то довольно сложно, почти всё время уходит на один-единственный файл; jfreechart ждёт оптимизированной jvm. Но те же самбы было бы здорово собирать по возможности во все доступные потоки. Да, порой апстримная сборочная система косовата и сваливается из-за недостатка явно указанной зависимости где-нить посреди сборки; порой удаётся найти и поправить, порой всё-таки ставлю что-то вроде %make_build || %make, поскольку в сумме быстрей. В общем, присоединяйтесь при удобном случае. Как минимум "параллельные" пакеты станут быстрее собираться в сизиф, поскольку aarch64 -- это много относительно медленных ядер; а ещё поможете другим архитектурам, даже если сразу этого сами не заметите. --  ---- WBR, Michael Shigorin / http://altlinux.org   ------ http://opennet.ru / http://anna-news.info --VS++wcV0S1rZb1Fb Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="bt3-300.txt" python-module-lxml: 26005 99% wesnoth1.12: 24803 99% samba-DC: 16774 94% samba: 14355 94% jfreechart: 10732 88% Mesa: 10110 96% python-module-wx: 7734 74% python-module-numpy: 7346 98% bouncycastle: 6833 99% chicken: 6529 97% php7: 5770 96% openfire: 5734 99% mysql-connector-c++: 5569 99% jna: 5229 99% kde5-smplayer: 4436 98% clutter: 4326 97% php5: 4284 94% pidgin: 4256 72% python-module-matplotlib: 4044 93% gcompris: 3989 91% tetex: 3878 83% net-snmp30: 3875 84% cyrus-imapd: 3745 81% git: 3588 89% fonts-ttf-sazanami: 3378 99% libgtk+2: 3373 95% PDFlib-Lite: 2925 90% sympy: 2919 96% projectlibre: 2671 99% argyllcms: 2544 95% doxygen: 2527 98% python-module-OpenGL: 2497 85% ruby: 2191 96% krb5: 2148 90% perl: 2138 94% python-module-docutils: 2101 98% xonotic-data: 2098 99% setools: 2075 97% openjade: 2044 97% coreutils: 1975 97% frei0r-plugins: 1936 73% libdb6.1: 1897 95% sssd: 1815 82% blas: 1734 99% Inventor: 1696 91% openldap: 1677 84% libgnomeui: 1666 76% highlight: 1665 94% vala: 1648 95% tcl: 1644 65% milkytracker: 1643 41% nexuiz: 1607 43% zabbix: 1598 92% netpbm: 1593 80% fonts-ttf-google-noto: 1574 81% gettext: 1562 95% findutils: 1560 84% bind: 1548 95% zookeeper: 1547 99% texlive-lang: 1544 72% libleptonica: 1534 96% openmotif: 1531 85% hamlib: 1529 93% openssl10: 1464 88% groff: 1433 87% lvm2: 1418 87% exim: 1407 93% python-module-boto: 1392 95% cups: 1386 83% gnutls30: 1368 87% nss: 1340 92% fonts-ttf-gnu-freefont: 1335 98% libmemcached: 1292 86% libraw: 1266 91% quagga: 1257 92% squeak-vm: 1247 96% e2fsprogs: 1231 85% python-module-Reportlab: 1217 95% python-module-4Suite-XML: 1185 96% libtdb: 1175 88% unbound: 1147 93% nethack: 1140 60% jss: 1139 97% python-module-Pillow: 1133 94% lincity-ng: 1115 97% python-module-django: 1105 87% cups-filters: 1094 84% LibreSSL: 1090 81% postgis: 1088 94% lp_solve: 1084 98% xapian-bindings: 1083 98% libgtksourceviewmm3: 1060 94% python-module-nltk: 1055 60% python-module-pip: 1042 70% netsurf: 1039 93% grep: 1035 75% libgnomecanvasmm: 1034 90% splint: 1016 78% youtube-dl: 1012 94% qpdf: 985 95% python-module-sphinx: 969 92% mpv: 968 95% foomatic-db: 968 99% tar: 956 89% hplip: 948 79% rabbitmq-java-client: 946 99% uqm-bin: 944 95% python-module-pycrypto: 930 89% libbonoboui: 920 91% freeradius: 913 84% numptyphysics: 905 98% wpa_supplicant: 900 97% freeswitch-sounds: 898 96% libisc-export-dhcp: 889 93% pacemaker: 869 77% openipmi: 863 88% apache2: 863 77% zsh: 861 83% ganttproject: 859 99% proguard: 836 99% ispell-ru-lebedev: 830 97% bison: 830 95% libodfgen: 812 83% perl-Wx-Scintilla: 809 94% diffutils: 797 77% perl-DateTime-TimeZone: 791 97% libarchive: 775 85% flite: 761 94% libpeas: 739 54% apache: 734 68% libgdk-pixbuf: 723 87% rocksndiamonds: 718 98% cppunit: 717 88% elinks: 715 88% python-module-Pygments: 703 94% python-module-timelib: 697 96% gobject-introspection: 680 91% m4: 679 37% libgtk-engines-default: 674 82% ccrtp: 672 92% bash4: 663 88% libnetcdf11-seq: 659 82% moodle2.5: 658 72% libbonobo: 650 86% xmms: 649 71% perl-Boost-Geometry-Utils: 649 94% libformula: 648 96% python-module-pyglet: 645 95% ORBit2: 642 80% flac: 639 70% libgtest: 638 95% perl-SDL: 626 85% firmware-linux: 626 99% parole: 624 92% collectd: 614 90% liboil: 611 91% python-module-gssapi: 610 91% libgmime2.6: 606 90% jack-audio-connection-kit: 598 74% python-module-xlsxwriter: 597 96% fonttools: 582 96% sed: 580 66% mate-desktop: 576 94% bash: 573 87% mediawiki: 572 99% python-module-babel: 565 96% trigger: 563 95% python-module-genshi: 562 93% libldb: 561 84% iptables: 552 83% w3c-libwww: 548 72% xmlrpc-c: 541 82% gzip: 540 82% tla: 535 96% rsyslog: 535 96% libquicktime111: 534 97% dnsjava: 533 98% superlu4.0: 530 88% gst-plugins-ugly1.0: 528 49% ntfs-3g: 527 84% GConf: 521 82% python-module-mistune: 520 96% w3m: 513 89% perl-Compiler-Lexer: 502 92% libnl3: 496 70% nagios-plugins: 495 97% python-module-html5lib: 490 97% xfsprogs: 487 84% libqwt: 484 91% xpilot-ng: 482 91% autogen: 480 72% cpio: 479 96% python-module-dbus: 473 96% patch: 460 82% uucp: 453 86% python-module-BTrees: 453 87% xfig: 452 95% audiofile: 450 90% imlib: 447 81% fonts-type1-cm-super: 447 94% xfdesktop: 442 74% python-module-SQLAlchemy: 441 91% po4a: 433 94% libgtop: 432 98% ORBit: 427 75% python-module-zmq: 423 85% python-module-psycopg2: 422 69% shadow: 420 92% texlive-doc: 412 91% augeas: 412 88% gimp-help: 410 98% lynx: 405 93% freeswitch-sounds-music: 403 98% mgetty: 399 85% jabberd2: 399 86% alsa-tools: 394 81% python-module-cryptography: 393 94% python-module-paste: 392 95% libmng: 389 97% mate-icon-theme-faenza: 383 46% uqm-voice: 382 99% xchat: 381 90% python-module-future: 381 90% libxslt: 381 82% libappstream-glib: 380 95% kernel-modules-ipset-elbrus-8c: 380 97% yasm: 377 93% gsasl: 377 92% mathjax: 375 90% libxcb: 375 75% upower: 365 59% ansible: 365 85% mercurial: 361 89% perl-Date-Manip: 359 41% lz4: 351 93% libgeoclue: 351 76% python-module-tornado: 350 93% python-module-fake-factory: 349 90% timeshift: 348 92% avahi: 347 85% lirc: 345 69% gnome-icon-theme: 345 54% rrd: 343 93% foo2zjs: 342 87% python-module-unittest2: 339 97% libtool_1.5: 339 56% wordnet: 336 86% python-module-zodbpickle: 331 91% kernel-modules-ipset-elbrus-4c: 330 93% kernel-modules-ipset-elbrus-1cp: 327 95% re2c: 326 95% speech-dispatcher: 325 85% perl-HTML-FormFu: 324 99% libatk: 324 93% libimobiledevice: 323 52% bsc: 322 95% lcc1.21: 321 90% python-module-yaml: 319 92% t1lib: 315 79% raptor: 315 86% nas: 314 77% libiodbc: 313 47% kf5-kidletime: 312 63% sg3_utils: 309 97% libunique: 308 50% procps: 307 97% mate-icon-theme: 307 61% libtasn1: 307 83% cryptsetup: 304 50% libselinux: 303 98% meson: 300 96% --VS++wcV0S1rZb1Fb Content-Type: application/x-sh Content-Disposition: attachment; filename="buildtimes-1way.sh" Content-Transfer-Encoding: quoted-printable #!/bin/bash=0A# provide non-paralleled build statistics=0A=0Alogs=3D$TMP/bt= =0Apushd .=0Arm -rf $logs=0Amkdir $logs=0A=0Acd /tasks &&=0Afor t in [0-9]*= archive/done/_*/[0-9]*; do=0A read repo < /tasks/$t/task/repo &&=0A [ "$re= po" =3D "sisyphus_e2k" ] || continue=0A cd /tasks/$t/build 2>/dev/null &&= =0A for i in [0-9]*; do=0A [ -f $i/pkgname ] || continue=0A read name < $= i/pkgname 2>/dev/null &&=0A case "$name" in=0A dummy-*)=0A continue;;= =0A esac &&=0A read usr sys cpu < <(tail -qn2 $i/e2k*/log | sed -rn 's,^(= [0-9]+)\...user ([0-9]+)\...system .* ([0-9]+)%CPU.*$,\1 \2 \3,p') &&=0A c= ase "$cpu" in=0A [0-9][0-9])=0A echo "$(($usr+$sys)) $cpu%" >> $logs/$na= me=0A esac=0A done=0Adone=0A=0Apopd=0A --VS++wcV0S1rZb1Fb Content-Type: application/x-sh Content-Disposition: attachment; filename="buildtimes-analyze.sh" Content-Transfer-Encoding: quoted-printable #!/bin/sh=0A# try neglecting the worst result as it might be due to build c= ontention=0A=0Acd $TMP/bt || exit 1=0Afor i in *; do=0A echo -n "$i: "=0A s= ort -n < $i | tail -2 | head -1=0Adone |=0Asort -rnk2=0A --VS++wcV0S1rZb1Fb--