ALT Linux Distributions development
 help / color / mirror / Atom feed
From: Michael Shigorin <mike@altlinux.org>
To: devel-distro@lists.altlinux.org
Subject: Re: [devel-distro] [PATCH] reports.mk: Generate rpm and srpms lists
Date: Wed, 28 Apr 2021 15:38:53 +0300
Message-ID: <20210428123853.GA16601@imap.altlinux.org> (raw)

[-- Attachment #1: Type: text/plain, Size: 2144 bytes --]

On Fri, Apr 16, 2021 at 03:34:11PM +0700, Anton Midyukov wrote:
> --- a/reports.mk
> +++ b/reports.mk
> +	@cat $(BUILDLOG) | grep -E 'chroot/.in/[^/]*.rpm' | cut -d' ' -f 1 | tr -d "'"'`' | \
> +		rev | cut -d'/' -f 1 | rev | sort -u > "$(REPORTDIR)/list-rpms.txt"
> +	@cat $(BUILDLOG) | grep -E 'chroot/.in/[^/]*.rpm' | cut -d' ' -f 1 | tr -d "'"'`' | \
> +		xargs rpm -qp --queryformat '%{sourcerpm}\n' | sort -u > "$(REPORTDIR)/list-srpms.txt"

Типовой useless use of cat(1): можно же делать сразу

grep ... < $(BUILDLOG) | cut ...

:-)

Заодно причесал и выделил отдельной целью reports/packages,
она может быть самоценна.

PS: ещё немножко посмотрел этот скрипт -- не настаиваю на мерже
последнего патча из приложенных трёх, но мне кажется, что так
чуточку предпочтительней и на двадцатую долю секунды быстрей:

[ прогон для разогрева дискового кэша ]
$ time sh reports1.sh  
sh reports1.sh  0,72s user 0,13s system 109% cpu 0,768 total
$ time bash reports2.sh
bash reports2.sh  0,65s user 0,12s system 106% cpu 0,723 total
$ md5sum list-*
e592b01a6c506685aac3b1026eb92b93  list-rpms.1.txt
e592b01a6c506685aac3b1026eb92b93  list-rpms.2.txt
aef452a7062c2be24b0e23ba418e160b  list-srpms.1.txt
aef452a7062c2be24b0e23ba418e160b  list-srpms.2.txt
$ head reports*
==> reports1.sh <==
BUILDLOG=$HOME/out/alt-server-20210428-x86_64.iso.log; REPORTDIR=$TMP;
grep -E 'chroot/.in/[^/]*.rpm' < $BUILDLOG | cut -d' ' -f 1 | tr -d "'"'`' | sed 's,^.*/,,' | sort -u > "$REPORTDIR/list-rpms.1.txt"
grep -E 'chroot/.in/[^/]*.rpm' < $BUILDLOG | cut -d' ' -f 1 | tr -d "'"'`' | xargs rpm -qp --queryformat '%{sourcerpm}\n' | sort -u > "$REPORTDIR/list-srpms.1.txt"

==> reports2.sh <==
BUILDLOG=$HOME/out/alt-server-20210428-x86_64.iso.log; REPORTDIR=$TMP; grep -E 'chroot/.in/[^/]*.rpm' < $BUILDLOG | cut -d' ' -f 1 | tr -d "'"'`' |tee /dev/stderr 2> >(sed 's,^.*/,,' | sort -u > "$REPORTDIR/list-rpms.2.txt") | xargs rpm -qp --queryformat '%{sourcerpm}\n' | sort -u > "$REPORTDIR/list-srpms.2.txt"

PPS: когда копипаст махровый лезет -- пиши, гляну.

-- 
 ---- WBR, Michael Shigorin / http://altlinux.org
  ------ http://opennet.ru / http://anna-news.info

[-- Attachment #2: 0001-reports.mk-factor-out-reports-packages.patch --]
[-- Type: text/x-patch, Size: 1723 bytes --]

>From 14ecaee2cbb65ef5e37b379181f9bcd25c35c455 Mon Sep 17 00:00:00 2001
From: Michael Shigorin <mike@altlinux.org>
Date: Wed, 28 Apr 2021 14:48:59 +0300
Subject: [PATCH 1/3] reports.mk: factor out reports/packages

...making it fit 80 columns while at that
to avoid horizontal spaghetti code.
---
 reports.mk | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/reports.mk b/reports.mk
index 9efa4f885..0e44c4471 100644
--- a/reports.mk
+++ b/reports.mk
@@ -26,7 +26,8 @@ define report_body
 fi; }
 endef
 
-all: reports/targets reports/scripts reports/cleanlog reports/contents
+all: reports/targets reports/scripts reports/cleanlog \
+	reports/contents reports/packages
 	@if [ -n "$(IMAGE_OUTPATH)" ]; then \
 		rm -fr "$(LOGDIR)/$(IMAGE_OUTFILE).reports"; \
 		cp -a "$(REPORTDIR)" "$(LOGDIR)/$(IMAGE_OUTFILE).reports"; \
@@ -75,10 +76,16 @@ reports/contents: reports/prep
 			echo "reports.mk: missing isoinfo" >&2; \
 		fi; \
 	esac
-	@cat $(BUILDLOG) | grep -E 'chroot/.in/[^/]*.rpm' | cut -d' ' -f 1 | tr -d "'"'`' | \
-		rev | cut -d'/' -f 1 | rev | sort -u > "$(REPORTDIR)/list-rpms.txt"
-	@cat $(BUILDLOG) | grep -E 'chroot/.in/[^/]*.rpm' | cut -d' ' -f 1 | tr -d "'"'`' | \
-		xargs rpm -qp --queryformat '%{sourcerpm}\n' | sort -u > "$(REPORTDIR)/list-srpms.txt"
+
+reports/packages: reports/prep
+	@grep -E 'chroot/.in/[^/]*.rpm' < $(BUILDLOG) | \
+		cut -d' ' -f 1 | tr -d "'"'`' | \
+		rev | cut -d'/' -f 1 | rev | \
+		sort -u > "$(REPORTDIR)/list-rpms.txt"
+	@grep -E 'chroot/.in/[^/]*.rpm' < $(BUILDLOG) | \
+		cut -d' ' -f 1 | tr -d "'"'`' | \
+		xargs rpm -qp --queryformat '%{sourcerpm}\n' | \
+		sort -u > "$(REPORTDIR)/list-srpms.txt"
 
 else
 all:; @:
-- 
2.25.4


[-- Attachment #3: 0002-reports.mk-streamline-basedir-1-equivalent.patch --]
[-- Type: text/x-patch, Size: 1047 bytes --]

>From 5eca87b3314d00a11009a5b5f3b57e1a17dec949 Mon Sep 17 00:00:00 2001
From: Michael Shigorin <mike@altlinux.org>
Date: Wed, 28 Apr 2021 14:50:06 +0300
Subject: [PATCH 2/3] reports.mk: streamline basedir(1) equivalent

That "clever hack" with two rev(1) invocations
seems pretty stupid to me: too much processes
and processing instead of simple prefix removal.

One can also have that as ${var##*/} when dealing
with a shell variable and avoiding extra forks, BTW.
---
 reports.mk | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/reports.mk b/reports.mk
index 0e44c4471..7b81e785a 100644
--- a/reports.mk
+++ b/reports.mk
@@ -79,8 +79,7 @@ reports/contents: reports/prep
 
 reports/packages: reports/prep
 	@grep -E 'chroot/.in/[^/]*.rpm' < $(BUILDLOG) | \
-		cut -d' ' -f 1 | tr -d "'"'`' | \
-		rev | cut -d'/' -f 1 | rev | \
+		cut -d' ' -f 1 | tr -d "'"'`' | sed 's,^.*/,,' | \
 		sort -u > "$(REPORTDIR)/list-rpms.txt"
 	@grep -E 'chroot/.in/[^/]*.rpm' < $(BUILDLOG) | \
 		cut -d' ' -f 1 | tr -d "'"'`' | \
-- 
2.25.4


[-- Attachment #4: 0003-reports.mk-use-process-substitution-to-dedup.patch --]
[-- Type: text/x-patch, Size: 1346 bytes --]

>From 20642ae4aa3847403d3462e74ace4c00ca3240b3 Mon Sep 17 00:00:00 2001
From: Michael Shigorin <mike@altlinux.org>
Date: Wed, 28 Apr 2021 15:06:57 +0300
Subject: [PATCH 3/3] reports.mk: use process substitution to dedup

The first half of both pipes was clearly a copy-pasted
initial logfile processing; the file can be of considerable
size (e.g. several megabytes) so it might be slightly more
efficient and cool (but a bit more arcane) to use bash(1)'s
process substitution along with good ol' tee(1) like this:

$ echo -e '1\n2\n3' |tee /dev/stderr 2> >(grep 2 >STDERR) |grep 1 >STDOUT
$ head STD*
==> STDERR <==
2

==> STDOUT <==
1
---
 reports.mk | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/reports.mk b/reports.mk
index 7b81e785a..247c722b5 100644
--- a/reports.mk
+++ b/reports.mk
@@ -78,11 +78,10 @@ reports/contents: reports/prep
 	esac
 
 reports/packages: reports/prep
-	@grep -E 'chroot/.in/[^/]*.rpm' < $(BUILDLOG) | \
-		cut -d' ' -f 1 | tr -d "'"'`' | sed 's,^.*/,,' | \
-		sort -u > "$(REPORTDIR)/list-rpms.txt"
 	@grep -E 'chroot/.in/[^/]*.rpm' < $(BUILDLOG) | \
 		cut -d' ' -f 1 | tr -d "'"'`' | \
+		tee /dev/stderr 2> >(sed 's,^.*/,,' | \
+			sort -u > "$(REPORTDIR)/list-rpms.txt") | \
 		xargs rpm -qp --queryformat '%{sourcerpm}\n' | \
 		sort -u > "$(REPORTDIR)/list-srpms.txt"
 
-- 
2.25.4


             reply	other threads:[~2021-04-28 12:38 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-28 12:38 Michael Shigorin [this message]
2021-04-28 13:52 ` Michael Shigorin
2021-04-29 14:39   ` Антон Мидюков

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210428123853.GA16601@imap.altlinux.org \
    --to=mike@altlinux.org \
    --cc=devel-distro@lists.altlinux.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

ALT Linux Distributions development

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://lore.altlinux.org/devel-distro/0 devel-distro/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 devel-distro devel-distro/ http://lore.altlinux.org/devel-distro \
		devel-distro@lists.altlinux.org devel-distro@lists.altlinux.ru devel-distro@lists.altlinux.com
	public-inbox-index devel-distro

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://lore.altlinux.org/org.altlinux.lists.devel-distro


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git