[devel] Optimize rebuilt subtask packages identical to the packages in the target repo

[devel] Optimize rebuilt subtask packages identical to the packages in the target repo

[Vladimir D. Seleznev]

Hi, devel@!

The first step of binary package optimization. This patch is only about
subtask package rebuild within a particular repository.

--
  WBR,
  Vladimir D. Seleznev

[devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Vladimir D. Seleznev]

Introduce task post-build processing. It finds subtasks with package
rebuild and if the rebuilt packages identical to the same packages in
the target repo it optimizes them.

Whether subtasks in the particular repository should be check or not for
such optimization is controlling via GB_OPTIMIZE_IDENTICAL_REBUILD
option.

* gb/gb-build-task: Add gb-task-build-post.
* gb/gb-sh-conf: Add GB_OPTIMIZE_IDENTICAL_REBUILD option.
* gb/gb-task-build-post: New file.
* gb/gb-task-build-post-i: Likewise.
---
 gb/gb-build-task        |   1 +
 gb/gb-sh-conf           |   1 +
 gb/gb-task-build-post   |   9 ++++
 gb/gb-task-build-post-i | 114 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 125 insertions(+)
 create mode 100755 gb/gb-task-build-post
 create mode 100755 gb/gb-task-build-post-i

diff --git a/gb/gb-build-task b/gb/gb-build-task
index 0fb076e..5f4b23e 100755
--- a/gb/gb-build-task
+++ b/gb/gb-build-task
@@ -61,6 +61,7 @@ gb-task-set-summary
 	gb-task-copy-packages
 	gb-task-build-prep
 	gb-task-build
+	gb-task-build-post
 	fail_if_task_abort_requested
 	gb-x-girar task-make-index-html "$id" ||:
 
diff --git a/gb/gb-sh-conf b/gb/gb-sh-conf
index f1366fe..52bc79e 100644
--- a/gb/gb-sh-conf
+++ b/gb/gb-sh-conf
@@ -38,3 +38,4 @@ GB_AREPO_DIR="$TMPDIR/gb-arepo-$USER-$GB_REPO_NAME"
 	remote_host="build-$USER-$GB_REPO_NAME-$arch"
 
 : ${GB_ALLOW_SAME_NEVR=}
+: ${GB_OPTIMIZE_IDENTICAL_REBUILD=}
diff --git a/gb/gb-task-build-post b/gb/gb-task-build-post
new file mode 100755
index 0000000..e0af1ff
--- /dev/null
+++ b/gb/gb-task-build-post
@@ -0,0 +1,9 @@
+#!/bin/sh -efu
+
+. gb-sh-functions
+
+fail_if_task_abort_requested
+
+for i in $(src_nums); do
+	$0-i "$i"
+done
diff --git a/gb/gb-task-build-post-i b/gb/gb-task-build-post-i
new file mode 100755
index 0000000..104c294
--- /dev/null
+++ b/gb/gb-task-build-post-i
@@ -0,0 +1,114 @@
+#!/bin/sh -efu
+
+. gb-sh-functions
+. gb-sh-tmpdir
+
+[ -n "$GB_OPTIMIZE_IDENTICAL_REBUILD" ] ||
+	exit 0
+
+i="$1"; shift
+query_filename='%{name}-%{version}-%{release}.%{arch}.rpm'
+
+is_rebuild=
+st_is_rebuild()
+{
+	[ -z "$is_rebuild" ] ||
+		return 0
+
+	local arch="$1"; shift
+
+	srpm="$(find "build/$i/$arch/srpm/" \
+		-mindepth 1 -maxdepth 1 -type f \
+		-name '*.rpm' -printf '%f')"
+
+	[ -f "$GB_REPO_DIR/files/SRPMS/$srpm" ] ||
+		return 1
+
+	is_rebuild=1
+
+	return 0
+}
+
+mixed_arches=
+narchs=
+get_narchs()
+{
+	[ -z "$narchs" ] ||
+		return 0
+
+	narchs="$(find "build/$i/$arch/rpms/" \
+		-mindepth 1 -maxdepth 1 \
+		-type f -name '*.rpm' \
+		-execdir rpmquery --qf '%{arch}\n' -p '{}' '+' |
+		sort -u |wc -l)"
+
+	[ "$narchs" -eq 1 ] ||
+		mixed_arches=1
+}
+
+stamp_echo >&2 "#$i: Starting post build processing"
+
+first_arch=1
+for arch in $GB_ARCH; do
+	[ ! -s "build/$i/$arch/excluded" ] ||
+		continue
+
+	# exit if subtask is not a rebuild
+	st_is_rebuild "$arch" || {
+		stamp_echo >&2 "#$i: subtask is not a rebuild, skipping"
+		exit 0
+	}
+
+	get_narchs
+
+	find "build/$i/$arch/rpms/" -mindepth 1 -maxdepth 1 -type f -name '*.rpm' \
+		-execdir rpmquery --qf "$query_filename %{arch}\n" \
+		-p '{}' '+' >"$tmpdir/built.$arch.pkgs"
+
+	while read -r brpm barch; do
+		# debuginfo rpms are out of our interest
+		case "$brpm" in
+		*-debuginfo-*) continue ;;
+		esac
+
+		# skip non-first built noarch rpm in mixed arches subtask
+		[ -z "$mixed_arches" ] || [ -z "$first_arch" ] || [ "$barch" != "noarch" ] ||
+			continue
+
+		bid="$(rpmidentity -p "build/$i/$arch/rpms/$brpm")"
+		printf "%s %s %s\n" "$brpm" "$barch" "$bid" >>"$tmpdir/built.identity"
+		rid="$(rpmidentity -p "$GB_REPO_DIR/files/$barch/RPMS/$brpm")"
+		printf "%s %s %s\n" "$brpm" "$barch" "$rid" >>"$tmpdir/repo.identity"
+	done <"$tmpdir/built.$arch.pkgs"
+
+	first_arch=
+done
+
+sort -k 3 -o "$tmpdir/built.identity"{,}
+sort -k 3 -o "$tmpdir/repo.identity"{,}
+join -j 3 -v2 "$tmpdir/built.identity" "$tmpdir/repo.identity" |
+	cut -d' ' -f2 >"$tmpdir/nonidentical.arches"
+
+# Mixed noarch and arches build is a special case: we have to optimize either
+# all packages of all architectures or none of them
+if [ -n "$mixed_arches" -a -s "$tmpdir/nonidentical.arches" ]; then
+	echo >&2 "#$i: non-identical rebuild with mixed arches, skip optimizing"
+	exit 0
+fi
+
+for arch in $GB_ARCH; do
+	[ ! -s "build/$i/$arch/excluded" ] ||
+		continue
+
+	if grep -q "$arch" "$tmpdir/nonidentical.arches"; then
+		echo >&2 "[$arch] #$i rebuild is non-identical, skip optimizing"
+		continue
+	fi
+
+	echo >&2 "[$arch] #$i rebuild is identical, optimize packages"
+	while read brpm barch; do
+		cp -p "$GB_REPO_DIR/files/$barch/RPMS/$brpm" "build/$i/$arch/rpms/$brpm"
+		echo >&2 "[$arch] #$i $brpm is optimized"
+	done <"$tmpdir/built.$arch.pkgs"
+done
+
-- 
2.25.4

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Alexey Tourbin]

On Thu, Jun 4, 2020 at 10:58 PM Vladimir D. Seleznev
<vseleznv@altlinux.org> wrote:
> Introduce task post-build processing. It finds subtasks with package
> rebuild and if the rebuilt packages identical to the same packages in
> the target repo it optimizes them.

It doesn't make much sense. When we rebuild a package without changing
the release, we expect something else in the package to change because
of the rebuild (e.g. a binary will be linked with a new library
version). If the package hasn't changed, it is an alarming condition
which indicates that some of the packager's assumptions were wrong
(e.g. the binary actually doesn't link with the library). So should we
really "optimize" this case? We might as well prohibit it! The
packager should be cognizant that some of his efforts aren't making
any difference. :)

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Vladimir D. Seleznev]

On Fri, Jun 05, 2020 at 04:40:52PM +0300, Alexey Tourbin wrote:
> On Thu, Jun 4, 2020 at 10:58 PM Vladimir D. Seleznev
> <vseleznv@altlinux.org> wrote:
> > Introduce task post-build processing. It finds subtasks with package
> > rebuild and if the rebuilt packages identical to the same packages in
> > the target repo it optimizes them.
> 
> It doesn't make much sense. When we rebuild a package without changing
> the release, we expect something else in the package to change because
> of the rebuild (e.g. a binary will be linked with a new library
> version). If the package hasn't changed, it is an alarming condition
> which indicates that some of the packager's assumptions were wrong
> (e.g. the binary actually doesn't link with the library). So should we
> really "optimize" this case? We might as well prohibit it!

By "prohibit" you mean make task build fail? I would say that it is
unnecessary. It'd produce additional difficulties for maintainers
without any profit.

> The packager should be cognizant that some of his efforts aren't
> making any difference. :)

But packager can know that this kind of effort aren't making any
difference because of build log warning. And this can be convenient in
case of mass rebuild, for example.

P.S. This suggestion seems like a joke to me :)

-- 
   WBR,
   Vladimir D. Seleznev

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Alexey Tourbin]

On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev
<vseleznv@altlinux.org> wrote:
> > > Introduce task post-build processing. It finds subtasks with package
> > > rebuild and if the rebuilt packages identical to the same packages in
> > > the target repo it optimizes them.
> >
> > It doesn't make much sense. When we rebuild a package without changing
> > the release, we expect something else in the package to change because
> > of the rebuild (e.g. a binary will be linked with a new library
> > version). If the package hasn't changed, it is an alarming condition
> > which indicates that some of the packager's assumptions were wrong
> > (e.g. the binary actually doesn't link with the library). So should we
> > really "optimize" this case? We might as well prohibit it!
>
> By "prohibit" you mean make task build fail? I would say that it is
> unnecessary. It'd produce additional difficulties for maintainers
> without any profit.

The difficulties are all in the maintainers' heads.
There must be a valid reason for rebuilding a package.

> > The packager should be cognizant that some of his efforts aren't
> > making any difference. :)
>
> But packager can know that this kind of effort aren't making any
> difference because of build log warning. And this can be convenient in
> case of mass rebuild, for example.

So are we to permit the tasks that don't change the repo? Then you
should also optimize base/pkglist and the and release files, their
timestamps should not be clobbered. So that the end user immediately
sees there's no need to update from the repo, although a few tasks
have been committed.  Nice optimization this, eh?

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Dmitry V. Levin]

On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > Introduce task post-build processing. It finds subtasks with package
> > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > the target repo it optimizes them.
> > >
> > > It doesn't make much sense. When we rebuild a package without changing
> > > the release, we expect something else in the package to change because
> > > of the rebuild (e.g. a binary will be linked with a new library
> > > version). If the package hasn't changed, it is an alarming condition
> > > which indicates that some of the packager's assumptions were wrong
> > > (e.g. the binary actually doesn't link with the library). So should we
> > > really "optimize" this case? We might as well prohibit it!
> >
> > By "prohibit" you mean make task build fail? I would say that it is
> > unnecessary. It'd produce additional difficulties for maintainers
> > without any profit.
> 
> The difficulties are all in the maintainers' heads.
> There must be a valid reason for rebuilding a package.

Given that rebuilding a package costs so little for the maintainer,
we definitely should reject rebuilds that do not result to changed
packages.


-- 
ldv

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Andrey Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

On Sat, 13 Jun 2020 20:45:19 +0300 Dmitry V. Levin wrote:
> On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> > On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > > Introduce task post-build processing. It finds subtasks with package
> > > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > > the target repo it optimizes them.
> > > >
> > > > It doesn't make much sense. When we rebuild a package without changing
> > > > the release, we expect something else in the package to change because
> > > > of the rebuild (e.g. a binary will be linked with a new library
> > > > version). If the package hasn't changed, it is an alarming condition
> > > > which indicates that some of the packager's assumptions were wrong
> > > > (e.g. the binary actually doesn't link with the library). So should we
> > > > really "optimize" this case? We might as well prohibit it!
> > >
> > > By "prohibit" you mean make task build fail? I would say that it is
> > > unnecessary. It'd produce additional difficulties for maintainers
> > > without any profit.
> > 
> > The difficulties are all in the maintainers' heads.
> > There must be a valid reason for rebuilding a package.
> 
> Given that rebuilding a package costs so little for the maintainer,
> we definitely should reject rebuilds that do not result to changed
> packages.

There are valid cases when it is impossible to determine beforehand
if rebuild will result in changed package or not, e.g. during boost
updates.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Dmitry V. Levin]

On Sat, Jun 13, 2020 at 09:50:37PM +0300, Andrey Savchenko wrote:
> On Sat, 13 Jun 2020 20:45:19 +0300 Dmitry V. Levin wrote:
> > On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> > > On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > > > Introduce task post-build processing. It finds subtasks with package
> > > > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > > > the target repo it optimizes them.
> > > > >
> > > > > It doesn't make much sense. When we rebuild a package without changing
> > > > > the release, we expect something else in the package to change because
> > > > > of the rebuild (e.g. a binary will be linked with a new library
> > > > > version). If the package hasn't changed, it is an alarming condition
> > > > > which indicates that some of the packager's assumptions were wrong
> > > > > (e.g. the binary actually doesn't link with the library). So should we
> > > > > really "optimize" this case? We might as well prohibit it!
> > > >
> > > > By "prohibit" you mean make task build fail? I would say that it is
> > > > unnecessary. It'd produce additional difficulties for maintainers
> > > > without any profit.
> > > 
> > > The difficulties are all in the maintainers' heads.
> > > There must be a valid reason for rebuilding a package.
> > 
> > Given that rebuilding a package costs so little for the maintainer,
> > we definitely should reject rebuilds that do not result to changed
> > packages.
> 
> There are valid cases when it is impossible to determine beforehand
> if rebuild will result in changed package or not, e.g. during boost
> updates.

So what?  Failed build is not a crime, let it fail.


-- 
ldv

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Andrey Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2027 bytes --]

On Sat, 13 Jun 2020 23:48:38 +0300 Dmitry V. Levin wrote:
> On Sat, Jun 13, 2020 at 09:50:37PM +0300, Andrey Savchenko wrote:
> > On Sat, 13 Jun 2020 20:45:19 +0300 Dmitry V. Levin wrote:
> > > On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> > > > On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > > > > Introduce task post-build processing. It finds subtasks with package
> > > > > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > > > > the target repo it optimizes them.
> > > > > >
> > > > > > It doesn't make much sense. When we rebuild a package without changing
> > > > > > the release, we expect something else in the package to change because
> > > > > > of the rebuild (e.g. a binary will be linked with a new library
> > > > > > version). If the package hasn't changed, it is an alarming condition
> > > > > > which indicates that some of the packager's assumptions were wrong
> > > > > > (e.g. the binary actually doesn't link with the library). So should we
> > > > > > really "optimize" this case? We might as well prohibit it!
> > > > >
> > > > > By "prohibit" you mean make task build fail? I would say that it is
> > > > > unnecessary. It'd produce additional difficulties for maintainers
> > > > > without any profit.
> > > > 
> > > > The difficulties are all in the maintainers' heads.
> > > > There must be a valid reason for rebuilding a package.
> > > 
> > > Given that rebuilding a package costs so little for the maintainer,
> > > we definitely should reject rebuilds that do not result to changed
> > > packages.
> > 
> > There are valid cases when it is impossible to determine beforehand
> > if rebuild will result in changed package or not, e.g. during boost
> > updates.
> 
> So what?  Failed build is not a crime, let it fail.

Intentionally wasted maintainer's time is a huge crime. Let a
machine work instead of a human, this way we can be more productive.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Dmitry V. Levin]

On Sat, Jun 13, 2020 at 11:57:08PM +0300, Andrey Savchenko wrote:
> On Sat, 13 Jun 2020 23:48:38 +0300 Dmitry V. Levin wrote:
> > On Sat, Jun 13, 2020 at 09:50:37PM +0300, Andrey Savchenko wrote:
> > > On Sat, 13 Jun 2020 20:45:19 +0300 Dmitry V. Levin wrote:
> > > > On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> > > > > On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > > > > > Introduce task post-build processing. It finds subtasks with package
> > > > > > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > > > > > the target repo it optimizes them.
> > > > > > >
> > > > > > > It doesn't make much sense. When we rebuild a package without changing
> > > > > > > the release, we expect something else in the package to change because
> > > > > > > of the rebuild (e.g. a binary will be linked with a new library
> > > > > > > version). If the package hasn't changed, it is an alarming condition
> > > > > > > which indicates that some of the packager's assumptions were wrong
> > > > > > > (e.g. the binary actually doesn't link with the library). So should we
> > > > > > > really "optimize" this case? We might as well prohibit it!
> > > > > >
> > > > > > By "prohibit" you mean make task build fail? I would say that it is
> > > > > > unnecessary. It'd produce additional difficulties for maintainers
> > > > > > without any profit.
> > > > > 
> > > > > The difficulties are all in the maintainers' heads.
> > > > > There must be a valid reason for rebuilding a package.
> > > > 
> > > > Given that rebuilding a package costs so little for the maintainer,
> > > > we definitely should reject rebuilds that do not result to changed
> > > > packages.
> > > 
> > > There are valid cases when it is impossible to determine beforehand
> > > if rebuild will result in changed package or not, e.g. during boost
> > > updates.
> > 
> > So what?  Failed build is not a crime, let it fail.
> 
> Intentionally wasted maintainer's time is a huge crime. Let a
> machine work instead of a human, this way we can be more productive.

I suppose packages are built for a reason, so every unchanged build
must be a mistake, and I read this as an argument to fail such builds.


-- 
ldv

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Andrey Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2603 bytes --]

On Sun, 14 Jun 2020 00:10:49 +0300 Dmitry V. Levin wrote:
> On Sat, Jun 13, 2020 at 11:57:08PM +0300, Andrey Savchenko wrote:
> > On Sat, 13 Jun 2020 23:48:38 +0300 Dmitry V. Levin wrote:
> > > On Sat, Jun 13, 2020 at 09:50:37PM +0300, Andrey Savchenko wrote:
> > > > On Sat, 13 Jun 2020 20:45:19 +0300 Dmitry V. Levin wrote:
> > > > > On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> > > > > > On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > > > > > > Introduce task post-build processing. It finds subtasks with package
> > > > > > > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > > > > > > the target repo it optimizes them.
> > > > > > > >
> > > > > > > > It doesn't make much sense. When we rebuild a package without changing
> > > > > > > > the release, we expect something else in the package to change because
> > > > > > > > of the rebuild (e.g. a binary will be linked with a new library
> > > > > > > > version). If the package hasn't changed, it is an alarming condition
> > > > > > > > which indicates that some of the packager's assumptions were wrong
> > > > > > > > (e.g. the binary actually doesn't link with the library). So should we
> > > > > > > > really "optimize" this case? We might as well prohibit it!
> > > > > > >
> > > > > > > By "prohibit" you mean make task build fail? I would say that it is
> > > > > > > unnecessary. It'd produce additional difficulties for maintainers
> > > > > > > without any profit.
> > > > > > 
> > > > > > The difficulties are all in the maintainers' heads.
> > > > > > There must be a valid reason for rebuilding a package.
> > > > > 
> > > > > Given that rebuilding a package costs so little for the maintainer,
> > > > > we definitely should reject rebuilds that do not result to changed
> > > > > packages.
> > > > 
> > > > There are valid cases when it is impossible to determine beforehand
> > > > if rebuild will result in changed package or not, e.g. during boost
> > > > updates.
> > > 
> > > So what?  Failed build is not a crime, let it fail.
> > 
> > Intentionally wasted maintainer's time is a huge crime. Let a
> > machine work instead of a human, this way we can be more productive.
> 
> I suppose packages are built for a reason, so every unchanged build
> must be a mistake, and I read this as an argument to fail such builds.

Filter them out automatically if you can, keep them if you can't.
But wasting human time for such type of work is nonsense and should
not be allowed.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Vladimir D. Seleznev]

On Sat, Jun 13, 2020 at 08:45:19PM +0300, Dmitry V. Levin wrote:
> On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> > On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > > Introduce task post-build processing. It finds subtasks with package
> > > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > > the target repo it optimizes them.
> > > >
> > > > It doesn't make much sense. When we rebuild a package without changing
> > > > the release, we expect something else in the package to change because
> > > > of the rebuild (e.g. a binary will be linked with a new library
> > > > version). If the package hasn't changed, it is an alarming condition
> > > > which indicates that some of the packager's assumptions were wrong
> > > > (e.g. the binary actually doesn't link with the library). So should we
> > > > really "optimize" this case? We might as well prohibit it!
> > >
> > > By "prohibit" you mean make task build fail? I would say that it is
> > > unnecessary. It'd produce additional difficulties for maintainers
> > > without any profit.
> > 
> > The difficulties are all in the maintainers' heads.
> > There must be a valid reason for rebuilding a package.
> 
> Given that rebuilding a package costs so little for the maintainer,
> we definitely should reject rebuilds that do not result to changed
> packages.

So you think than tasks should fail if rebuild do not result to changed
packages. I argue with that. Packages optimization in that case is more
graceful solution and less disturbing. And imaging the case when you
have a big task with a lot of subtasks caused by brand new Python
update.  Some of these subtasks may be unchanged after rebuild and I
don't see why they should be failed. But if they will, this will
produce too much needless work for the poor maintainer.

Now subtasks do not fail if somebody just rebuilds packages if their
builds aren't broken, and if their results unchange packages, we have
wasted storage space. If packages will be optimized, this storage space
will be saved.

But I like the idea to optimize repo updates for optimized subtasks. And
I think there will be no harm if task will fail if all of its subtasks
will have unchanged packages after rebuild.

-- 
   WBR,
   Vladimir D. Seleznev

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Dmitry V. Levin]

On Sat, Jun 13, 2020 at 10:32:13PM +0300, Vladimir D. Seleznev wrote:
> On Sat, Jun 13, 2020 at 08:45:19PM +0300, Dmitry V. Levin wrote:
> > On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> > > On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > > > Introduce task post-build processing. It finds subtasks with package
> > > > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > > > the target repo it optimizes them.
> > > > >
> > > > > It doesn't make much sense. When we rebuild a package without changing
> > > > > the release, we expect something else in the package to change because
> > > > > of the rebuild (e.g. a binary will be linked with a new library
> > > > > version). If the package hasn't changed, it is an alarming condition
> > > > > which indicates that some of the packager's assumptions were wrong
> > > > > (e.g. the binary actually doesn't link with the library). So should we
> > > > > really "optimize" this case? We might as well prohibit it!
> > > >
> > > > By "prohibit" you mean make task build fail? I would say that it is
> > > > unnecessary. It'd produce additional difficulties for maintainers
> > > > without any profit.
> > > 
> > > The difficulties are all in the maintainers' heads.
> > > There must be a valid reason for rebuilding a package.
> > 
> > Given that rebuilding a package costs so little for the maintainer,
> > we definitely should reject rebuilds that do not result to changed
> > packages.
> 
> So you think than tasks should fail if rebuild do not result to changed
> packages. I argue with that. Packages optimization in that case is more
> graceful solution and less disturbing. And imagine the case when you
> have a big task with a lot of subtasks caused by brand new Python
> update.  Some of these subtasks may be unchanged after rebuild and I
> don't see why they should be failed. But if they will, this will
> produce too much needless work for the poor maintainer.

No, I don't buy these arguments.  If a subtask produced a non-changer,
just remove it.


-- 
ldv

Re: [devel] [PATCH] gb: add gb-task-build-post, optimize packages with identical rebuild

[Vladimir D. Seleznev]

On Sun, Jun 14, 2020 at 12:03:40AM +0300, Dmitry V. Levin wrote:
> On Sat, Jun 13, 2020 at 10:32:13PM +0300, Vladimir D. Seleznev wrote:
> > On Sat, Jun 13, 2020 at 08:45:19PM +0300, Dmitry V. Levin wrote:
> > > On Sat, Jun 06, 2020 at 04:42:21PM +0300, Alexey Tourbin wrote:
> > > > On Fri, Jun 5, 2020 at 5:23 PM Vladimir D. Seleznev wrote:
> > > > > > > Introduce task post-build processing. It finds subtasks with package
> > > > > > > rebuild and if the rebuilt packages identical to the same packages in
> > > > > > > the target repo it optimizes them.
> > > > > >
> > > > > > It doesn't make much sense. When we rebuild a package without changing
> > > > > > the release, we expect something else in the package to change because
> > > > > > of the rebuild (e.g. a binary will be linked with a new library
> > > > > > version). If the package hasn't changed, it is an alarming condition
> > > > > > which indicates that some of the packager's assumptions were wrong
> > > > > > (e.g. the binary actually doesn't link with the library). So should we
> > > > > > really "optimize" this case? We might as well prohibit it!
> > > > >
> > > > > By "prohibit" you mean make task build fail? I would say that it is
> > > > > unnecessary. It'd produce additional difficulties for maintainers
> > > > > without any profit.
> > > > 
> > > > The difficulties are all in the maintainers' heads.
> > > > There must be a valid reason for rebuilding a package.
> > > 
> > > Given that rebuilding a package costs so little for the maintainer,
> > > we definitely should reject rebuilds that do not result to changed
> > > packages.
> > 
> > So you think than tasks should fail if rebuild do not result to changed
> > packages. I argue with that. Packages optimization in that case is more
> > graceful solution and less disturbing. And imagine the case when you
> > have a big task with a lot of subtasks caused by brand new Python
> > update.  Some of these subtasks may be unchanged after rebuild and I
> > don't see why they should be failed. But if they will, this will
> > produce too much needless work for the poor maintainer.
> 
> No, I don't buy these arguments.  If a subtask produced a non-changer,
> just remove it.

Ok, that's fine with me, if this remove will be done automatically. But
theoretically there can be cases where in non-mixed arches you have
non-changed rebuilt packages on some architectures, but changed packages
on anoter. I would prefer to optimize non-changed than to pretend that
this whole rebuild produced changed packages.

Non-mixed arches build means either all the built packages are
arch-specific, or they all are noarch.

-- 
   WBR,
   Vladimir D. Seleznev