* tests/rm/dangling-symlink.sh: The effective delay was only
1.5s. So delay before the operation, and increase the iteration
count by 1 to increase the delay to 6.3s. This failure was
noticed once on a FreeBSD 10.2 x86_64 virtual machine.
* tests/dd/stats.sh: Remove quotes on sleep argument
to ensure arguments are accumulated appropriately.
* tests/du/move-dir-while-traversing.sh: sleep before operation,
to increase the effective delay from 1.6s to 3.2s.
* tests/tail-2/flush-initial.sh: Likewise.
* tests/tail-2/pipe-f2.sh: Sleep first so that the effective delay
matches the commented value.
At least the false positive in tail-2/follow-stdin.sh could be seen
on a 4-core i5 system with -j8. Fix similar cases. too.
* tests/tail-2/follow-stdin.sh: Empty the 'out' file in each iteration.
Otherwise, under heavy system load, 'check_tail_output' would see the
expected output from the previous round before tail would have the
chance to come up in the background.
While at it, move the creation of the 'exp' file out of the loop.
* tests/dd/stats.sh: Empty the output file of the background process
here, too.
* tests/misc/cat-buf.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/tail-2/wait.sh: Likewise.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
Problem reported by Linda Walsh in: http://bugs.gnu.org/17505
* NEWS: Document this.
* doc/coreutils.texi (dd invocation): Use a simpler script.
Adjust output example to match new behavior.
* src/dd.c (human_size): Remove.
All uses changed to use human_readable and ...
(human_opts): ... this new constant.
(abbreviation_lacks_prefix): New function.
(print_xfer_stats): Use it. Output both --si and --human-readable
summaries, but only if they have prefixes.
* tests/dd/reblock.sh, tests/dd/stats.sh: Test new behavior.
programs may not be built due to missing system dependencies,
or any program can be excluded at configure time with
--enable-no-install-program. So ensure we're not testing the
system versions in these cases.
* init.cfg (print_ver_): Call require_built_ first.
* tests/misc/tty-eof.pl: Skip programs not built.
* tests/Coreutils.pm (run_tests): Likewise.
* tests/misc/ls-misc.pl: Use 'env test' rather than abs path.
* tests/misc/test-diag.pl: Likewise.
* tests/local.mk: Adjust include order for dependencies.
* tests/misc/arch.sh: Remove redundant calls to require_built_.
* tests/misc/chroot-fail.sh: Likewise.
* tests/misc/groups-dash.sh: Likewise.
* tests/misc/groups-version.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/cp/acl.sh: Remove problematic call to print_ver_ [gs]etfacl.
* tests/mv/acl.sh: Likewise.
* cfg.mk (sc_env_test_dependencies): A new syntax check to enforce
specifying dependencies with print_ver_ for programs
specified through the env command.
* du/bigtime.sh: Add new print_ver_ dependencies.
* du/max-depth.sh: Likewise.
* dd/ascii.sh: Likewise.
* tests/ls/capability.sh: Likewise.
* tests/ls/root-rel-symlink-color.sh: Likewise.
* tests/misc/chroot-fail.sh: Likewise.
* tests/misc/readlink-fp-loop.sh: Likewise.
* tests/misc/sort-debug-keys.sh: Likewise.
* tests/readlink/can-e.sh: Likewise.
* tests/readlink/can-f.sh: Likewise.
* tests/readlink/can-m.sh: Likewise.
* tests/tail-2/inotify-race.sh: Likewise.
* tests/tail-2/inotify-race2.sh: Likewise.
* tests/touch/no-create-missing.sh: Likewise.
* tests/touch/no-dereference.sh: Likewise.
* tests/misc/printenv.sh: Tweak to avoid syntax check trigger.
* tests/misc/help-version.sh: Likewise.
* tests/misc/yes.sh: Likewise.
* tests/misc/printf-quote.sh: Use previously unused $prog.
* configure.ac (EXTRA_MANS): Add $gl_no_install_prog to the list
so that check-x-vs-1 syntax check is satisfied.
A side effect of this cleanup is we no longer
depend on our own kill command being built.
* init.cfg (require_trap_signame_): A new function to verify
that the shell supports specifying traps by signal name.
(require_kill_group_): A new function to ensure the shell
supports sending a signal to a group.
* tests/du/move-dir-while-traversing.sh: Ensure trap supports
signal names.
* tests/misc/stty-invalid.sh: Likewise.
* tests/misc/stty-pairs.sh: Likewise.
* tests/misc/stty-row-col.sh: Likewise.
* tests/misc/stty.sh: Likewise.
* tests/misc/sort-compress.sh: Likewise. Also simplify trap call.
* tests/install/trap.sh: Likewise.
* tests/misc/timeout.sh: Likewise.
* tests/dd/stats.sh: Likewise. Also use default kill command.
* tests/misc/timeout-group.sh: Likewise.
* tests/dd/no-allocate.sh: Account for timeout(1) when
determining the required mem, as timeout has additional shared libs.
This avoids the need for the hardcoded 4M addition to the limit.
* tests/misc/head-c.sh: Increase the base limit, to account for
the fact that head(1) will allocate some additional mem in this case.
* tests/misc/cut-huge-range.sh: Remove mention of specific limits.
* tests/misc/printf-surprise.sh: Likewise.
Reported by Dmitry V. Levin
When configured with either 'symlinks' or 'shebangs' as value for
the --enable-single-binary option, tests based on `ulimit -v` are
skipped. The reason is that the multicall 'coreutils' binary requires
much more memory due to shared libraries being loaded, and the size of
the 'date' binary (~290KiB) compared to the multicall binary (~5MiB),
of course. Finally, in the case of 'shebangs', the starting shell
requires more memory, too
Instead of using hard-coded values for the memory limit, use an
adaptive approach: first determine the amount of memory for a similar,
yet more trivial invocation of the command, and then do the real test
run using that limit (plus some buffer in some cases).
* init.cfg (require_ulimit_v_): Remove function.
(get_min_ulimit_v_): Add function to determine the minimum memory limit
required for a given command in an adaptive way.
* cfg.mk (sc_prohibit_test_ulimit_without_require_): Change the name
of the above function in the syntax-check rule.
* tests/cp/link-heap.sh: Use the above function to determine the
minimum memory required to run a command simpler than in the real test
run. Use that limit plus a buffer there. While at it, change to list
of commands in the subshell to fail also if the beginning `ulimit -v`
fails.
* tests/dd/no-allocate.sh: Likewise.
* tests/misc/csplit-heap.sh: Likewise.
* tests/misc/cut-huge-range.sh: Likewise.
* tests/misc/head-c.sh: Likewise.
* tests/misc/printf-surprise.sh: Likewise.
* tests/split/line-bytes.sh: Likewise.
* tests/rm/many-dir-entries-vs-OOM.sh: Likewise - doing it separately
for each program under test.
* tests/dd/stats.sh: Wait 20s for dd to write 250MB through a fifo,
rather than 10s for 500MB. The failure was seen often on
a lightly loaded SPARC-Enterprise-T5220 running Solaris 10.
* tests/dd/sparse.sh: Sync files before checking allocations,
which may be done asynchronously on NFS and BTRFS at least.
Also mark this test as very expensive on remote file systems.
* tests/du/2g.sh: Likewise, also use fallocate if available
to efficiently allocate the large file, otherwise skip
on remote file systems.
* tests/tail-2/inotify-rotate-resources.sh: Use the more
standard is_local_dir_() to check remoteness.
* tests/cp/fiemap-empty.sh: Comment on the sync issue
for this currerntly unused test.
Fixes http://bugs.gnu.org/20570
When some program produces unexpected output, that use of
compare-vs-/dev/null will ensure that the surprising output is
printed in the test's output. With "test -s err" only, one
would have to instrument and rerun in order to see the offending
output.
* cfg.mk (sc_prohibit_and_fail_1): Exempt 'compare' from this check.
* tests/dd/misc.sh: Change "tests -s ... || fail=1" to
"compare /dev/null ... && fail=1".
* tests/misc/nice.sh: Likewise.
* tests/rm/read-only.sh: Likewise.
* tests/tail-2/inotify-race.sh: Likewise.
* tests/touch/no-dereference.sh: Likewise.
Suggested by Jim Meyering in
http://lists.gnu.org/archive/html/coreutils/2015-01/msg00042.html
Many tests use `program ... && fail=1` to ensure expected
error situations are indicated. However that would mask
an unexpected exit (like a crash). Therefore explicitly
check the expected exit code.
Note where error messages are also verified, the extra
protection is not added.
* tests/init.sh (returns_): A new helper function to
check the return code of a command, and used
throughout the tests.
* cfg.mk (sc_prohibit_and_fail_1): Add a syntax check
to avoid new instances of this issue.
Run "make update-copyright" and then...
* tests/sample-test: Adjust to use the single most recent year.
* tests/du/bind-mount-dir-cycle-v2.sh: Fix case in copyright message,
so that year is updated automatically in future.
Solaris 8 was seen to issue this error:
"printf: `&': illegal format character"
* test/dd/ascii.sh: Use the coreutils printf in this test
rather than the system one, to avoid portability issues.
* src/dd.c: Report the transfer progress every second when the
new status=progress level is used. Adjust the handling and
description of the status= option so that they're treated as
mutually exclusive levels, rather than flags with implicit precedence.
* doc/coreutils.texi (dd invocation): Document the new progress
status level. Reference the new level in the description of SIGUSR1.
* tests/dd/stats.sh: Add new test for status=progress.
* tests/dd/misc.sh: Change so status=none only takes precedence
if it's the last level specified.
* NEWS: Mention the feature.
* src/dd.c (ifd_reopen): A new wrapper to ensure we
don't exit upon receiving a SIGUSR1 in a blocking open()
on a fifo for example.
(iftruncate): Likewise for ftruncate().
(iread): Process signals also after a short read.
(install_signal_handlers): Install SIGINFO/SIGUSR1 handler
even if set to SIG_IGN, as this is what the parent can easily
set from a shell script that can send SIGUSR1 without the
possiblity of inadvertently killing the dd process.
* doc/coreutils.texi (dd invocation): Improve the example to
show robust usage wrt signal races and short reads.
* tests/dd/stats.sh: A new test for various signal races.
* tests/local.mk: Reference the new test.
* NEWS: Mention the fix.
* tests/dd/ascii.sh: Quote so that '\\' is passed
to printf rather than a single '\', as that's not portable.
bash, dash, zsh and external solaris printf were seen to support
printf '\%03o' 1 2
while ksh, freebsh shell and external GNU printf need the more portable
printf '\\%03o' 1 2
Note we don't use env here to call the coreutils printf implementation,
as there are many printf calls, so relying on portable shell
implementations will be faster.
Avoid complicated and error-prone parsing of df's output via
sed(1), cut(1), etc., and instead use df's more modern --output
option.
* src/ioblksize.h (in a comment): Simplify the extraction of the
device name of the mounted file system from df's output.
* tests/dd/skip-seek-past-dev.sh: Likewise.
* tests/du/2g.sh: Likewise for the 'avail' column here.
Also avoid the deprecated use of "tail -NUM".
* tests/misc/stat-mount.sh: While at it, remove the determination
of the mount point of "." via df(1) plus sed(1) as it is unused
since commit v8.5-159-gf57cb37 anyway. Instead, improve this test
by verifying that the output of "stat -c%m ." at least starts with
a slash '/'.
* tests/dd/ascii.sh: Avoid unnecessary subshells. Catch dd's
exit code. Remove testing artifact. In the case of a comparison
failure, show the differences in octal format in addition to
"binary files differ". Simplify the creation of the 'in' file.
Problem reported by Don Baggett in <http:/bugs.gnu.org/17422>.
* NEWS:
* doc/coreutils.texi (dd invocation): Document this.
* src/dd.c (conversions): conv=ascii implies conv=unblock.
conv=ebcdic and conv=ibm imply conv=block.
(ascii_to_ebcdic, ebcdic_to_ascii): Correct to match
POSIX 1003.1-2013.
* tests/dd/ascii.sh: New file.
* tests/local.mk (all_tests): Add it.
* tests/dd/sparse.sh: When testing that a hole is created,
use an existing sparse destination file, so that we're
not write extending the file size, and thus avoiding
speculative preallocation which can result in smaller
holes than requested.
Workaround suggested by Brian Foster
Run "make update-copyright", but then also run this,
perl -pi -e 's/2\d\d\d-//' tests/sample-test
to make that one script use the single most recent year number.
* test/dd/no-allocate.sh: Use 'wait' to ensure we don't have
multiple writers to the fifo, which was seen to trigger
a very hard to reproduce deadlock with make -j20 on solaris.
Also avoid writing to the fifo with the shell; instead using dd.
(check_dd_seek_alloc): A new function refactored from the various
cases, which are now constructed from function parameters.
* src/dd.c (STATUS_NONE): Simplify the enum so that
it's more general than just suppressing transfer counts.
Then test this in all locations where non fatal diagnostics
are output.
* tests/dd/misc.sh: Ensure the diagnostic about
being unable to skip past the end of input is suppressed.
* NEWS: Mention the change in behavior.
Fixes http://bugs.gnu.org/14897
* init.cfg (require_ulimit_v_): Renamed from require_ulimit_
as this only checks for ulimit -v support. Other uses of
ulimit -t and ulimit -n in tests shouldn't cause false failures
if not supported.
* cfg.mk (sc_prohibit_test_ulimit_without_require_): A new syntax check
to ensure that require_ulimit_v_() is used iff required.
* tests/misc/head-c.sh: Add missing call to require_ulimit_v_.
* tests/rm/many-dir-entries-vs-OOM.sh: Likewise.
* tests/split/r-chunk.sh: Remove non mandatory require_ulimit_ call.
* tests/misc/sort-merge-fdlimit.sh: Likewise.
* tests/cp/link-heap.sh: Adjust to renamed require_ulimit_v_.
* tests/dd/no-allocate.sh: Likewise.
* tests/misc/csplit-heap.sh: Likewise.
* tests/misc/cut-huge-to-eol-range.sh: Likewise.
* tests/misc/printf-surprise.sh: Likewise.
* src/dd.c: Add new static global variable ibuf.
(alloc_ibuf, alloc_obuf): New functions factored from dd_copy().
(dd_copy): Call the new functions to allocate memory for
ibuf and obuf when necessary.
(skip): Likewise.
* tests/dd/no-allocate.sh: New test.
* tests/local.mk: Reference the test.
Run "make update-copyright", but then also run this,
perl -pi -e 's/2\d\d\d-//' tests/sample-test
to make that one script use the single most recent year number.
* src/dd.c (STATUS_NONE): A new bitmask combining all STATUS_
options, thus used to suppress all informational output.
(struct symbol_value statuses): Expose the "none" option,
corresponding to the STATUS_NONE bitmask above.
(print_stats): Return early if STATUS_NONE is specified.
Also move the call to gethrxtime() down so that it's only
called when needed.
(usage): Describe the new options.
* doc/coreutils.texi (dd invocation): Likewise.
* NEWS: Mention the new feature.
* tests/dd/misc.sh: Ensure the new option works.
* Makefile.am (SUBDIRS): Remove 'tests'.
(include): The '$(top_srcdir)/tests/local.mk' file.
(check-root): Remove this convenience target, it's no longer needed
now that the "real" check-root target once in 'tests/Makefile' will
land in the top-level makefile.
* configure.ac (AC_CONFIG_FILES): Remove 'tests/Makefile'.
* tests/Makefile.am: Rename ...
* tests/local.mk: ... like this, with a lot of adjustments.
* tests/init.cfg: Move ...
* init.cfg: ... here. This is necessary, for a limitation of the
gnulib-provided 'tests/init.sh', which unconditionally look for
'init.cfg' in the $(srcdir) directory.
* tests/*/*.sh: Adjust: expect init.sh to be in '$srcdir/tests',
not in '$srcdir', and extend $PATH with './src', not with '../src'.
* tests/Coreutils.pm: Adjust similarly.
* tests/pr/pr-tests.pl ($pfx): Likewise.
Not only this shrinks the size of the generated Makefile (from > 6300
lines to ~3000), but will allow further simplifications in future
changes.
* tests/Makefile.am (TEST_EXTENSIONS): Add '.sh' and '.pl'.
(PL_LOG_COMPILER, SH_LOG_COMPILER): New, still defined simply to
$(LOG_COMPILER) for the time being.
(TESTS, root_tests): Adjust as described.
* All tests: Rename as described.
Exempt init.sh because it runs before we're assured to have a
shell that groks $(...). Exempt *.mk because "$" would have to
be doubled, and besides, any `...` expression in a .mk file is
almost certainly evaluated before init.sh is run. Finally, also
exempt the perl-based tests, because perl's `...` cannot be
converted to $(...). Do that by running this command:
git grep -l '`.*`' tests \
| grep -Ev 'init\.sh|\.mk$' | xargs grep -Lw perl \
| xargs perl -pi -e 's/`(.*?)`/\$($1)/g'
One minor fix-up change was required after that, due to how
quoting differs:
diff --git a/tests/chmod/equals b/tests/chmod/equals
- expected_perms=$(eval 'echo \$expected_'$dest)
+ expected_perms=$(eval 'echo $expected_'$dest)
Another was to make these required quoting adjustments:
diff --git a/tests/misc/stty b/tests/misc/stty
...
- rev=$(eval echo "\\\$REV_$opt")
+ rev=$(eval echo "\$REV_$opt")
...
- rev1=$(eval echo "\\\$REV_$opt1")
- rev2=$(eval echo "\\\$REV_$opt2")
+ rev1=$(eval echo "\$REV_$opt1")
+ rev2=$(eval echo "\$REV_$opt2")
Also, transform two files that were needlessly excluded above:
(both use perl, but are mostly bourne shell)
perl -pi -e 's/`(.*?)`/\$($1)/g' \
tests/du/long-from-unreadable tests/init.cfg
* tests/dd/sparse: The last two parts of this test would fail due to
the underlying file system at least on Solaris 10 with NFS. That file
system would report that a 3MiB file was occupying <= 1KiB of space
for nearly 50 seconds after creation.
Improved-by: Bernhard Voelker
Prompted by the continuous integration build failure at:
http://hydra.nixos.org/build/2188210 (which uses XFS).
* tests/dd/sparse (alloc_equal): Add a block allocation
comparison function that accounts for variations due
to alignment.
Notes:
Small seeks are not coalesced to larger ones,
like is done in cache_round() for example.
conv= is used rather then oflag= for FreeBSD compatibility.
* src/dd.c (final_op_was_seek): A new global boolean to flag
whether the final "write" was converted to a seek.
(usage): Describe the new conf=sparse option.
(iwrite): Convert a write of a NUL block to a seek if requested.
(do_copy): Initialize the output buffer to have a sentinel,
to allow for efficient testing for NUL output blocks.
If the last block in the file was converted to a seek,
then convert back to a write so the size is updated.
* NEWS: Mention the new feature.
* tests/dd/sparse: A new test for the feature.
* tests/Makefile.am: Reference the new test.
These edge cases were missed in the previous commit 140eca15c.
* src/dd.c (main): Include the bytes slop when truncating
without further I/O. Don't invalidate the whole file cache
in the case where 0 < count < ibs.
* tests/dd/bytes: Change to using the independent truncate
command to generate the file for comparison. Remove a redundant
test case and replace with one testing the truncation only logic.
dd now accepts the count_bytes and skip_bytes input flag and the
seek_bytes output flag, to more easily allow processing portions of a
file.
* src/dd.c (scanargs): Compute skip_records and skip_bytes when
'skip_bytes' iflag is used. Compute max_records and max_bytes when
'count_bytes' iflag is used. Compute seek_records and seek_bytes
when 'seek_bytes' oflag is used.
(skip_via_lseek): Use new 'bytes' parameter and handle potential
'records' equals to zero. Update the bytes parameter when called with
'fdesc' equal to STDOUT_FILENO. Update the header comments.
(dd_copy): Skip accordingly to skip_records AND skip_bytes. Count
accordingly to max_records AND max_bytes. Seek on output accordingly
to seek_records AND seek_bytes.
* NEWS (New features): Mention it.
* doc/coreutils.texi (dd invocation): Detail new flags and behaviors.
* tests/dd/bytes: New file. Tests for these new flags.
* tests/Makefile.am (TESTS): Add it.
Exempt lines with '$' or '=', since those are prone to improper
conversion. Run this:
git grep -l "\`[^']*'" tests \
|xargs perl -pi -e '/[=\$]/ and next;s/\`([^'\''"]*?'\'')/'\''$1/g'
These changes were mostly mechanical, made by running the
following command:
git grep -lw framework_failure | grep -v ChangeLog \
| xargs perl -pi -e 's/\b(framework_failure)\b/${1}_/'
and then editing init.cfg and `tests/cp/cp-a-selinux' by hand.
* tests/init.cfg (framework_failure): Remove, `framework_failure_'
from init.sh should be used instead in the tests.
Remove now-obsolete "FIXME" comment.
(is_local_dir_, require_strace_, require_membership_in_two_groups_,
require_sparse_support_, skip_if_mcstransd_is_running_,
mkfifo_or_skip_) Use `framework_failure_', not `framework_failure'.
* Many test scripts: Likewise.
* tests/init.cfg (skip_test_): Remove function.
Use skip_ in place of skip_test_ everywhere else.
* cfg.mk (sc_prohibit_skip_): Remove rule.
* tests/**: Use skip_, not skip_test_, everywhere.
* tests/dd/nocache: Relax the test, as the system
may return various errors from posix_fadvise().
HPUX 11.31 returns ENOTTY for example.
Reported by Bruno Haible
