* tests/misc/tty-eof.pl: Ensure we don't erroneously
skip commands with parameters.
Comment as to why cut(1) is treated differently.
Adjust expect calls to not wait needlessly for cut output.
* tests/cp/sparse-2.sh: Don't depend on the copy taking
<= allocation of the source. Instead leverage --debug
to check that zero detection is being enabled.
Fix a build failure seen on gcc 3.4 on Solaris 10 at least.
* src/crctab.c: Ensure we include config.h for all compilation units.
This is now required for new _Noreturn usage in gnulib for stdint.h.
* src/cksum.c: Update generation code to ensure config.h included.
* cfg.mk: Remove crctab.c exclusion from the config.h check.
* src/wc.c (wc): Update the offset when not reading,
and do read if we can't update the offset.
* tests/misc/wc-proc.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/61300
* NEWS: Mention the change in behavior to issue a "not replaced"
error diagnostic with -n, and the "skipped" message with -v.
* src/copy.c (copy_internal): Adjust to output the "skipped" messages
depending on -i, -n, -u.
* tests/cp/cp-i.sh: Adjust accordingly.
* tests/mv/mv-n.sh: Likewise.
Add --update=none which is equivalent to the --no-clobber behavior
from before coreutils 9.2. I.e. existing files are unconditionally
skipped, and them not being replaced does not affect the exit status.
* src/copy.h [enum Update_type]: A new type to support parameters
to the --update command line option.
[enum Interactive]: Add I_ALWAYS_SKIP.
* src/copy.c: Treat I_ALWAYS_SKIP like I_ALWAYS_NO (-n),
except that we don't fail when skipping.
* src/system.h (emit_update_parameters_note): A new function
to output the description of the new --update parameters.
* src/cp.c (main): Parse --update arguments, ensuring that
-n takes precedence if specified.
(usage): Describe the new option. Also allude that
-u is related in the -n description.
* src/mv.c: Accept the new --update parameters and
update usage() accordingly.
* doc/coreutils.texi (cp invocation): Describe the new --update
parameters. Also reference --update from the --no-clobber description.
(mv invocation): Likewise.
* tests/mv/update.sh: Test the new parameters.
* NEWS: Mention the new feature.
Addresses https://bugs.gnu.org/62572
* gnulib: Reference the latest gnulib including the
fix to the backupfile module in commit 94496522.
* tests/cp/backup-dir.sh: Add a test to ensure
we rename appropriately when backing up through subdirs.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/62607
* tests/misc/tee.sh: Call cleanup_ in all cases to ensure
there are no overlapping interactions on the fifo that
might impact later parts of the test. This was seen to
cause issue with dash on musl libc.
Addresses https://bugs.gnu.org/62542
* tests/misc/csplit-heap.sh: More memory is required to avoid
a false failure on some systems. Noticed with musl libc
with bash as the shell. This is confirmed to still easily
trigger with the original memory leak being tested.
Addresses https://bugs.gnu.org/62542
* src/wc.c (wc): Use INT_ADD_WRAPV() to detect overflow.
(main): Upon overflow, saturate the total, print a diagnostic,
and set exit status.
* tests/misc/wc-total.sh: Add a test case, which operates
on BTRFS and 64 bit systems at least.
Reported at https://bugs.debian.org/1027100
* NEWS: Mention the fix.
* src/dircolors.c: Fail upon read error from getline().
* tests/misc/dircolors.sh: Add a new test.
* tests/local.mk: Reference the new test.
On restricted systems like android or some containers,
FICLONE could return EPERM, EACCES, or ENOTTY,
which would have induced the command to fail to copy
rather than falling back to a more standard copy.
* src/copy.c (is_terminal_failure): A new function refactored
from handle_clone_fail().
(is_CLONENOTSUP): Merge in the handling of EACCES, ENOTTY, EPERM
as they also pertain to determination of whether cloning is supported
if we ever use this function in that context.
(handle_clone_fail): Use is_terminal_failure() in all cases,
so that we assume a terminal failure in less errno cases.
* NEWS: Mention the bug fix.
Addresses https://bugs.gnu.org/62404
This applies to all checksumming utilities,
where we incorrectly report all subsequent files as checking 'OK'
once any file has passed a digest check.
The exit status was not impacted, only the printed status.
* src/digest.c (digest_check): Use the correct state variable
to determine if the _current_ file has passed or not.
* tests/misc/md5sum.pl: Add a test case.
Fixes https://bugs.gnu.org/62403
Similarly to the fix to tests/rmdir/ignore.sh in c0e5f8c59,
tee should not be expected to fail when run with read-only outputs
when run as root.
* tests/misc/tee.sh: Add uid_is_privileged_ guard around test for
read-only outputs.
* tests/ls/stat-free-symlinks.sh: Filter out syscalls that
return ENOSYS, as that was seen with statx() on Debian 10.13
on mips64, and resulted in overcounting of stat calls.
* src/stty.c (main): Use static structures to ensure
they're initialized (to zero), so that random data is
not displayed, or compared resulting in a inaccurate
failure reported to users. This was seen on musl libc
where some parts of the termios c_cc array were
not initialized by tcgetattr().
Reported by Bruno Haible.
* tests/misc/tee.sh: AIX doesn't support detecting
closed outputs either with poll() or select() so avoid
testing that functionality.
* tests/tail-2/pipe-f.sh: Likewise.
Since SELinux version 3.5, the return value of context_str(3) is
declared as const; see:
https://github.com/SELinuxProject/selinux/commit/dd98fa322766
Therefore, GCC complains (here with -Werror):
src/selinux.c: In function 'defaultcon':
src/selinux.c:152:16: error: assignment discards 'const' qualifier \
from pointer target type [-Werror=discarded-qualifiers]
152 | if (!(constr = context_str (tcontext)))
| ^
src/selinux.c: In function 'restorecon_private':
src/selinux.c:252:16: error: assignment discards 'const' qualifier \
from pointer target type [-Werror=discarded-qualifiers]
252 | if (!(constr = context_str (tcontext)))
| ^
* src/selinux.c (defaultcon): Define CONSTR as const.
(restorecon_private): Likewise.
* src/sum.c (output_bsd): On sparc64 for example,
a crc of 0 was output due to casting an int variable
to uint16_t and thus operating on the wrong end of the variable.
Instead use explicit assignment to the narrower type
to ensure we get the appropriate data.
(output_sysv): Likewise.
Reported by Bruno Haible.
* iopoll.c (fclose_wait): Rename from confusing fclose_nonblock name.
Also adjust to do no operations on the stream after fclose()
as this is undefined. Instead use fflush() to determine EAGAIN status.
(fwrite_wait): Renamed from confusing fwrite_nonblock name.
* src/dircolors.hin: Make the separate sections of the self
documenting dircolors database more apparent,
by adding heading comments, and appropriate separation.
Following on from commit v8.29-45-g24053fbd8 which unconditionally
used case insensitive extension matching, support selective
case sensitive matching when there are separate extension cases
defined with different display sequences.
* src/dircolors.hin: Document how file name suffixes are matched.
Note this is displayed with `dircolors --print-database` which
the texi info recommends to use for details.
* src/ls.c (parse_ls_color): Postprocess the list to
mark entries for case sensitive matching,
and also adjust so that unmatchable entries are more quickly ignored.
(get_color_indicator): Use exact matching rather than
case insensitive matching if so marked.
* tests/ls/color-ext.sh: Add test cases.
* NEWS: Mention the change in behavior.
Addresses https://bugs.gnu.org/33123
* tests/du/threshold.sh: Directories are assumed to be
of size 0 with --apparent since commit v9.1-187-g110bcd283
so remove --apparent cases from this test.
Non blocking outputs can be seen for example
when piping telnet through tee to a terminal.
In that case telnet sets its input to nonblocking mode,
which results in tee's output being nonblocking,
in which case in may receive an EAGAIN error upon write().
The same issue was seen with mpirun.
The following can be used to reproduce this
locally at a terminal (in most invocations):
$ { dd iflag=nonblock count=0 status=none;
dd bs=10K count=10 if=/dev/zero status=none; } |
tee || echo fail >/dev/tty
* src/iopoll.c (iopoll_internal): A new function refactored from
iopoll(), to also support a mode where we check the output
descriptor is writeable.
(iopoll): Now refactored to just call iopoll_internal().
(fwait_for_nonblocking_write): A new internal function which
uses iopoll_internal() to wait for writeable output
if an EAGAIN or EWOULDBLOCK was received.
(fwrite_nonblock): An fwrite() wrapper which uses
fwait_for_nonblocking_write() to handle EAGAIN.
(fclose_nonblock): Likewise.
src/iopoll.h: Add fclose_nonblock, fwrite_nonblock.
src/tee.c: Call fclose_nonblock() and fwrite_nonblock wrappers,
instead of the standard functions.
* tests/misc/tee.sh: Add a test case.
* NEWS: Mention the improvement.
The idea was suggested by Kamil Dudka in
https://bugzilla.redhat.com/1615467
* bootstrap.conf (gnulib_modules): Add free-posix, tmpfile.
* src/split.c (copy_to_tmpfile): New function.
(input_file_size): Use it to split larger files when sizes cannot
easily be determined via fstat or lseek. See Bug#61386#235.
* tests/split/l-chunk.sh: Mark tests of /dev/zero as
very expensive since they exhaust /tmp.
This was introduced recently with commit v9.1-166-g6b12e62d9
* src/tee.c (tee_files): Check the return from fopen()
before passing to fileno() etc.
* tests/misc/tee.sh: Add a test case.
Problem reported by Pádraig Brady (Bug#61386#226).
* src/split.c (parse_chunk): Use die instead of error.
(main): Quote a string.
* tests/local.mk (all_root_tests): Move du/apparent.sh from here ...
(all_tests): ... to here.
Problem reported by Christoph Anton Mitterer (Bug#61884).
* src/du.c (process_file): When counting apparent sizes, count
only usable st_size members.
* tests/du/apparent.sh: New file.
* tests/local.mk (all_root_tests): Add it.
* src/split.c (create): Avoid fstat + ftruncate in the usual case
where the output file does not already exist, by trying
to create it with O_EXCL first. This costs a failed open
in the unusual case where the output file already exists,
but that’s OK.
Prefer signed types to uintmax_t, as this allows for better
runtime checking with gcc -fsanitize=undefined.
Also, when an integer overflows just use the maximal value
when the code will do the right thing anyway.
* src/split.c (set_suffix_length, bytes_split, lines_split)
(line_bytes_split, lines_chunk_split, bytes_chunk_extract)
(lines_rr, parse_chunk, main):
Prefer a signed type (typically intmax_t) to uintmax_t.
(strtoint_die): New function.
(OVERFLOW_OK): New macro. Use it elsewhere, where we now allow
LONGINT_OVERFLOW because the code then does the right thing on all
practical platforms (they have int wide enough so that it cannot
be practically exhausted). We can do this now that we can safely
assume intmax_t has at least 64 bits.
(parse_n_units): New function.
(parse_chunk, main): Use it.
(main): Do not worry about integer overflow when the code
will do the right thing anyway with the extreme value.
Just use the extreme value.
* tests/split/fail.sh: Adjust to match new behavior.
* src/split.c (bytes_split, lines_chunk_split)
(bytes_chunk_extract, main): Prefer ssize_t to size_t when
representing the return value of ‘read’. Use a negative value
instead of SIZE_MAX to indicate a missing value.
* src/split.c: Include sys-limits.h, not safe-read.h.
(input_file_size, bytes_split, lines_split, line_bytes_split)
(lines_chunk_split, bytes_chunk_extract, lines_rr): Call read, not
safe_read, since safe_read no longer buys us anything.
(main): Reject outlandish buffer sizes right away,
rather than allocating huge buffers and never using them.
* src/split.c (closeout): There should be no need for a special
case for ECHILD, since we never wait for the same child twice.
Simplify with this in mind.
Ignore and default SIGPIPE, rather than blocking and unblocking it.
* src/split.c (default_SIGPIPE):
New static var, replacing oldblocked and newblocked.
(create): Use it.
(main): Set it.
* src/split.c (input_file_size): Do not bother with lseek if the
initial read probe reaches EOF, since the file size is known then.
This works better on macOS, which doesn’t allow lseek on /dev/null.
Do not special-case size-zero files, as the issue can occur
with any size file (though /proc files are the most common).
If the current position is past end of file, treat this as
size zero regardless of whether the file has a usable st_size.
Pass through lseek -1 return values rather than using ‘return -1’;
this makes the code a bit easier to analyze (and a bit faster).
Avoid undefined behavior if the size calculation overflows.
(lines_chunk_split): Do not bother with lseek if it would have
no effect if successful. This works better on macOS, which
doesn’t allow lseek on /dev/null.
* tests/split/l-chunk.sh: Adjust to match fixed behavior.
* src/split.c (bytes_split): New arg REM_BYTES.
Use this to split more evenly. All callers changed.
(lines_chunk_split, bytes_chunk_extract):
Be consistent with new byte_split.
* tests/split/b-chunk.sh, tests/split/l-chunk.sh: Test new behavior.
* src/split.c (lines_chunk_split): Simplify by having chunk_end
point to the first byte after the chunk, rather than to the last
byte of the chunk. This will reduce confusion once we allow
chunks to be empty.
* src/tee.c (pipe_check): Make this a local var instead
of a static var. This suppresses a -Wmaybe-uninitialized
diagnostic with gcc 12.2.1 20221121 (Red Hat 12.2.1-4).
(main): Don’t set pipe_check unnecessarily if a later
-p option overrides an earlier one that wants pipe_check.
Problem discovered when I investigated the GCC warning.
* src/tail.c (check_output_alive): Reuse iopoll()
rather than directly calling poll() or select().
* src/iopoll.c (iopoll): Refactor to support non blocking operation,
or ignoring descriptors by passing a negative value.
* src/iopoll.h (iopoll): Adjust to support a BLOCK parameter.
* src/tee.c (tee_files): Adjust iopoll() call to explicitly block.
* src/local.mk: Have tail depend on iopoll.c.
* src/tee.c (usage): Change from describing one (non pipe) aspect
to the more general point of being the option to use if working with
pipes, and referencing the more detailed info below.
* doc/coreutils.texi (tee invocation): s/standard/appropriate/ since
the standard operation with pipes is to exit immediately upon write
error. s/early/immediately/ as it's ambiguous as to what "early"
is in relation to.
If input is intermittent (a tty, pipe, or socket), and all remaining
outputs are pipes (eg, >(cmd) process substitutions), exit early when
they have all become broken pipes (and thus future writes will fail),
without waiting for more input to become available, as future write
attempts to these outputs will fail (SIGPIPE/EPIPE).
Only provide this enhancement when pipe errors are ignored (-p mode).
Note that only one output needs to be monitored at a time with iopoll(),
as we only want to exit early if _all_ outputs have been removed.
* src/tee.c (pipe_check): New global for iopoll mode.
(main): enable pipe_check for -p, as long as output_error ignores EPIPE,
and input is suitable for iopoll().
(get_next_out): Helper function for finding next valid output.
(fail_output, tee_files): Break out write failure/output removal logic
to helper function.
(tee_files): Add out_pollable array to track which outputs are suitable
for iopoll() (ie, that are pipes); track first output index that is
still valid; add iopoll() broken pipe detection before calling read(),
removing an output that becomes a broken pipe.
* src/local.mk (src_tee_SOURCES): include src/iopoll.c.
* NEWS: Mention tee -p enhancement in Improvements.
* doc/coreutils.texi: Mention the new early exit behavior in the nopipe
modes for the tee -p option.
Suggested-by: Arsen Arsenović <arsen@aarsen.me>
When a program's output becomes a broken pipe, future attempts to write
to that ouput will fail (SIGPIPE/EPIPE). Once it is known that all
future write attepts will fail (due to broken pipes), in many cases it
becomes pointless to wait for further input for slow devices like ttys.
Ideally, a program could use this information to exit early once it is
known that future writes will fail.
Introduce iopoll() to wait on a pair of fds (input & output) for input
to become ready or output to become a broken pipe.
This is relevant when input is intermittent (a tty, pipe, or socket);
but if input is always ready (a regular file or block device), then
a read() will not block, and write failures for a broken pipe will
happen normally.
Introduce iopoll_input_ok() to check whether an input fd is relevant
for iopoll().
Experimentally, broken pipes are only detectable immediately for pipes,
but not sockets. Errors for other file types will be detected in the
usual way, on write failure.
Introduce iopoll_output_ok() to check whether an output fd is suitable
for iopoll() -- namely, whether it is a pipe.
iopoll() is best implemented with a native poll(2) where possible, but
fall back to a select(2)-based implementation platforms where there are
portability issues. See also discussion in tail.c.
In general, adding a call to iopoll() before a read() in filter programs
also allows broken pipes to "propagate" backwards in a shell pipeline.
* src/iopoll.c, src/iopoll.h (iopoll): New function implementing broken
pipe detection on output while waiting for input.
(IOPOLL_BROKEN_OUTPUT, IOPOLL_ERROR): Return codes for iopoll().
(IOPOLL_USES_POLL): Macro for poll() vs select() implementation.
(iopoll_input_ok): New function to check whether an input fd is relevant
for iopoll().
(iopoll_output_ok): New function to check whether an input fd is
suitable for iopoll().
* src/local.mk (noinst_HEADERS): add src/iopoll.h.
* NEWS: Mention the fts fix to avoid the following assert
in rm on mem pressure:
Program terminated with signal SIGSEGV, Segmentation fault.
at ../lib/cycle-check.c:60
assure (state->magic == CC_MAGIC);
* gnulib: Update to the latest to pick up fts commit f17d3977.
* tests/rm/empty-inacc.sh: Ensure we're not reading from stdin
when we're relying on no prompt to proceed. Also change the
file being tested so that a failure in one test doesn't impact
following tests causing a framework failure.
gdb was seen to hang intermittently on macOS 12.
Also gdb requires signing on newer macOS systems:
https://sourceware.org/gdb/wiki/PermissionsDarwin
So restrict its use on macOS systems for now.
* tests/rm/r-root.sh: Skip on darwin systems.
* tests/tail-2/inotify-race.sh: Restrict the test to
inotify capable systems to avoid the hang with some gdbs.
* tests/tail-2/inotify-race.sh: Likewise.
Upcomming gnulib changes may disable SEEK_HOLE
even if the system supports it, so dynamically
check if we've SEEK_HOLE enabled.
* init.cfg (seek_data_capable_): SEEK_DATA may be disabled in the build
if the system support is deemed insufficient, so also use `cp --debug`
to determine if it's enabled.
* tests/cp/sparse-2.sh: Adjust to a more general diagnostic.
* tests/cp/sparse-extents-2.sh: Likewise.
* tests/cp/sparse-extents.sh: Likewise.
* tests/cp/sparse-perf.sh: Likewise.
How a file is copied is dependent on the sparseness of the file,
what file system it is on, what file system the destination is on,
the attributes of the file, and whether they're being copied or not.
Also the --reflink and --sparse options directly impact the operation.
Given it's hard to reason about the combination of all of the above,
the --debug option is useful for users to directly identify if
copy offloading, reflinking, or sparse detection are being used.
It will also be useful for tests to directly query if
these operations are supported.
The new output looks as follows:
$ src/cp --debug src/cp file.sparse
'src/cp' -> 'file.sparse'
copy offload: yes, reflink: unsupported, sparse detection: no
$ truncate -s+1M file.sparse
$ src/cp --debug file.sparse file.sparse.cp
'file.sparse' -> 'file.sparse.cp'
copy offload: yes, reflink: unsupported, sparse detection: SEEK_HOLE
$ src/cp --reflink=never --debug file.sparse file.sparse.cp
'file.sparse' -> 'file.sparse.cp'
copy offload: avoided, reflink: no, sparse detection: SEEK_HOLE
* doc/coreutils.texi (cp invocation): Describe the --debug option.
(mv invocation): Likewise.
(install invocation): Likewise.
* src/copy.h: Add a new DEBUG member to cp_options, to control
whether to output debug info or not.
* src/copy.c (copy_debug): A new global structure to
unconditionally store debug into from the last copy_reg operations.
(copy_debug_string, emit_debug): New functions to print debug info.
* src/cp.c: if ("--debug") x->debug=true;
* src/install.c: Likewise.
* src/mv.c: Likewise.
* tests/cp/debug.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the new feature.
* src/remove.c (prompt, rm_fts): In the dir-handling code of both of
these functions, relax a "get_dir_status (...) == DS_EMPTY" condition
to instead test only "get_dir_status (...) != 0", enabling flow control
to reach the prompt function also for unreadable directories. However,
that function itself also needed special handling for this case:
(prompt): Handle empty, inaccessible directories properly,
deleting them with -d (--dir), and prompting about whether to delete
with -i (--interactive).
* tests/rm/empty-inacc.sh: Add tests for the new code.
Reported by наб <nabijaczleweli@nabijaczleweli.xyz> in
bugs.debian.org/1015273
* NEWS (Bug fixes): Mention this.
* tests/chmod/setgid.sh: Try all the groups you’re a member of,
in case id -g returns 4294967295 (nogroup) which is special
and does not let you chgrp a file to it.
* init.cfg (groups): Port better to macOS 12, where
group 4294967295 (nogroup) is special: you can be a member
without being able to chgrp files to the group.
* src/copy.c: Some changes if HAVE_FCLONEFILEAT && !USE_XATTR.
(fd_has_acl): New function.
(CLONE_ACL): Default to 0.
(copy_reg): Use CLONE_NOFOLLOW to avoid races like CVE-2021-30995
<https://www.trendmicro.com/en_us/research/22/a/
analyzing-an-old-bug-and-discovering-cve-2021-30995-.html>.
Use CLONE_ACL if available and working, falling back to cloning
without it if it fails due to EINVAL.
If the only problem with fclonefileat is that it would create the
file with the wrong timestamp, or with too few permissions,
do that but fix the timestamp and permissions afterwards,
rather than falling back on a traditional copy.
* src/copy.c (infer_scantype): Do not set *SCAN_INFERENCE
when returning a value other than LSEEK_SCANTYPE.
This is just minor refactoring; it simplifies the code a bit.
Callers are uneffected.
doc: document --preserve=mode better
* src/tail (tail_forever): Attempt to read() from non blocking
single non regular file, which shouldn't block, but also
read data even when the mtime doesn't change.
* NEWS: Mention the improvement.
* THANKS.in: Thanks for detailed testing.
This was seen to be an issue when following a
symlink that was being updated to point to
different underlying devices.
* src/tail.c (recheck): Guard the lseek() call to only
be performed for regular files.
* NEWS: Mention the bug fix.
--raw output is the most composable format, and also is a
robust way to discard the file name without parsing (escaped) output.
Examples:
$ cksum --raw -a crc "$afile" | basenc --base16
4ACFC4F0
$ cksum --raw -a crc "$afile" | basenc --base2msbf
01001010110011111100010011110000
$ cksum --raw -a sha256 "$bfile" | basenc --base32
AAAAAAAADHLGRHAILLQWLAY6SNH7OY5OI2RKNQLSWPY3MCUM4JXQ====
* doc/coreutils.texi (cksum invocation): Describe the new feature.
* src/digest.c (output_file): Inspect the new RAW_DIGEST global,
and output the bytes directly if set.
* src/cksum.c (output_crc): Likewise.
* src/sum.c (output_bsd, output_sysv): Likewise.
* tests/misc/cksum-raw.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the new feature.
* src/uptime.c (print_uptime): Following gnulib commit 9041103
HAVE_UTMP_H will always be defined. Therefore key on whether
the utmp.ut_type member is present.
* boottime.m4 (GNULIB_BOOT_TIME): Assume utmp.h is present.
* src/digest.c [HASH_ALGO_CKSUM]: Include "base64.h"
[HASH_ALGO_CKSUM] (base64_digest): New global.
[HASH_ALGO_CKSUM] (enum BASE64_DIGEST_OPTION): New enum.
[HASH_ALGO_CKSUM] (long_options): Add "base64".
(valid_digits): Rename from hex_digits, now taking an input length argument.
Adjust callers.
(bsd_split_3): Rename arg from hex_digits to digest.
Add new *d_len parameter for length of extracted digest.
Move "i" declaration down to first use.
(split_3): Rename arg from hex_digits to digest.
Add new *d_len parameter for length of extracted digest.
Instead of relying on "known" length of digest to find the following
must-be-whitespace byte, search for the first whitespace byte.
[HASH_ALGO_CKSUM] (output_file): Handle base64_digest.
[HASH_ALGO_CKSUM] (main): Set base64_digest.
[HASH_ALGO_CKSUM] (b64_equal): New function.
(hex_equal): New function, factored out of digest_check.
(digest_check) Factored part into b64_equal and hex_equal.
Rename local hex_digest to digest.
* tests/misc/cksum-base64.pl: Add tests.
* tests/local.mk (all_tests): Add to the list.
* cfg.mk (_cksum): Define.
(exclude_file_name_regexp--sc_prohibit_test_backticks): Exempt new test.
(exclude_file_name_regexp--sc_long_lines): Likewise.
* doc/coreutils.texi (cksum invocation): Document it.
(md5sum invocation) [--check]: Mention digest encoding auto-detect.
* NEWS (New Features): Mention this.
This reverts the previous change, so that when a file
is skipped due to -u, this is not considered a failure.
* doc/coreutils.texi: Document this.
* src/copy.c (copy_internal): If --update says to skip,
treat this as success instead of failure.
* tests/mv/update.sh, tests/cp/slink-2-slink.sh:
Revert previous change, to match reverted behavior.
* NEWS, doc/coreutils.texi: Document this.
* src/copy.c (copy_internal):
* src/ln.c (do_link): Return false when skipping action due to
--interactive or --no-clobber.
* tests/cp/cp-i.sh, tests/cp/preserve-link.sh:
* tests/cp/slink-2-slink.sh, tests/mv/i-1.pl, tests/mv/i-5.sh:
* tests/mv/mv-n.sh, tests/mv/update.sh:
Adjust expectations of exit status to match revised behavior.
* src/digest.c (digest_check): Locals n_misformatted_lines and
n_improperly_formatted_lines were declared and set/incremented
identically. Remove declaration of the latter. Use the other instead.
Wishlist item from Mike Frysinger (Bug#61050).
* src/copy.c (copy_internal):
Do not fall back on copying if x->no_copy.
* src/copy.h (struct cp_options): New member no_copy.
* src/mv.c (NO_COPY_OPTION): New constant.
(long_options, usage, main): Support --no-copy.
* tests/mv/no-copy.sh: New test.
* tests/local.mk (all_tests): Add it.
* src/csplit.c (usage): Use "suppress" rather than "remove"
when describing -z so it's more apparent that the effect
is a particular numbered file is not created, rather than
being removed later. I.e., don't suggest -z may induce
gaps in file numbering.
Reported at https://bugs.debian.org/1029103
* src/copy.c (sparse_copy): Fallback to standard copy upon ENOENT,
which was seen intermittently across CIFS file systems.
* NEWS: Mention the bug fix, though qualify it as an "issue"
rather than a bug, as coreutils is likely only highlighting
a CIFS bug in this case.
Fixes https://bugs.gnu.org/60455
* src/copy.c (handle_clone_fail): A new function refactored
from copy_reg() to handle failures from FICLONE or fclonefileat().
Fail with all errors from FICLONE, unless they're from the set
indicating the file system or file do not support the clone operation.
Also fail with errors from fclonefileat() (dest_dest < 0)
if they're from the set indicating a transient failure for the file.
(copy_ref): Call handle_clone_fail() after fclonefileat() and FICLONE.
(sparse_copy): Call the refactored is_CLONENOTSUP()
which is now also used by the new handle_clone_fail() function.
* NEWS: Mention the bug fix. Also mention explicitly
the older --reflink=auto default change to aid searching.
* cfg.mk: Adjust old_NEWS_hash with `make update-NEWS-hash`.
Fixes https://bugs.gnu.org/60489
* src/dd.c (parse_integer): Support Q,R suffixes.
* src/od.c (main): Likewise.
* src/split.c (main): Likewise.
* src/stdbuf.c (parse_size): Likewise.
* src/truncate.c (main): Likewise.
* src/sort.c (specify_size_size): Likewise.
Also line length syntax check fix.
* tests/misc/numfmt.pl: Adust top end large number checks
to the new largest values.
* doc/coreutils.texi (numfmt invocation): Add a numfmt example.
* NEWS: Tweak to aid searchability.
* src/dd, src/head.c, src/od.c, src/sort.c, src/stdbuf.c, src/tail.c:
(usage):
* src/system.h (emit_size_note):
Mention new SI prefixes.
* src/du.c (main):
* src/head.c (head_file):
* src/numfmt.c (suffix_power, suffix_power_char, prepare_padded_number):
* src/shred.c (main):
* src/sort.c (unit_order):
* src/tail.c (parse_options):
Support new SI prefixes.
* src/numfmt.c (MAX_ACCEPTABLE_DIGITS): Increase to 33.
(zero_and_valid_suffixes, valid_suffixes): New constants,
with new SI prefixes.
(valid_suffix, unit_to_umax): Use them.
(prepare_padded_number): Diagnose "999Q" instead of "999Y".
* tests/misc/numfmt.pl, tests/misc/sort.pl:
Adjust tests to match new max.
Newer grep(1) complains:
$ make sc_prohibit_test_minus_ao
/usr/bin/grep: warning: * at start of expression
prohibit_test_minus_ao
* cfg.mk (exclude_file_name_regexp--sc_prohibit_test_minus_ao): Fix
expression inroduced in v8.24-120-g3205bb178, and narrow down the file
pattern to the 'doc/' directory.
* bootstrap.conf (gnulib_modules): Add count-leading-zeros,
which was already an indirect dependency, since ioblksize.h
now uses it directly.
* src/ioblksize.h: Include count-leading-zeros.h.
(io_blksize): Treat impossible blocksizes as IO_BUFSIZE.
When growing a blocksize to IO_BUFSIZE, keep it a multiple of the
stated blocksize. Work around the ZFS performance bug.
* NEWS: Mention the bug fix.
Problem reported by Korn Andras at https://bugs.gnu.org/59382
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Manually update copyright year,
until we fully sync with gnulib at a later stage.
* tests/sample-test: Adjust to use the single most recent year.
* src/stty.c (wrapf): Adjust the comparison by 1,
to account for the space we're adding.
* tests/misc/stty.sh: Add a test case.
* NEWS: Mention the fix.
Reported in https://bugs.debian.org/1027442
This was seen to vastly improve performance
on NFS 4.2 systems by allowing server side copies,
with partially sparse files (avidemux generated mp4 files).
* src/copy.c (lseek_copy): Also set hole_size to 0,
i.e. enable copy_file_range(), with --sparse=auto (the default),
to enable copy offload in this case, as we've strong signal
from SEEK_DATA that we're operating on actual data and not holes here.
* NEWS: Mention the improvement.
Fixes https://bugs.gnu.org/60416
* src/wc.c (wc): Use off_t rather than size_t
when calculating where to seek to, so that
we don't seek to a too low offset on systems
where size_t < off_t, which would result in
many read() calls to determine the file size.
* tests/misc/wc-proc.sh: Add a test case
sufficient for 32 bit systems at least.
* NEWS: Mention the bug fix.
Reported at https://bugs.debian.org/1027101
* tests/cp/proc-short-read.sh: Kernel on ARMv7 Processor rev 3 (v7l)
spells it "BogoMIPS", so allow any capitalization. Patch from
Zach van Rijn in <https://bugs.gnu.org/60339>.
* doc/coreutils.texi, doc/sort-version.texi: Prefer on "x -- y" to
"x---y" in prose, as the result is more readable in Emacs.
Fix some instances of unescaped ‘-’ that should be minus, not
hyphen. Fix some other instances that should be en dash. No
spaces around en dash when it’s a range.
* cfg.mk (sc_texi_long_option_escaped): A new check to
avoid future instances of this.
* doc/coreutils.texi (Common options): Rearrange this menu
to be less repetitive in each description, and avoid long lines.
Addresses https://bugs.gnu.org/59262
Problem reported by Antonio Diaz Diaz (bug#59262).
* doc/coreutils.texi: Use markup in menus to prevent
‘--’ from turning into an em dash, and to be more
consistent.
Note using iconv(1) rather than recode(1) is not appropriate
for this example, as the required functionality is only
available on libiconv's iconv implementation, which is
not installed on most systems.
* doc/coreutils.texi (printf invocation): Use env rather than
/usr/local/bin for the printf command. Escape '%' so more robust.
Also use a locale that exists on modern systems.
Previously this was restricted to the C99 universal character subset,
which restricted most values <= 0x9F, as that simplifies the C lexer.
However printf(1) doesn't need this restriction.
Note also the bash builtin printf already supports all values <= 0x9F.
* src/printf.c (main): Relax the restriction on points <= 0x9F.
* doc/coreutils.texi (printf invocation): Adjust description.
* tests/misc/printf-cov.pl: Adjust accordingly. Add new cases.
* NEWS: Mention the change in behavior.
Reported at https://bugs.debian.org/1022857
* doc/coreutils.texi (sort invocation): Mention in the
multi invocation sort example that the -V GNU extension
could be used to sort IPv4 addresses, and thus simplify
to a single invocation.
* src/system.h (emit_exec_status): A new function to
output standard "Exit status:" info for commands that exec others.
* doc/coreutils.texi (Exit status): Add "ls" and "runcon"
to the list of commands with non standard exit status.
* src/numfmt.c (main): Call initialize_exit_failure() explicitly
to better indicate this utility may exit with something other than
EXIT_FAILURE.
* src/timeout.c (usage): Use more consistent capitalization.
* src/chroot.c: Call emit_exec_status().
* src/env.c: Likewise.
* src/nice.c: Likewise.
* src/nohup.c: Likewise.
* src/runcon.c: Likewise.
* src/stdbuf.c: Likewise.
* src/runcon.c (main): Call initialize_exit_failure(),
so we use an appropriate exit status upon failure to close stdout.
This should have been part of recent commit ea3ee6df.
* tests/misc/help-version.sh: Adjust test case accordingly.
* src/getlimits.c: Don't call initialize_exit_failure()
as it's not needed for standard EXIT_FAILURE returns.
Also use the function variant that diagnoses invalid options.
without this option, control of when the total is output
is quite awkward. Consider trying to suppress the total line,
which could be achieved with something like:
wc-no-total() { wc "$@" /dev/null | head -n-2; }
As well as being non obvious, it's also non general.
It would give a non failure, but zero count if passed a file on stdin.
Also it doesn't work in conjunction with the --files0-from option,
which would need to be handled differently with something like:
{ find files -print0; printf '%s\0' /dev/null; } |
wc --files0-from=- |
head -n2
Also getting just the total can be awkward as file names
are only suppressed when processing stdin, and
also a total line is only printed if processing more than one file.
For completness this might be achieved currently with:
wc-only-total() {
wc "$@" |
tail -n1 |
sed 's/^ *//; s/ [^ 0-9]*$//'
}
* src/wc.c: Add new --total option.
* tests/misc/wc-total.sh: New test suite for the new option.
* tests/local.mk: Reference the new test.
* doc/coreutils.texi (wc invocation): Document the new option.
* THANKS.in: Add suggestor.
* NEWS: Mention the new feature.
* .gitignore: Add new headers from gnulib.
* src/basenc.c: Adjust line length due to replacement
of 'verify' with 'static_assert'.
* src/od.c: Likewise.
I ran into this problem when attempting to recursively
remove a directory in a filesystem on flaky hardware.
Although the underlying readdir syscall failed with errno == EIO,
rm issued no diagnostic about the I/O error.
Without this patch I see this behavior:
$ rm -fr baddir
rm: cannot remove 'baddir': Directory not empty
$ rm -ir baddir
rm: descend into directory 'baddir'? y
rm: remove directory 'baddir'? y
rm: cannot remove 'baddir': Directory not empty
With this patch I see the following behavior, which
lets the user know about the I/O error when rm tries
to read baddir's directory entries:
$ rm -fr baddir
rm: cannot remove 'baddir': Input/output error
$ rm -ir baddir
rm: cannot remove 'baddir': Input/output error
* src/remove.c (Ternary): Remove. All uses removed.
(get_dir_status): New static function.
(prompt): Last arg is now directory status, not ternary.
Return RM_USER_ACCEPTED if user explicitly accepted.
All uses changed.
Report any significant error in directory status right away.
(prompt, rm_fts): Use get_dir_status to get directory status lazily.
(excise): Treat any FTS_DNR errno as being more descriptive, not
just EPERM and EACCESS. For example, EIO is more descriptive.
(rm_fts): Distinguish more clearly between explicit and implied
user OK.
* src/remove.h (RM_USER_ACCEPTED): New constant.
(VALID_STATUS): Treat it as valid.
* src/system.h (is_empty_dir): Remove, replacing with ...
(directory_status): ... this more-general function.
All uses changed. Avoid undefined behavior of looking at
a non-null readdir pointer after corresponding closedir.
* tests/rm/rm-readdir-fail.sh: Adjust test of internals
to match current behavior.
* bootstrap.conf (gnulib_modules): Remove alignof, which isn’t
needed since coreutils source modules don’t include alignof.h.
Add stdalign, since they depend on alignof working without
stdalign.h.
* gl/lib/fadvise.h, gl/lib/smack.h, src/blake2/blake2-impl.h:
Do not include config.h from a .h file. config.h is supposed
to be included once, at the start of compilation and before
any other file.
* src/stty.c (check_speed): If difference input and output speeds
are specified, then validate the system supports that, before
interacting with the device.
* src/stty.c (eq_mode): A new function to compare
equivalence of two modes.
(main): Use eq_mode() rather than memcmp() to compare
two modes. Also use stack variables rather than implicitly
initialized static variables. Also remove all uses of
the SPEED_WAS_SET hack since we now more robustly compare modes.
* NEWS: Update the [io]speed fix entry.
Reported at https://bugs.debian.org/1019468
* src/stty.c (main): Move internal TESTING code that showed
the new and old mode, upon failure to apply the new mode,
to being runtime controlled with the ---debug option.
Also augment the display to show which items were not
set as expected.
* doc/coreutils.texi (stty invocation): Say that "drain"
is treated as an option, rather than a line setting,
and so option processing rules apply to it.
Reported in https://bugs.debian.org/1018803
* src/stty.c (apply_settings): Validate [io]speed arguments
against the internal accepted set.
(set_speed): Check the cfset[io]speed() return value so
that we validate against the system supported set.
* tests/misc/stty-invalid.sh: Add a test case.
* NEWS: Mention the bug fix.
Reported in https://bugs.debian.org/1018790
* src/tail.c (check_output_alive): Add a guard that would
trigger on most platforms, to detect if we're using the
gnulib poll module. That's currently problematic in the
way it emulates poll() using select() and would cause
issues on macOS and AIX at least as poll() is replaced there.
* src/comm.c (compare_files): Handle the single character
--output-delimeter case separately so that NUL is appropriately
handled.
* doc/coreutils.texi (comm invocation): Fix the description
of --output-delimiter to say an empty delimeter is treated
as a NUL separator, rather than being disallowed.
* tests/misc/comm.pl: Add a test case.
Reported at https://bugs.debian.org/1014008
* src/runcon.c: Use EXIT_CANCELED (125) instead of EXIT_FAILURE (1),
so that errors specific to runcon can be distinguished,
from those of the invoked program.
* doc/coreutils.texi (runcon invocation): Fix the Exit status
description to say we return 125 (not 127) for internal errors.
* tests/misc/runcon-no-reorder.sh: Add a test case.
The README was becoming too long and contained
quite a bit of info only pertaining to rarely used systems, so...
* README: Split out install specific info to README-install.
Also remove a few stale lines, and reorder a few items.
* README-install: A new file split from README.
* Makefile.am [EXTRA_DIST]: Explicitly reference new README-install
file for distribution, since automake only auto adds README.
* TODO: Reference the HPUX info now in README-install.
* src/ls.c (usage): Don't mention "modification" in the
description of ctime (-c), as it's confusing with mtime.
Mention "metadata" when discussing "change" time to
disambiguate from data change time.
* doc/coreutils.texi (ls invocation): State that --time=creation
falls back to using mtime where not available.
This behaviour is correctly documented when doing `cp --help`.
There is no `--reflink=when` option.
* doc/coreutils.texi (cp invocation): Fix document stating
that `--reflink` is equivalent to `--reflink=always`.
It's useful to treat empty and missing arguments differently.
Missing means all signals, while empty means no signals and
so is a no-op. It's useful to treat empty arguments like
this, so that dynamically specified arguments like the following
are supported
env --ignore-signals "$SIGS_TO_IGNORE"
Note `env --ignore-signals=` is treated as an empty argument.
* doc/coreutils.texi (env invocation): Empty args are treated
differently to missing arguments, so call that out explicitly.
* src/env.c (usage): Likewise.
Addresses https://bugs.debian.org/1016049
* src/date.c (usage): Specify that --date, --file, --reference,
and --resolution are mutually exclusive. This is also useful
documentation to group similar options.
* doc/coreutils.texi (Options for date): Likewise.
Addresses https://bugs.gnu.org/55401
* src/date.c: (main): Track and diagnose whether any
-d or -s options are dropped, as users may think
multiple options are supported, given they can be relative.
* tests/misc/date-debug.sh: Add a test case.
* NEWS: Mention the improvement.
* src/runcon.c (main): With -c avoid searching the path
to ensure the file specified to --compute is executed.
* tests/misc/runcon-compute.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Reported in https://bugs.debian.org/1013924
* src/remove.c: Include stat-time.h.
(cache_fstatat, cache_stat_init): Use negative st->st_atim.tv_sec to
determine whether the stat is cached, not negative st->st_size.
On non-POSIX platforms that lack st_atim.tv_sec, don’t bother to cache.
This follows up on comments by Pádraig Brady (bug#56391).
* src/copy.c (copy_reg): When --reflink=always removes a file
due to an FICLONE failure, do not remove a nonempty file.
* src/shuf.c: Do not include xdectoint.h.
(main): Improve diagnostic for ‘shuf -i -10-10’. Without this
patch, the diagnostic was “shuf: invalid input range: ‘’” which is
not helpful. Now it is “shuf: invalid input range: ‘-10-10’”.
* cfg.mk (begword, endword): New macros.
(sc_prohibit_stat_macro_address, sc_prohibit_fail_0)
(sc_prohibit_short_facl_mode_spec, sc_require_stdio_safer)
(sc_prohibit_sleep, sc_prohibit_framework_failure)
(sc_marked_devdiagnostics):
* build-aux/gen-single-binary.sh:
Prefer POSIX-compatible EREs to GNU extensions like \w and \<.
Problem reported by pkoraou@gmail.com (Bug#55910).
* src/copy.c (copy_internal): Treat a relative destination name ""
as if it were "." for the purpose of directory-relative syscalls
like fstatat that might might refer to the destination directory.
* src/sort.c (keycompare, compare): Don’t overflow if -r is
specified and a comparison function returns INT_MIN, as this
causes the comparison to have undefined behavior (typically the
reverse of correct). glibc memcmp on s390x reportedly returns
INT_MIN in some cases, so this is not a purely academic issue.
* src/comm.c (compare_files):
* src/join.c (keycmp):
* src/ls.c (off_cmp):
* src/ptx.c (compare_words, compare_occurs):
* src/set-fields.c (compare_ranges):
Prefer ((a > b) - (a < b)) to variants like (a < b ? -1 : a > b)
as it’s typically faster these days.
* src/sort.c (keycompare): Rework to avoid gotos.
This also shrinks the machine code a bit (112 bytes)
with GCC 12 x86-64 -O2. Nowadays compilers are smart
enough to coalesce jumps so we need not do it by hand.
* src/sort.c (keycompare): Rework to pacify a GCC 12
-Wmaybe-uninitialized false positive, by coalescing some minor
duplicate code and eliminating a branch. This should execute an
insn or two less in the usual case.
When factoring numbers that have a large 2^n factor, it can be hard to
eyeball just how many 2's there are. Add an option to print each prime
power factor in the p^e format (omitting the exponent when it is 1).
* src/factor.c: Add -h, --exponents option for printing in p^e format.
* doc/coreutils.texi (factor invocation): Document the new option.
* tests/misc/factor.pl: Add test case.
* THANKS.in: Add previous suggester
(https://lists.gnu.org/r/coreutils/2017-11/msg00015.html).
Suggested-by: Emanuel Landeholm <emanuel.landeholm@gmail.com>
Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Problem reported by Giulio Genovese (Bug#55212).
* src/sort.c (nan_compare): To compare NaNs, simply printf+strcmp.
This avoids the problem of padding bits and unspecified behavior.
Args are now long double instead of char *; caller changed.
Found with -flto and --enable-gcc-warnings.
* src/pr.c (getoptarg): Fix misuse of xstrtol, which does not
necessarily set tmp_long on errror, and does not set errno in any
reliable way. The previous code might access uninitialized
storage; on typical platforms this merely causes it to possibly
print the wrong diagnostic.
* doc/coreutils.texi: Set txicodequoteundirected and
txicodequotebacktick so that ' and ` in code examples appear
as-is, rather than being transliterated to ’ and ‘. E.g., prefer
“... this is equivalent to ‘tr '\303\266' '\305\201'’ and ...” to
“... this is equivalent to ‘tr ’\303\266’ ’\305\201’’ and ...”
in PDF output.
* doc/coreutils.texi (Character arrays): Avoid using shell
notation like $'\u7530' since this isn’t in POSIX yet. Instead,
use ö and Ł which should work in all texinfo output formats.
This option has changed from ignoring only ENOTEMPTY|EEXIST
(i.e. ignore errors _solely_ due to dir not empty),
to ignoring some other errors from more protected dirs
that are not empty. That adjustment was made to better
support use with --parents, to essentially remove as much of
a hierarchy as possible, without erroring as we hit more
protected non empty parent dirs.
That functionality adjustment was originally discussed at:
https://lists.gnu.org/r/bug-coreutils/2008-01/msg00283.html
* src/rmdir.c (usage): Adjust to be more accurate to current behavior.
Also adjust --parents option to be easier to read.
* doc/coreutils.texi (rmdir invocation): Likewise.
Reported at https://github.com/coreutils/coreutils/issues/40
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
