* tests/cp/sparse-perf.sh: Avoid the case where
we saw SEEK_DATA take 35s to return a result
against a 1TB sparse file. This happened on
a FreeBSD 9.1 VM at least.
Reported by Nelson H. F. Beebe.
* src/cksum.c (crc_sum_stream): On sparc64 for example,
a crc of 0 was printed due to mismatch in size of
variable copied between generator and output functions.
uint_fast32_t is generally 64 bits on 64 bit systems,
so we copy through an int to ensure we don't use the wrong
end of a 64 bit variable.
Reported by Nelson H. F. Beebe
* bootstrap.conf: We only need poll on Linux and AIX
where poll is not replaced. Also resinstate dependence
on select so we can use it unconditionally.
* src/tail.c (check_output_alive): Reinstate use of select()
by default as poll was seen to be ineffective for this
application on macOS.
Fixes https://bugs.gnu.org/50714
* src/digest.c: Allow using the --untagged option with --check,
so that `cksum -a md5 --untagged` used to emulate md5sum for example,
may be augmented with the --check option. Also support the --tag
option with cksum, to allow overriding a previous --untagged setting.
* doc/coreutils.texi: Adjust accordingly.
* tests/misc/cksum-a.sh: Likewise.
* tests/tail-2/F-vs-rename.sh: Keep stdout and stderr separate,
so that interspersion doesn't impact regex checks. Also wait
for each file's data to be printed to avoid multiple writes
to a file to be printed in a single iteration, which would
impact the regex checks. Also we refactor the check function,
rather than repeatedly redefining variations.
This is wrong fix really, as only introducing delay I think.
* tests/tail-2/F-vs-rename.sh: Avoid a rare false failure
due to a race in the test. Now wait until tail has noticed
that b is replaced before writing to a, so that the subsequent
write of "y" to b will be displayed independently from
current contents of b ("x").
* tests/ls/removed-directory.sh: On FreeBSD 9.1 at least,
one gets ENOENT when trying to traverse the current removed dir
with ../, so instead reference the parent dir directly.
Here are the warnings:
src/chmod.c:175:3: error: 'ch.new_mode' may be used uninitialized in\
this function [-Werror=maybe-uninitialized]
175 | strmode (ch->new_mode, perms);
src/chmod.c:178:3: error: 'ch.old_mode' may be used uninitialized in\
this function [-Werror=maybe-uninitialized]
178 | strmode (ch->old_mode, old_perms);
* src/chmod.c (process_file): Initialize ch. Its new_mode and
old_mode fields could indeed be used uninitialized to form mode
strings, but those are used only when built from initialized members.
* src/digest.c (digest_check): Treat empty lines like comments,
as commented checksum files very often have empty lines.
* tests/misc/md5sum.pl: Adjust accordingly.
* src/digest.c: Always set the digest_length, so that
we check the correct number of hex digits when parsing
non tagged format checksums.
* tests/misc/cksum-a.sh: Add a test case. Also fix
up this test which was ineffective due to fail=1
being set in a subshell and ignored.
Support checksum files with CRLF line endings,
which is a common gotcha for using --check on windows,
or with checksum files generated on windows.
Note we escape \r here to support the original coreutils format
(with file name at EOL), and file names with literal
\r characters as the last character of their name.
* src/digest.c (filename_unescape): Convert \\r -> \r.
(print_filename): Escape \r -> \\r.
(output_file): Detect \r chars in file names.
(digest_check): Ignore literal \r char at EOL.
* tests/misc/md5sum.pl: Add a test case.
* tests/misc/sha1sum.pl: Likewise.
* NEWS: Mention the improvement.
This only practically matters on windows.
But given there are separate text handling options in cygwin,
keep the interface simple, and avoid exposing the
confusing binary/text difference here.
* doc/coreutils.texi (md5sum invocation): Mention that
--binary and --text are not supported by the cksum command.
* src/digest.c: Set flag to use binary mode by default.
(output_file): Don't distinguish text and binary modes with
' ' and '*', and just use ' ' always.
This format is a better default, since it results in simpler usage,
as you don't need to specify --tag on generation or -a on
checking invocations. Also it's a more general format supporting
mixed and length adjusted digests.
* doc/coreutils.texi (cksum invocation): Document a new --untagged
option, to use the older coreutils format.
(md5sum invocation): Mention that cksum doesn't support --tag.
* src/digest.c: Adjust cksum(1) to default to --tag,
and accept the new --untagged option.
* tests/misc/b2sum.sh: Adjust accordingly.
* tests/misc/cksum-a.sh: Likewise.
* tests/misc/cksum-c.sh: Likewise.
* src/cksum.h: Thread DELIM through the output functions.
* src/digest.c: Likewise.
* src/sum.c: Likewise.
* src/sum.h: Likewise.
* src/cksum.c: Likewise. Also adjust check to allow -z
with traditional output modes. Also ajust the global variable
name to avoid shadowing warnings.
* tests/misc/cksum-a.sh: Adjust accordingly.
This will be generally useful going forward, for sha3-256 etc.
* src/digest.c: Rename b2_length to digest_length, and
adjust/simplify the code to operate on this for both
b2sum and cksum -a blake2b.
Support `cksum --check FILE` without having to specify a digest
algorithm, allowing for more generic file check instructions.
This also supports mixed digest checksum files, supporting
more robust multi digest checks.
* src/digest.c (algorithm_from_tag): A new function to
identify the digest algorithm from a tagged format line.
(split3): Set the algorithm depending on tag, and update
the expected digest length accordingly.
* tests/misc/cksum-c.sh: Add a new test.
* tests/local.mk: Reference the new test.
* tests/misc/md5sum.pl: Adjust to more generic error.
* tests/misc/sha1sum.pl: Likewise.
* doc/coreutils.texi (md5sum invocation): Mention the new -c feature.
* NEWS: Mention the new feature.
Add message digest sm3, which uses the OSCCA SM3 secure
hash (OSCCA GM/T 0004-2012 SM3) generic hash transformation.
* bootstrap.conf: Add the sm3 module.
* doc/coreutils.texi: Mention the cksum -a option.
* src/digest.c: Provide support for --algorithm='sm3'.
* tests/misc/sm3sum.pl: Add a new test (from Tianjia Zhang)
* tests/local.mk: Reference the new test.
* NEWS: Mention the new feature.
Tested-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
* src/digest.c: Organize HASH_ALGO_CKSUM to be table driven,
and amalgamate all digest algorithms.
(main): Parse all options if HASH_ALGO_CKSUM, and disallow
--tag, --zero, and --check with the traditional bsd, sysv, and crc
checksums for now.
* src/local.mk: Reorganize to include all digest modules in cksum.
* tests/misc/cksum-a.sh: Add a new test.
* tests/misc/b2sum.sh: Update to default to checking with cksum,
as b2sum's implementation diverges a bit from the others.
* tests/local.mk: Reference the new test.
* doc/coreutils.texi (cksum invocation): Adjust the summary to
identify the new mode, and document the new --algorithm option.
* man/cksum.x: Adjust description to be more general.
* man/*sum.x: Add [See Also] section referencing cksum(1).
* NEWS: Mention the new feature.
* cfg.mk: Adjust cksum.c to not require config.h
and support a main (for crctab) without calling bindtextdomain().
* po/POTFILES.in: Remove cksum_pclmul.c since it no longer
concerns itself with diagnostics.
* src/cksum.c: Refactor to just providing stream digest,
and digest printing functionality.
* src/cksum.h: Adjust to the new interface.
* src/cksum_pclmul.c: Remove diagnostics, and determine errors
internally.
* src/crctab.c: Separate from cksum.h since that's now included
multiple times.
* src/digest.c: Provide cksum(1) functionality if -DHASH_ALGO_CKSUM
* src/local.mk: Adjust to new crctab.c and HASH_ALGO_CKSUM define.
This should have been part of commit v8.32-113-gb73b9fcb1
* doc/coreutils.texi (cksum invocation): Add the --debug description.
* src/cksum.c (usage): Likewise.
(main): Also give explicit indication when using generic hardware.
Since digest will be providing all digest functionality,
refactor sum.c into it.
* po/POTFILES.in: sum.c no longer has translatable strings so remove.
* src/digest.c: Call out to new stream interfaces in sum.c
* src/local.mk: Adjust sources for the sum binary.
* src/sum.c: Provide a stream interface for BSD and SYSV digests.
* src/sum.h: A new file to declare the exported functions in sum.c
md5sum.c will be the base for all digest functions,
so rename accordingly.
* src/md5sum.c: Rename to ...
* src/digest.c: ... renamed from md5sum.c
* scripts/git-hooks/commit-msg: Allow digest: commit prefix.
* po.POTFILES.in: Adjust to new name.
* src/local.mk: Likewise.
* doc/coreutils.texi (whoami invocation): Clarify it prints names,
not numeric IDs.
* man/whoami.x: Likewise.
* man/logname.x: Reference getlogin(3).
* src/logname.c: Clarify that it prints the login name,
rather than the name of the effective user ID.
Fixes https://bugs.gnu.org/48894
* tests/ls/stat-vs-dirent.sh: Skip the test if we can't stat(1),
as the file may have been removed, or have a malformed name
due to '\n' etc. in the file name.
* gnulib: Update to latest. This fixes a gnulib test failure in base64,
among other fixes.
* cfg.mk: Disable sc_indent as auto indent is too invasive for now.
Adjust to output the file name if any name parameter is passed.
This is consistent with sum -s, cksum, and sum implementations
on other platforms. This should not cause significant compat
issues, as multiple fields are already output, and so already
need to be parsed.
* src/sum.c (bsd_sum_file): Output the file name
if any name parameter is passed.
* tests/misc/sum.pl: Adjust accordingly.
* doc/coreutils.texi (sum invocation): Likewise.
* NEWS: Mention the change in behavior.
* tests/misc/help-version.sh: Test that /dev/full causes
shell printf to fail. This ports better to NetBSD 9.88.46,
where it doesn’t. Problem reported by Nelson H. F. Beebe.
* tests/misc/help-version.sh: Merge gzip-related changes
back from gzip/tests/help-version. This fixes problems
when TERM is not 'dumb', and should simplify maintenance.
Emil Lundberg <lundberg.emil@gmail.com> reports in
https://bugs.gnu.org/49741 about a 'basenc --base64 -d' decoding bug.
The input buffer length was not divisible by 3, resulting in
decoding errors.
* NEWS: Mention fix.
* src/basenc.c (DEC_BLOCKSIZE): Change from 1024*5 to 4200 (35*3*5*8)
which is divisible by 3,4,5,8 - satisfying both base32 and base64;
Use compile-time verify() macro to enforce the above.
* tests/misc/basenc.pl: Add test.
This patch modifies basenc to prefer signed integers to
unsigned, as signed are less error-prone.
This patch also updates Gnulib to to latest, which updates Gnulib’s
base32 and base64 modules to prefer signed to unsigned integers.
* src/basenc.c: Include idx.h.
(struct base2_decode_context): Use unsigned char, not unsigned
for an octet that must fit in an unsigned char.
(base_encode, struct base_decode_context)
(base64_decode_ctx_wrapper, prepare_inbuf, base64url_encode)
(base64url_decode_ctx_wrapper, base32_decode_ctx_wrapper)
(base32hex_encode, base32hex_decode_ctx_wrapper, base16_encode)
(base16_decode_ctx, z85_encode, Z85_HI_CTX_TO_32BIT_VAL)
(z85_decoding, z85_decode_ctx, base2msbf_encode)
(base2lsbf_encode, base2lsbf_decode_ctx, base2msbf_decode_ctx)
(wrap_write, do_encode, do_decode, main):
Prefer signed integers to unsigned.
(main): Treat extremely large wrap columns as if they were
infinite; that’s good enough. Since we’re now using xstrtoimax,
this allows ‘-w -0’ (same as ‘-w 0’).
* tests/misc/base64.pl (gen_tests): -w-0 is no longer an error.
find-mount-point.h rightly includes <stdlib.h> for its use
of _GL_ATTRIBUTE_DEALLOC_FREE, which uses free, yet that new
inclusion provoked a syntax-check failure. Exempt this header
file as we've done for others.
* cfg.mk (exclude_file_name_regexp--sc_system_h_headers):
Add find-mount-point.h to the regexp.
(sc_system_h_headers): Use grep -E, for a more readable regexp.
This better tests the SEEK_HOLE logic which
replaced the original fiemap hole identification logic.
Also it avoids a false failure in sparse-2.sh
on reflink supporting file systems, where we
try to correlate the file sizes produced by cp and dd.
* tests/cp/sparse-2.sh: s/cp/cp --reflink=never/
* tests/cp/sparse-extents-2.sh: Likewise.
* tests/cp/sparse-extents.sh: Likewise.
* tests/cp/sparse-perf.sh: Likewise.
* tests/cp/sparse.sh: Likewise.
Fixes https://github.com/coreutils/coreutils/issues/54
This is so that commands like ‘fmt - -’ read from stdin
both times, even when it is a tty. Fix some other minor
issues that are related.
* src/blake2/b2sum.c (main):
* src/cksum.c (cksum):
* src/cut.c (cut_file):
* src/expand-common.c (next_file):
* src/fmt.c (fmt):
* src/fold.c (fold_file):
* src/md5sum.c (digest_file, digest_check):
* src/nl.c (nl_file):
* src/od.c (check_and_close):
* src/paste.c (paste_parallel, paste_serial):
* src/pr.c (close_file):
* src/sum.c (bsd_sum_file):
Use clearerr on stdin so that stdin can be read multiple times
even if it is a tty. Do not assume that ferror preserves errno as
POSIX does not guarantee this. Coalesce duplicate diagnostic
calls.
* src/blake2/b2sum.c (main):
* src/fmt.c (main, fmt):
Report read error, even if it's merely fclose failure.
* src/fmt.c: Include die.h.
(fmt): New arg FILE. Close input (reporting error) if not stdin.
All callers changed.
* src/ptx.c (swallow_file_in_memory): Clear stdin's EOF flag.
* src/sort.c (xfclose): Remove unnecessary feof call.
Problem reported by Michael Debertol (Bug#50070).
* NEWS: Mention the fix.
* src/chmod.c (struct change_status): New struct, replacing the
old enum Change_status. All uses changed.
(describe_change): Distinguish between cases depending on
whether 'stat' or its equivalent succeeded. Report a line
of output even if 'stat' failed, as that matches the documentation.
Rework to avoid casts.
(process_file): Do not output nonsense modes computed from
uninitialized storage, removing a couple of IF_LINTs. Simplify by
defaulting to CH_NO_STAT.
If the command-line argument is automounted, df would use
stat info that became wrong after the following open.
* NEWS: Mention the fix (bug#50012).
* src/df.c (automount_stat_err): New function.
This fixes the hang on fifos in a better way, by using O_NONBLOCK.
(main): Use it.
We must delay handling when \r is the last character
of the buffer being processed, as the next character
may or may not be \n.
* src/cat.c (pending_cr): A new global to record whether
the last character processed (in -E mode) is '\r'.
(cat): Honor pending_cr when processing the start of the buffer.
(main): Honor pending_cr if no more files to process.
* tests/misc/cat-E.sh: Add test cases.
Fixes https://bugs.gnu.org/49925
Pacify GCC 11.1 -fanalyzer.
* src/uniq.c (check_file): Use simpler test to check whether this
is the first time through the loop. Although the old test was
correct, the new one is easier to understand and perhaps a tiny
bit more efficient.
Caught by GCC 11.1 -fanalyzer.
* src/numfmt.c (simple_strtod_int): Remove unnecessary test of
*endptr vs NULL. Presumably this was a typo and **endptr was
intended instead of *endptr, but an **endptr test is also
unnecessary since c_isdigit (0) returns false.
* doc/coreutils.texi (tr invocation): Provide a summary
list of the available options, which is useful to
provide a quick reminder for those already familiar
with the functionality of tr.
Fixes https://bugs.gnu.org/49764
In documentation and comments, don’t assume that secondary storage
devices are disk devices. Similarly, don’t assume that main memory
uses magnetic cores, which became obsolete in the 1970s.
* src/du.c (usage):
* src/ls.c (usage):
* src/shred.c (usage): Reword to avoid “disk” in usage messages.
* doc/coreutils.texi (ls invocation): Document implementation more
closely. Be more consistent about style. Omit some needless words.
* src/ls.c (usage): Don’t overdocument -f, as the details were wrong.
Omit -1 advice as it’s a bit obsolete now that we have --zero and
is a bit much for --usage output anyway.
* NEWS, doc/coreutils.texi (General output formatting):
* src/ls.c (usage):
Document this.
* src/ls.c (ZERO_OPTION): Rename from NULL_OPTION.
All uses changed.
(long_options): Rename --null to --zero.
(dired_dump_obstack, main, print_dir): Use '\n' instead of
eolbyte where eolbyte must equal '\n'.
(decode_switches): Decode --zero instead of --null.
--zero also implies -1, -N, --color=none, --show-control-chars.
Use easier-to-decipher code to set ‘format’ and ‘dired’.
Reject attempts to combine --dired and --zero.
* tests/local.mk: Adjust to test script renaming.
* tests/ls/zero-option.sh: Rename from tests/ls/null-option.sh,
and test --zero instead of --null.
* src/ls.c (enum time_type, enum sort_type, enum indicator_style)
(enum Dereference_symlink, ignore_mode):
Put ‘= 0’ after default values, since the code relies
on static storage defaulting to zero.
(enum sort_type): Reorder so that -1 can be used to represent unset.
(main): Test print_with_color after parse_ls_color may have reset it.
(decode_line_length): Return the line length instead of setting
static storage. All uses changed. Treat line lengths exceeding
PTRDIFF_MAX as infinite, to avoid pointer-subtraction glitches.
(stdout_isatty): New function, to avoid calling isatty twice.
(decode_switches): Calculate defaults more lazily, to avoid using
syscalls or getenv during startup unless the results are more
likely to be needed. Use -1 to indicate options that haven’t been
set on the command line yet. Move print_with_color test from
here to ‘main’. Suppress bogus GCC warning.
(getenv_quoting_style): Return the quoting style instead of
setting static storage.
(init_column_info): New arg MAX_COLS, to avoid recalculating it.
Caller changed.
* src/ls.c (dired_pos): Now off_t, not size_t, since it counts
output file offsets.
(dired_dump_obstack): This obstack's file offsets are now
off_t, not size_t.
(format_user_or_group, format_user_or_group_width):
ID arg is now uintmax_t, not unsigned long, since uid_t and
gid_t values might exceed ULONG_MAX.
(format_user_or_group_width): Use snprintf with NULL instead of
sprintf with a discarded buffer. This avoids a stack buffer,
and so should be safer.
Prefer functions or constants to macros where either will do.
That’s cleaner, and nowadays there’s no performance reason to
prefer macros. All uses changed.
* src/ls.c (INITIAL_TABLE_SIZE, MIN_COLUMN_WIDTH):
Now constants instead of macros.
(file_or_link_mode): New function, replacing the old macro
FILE_OR_LINK_MODE.
(dired_outbyte): New function, replacing the old macro DIRED_PUTCHAR.
(dired_outbuf): New function, replacing the old macro DIRED_FPUTS.
(dired_outstring): New function, replacing the old macro
DIRED_FPUTS_LITERAL.
(dired_indent): New function, replacing the old macro DIRED_INDENT.
(push_current_dired_pos): New function, replacing the old macro
PUSH_CURRENT_DIRED_POS.
(assert_matching_dev_ino): New function, replacing the old macro
ASSERT_MATCHING_DEV_INO.
(do_stat, do_lstat, stat_for_mode, stat_for_ino, fstat_for_ino)
(signal_init, signal_restore, cmp_ctime, cmp_mtime, cmp_atime)
(cmp_btime, cmp_size, cmp_name, cmp_extension)
(fileinfo_name_width, cmp_width, cmp_version):
No longer inline; compilers can deduce this well enough nowadays.
(main): Protect unused assert with ‘if (false)’ rather than
commenting it out, so that the compiler checks the code.
(print_dir): Output the space and newline in the same buffer
as the human-readable number they surround.
(dirfirst_check): New function, replacing the old macro
DIRFIRST_CHECK. Simplify by using subtraction.
(off_cmp): New function, replacing the old macro longdiff.
(print_long_format): No need to null-terminate the string now.
(format_user_or_group): Let printf count the bytes.
* src/ls.c (format_user_or_group_width, print_long_format):
Use return value from sprintf instead of calling strlen on
the resulting buffer, or inferring the length some other way.
As originally reported in <https://bugzilla.redhat.com/1962515>,
df invoked without -a printed duplicated entries for NFS mounts
of bind mounts. This is a regression from commit v8.25-54-g1c17f61ef99,
which introduced the use of a hash table.
The proposed patch makes sure that the devlist entry seen the last time
is used for comparison when eliminating duplicated mount entries. This
way it worked before introducing the hash table.
Patch co-authored by Roberto Bergantinos.
* src/ls.c (struct devlist): Introduce the seen_last pointer.
(devlist_for_dev): Return the devlist entry seen the last time if found.
(filter_mount_list): Remember the devlist entry seen the last time for
each hashed item.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/49298
This fixes an unlikely stack out-of-bounds write reported by
Stepan Broz via Kamil Dudka (Bug#49209).
* bootstrap.conf (gnulib_modules): Replace select with poll.
* src/tail.c: Do not include <sys/select.h>.
[!_AIX]: Include poll.h.
(check_output_alive) [!_AIX]: Use poll instead of select.
(tail_forever_inotify): Likewise. Simplify logic, as there is no
need for a ‘while (len <= evbuf_off)’ loop.
* src/stat.c (default_format): Use decomposed decimal
representation (major,minor) in the default format.
This is least ambiguous for human interpretation,
and more consistent with ls for example.
Fixes https://bugs.gnu.org/48960
In preparation for changing the default device number
representation (to decomposed decimal), provide more
formatting options for device numbers.
These new (FreeBSD compat) formatting options are added:
%Hd major device number in decimal (st_dev)
%Ld minor device number in decimal (st_dev)
%Hr major device type in decimal (st_rdev)
%Lr minor device type in decimal (st_rdev)
%r (composed) device type in decimal (st_rdev)
%R (composed) device type in hex (st_rdev)
* doc/coreutils.texi (stat invocation): Document new formats.
* src/stat.c (print_it): Handle the new %H and %L modifiers.
(print_statfs): Adjust to passing the format as two chars
rather than an int. Using an int was introduced in commit db42ae78,
but using separate chars is cleaner and more extensible.
(print_stat): Likewise. Handle any modifiers and the new 'r' format.
(usage): Document the new formats.
* tests/misc/stat-fmt.sh: Add a test case for new modifiers.
Addresses https://bugs.gnu.org/48960
Coreutils mistakenly did not list xstrndup as a module
that it depends on directly. When the latest Gnulib removed
the dirname module's dependency on xstrndup, this mistake
caused coreutils to not build. Since all of Coreutils's
uses of xstrndup know the string length, xmemdup0 is a better
match for what's needed. Since the size args are typically
signed or derived from subtracting pointers, the new Gnulib
ximemdup0 function is a better match yet.
So, use ximemdup0 instead of xstrndup.
* src/cut.c, src/dircolors.c, src/expand-common.c, src/expand.c:
* src/numfmt.c, src/set-fields.c, src/unexpand.c:
Do not include xstrndup.h; no longer needed.
* src/dircolors.c (parse_line):
* src/expand-common.c (parse_tab_stops):
* src/numfmt.c (parse_format_string):
* src/set-fields.c (set_fields):
Use ximemdup0 instead of xstrndup.
This is now only used on 10 year old linux kernels,
and performs a sync before each copy.
* src/copy.c (extent_copy): Remove function and all callers.
* src/extent-scan.c: Remove.
* src/extent-scan.h: Remove.
* src/fiemap.h: Remove.
* src/local.mk: Adjust for removed files.
* NEWS: Adjust to say fiemap is removed.
copy_file_range() before Linux kernel release 5.3 had many issues,
as described at https://lwn.net/Articles/789527/, which was
referenced from https://lwn.net/Articles/846403/; a more general
article discussing the generality of copy_file_range().
Linux kernel 5.3 was released in September 2019, which is new enough
that we need to actively avoid older kernels.
* src/copy.c (functional_copy_file_range): A new function
that returns false for Linux kernels before version 5.3.
(sparse_copy): Call this new function to gate use of
copy_file_range().
* tests/cp/sparse-2.sh: Double check cp --sparse=always,
with dd conv=sparse, in the case where the former didn't
create a sparse file. Now that this test is being newly run
on macos, we're seeing a failure due to seek() not creating
holes on apfs unless the size is >= 16MiB.
fiemap is no longer the default copy implementation,
so check for SEEK_DATA support instead as that's preferred.
This will ensure better test coverage on systems without fiemap.
* init.cfg: Replace fiemap_capable_ with seek_data_capable_.
This is best supported with python 3 so prefer that.
* tests/seek-data-capable: A new test script checking for
SEEK_DATA support on the passed file name,
called from seek_data_capable_.
* tests/fiemap-capable: Remove no longer used probing script.
* tests/cp/fiemap-perf.sh: Renamed to tests/cp/sparse-perf.sh
* tests/cp/fiemap-2.sh: Renamed to tests/cp/sparse-2.sh
* tests/cp/fiemap-extents.sh: Renamed to tests/cp/sparse-extents.sh
* tests/cp/sparse-fiemap.sh: Renamed to tests/cp/sparse-extents-2.sh
* tests/cp/fiemap-FMR.sh: Renamed to tests/cp/copy-FMR.sh
* tests/local.mk: Reference the renamed tests.
* src/copy.c (sparse_copy): Upon EPERM from copy_file_range(),
fall back to a standard copy, which will give a more accurate
error as to whether the issue is with the source or destination.
Also this will avoid the issue where seccomp or apparmor are
not configured to handle copy_file_range(), in which case
the fall back standard copy would succeed without issue.
This specific issue with seccomp was noticed for example in:
https://github.com/golang/go/issues/40900
* src/copy.c (infer_scantype): Ensure we don't error out
if SEEK_DATA returns EOPNOTSUPP, on systems where this value
is distinct from ENOTSUP. Generally both of these should be checked.
* src/copy.c (sparse_copy): Ensure we fall back to
a standard copy if copy_file_range() returns ENOTSUP.
This generally is best checked when checking ENOSYS,
but it also seems to be a practical concern on Centos 7,
as a quick search gave https://bugzilla.redhat.com/1840284
Fixes a bits/long-double.h include build issue on some systems.
* bootstrap: Sync new --version option from gnulib.
* gnulib: Update to lastest.
Reported by Carl Edquist
The test program will compile successfully even if __get_cpuid_count
is not declared. The error for the missing symbol will only show up
at link time. Thus, use AC_LINK_IFELSE instead of AC_COMPILE_IFELSE.
* configure.ac (__get_cpuid_count check): Use C_LINK_IFELSE instead
of AC_COMPILE_IFELSE.
(__get_cpuid check): Likewise.
Ensure we call hash_free() to avoid valgrind and leak_sanitizer
"definitely lost" warnings. These were not real leaks as
we terminate immediately after, but we should avoid these
"definitely lost" warnings where possible.
* src/copy.c: Add dest_info_free() and src_info_free().
* src/copy.h: Declare the above.
* src/cp-hash.c: Don't define unless "lint" is defined.
* src/install.c: Call dest_info_free() in dev mode.
* src/mv.c: Likewise.
* src/cp.c: Likewise. Also call src_info_free().
* src/ln.c: Call hash_free() in dev mode.
* src/tail.c: Call hash_free() even if about to exit, in dev mode.
Fixes https://bugs.gnu.org/48189
* src/copy.c (sparse_copy): Don't use copy_file_range()
with --reflink=never as copy_file_range() may implicitly
use acceleration techniques like reflinking.
(extent_copy): Pass through whether we allow reflinking.
(lseek_copy): Likewise.
Fixes https://bugs.gnu.org/48164
* src/wc.c: (main): Handle the new --debug option.
Only call avx2_supported if needed.
(avx2_supported): Diagnose various failures and attempts.
* NEWS: Mention the new wc improvement and --debug option.
Use cpuid to detect CPU support for avx2 instructions.
Performance was seen to improve by 5x for a file with only newlines,
while the performance for a file with no such characters is unchanged.
* configure.ac [USE_AVX2_WC_LINECOUNT]: A new conditional,
set when __get_cpuid_count() and avx2 compiler intrinsics are supported.
* src/wc.c (avx2_supported): A new function using __get_cpuid_count()
to determine if avx2 instructions are supported.
(wc_lines): A new function refactored from wc(),
which implements the standard line counting logic,
and provides the fallback implementation for when avx2 is not supported.
* src/wc_avx2.c: A new module to implement using avx2 intrinsics.
* src/local.mk: Reference the new module. Note we build as a separate
lib so that it can be portably built with separate -mavx2 etc. flags.
Problem reported by Roland (Bug#48106).
* src/touch.c (touch): Take more care when deciding whether
to use open_errno or utime_errno in the diagnostic.
Stop worrying about SunOS 4 (which as part of the problem),
as it’s long obsolete. For Solaris 10, verify that EINVAL
really means the file was a directory.
* configure.ac: Use AC_PROG_CC, not AC_PROG_CC_STDC.
* gl/modules/smack (configure.ac):
* m4/jm-macros.m4 (coreutils_MACROS):
* m4/xattr.m4 (gl_FUNC_XATTR):
Use AS_HELP_STRING, not AC_HELP_STRING.
* m4/check-decl.m4 (gl_CHECK_DECLS):
Do not require AC_HEADER_TIME; we no longer care about it directly.
* m4/jm-macros.m4 (coreutils_MACROS):
Do not require AC_ISC_POSIX, which became obsolete in 2006.
Use AC_LINK_IFELSE instead of AC_TRY_LINK.
* src/csplit.c (load_buffer):
* src/pinky.c (create_fullname):
Use intprops-based checks rather than xalloc_oversized,
since Gnulib xalloc.h no longer includes xalloc-oversized.h.
* src/copy.c (sparse_copy): Fallback to read() if copy_file_range()
fails with ETXTBSY. Otherwise it would be impossible to copy files
that are being used as swap. This used to work before introducing
the support for copy_file_range() in coreutils. (Bug#48036)
On newer systems like Fedora 34 and openSUSE Tumbleweed, ls(1) calls
newfstatat(STDOUT_FILENO, ...), but only when there is something to
output.
* tests/ls/stat-free-color.sh: Add -a option to the reference invocation
of ls, thus enforcing something gets output.
* doc/coreutils.texi (ln invocation): State --symbolic is required.
* src/ln.c (usage): Explicitly state -s is not implied.
Fixes https://bugs.gnu.org/47703
* src/wc.c (usage): State that only printable characters are considered
when counting words. This also disambiguates wether we're talking
about bytes or characters in this context.
* doc/coreutils.texi (wc invocation): Likewise. Also clarify
that --characters counts valid locale aware characters,
and that --lines does not count a trailing "line" unless
it ends with a newline character.
Fixes https://bugs.gnu.org/47702
This is especially important now for --sort=width,
as that can greatly increase how often this
expensive quote_name_width() function is called per file.
This also helps the default invocation of ls,
or specifically the --format={across,vertical} cases
(when --width is not set to 0),
to avoid two calls to this function per file.
Note the only case where we later compute the width,
is for --format=commas. That's only done once though,
so we leave the computation close to use to
maximize hardware caching.
* src/ls.c (struct fileinfo): Add a WIDTH member to cache
the screen width of the file name.
(update_current_files_info): Set the WIDTH members for cases
they're needed multiple times. Note we do this explicitly here,
rather than caching at use, so that the fileinfo
structures can remain const in the sorting and presentation functions.
(sort_files): Call the new update_current_files_info() in this
initialization function.
(fileinfo_name_width): Renamed from fileinfo_width,
and adjusted to return the cached value if available.
This helps identify the outliers for long filenames, and also produces
a more compact display of columns when listing a directory with many
entries of various widths.
* src/ls.c (sort_type, sort_types, sort_width): New sort_width sort
type.
(sort_args): Add "width" sort arg.
(cmp_width, fileinfo_width): New sort function and helper for file name
width.
(quote_name_width): Add function prototype declaration.
(usage): Document --sort=width option.
* doc/coreutils.texi: Document --sort=width option.
* tests/ls/sort-width-option.sh: New test for --sort=width option.
* tests/local.mk: Reference new test.
* NEWS: Mention the new feature.
* bootstrap.conf (gnulib_modules): Add idx.
* src/env.c: Include idx.h, minmax.h.
Prefer idx_t to ptrdiff_t when values are nonnegative.
(valid_escape_sequence, escape_char, validate_split_str)
(CHECK_START_NEW_ARG):
Remove; no longer needed now that we validate as we go.
(struct splitbuf): New type.
(splitbuf_grow, splitbuf_append_byte, check_start_new_arg)
(splitbuf_finishup): New functions.
(build_argv): New arg ARGC. Validate and process in one go, using
the new functions; this is simpler and more reliable than the old
approach (as witness the recent bug). Avoid integer overflow in
the unlikely case where the string contains more than INT_MAX
arguments.
(parse_split_string): Simplify by exploiting the new build_argv.
The assertions didn’t help catch the most recent bug which
was in their area, and kind of get in the way.
* src/env.c: Do not include <assert.h>, and remove all assertions.
These seem to have been put in to pacify gcov, but surely there’s
a better way.
(escape_char): Pacify GCC with 'assume' instead.
Problem reported by Frank Busse (Bug#47412).
* src/env.c (C_ISSPACE_CHARS): New macro.
(shortopts, build_argv, main): Treate all C-locale space
characters like space and tab, for compatibility with FreeBSD.
(validate_split_str, build_argv, parse_split_string):
Use the C locale, not the current locale, to determine whether a
byte is a space character.
* src/cksum.c (cksum_slice8): Fix bug on little-endian
platforms lacking __bswap_32: the SWAP macro evaluates
its argument multiple times, but the macro has a side effect.
The diacrit module is obsolete, and ptx’s use of it is obsolete
too; it assumes an 8-bit locale (not that common these days) and
that TeX cannot process the 8-bit characters (nowadays, it can).
* NEWS, doc/coreutils.texi (Charset selection in ptx): Document this.
* bootstrap.conf (gnulib_modules): Remove diacrit.
* src/ptx.c: Do not include diacrit.h.
(print_field, fix_output_parameters): Remove obsolete support
for 8-bit diacritics.
This behavior was introduced in commit FILEUTILS-4_0_44-4-g519b707b4.
* src/cksum.c (cksum_slice8): Only report the overflow, and continue.
* src/cksum_pclmul.c (cksum_pclmul): Likewise.
* src/cksum.c: (main): Use getopt_long to parse options,
and handle the new --debug option.
(pclmul_supported): Diagnose various failures and attempts.
* NEWS: Mention the new option.
Use cpuid to detect CPU support for hardware instruction.
Fall back to slice by 8 algorithm if not supported.
A 500MiB file improves from 1.40s to 0.67s on an i3-2310M
* configure.ac [USE_PCLMUL_CRC32]: A new conditional,
set when __get_cpuid() and clmul compiler intrinsics are supported.
* src/cksum.c (pclmul_supported): A new function using __get_cpuid()
to determine if pclmul instructions are supported.
(cksum): A new function refactored from cksum_slice8(),
which calls pclmul_supported() and then cksum_slice8()
or cksum_pclmul() as appropriate.
* src/cksum.h: Export the crctab array for use in the new module.
* src/cksum_pclmul.c: A new module to implement using pclmul intrinsics.
* po/POTFILES.in: Reference the new cksum_pclmul module.
* src/local.mk: Likewise. Note we build it as a separate library
so that it can be portably built with separate -mavx etc. flags.
* tests/misc/cksum.sh: Add new test modes for pertinent buffer sizes.
build-aux/gen-single-binary.sh (override_single): A new function
to refactor the existing mappings for dir, vdir, and arch.
This function now also sets the DEPENDENCIES variable so that these
dependencies can be maintained later in the script, where
we now propagate the automake generated $(src_$cmd_DEPENDENCIES)
to our equivalent src_libsinglebin_$cmd_a_DEPENDENCIES.
This will ensure that any required libs are built,
which we require in a following change to cksum that
builds part of it as a separate library.
GNU/Linux is unusual here in that rmdir("symlink/") returns ENOTDIR,
whereas Solaris and FreeBSD at least, will follow the symlink
and remove the target directory. We don't make the behavior
on Linux kernels consistent, but at least clarify
the confusing error message.
* src/rmdir (main): Output a specific error message for the above case.
(remove_parents): In the error message, don't assume intermediate paths
are directories, as they could be symlinks.
* tests/rmdir/symlink-errors.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
This regression was introduced in commit COREUTILS-6_8-58-g553d347d3
* src/pr.c (init_parameters): Process tabs for multiple columns.
* tests/pr/pr-tests.pl: Add test cases.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/46422
- \r\n is common a line end combination
- catting such a file without options causes it to display normally
- overwriting the first char with $, loses info
* src/cat.c (cat): Convert \r preceeding a \n to ^M.
* tests/misc/cat-E.sh: New test.
* tests/local.mk: Reference new test.
* tests/misc/cat-proc.sh: Fix typo.
* doc/coreutils.texi (cat invocation): Mention the new behavior.
* NEWS: Mention the improvement.
This functionality regressed with the adjustments
in commit v8.25-4-g62e7af032
* src/split.c (bytes_chunk_extract): Account for already read data
when seeking into the file.
* tests/split/b-chunk.sh: Use the hidden ---io-blksize option,
to test this functionality.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/46048
Problem reported by David McCall (Bug#45886).
I introduced this problem when fixing Bug#14371.
* NEWS: Mention the fix.
* src/mkdir.c (struct mkdir_options): New members umask_ancestor,
umask_self, replacing umask_value.
(make_ancestor): Use them when temporarily adjusting umask.
(main): Set them, and set the umask to umask_self instead
of leaving it alone.
* tests/mkdir/perm.sh (tests): Add test case for bug.
* doc/coreutils.texi: Fix regexp cross-reference that had become
out-of-date (Bug#45749). Also, fix some obsolete references to
SunOS and to /usr/dict/words, and change “Linux” to “GNU/Linux”
where appropriate. Unfortunately the pipeline example gets more
complicated since /usr/share/dict/words is not sorted the way that
‘comm’ wants.
None of the coreutils man pages - but the two above - are using bold
setting for the references to other man pages in the SEE ALSO section.
* man/cat.x (SEE ALSO): Remove '\fB...\fP' setting.
* man/tac.x: Likewise, and add a reference to cat(1).
This file is a copy from gnulib and therefore should not get changed
by the yearly update.
* .x-update-copyright: Add pattern for the above file.
* doc/fdl.texi: Revert the previous change.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
A 100MB file improves from 2.50s to 1.80s on a Sparc T5220
A 100MB file improves from 0.54s to 0.13s on an i3-2310M
* bootstrap.conf: Explicitly depend on byteswap,
since now used directly by coreutils.
* src/cksum.c (cksum): Process in multiples of 8 bytes.
(main): Adjust for generation of expanded crctab.
* src/cksum.h: Split now larger crctab to separate header.
* src/local.mk: Reference the new header.
* NEWS: Mention the improvement.
* doc/coreutils.texi (seq invocation): Mention "inf" is supported,
and describe that it's handled specially to generate infinite
whole integer sequences. Also mention that such infinite generation
is supported for integer steps up to 200.
(sleep invocation): Give `sleep inf` as an example to sleep forever.
* src/seq.c: Add a comment on SEQ_FAST_STEP_LIMIT, to say it's
reflected in the texinfo description.
Chris Colohan wrote that the man page did not do enough to dispel
a common misunderstanding that “contributed to one of the scariest
outages Google has ever seen” (Bug#45258).
* doc/coreutils.texi (mkdir invocation):
* src/mkdir.c (usage): Document -m vs -p better.
* src/nl.c (main): Enforce the POSIX specified
behavior of assuming ':' is specified after a single
character argument to -d.
* tests/misc/nl.sh: Add a test case.
* NEWS: Mention the bug fix.
* doc/coreutils.texi (nl invocation): Mention the GNU extensions
of allowing arbitrary length and empty delimiter strings.
* src/nl.c (usage): Likewise.
* tests/misc/nl.sh: Add test cases for the GNU extensions.
* src/nl.c (main): Update the default delimiter characters
when passed two characters with --section-delimiter.
Avoid redundant copies for the body and footer delimiter strings,
and instead, just offset into the header string.
(check_section): Avoid redundant comparing of 2 bytes of memory
for an empty delimiter.
The format can be determined from --options or the locale,
so it's useful to output the format string being used.
* src/date.c (show_date): Show the output format
along with the date being shown.
* tests/misc/date-debug.sh: Adjust accordingly.
Addresses https://bugs.gnu.org/44960
When removing a directory fails for some reason, and that directory
is empty, the rm_fts code gets the return value of the excise call
confused with the return value of its earlier call to prompt,
causing fts_skip_tree to be called again and the next file
that rm would otherwise have deleted to survive.
* src/remove.c (rm_fts): Ensure we only skip a single fts entry,
when processing empty dirs. I.e. only skip the entry
having successfully removed it.
* tests/rm/empty-immutable-skip.sh: New root-only test.
* tests/local.mk: Add it.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/44883
This was needed before libselinux-2.3 (May 2014),
but modern releases have the correct const declarations.
* src/chcon.c: Remove se_const() wrapper.
* src/cp.c: Likewise.
* src/install.c: Likewise.
* src/mkdir.c: Likewise.
* src/mkfifo.c: Likewise.
* src/mknod.c: Likewise.
* src/system.h: Likewise.
* gnulib: update to pick up const correctness fixes in selinux stubs.
At least, I *think* they are false alarms. An SELinux expert eye
would be welcome.
* src/install.c (setdefaultfilecon): If selabel_lookup fails
due to either ENOTSUP or ENODATA, don’t diagnose the issue.
Problem reported by Kamil Dudka in:
https://lists.gnu.org/r/coreutils/2020-11/msg00050.html
* src/selinux.c: Don’t include die.h; no longer needed.
(computecon, defaultcon, restorecon): Propagate errno.
(defaultcon, restorecon): Don’t diagnose errors or exit, as that’s
the caller’s responsibility.
* src/selinux.c: selabel_lookup requires absolute paths
(while only older matchpathcon before libselinux < 2.1.5 2011-0826 did).
* po/POTFILES.in: Readd src/selinux.c since we now have
a translatable error message.
The previous commit introduced a couple of syntax-check failures.
* .gitignore (/lib/se-label.h): Add entry to silence the
sc_gitignore_missing check. Sort entries in C locale.
* po/POTFILES.in (src/selinux.c): Remove entry as this source doesn't
contain any translatable strings anymore; avoids a sc_po_check failure.
* src/mv.c: Replace tabs by spaces to avoid complaints by
sc_prohibit_tab_based_indentation.
Ubuntu 20.10 is using a newer version of libselinux that
complains that matchpathcon is obsolete. Rewrite the code
that it uses the recommended selabel_lookup instead.
* m4/jm-macros.m4 (coreutils_MACROS): Do not check for
matchpathcon_init_prefix, as it is no longer used.
* src/copy.c (set_file_security_ctx): Omit process_local arg,
as it is equivalent to !x->set_security_context. All callers changed.
* src/copy.h (struct cp_options): set_security_context is now of
type struct selabel_handle *, not bool. All uses changed.
* src/cp.c, src/install.c, src/mkdir.c, src/mkfifo.c, src/mknod.c:
* src/mv.c: Include selinux/label.h.
(main): Use selabel_open for set_security context.
* src/install.c (matchpathcon_init_prefix): Remove; now unused.
(get_labeling_handle): New static function.
(setdefaultfilecon, main): Use it.
(setdefaultfilecon): Do something regardless of
ENABLE_MATCHPATHCON, which seems to be a revenant macro.
(setdefaultfilecon): Use selabel_lookup instead of the obsolescent
matchpathcon. Report an error unless it fails due to ENOENT.
* src/local.mk (src_ginstall_CPPFLAGS): Remove.
* src/selinux.c: Include selinux/label.h
Do not include die.h, error.h, canonicalize.h.
(defaultcon, restorecon_private, restorecon):
New arg HANDLE. All callers changed.
Use selabel_lookup rather than matchpathcon.
(restorecon_private, restorecon): Don’t lose track of errno.
* src/selinux.c, src/selinux.h:
(restorecon): Don’t call ‘error’; that’s the caller’s job.
Use HAVE_SELINUX_LABEL_H, not HAVE_SELINUX_SELINUX_H,
in case there is some weird system with the former but not the latter.
* src/selinux.h (struct selinux_handle): Add forward decl.
* src/local.mk (src_ln_LDADD, src_mktemp_LDADD, src_tac_LDADD):
Add $(LIB_CLOCK_GETTIME), since these use tempname which uses
clock_gettime if getrandom fails. On platforms like Solaris 10,
clock_gettime is not in the standard C library.
* doc/coreutils.texi (sort invocation): Mention explicitly
that --general-numeric-sort supports arbitrary format hex numbers,
but also mention that consistent case/width hex numbers can
be sorted faster with a standard sort.
This crash was identified by Cyber Independent Testing Lab:
https://cyber-itl.org/2020/10/28/citl-7000-defects.html
and was introduced with commit v8.5-163-g3f48829c2
* src/tr.c (validate_case_classes): Don't apply these
extra case alignment checks in the --complement case,
which is even more restrictive as to the contents of SET2.
* tests/misc/tr-case-class.sh: Add a test case,
for a large SET1, which caused the length adjustment
in validate_case_classes to underflow and trigger the assert.
* NEWS: Mention the bug fix.
This crash was identified by Cyber Independent Testing Lab:
https://cyber-itl.org/2020/10/28/citl-7000-defects.html
and was introduced with commit v6.9.90-11-g4245876e2
* src/ls.c (gobble_file): Ensure scontext is initialized
in the case where files are not statable.
* tests/ls/selinux-segfault.sh: Renamed from proc-selinux-segfault.sh,
and added test case for broken symlinks.
* tests/local.mk: Adjust for the renamed test.
* NEWS: Mention the bug fix.
GCC 10.1.1 without optimization gives:
error: ‘strncat’ argument 2 declared attribute ‘nonstring’
[-Werror=stringop-overflow=]
strncat (comment, UT_ID (utmp_ent), utmpsize);
Note the strncat man page says that:
"src does not need to be null-terminated
if it contains n or more bytes."
And the POSIX spec says that the second (source) parameter
is an array not a string.
So I think it's incorrect for strncat to require src be a string type.
This constraint seems to be being added to the gcc builtin strncat,
as specifiying -fno-builtin also avoids the warning.
Note specifying any optimization level also avoids the warning.
* src/who.c (make_id_equals_comment): Avoid the issue by using
stpcpy + stzncpy, instead of strcpy + strncat.
This pattern is used elsewhere in who.c
* src/local.mk: Ensure we map 2 hex digits to 4,
so that we don't output already handled Z3FOLD file system (0x33).
Also hide the generation command for src/fs.h.
* src/basenc.c (usage): Remove extraneous blank line,
to be consistent with other utilities that have options.
* src/realpath.c: Likewise.
* src/runcon.c: Likewise.
Addresses https://bugs.gnu.org/44248
gcc version 11.0.0 20201025 (experimental) warns that
src/sort.c:1655:1: warning: function might be candidate for attribute \
'pure' if it is known to return normally [-Wsuggest-attribute=pure]
* src/sort.c (limfield): Mark as pure.
The workaround triggers warnings from newer kernel versions in case
a user does not have sufficient privileges for the MTIOCGET ioctl.
* src/dd.c (skip_via_lseek): Drop wrapper function no longer needed.
(skip): Use lseek() directly.
(advance_input_after_read_error): Likewise.
Reported-by: Nir Soffer at https://bugzilla.redhat.com/1876840
Fixes https://bugs.gnu.org/44235
Previously we would have failed immediately upon internal overflow,
which didn't output the full line being processed, and assumed
there would be another numbered line.
* src/nl.c (line_no_overflow): A new global to track overflow.
(print_lineno): Only fail if about to output an overflowed number.
(reset_lineno): A new function to refactor resetting of the number,
and which also clears line_no_overflow.
* tests/misc/nl.sh: Add a test case.
* man/help2man: sync to changes from version 1.47.16.
Note this doesn't materially change the generated man pages.
Addresses https://bugs.gnu.org/44105
* gl/lib/randperm.c, src/cp-hash.c, src/ls.c, src/sort.c, src/tail.c:
Change all instaces of hash_delete to hash_remove to accommodate
change to Gnulib API.
* src/tail.c: Remove FIXME to follow a file name in a recreated
directory. The comment was added in commit v8.5-191-g61b77891c
while the fix (albeit not using inotify) was added in
commit v8.27-21-gba5fe2d4b
* src/stat.c (usage): Replace a mistaken semicolon with a colon,
and replace mistaken backticks with single quotes. Also reorder
some words, for clarity.
Fixes https://bugs.gnu.org/43707
* doc/coreutils.texi (timeout invocation): Avoid any implication
that `timeout --foreground` could be used to retroactively
timeout commands not already invoked by timeout(1).
Fixes bug https://bugs.gnu.org/42831
* src/csplit.c (process_regexp): Process the line suppression
in all invocations so that the last match is suppressed.
Previously with a non infinite match count,
the last regex pattern was not suppressed.
* NEWS: Mention the bug fix.
* tests/misc/csplit-suppress-matched.pl: Add a test case.
Fixes https://bugs.gnu.org/42764
Parts of this test expect that the rmdir syscall returns with EPERM,
but the root user does not see that.
* tests/rmdir/ignore.sh: Add uid_is_privileged_ guards around parts
of the test which expect rmdir() to fail with EPERM.
Reported by Nick Alcock <nix@esperi.org.uk> in
https://bugs.gnu.org/42633
* doc/coreutils.texi (timeout invocation): Document that the the
duration of --kill-after=DURATION begins when sending the initial
signal. Also mention that -k does not have any effect if timeout's
duration is 0.
Suggested by Jonny Grant <jg@jguk.org>.
* src/factor.c (mp_factor_using_division): Use mpz_fdiv_q_2exp
instead of its no-longer-documented mpz_div_2exp alias.
(print_factors): Use mpz_out_str instead of gmp_printf.
* src/factor.c (strto2uintmax): Instead of here ...
(print_factors): ... skip spaces and '+' here, so that
bignums are treated like non-bignums.
* tests/misc/factor.pl (bug-gmp-plus_2_sup_128_plus_1): New test.
This lets use assume multiple-precision arithmetic on all
platforms, simplifying the code.
* bootstrap.conf (gnulib_modules): Add libgmp.
* configure.ac: Don’t call cu_GMP, as this is now done by Gnulib.
* m4/gmp.m4: Remove.
* src/expr.c, src/factor.c: Use gmp.h unconditionally.
* src/factor.c: Use the simpler ‘#ifndef mpz_inits’ to
determine whether there is an mpz_inits macro.
* tests/cp/fiemap-FMR.sh: Avoid FICLONE ioctl,
which would avoid the point of the test (fiemap testing).
Also it avoids a valgrind bug with this ioctl:
https://bugs.kde.org/show_bug.cgi?id=397605
If it works, prefer lseek with SEEK_DATA and SEEK_HOLE to FIEMAP,
as lseek is simpler and more portable (will be in next POSIX).
Problem reported in 2011 by Jeff Liu (Bug#8061).
* NEWS: Mention this.
* src/copy.c (lseek_copy) [SEEK_HOLE]: New function.
(enum scantype): New constants ERROR_SCANTYPE, LSEEK_SCANTYPE.
(union scan_inference): New type.
(infer_scantype): Last arg is now union scan_inference *,
not struct extent_scan *. All callers changed.
Prefer SEEK_HOLE to FIEMAP if both work, since
SEEK_HOLE is simpler and more portable.
(copy_reg): Do the fdadvise after initial scan, in case the scan
fails. Report an error if the initial scan fails.
(copy_reg) [SEEK_HOLE]: Use lseek_copy if scantype says so.
* src/copy.c (extent_copy): New arg SCAN, replacing
REQUIRE_NORMAL_COPY. All callers changed.
(enum scantype): New type.
(infer_scantype): Rename from is_probably_sparse and return
the new type. Add args FD and SCAN. All callers changed.
* src/stat.c (human_fstype): Add case for the 'vboxsf' file system type
which is used for VirtualBox Shared Folders mounted in VirtualBox guest
VMs.
* NEWS: Mention the Improvement.
Fixes https://bugs.gnu.org/41935
Since -LONG_MIN results in LONG_MIN again, the operation itself is
a signed integer overflow.
This can be observed with the following calls (best if compiled
with -ftrapv or -fsanitize=undefined):
$ numfmt --padding=-9223372036854775808
$ seq 1e-9223372036854775808
Technically, the change in seq "reduces" the precision, but a double
or long double that small would be represented as 0 anyway.
* src/numfmt.c: Explicitly disallow --padding=LONG_MIN.
* src/seq.c: Treat 1e$LONG_MIN as 1e-$LONG_MAX.
* tests/misc/numfmt.pl: Add a test case.
* tests/misc/seq-precision.sh: Likewise.
Fixes https://bugs.gnu.org/41850
* doc/coreutils.texi (timeout invocation): Document that the exit
status is 137 when the KILL signal is used, regardless of whether that
signal is sent to COMMAND or timeout.
* src/timeout.c (usage): Likewise. Also split out and expand
on the possible exit status values to a separate table.
Discussed at https://bugs.gnu.org/41634
This makes for one Gnulib module less, and at runtime there’s
typically just one getrandom syscall instead of several for large
nonces.
* gl/lib/randread.c: Include sys/random.h instead of sys/time.h
and unistd.h.
(get_nonce): Use getrandom, not getentropy.
* gl/modules/randread (Depends-on):
Depend on getrandom, not getentropy.
* src/shred.c (main):
* src/shuf.c (main):
* src/sort.c (random_md5_state_init):
Say "getrandom" rather than "getentropy" in (unlikely) diagnostic.
Update gnulib submodule to latest and use its new features.
Gnulib’s new getentropy module means coreutils can now assume
getentropy instead of approximating it, badly in some cases.
Gnulib’s improvements to the tempname module mean coreutils no
longer needs to maintain private patches.
* bootstrap.conf (gnulib_modules): Remove gettimeofday.
* gl/lib/randread.c (NAME_OF_NONCE_DEVICE): Remove.
(get_nonce): Return success indicator. Remove bytes_bound arg.
All callers changed. Rewrite by using getentropy instead of
reading the nonce device and falling back on gettimeofday.
Fail if getentropy fails.
(randread_new): Return NULL (setting errno) if get_nonce fails.
All callers changed.
* gl/lib/tempname.c.diff, gl/lib/tempname.h.diff:
* gl/modules/tempname.diff: Remove.
* gl/modules/randread (Depends-on):
Depend on getentropy, not gettimeofday.
* src/ptx.c (swallow_file_in_memory):
* src/shuf.c (read_input):
Adjust to read_file changes in Gnulib.
* src/shred.c (main):
* src/shuf.c (main):
* src/sort.c (random_md5_state_init):
Diagnose the new form of randread_new failures: randread_new can
fail now when !random_source, meaning getentropy failed.
Since the previous gnulib update, bootstrap outputs this warning:
Notice from module fdl:
Don't use this module! Instead, copy the referenced license file \
into your version control repository.
See gnulib commit:
https://git.sv.gnu.org/cgit/gnulib.git/commit/?id=88fc5afbccc9
* bootstrap.conf (gnulib_modules): Remove 'fdl'.
* doc/fdl.texi: Add file as a copy of 'gnulib/doc/fdl.texi'.
* doc/.gitignore (/fdl.texi): Remove entry.
* cfg.mk (FILTER_LONG_LINES): Add pattern for the 'fdl.texi' file.
The previous attempt to skip that test on NFS (commit 4181fc5183)
made the test fail; it introduced two problems:
a) In the good case, i.e., when the subshell returns with exit status 0,
the test ran into framework_failure_.
b) As the subshell also runs with 'set -x', the later comparison of
/dev/null with 'err' would fail.
* tests/ls/removed-directory.sh: Revert to the style without subshell,
and add 'test -d .' to verify that 'ls' can read the removed dir.
* src/chown-core.c, src/comm.c:
* src/tsort.c (record_relation):
Remove GCC 10 pragmas that are not needed in GCC 10.1.0 (the first
public GCC 10 release) and that in some cases cause diagnostics
with GCC 10.1.0. The tsort.c change fixes a bug that was
inadvertantly introduced when these pragmas were added.
* src/env.c (build_argv): Add an assert() to avoid:
warning: use of NULL 'n' where non-null expected
[CWE-690] [-Wanalyzer-null-argument]
note: argument 1 of 'getenv' must be non-null
* src/dd.c (alloc_ibuf): Don't discard the allocated pointer, to avoid:
[CWE-401] [-Wanalyzer-malloc-leak]
(alloc_obuf): Likewise.
(cleanup): Deallocate the now tracked buffers which
also avoids "possibly lost" warnings from valgrind.
* src/tsort.c (search_item): Add asserts to avoid:
[CWE-690] [-Wanalyzer-null-dereference]
(record_relation): An assert doesn't suffice here,
so disable the warning for this function.
* src/comm.c: Suppress the following false positive for the whole file:
[CWE-457] [-Wanalyzer-use-of-uninitialized-value]
* src/chown-core.c: Suppress the following false positive for the file:
[CWE-415] [-Wanalyzer-double-free]
Have the `ls` `--classify` option take an optional argument for when to
classify ("always", "auto", "never"), just like the optional argument
for `--color`. When the optional argument is not specified, default to
"always" for backwards compatibility.
* src/ls.c (usage): Update help text.
(decode_switches): Support an optional argument for --classify.
* tests/ls/classify.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the new feature.
Pull in a fix for FTS to avoid a crash when traversing a heavily
changed XFS file system:
> fts: remove NOSTAT_LEAF_OPTIMIZATION
* NEWS (Bug fixes): Mention the fix.
* gnulib: Update to latest.
* bootstrap: Sync from gnulib/build-aux/bootstrap.
Discussed at:
<https://lists.gnu.org/r/bug-gnulib/2020-04/msg00068.html>
* tests/ls/removed-directory.sh: Remove host_triplet test.
Skip this test if one cannot remove the working directory.
From a suggestion by Bernhard Voelker (Bug#39929).
* NEWS: Mention this.
* src/ls.c: Do not include <sys/sycall.h>
(print_dir): Don't worry about whether the directory is removed.
* tests/ls/removed-directory.sh: Adjust to match new (i.e., old)
behavior.
* tests/misc/env-S.pl: `env -i env` will call the system env
due to the path being cleared, so pass the absolute path
of our env binary under test to avoid that. This was seen
to be an issue on Guix where /usr/bin/env was not available.
* src/basenc.c (z85_decode_ctx_init): Ensure we're working
with unsigned, as otherwise ubsan triggers with:
src/basenc.c:767:18: runtime error: signed integer overflow:
43 * 52200625 cannot be represented in type 'int'
(z85_encode): Likewise to avoid the usban error:
src/basenc.c:630:26: runtime error:
left shift of 134 by 24 places cannot be represented in type 'int'
* tests/misc/timeout-parameters.sh: Split the large timeout
handling to ...
* tests/misc/timeout-large-parameters.sh: ... here, so that
the 3 second delay is contained in its own test, and if
the test is skipped due invalid handling within timeout(1),
it will be more apparent.
Also adjust the check so we skip whenever the kernel timer
fires immediately, to handle the buggy OpenIndiana 11 kernel also.
Reported by Bruno Haible.
* init.cfg (require_bash_as_SHELL_): A new function to replace
SHELL for the current test, with bash if available.
This is useful on OpenIndiana 11 where /bin/sh was seen
to have races in handling of SIGPIPE.
* tests/misc/seq-epipe.sh: Use the new function to enforce bash.
* tests/misc/env-signal-handler.sh: Likewise.
Reported by Bruno Haible
* tests/ls/stat-free-color.sh: Check for the availability
of various stat calls individually, and add statx() and fstatat64()
to the list to check. Fix the stat counting logic to
ignore lines like "+++ exited with 0 +++".
* tests/ls/stat-free-symlinks.sh: Check syscalls other than stat().
* init.cfg (gcc_shared_libs_): New variable.
(gcc_shared_): Use it, instead of hardcoding -ldl.
(require_gcc_shared_): Determine the suitable value
for gcc_shared_libs_.
With these adjustments, all tests pass on macOS Catalina.
* tests/dd/sparse.sh: Adjust so that systems like apfs that
don't create holes < 16 MiB do not fail erroneously.
* tests/touch/trailing-slash.sh: Darwin was seen to dereference
symlinks to files when given a trailing slash, so avoid
that particular case.
* configure.ac: Reenable distribution of gzip-compressed
tarballs, for Guix bootstrapping reasons as discussed at:
https://lists.gnu.org/r/coreutils/2020-02/msg00042.html
* THANKS.in: Remove me, as now a committer.
* NEWS (Build-related): Mention this.
If the current directory has been removed, then "ls" confusingly
produced no output and no error message, indistinguishable from
running on an empty directory.
* src/ls.c (print_dir): Report ENOENT on GNU/Linux if readdir
finds no directory entries at all, not even "." or "..",
and a recheck with the getdents syscall returns ENOENT.
We recheck with getdents() as POSIX states that
"The directory entries for dot and dot-dot are optional".
* tests/ls/removed-directory.sh: New file.
* tests/local.mk (all_tests): Add new test.
* NEWS: Mention the change in behavior.
Reported by Owen Thomas.
* src/blake2/blake2-impl.h: Sync load16() implementation,
which doesn't change code generation.
Also leverage (builtin) memcpy to more efficiently
move data on little endian systems,
giving a 2% win with GCC 9.2.1 on an i3-2310M.
* src/longlong.h: Sync changes from:
https://gmplib.org/repo/gmp/log/tip/longlong.h
mips64: Provide r6 asm code as default expression yields.
arm32: Define sub_ddmmss separately for non-thumb (no rsc instruction).
powerpc: Add "CLOBBER" descriptions for some registers.
x86: Fix criterion for when to use mulx in umul_ppmm.
strcoll() is only significant to uniq(1) if it returns 0,
and it generally only does so with buggy locales or mismatched
locales and data. Some systems may have strcoll()
return 0 for equivalent normalized unicode forms,
but for consistency across platforms strcoll() is avoided.
The various cases are defined in the new test.
This is consistent with newer POSIX standards as discussed at:
https://www.austingroupbugs.net/view.php?id=963
* src/uniq.c: s/xstrcoll/memcmp/.
* tests/local.mk: Reference the new test.
* tests/misc/uniq-collate.sh: Add a new test.
* NEWS: Mention the change in behavior.
Fixes https://bugs.gnu.org/38627
* src/stat.c (usage): Mention permission bits rather than
"access" so there is no confusion with ACLs etc.
Also indicate we output the file type with '%A'.
* doc/coreutils.texi (stat invocation): Likewise.
Also indicate '%A' is similar to `ls -ld` output.
Addresses https://bugs.gnu.org/39613
Pick up recent build fixes to avoid sysctl.h inclusion on glibc systems,
restrict the max file size supported by read-file to PTRDIFF_MAX,
and to avoid a -Werror=unused failure in test-canonicalize.
* tests/cp/proc-short-read.sh: Switch to using /proc/cpuinfo,
rather than /proc/kallsyms which was seen to vary in some cases.
Fixes https://bugs.gnu.org/39357
Since v6.10-21-ged5c4e7 `rmdir --ignore-fail-on-non-empty`
had reversed the failure status for directories that failed
to be removed for permissions reasons. I.E. it would have
returned a failure status for such non empty dirs, and vice versa.
* src/rmdir.c (errno_may_be_non_empty): Rename from the
more confusing errno_may_be_empty(), and remove the EEXIST
case (specific to Solaris), which is moot here since
handled in errno_rmdir_non_empty().
(ignorable_failure): Fix the logic error so that
_non_ empty dirs are deemed to have ignorable failures.
(main): Fix clobbering of errno by is_empty_dir().
(remove_parents): Likewise.
* tests/rmdir/ignore.sh: Add a test case.
* THANKS.in: Add reporter who fixed the errno handling.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/39364
* configure.ac: Add -Wno-vector-operation-performance to suppress the
following gcc-9.2 error in gl/lib/randperm.c:
error: vector operation will be expanded piecewise
* src/ls.c (usage): Reorganize help for --time,
and add description for --time=birth.
(do_statx): Store btime in mtime if available.
(get_stat_btime): A new function to read the creation time
from the appropriate stat structure member.
(cmp_btime): A new function to compare birth time.
(print_long_format): Output '?' when birth time unavailable.
* doc/coreutils.texi: Document --time={birth,creation}.
* tests/local.mk: Reference the new test.
* tests/ls/birthtime.sh: Add a new test.
* NEWS: Mention the new feature.
* src/yes.c (main): Convert for loop to do-while in order to indicate
that the loop will be run at least once.
This avoids the following warning after the second loop:
src/yes.c:110:20: error: writing 1 byte into a region of size 0
* configure.ac: Set --with-openssl=auto-gpl-compat as the default,
so that openssl is used for md5sum etc., with openssl >= 3,
which is newly licensed under ASL v2.
* gnulib: Update to include "auto-gpl-compat" support.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
* doc/coreutils.texi (sleep invocation): Add an example to demonstrate
how to use the floating-point and the scientific notation to sleep
for sub-second times, e.g. milli-, micro- and nanoseconds.
Inspired by Stephane Chazelas in:
https://lists.gnu.org/r/coreutils/2019-12/msg00005.html
* src/chcon.c (main): Skip call of security_check_context()
in case SELinux is disabled to avoid unnecessary failure.
Bug: https://bugzilla.redhat.com/1777831
* doc/sort-version.texi (Minus/Hyphen and Colon characters):
Rename from “Minus/Hyphen @samp{-} and Colon @samp{:} characters”,
as texi2any 6.6 complains about colons in node names.
* doc/coreutils.texi (shred invocation):
Modernize discussion to today’s technology (Bug#38168).
* src/shred.c (usage): Omit lengthy duplication of the manual’s
discussion of file systems and storage devices, as that became out
of sync with the manual. Instead, just cite the manual.
This addresses a longstanding "update all callers" FIXME in
lib/xstrtol.c, by having programs check that numbers do not
have unknown suffixes. The problem was also reported for
'shuf' by my student Maggie Huang while reimplementing a shuf
subset in Python as an exercise in UCLA Computer Science 35L:
https://web.cs.ucla.edu/classes/fall19/cs35L/assign/assign3.html
This patch also improves the portability of the code to unusual
platforms where ULONG_MAX < SIZE_MAX.
* NEWS: Mention user-visible changes.
* src/chgrp.c (parse_group):
* src/chroot.c (parse_additional_groups):
* src/du.c (main):
* src/install.c (get_ids):
* src/join.c (string_to_join_field):
* src/ls.c (decode_switches):
* src/md5sum.c (split_3):
* src/shuf.c (main):
* src/sort.c (specify_nthreads):
* src/uniq.c (size_opt, main):
Use uintmax_t instead of unsigned long, for portability
to oddball platforms where unsigned long is not wide enough.
* src/du.c (main):
* src/expr.c (mpz_init_set_str) [!HAVE_GMP]:
* src/install.c (get_ids):
* src/ls.c (decode_switches):
* src/mknod.c (main):
* src/ptx.c (main):
* src/shuf.c (main):
* src/sort.c (specify_nmerge, specify_nthreads):
Reject numbers with suffixes.
* src/md5sum.c (split_3): Simplify.
* gl/lib/randperm.c: Include randperm.h first, since it’s the API.
Include stdint.h, count-leading-zeros.h, verify.h.
(floor_lg): Rename from ceil_log (which was not actually
implementing the ceiling!) and implement the floor using
count_leading_zeros.
(randperm_bound): Use floor_lg, not ceil_log. Use uintmax_t
instead of size_t in case the size gets large on a 32-bit host.
* gl/modules/randperm (Depends-on): Add count-leading-zeros, stdint.
* configure.ac: When --enable-gcc-warnings is used, omit
-Wno-type-limits. The need for -Wno-type-limits has passed, now
that intprops.h uses builtin primitives for GCC 5 and later, given
that recent GCCs issue type-limits warnings only for non-constant
expressions. --enable-gcc-warnings is not intended for use with
old compilers, so we can drop -Wno-type-limits now.
‘shuf -r -n 0 file’ would mistakenly read from standard input.
Problem reported by my student Jingnong Qu while reimplementing a
shuf subset in Python as an exercise in UCLA Computer Science 35L:
https://web.cs.ucla.edu/classes/fall19/cs35L/assign/assign3.html
* NEWS: Mention the fix. Also, ASCIIfy a previous item.
* src/shuf.c (main): Fix bug.
* tests/misc/shuf.sh: Add a test case for the bug.
statx allows ls to indicate interest in only certain inode metadata.
This is potentially a win on networked/clustered/distributed
file systems. In cases where we'd have to do a full, heavyweight stat()
call we can now do a much lighter statx() call.
As a real-world example, consider a file system like CephFS where one
client is actively writing to a file and another client does an
ls --color in the same directory. --color means that we need to fetch
the mode of the file.
Doing that with a stat() call means that we have to fetch the size and
mtime in addition to the mode. The MDS in that situation will have to
revoke caps in order to ensure that it has up-to-date values to report,
which disrupts the writer.
This has a measurable affect on performance. I ran a fio sequential
write test on one cephfs client and had a second client do "ls --color"
in a tight loop on the directory that held the file:
Baseline -- no activity on the second client:
WRITE: bw=76.7MiB/s (80.4MB/s), 76.7MiB/s-76.7MiB/s (80.4MB/s-80.4MB/s),
io=4600MiB (4824MB), run=60016-60016msec
Without this patch series, we see a noticable performance hit:
WRITE: bw=70.4MiB/s (73.9MB/s), 70.4MiB/s-70.4MiB/s (73.9MB/s-73.9MB/s),
io=4228MiB (4433MB), run=60012-60012msec
With this patch series, we gain most of that ground back:
WRITE: bw=75.9MiB/s (79.6MB/s), 75.9MiB/s-75.9MiB/s (79.6MB/s-79.6MB/s),
io=4555MiB (4776MB), run=60019-60019msec
* src/stat.c: move statx to stat struct conversion to new header...
* src/statx.h: ...here.
* src/ls.c: Add wrapper functions for stat/lstat/fstat calls,
and add variants for when we are only interested in specific info.
Add statx-enabled functions and set the request mask based on the
output format and what values are needed.
* NEWS: Mention the Improvement.
* src/truncate.c (do_ftruncate): Simplify overflow checking,
and don’t rely on theoretically-nonportable assumptions
like assuming that OFF_MAX < UINTMAX_MAX.
* src/seq.c: (seq_fast): Accept STEP as a parameter and use that
to skip the output of generated numbers.
(main): Relax to using seq_fast for integer steps between 1 and 200.
For larger steps the throughput was faster using the standard
incrementing procedure.
(cmp): Use the equivalent but faster memcmp for equal len strings.
* tests/misc/seq.pl: Update fast path cases.
Addresses https://bugs.gnu.org/37241
The previous commit introduced a bug into the following syntax-check,
and thus effectively turned it off:
$ make sc_prohibit_test_calls_print_ver_with_irrelevant_argument; \
echo $?
prohibit_test_calls_print_ver_with_irrelevant_argument
fatal: cannot change to 'grep': No such file or directory
0
* cfg.mk (sc_prohibit_test_calls_print_ver_with_irrelevant_argument):
Remove changing directory, and pass $(srcdir) as argument to 'git -C'.
* cfg.mk (sc_prohibit_colon_redirection): Don't expect `|` to denote
the pipe character in git grep.
(sc_tests_executable)
(sc_case_insensitive_file_names)
(sc_some_programs_must_avoid_exit_failure)
(sc_prohibit_test_background_without_cleanup_)
(sc_prohibit_test_calls_print_ver_with_irrelevant_argument)
(sc_prohibit_test_ulimit_without_require_)
(sc_prohibit_test_background_without_cleanup_)
(sc_THANKS_in_duplicates)
*sc_prohibit_test_calls_print_ver_with_irrelevant_argument):
Don't expect builddir to be a descendant of srcdir.
(sc_strftime_check): Don't check file size against 0 when "N\nq\n" was
already put in the file.
* THANKS.in: Remove me.
Under certain circumstances seq prints an extra line when the output
format has custom format with characters following the printed numbers:
$ seq -f "%g " 1000000 1000000
1e+06
1e+06
This is due to the "print_extra_number" logic using strings to determine
whether a 'extra number' is needed, but only one string was trimmed
when using a custom printf format.
Prompted by https://lists.gnu.org/r/coreutils/2019-08/msg00001.html
* NEWS: Mention fix.
* src/seq.c (print_numbers): Trim the 'x0_str' string before comparing
it to the previous 'x_str' string.
* tests/misc/seq-extra-number.sh: Add this scenario.
* tests/local.mk (all_tests): Add new test.
* doc/sort-version.texi: Fix some typos, avoid overly long lines in
the generated PDF, enclose some sample strings in @samp{...} for better
readability, etc. This also avoids an sc-avoid-builtin error:
s/builtin/built-in/
* doc/sort-version.texi: New file.
* doc/local.mk (doc_coreutils_TEXINFOS): Add new file.
* doc/coreutils.texi: @include new file, replace previous "Details about
version sort" section.
When calling 'stat -c %N' to print the filename, don't explicitly
request the size of the file via statx(), as it may add overhead on
some filesystems. The size is only needed to optimize an allocation
for the relatively rare case of reading a symlink name, and the worst
effect is a somewhat-too-large temporary buffer may be allocated for
areadlink_with_size(), or internal retries if buffer is too small.
The file size will be returned by statx() on most filesystems, even
if not requested, unless the filesystem considers this to be too
expensive for that file, in which case the tradeoff is worthwhile.
* src/stat.c: Don't explicitly request STATX_SIZE for filenames.
Problem reported by Szőts Ákos (Bug#36291).
* NEWS: Mention this.
* src/od.c (skip): Try fseek even on files that do not have usable
sizes, falling back on fread if fseek fails.
When debugging an invalid date due to DST switching, the intermediate
'normalized time' should not be checked - its value can differ between
systems (e.g. glibc vs musl).
Reported by Niklas Hambüchen in
https://lists.gnu.org/r/coreutils/2019-05/msg00031.html
Analyzed by Rich Felker in
https://lists.gnu.org/r/coreutils/2019-05/msg00039.html
* tests/misc/date-debug.sh: Replace the exact normalized time
with 'XX:XX:XX' so different values would not trigger test failure.
* src/stat.c: Drop statbuf argument from out_epoch_sec().
Use statx() rather than [lf]stat() where available,
so a separate call is not required to get birth time.
Set STATX_* mask bits only for things we want to print,
which can be more efficient on some file systems.
Add a new --cache= command-line option that sets the appropriate hint
flags in the statx call. These are primarily used with network
file systems to indicate what level of cache coherency is desired.
The new option is available unconditionally for better portability,
and ignored where not implemented.
* doc/coreutils.texi: Add documention for --cached.
* man/stat.x (SEE ALSO): Mention statx().
* NEWS: Mention the new feature.
* src/split.c (set_suffix_length): Use a more standard
zero based logN calculation for the number of units.
* tests/split/suffix-auto-length.sh: Add a test case.
* THANKS.in: Mention the reporter.
* NEWS: Mention the fix.
Fixes https://bugs.gnu.org/35291
Problem reported by Hans Henrik Bergan (Bug#36007).
* NEWS: Mention this.
* src/dd.c (iclose, ifdatasync, ifstat, ifsync):
New functions, which are more careful about SIGINT.
(cleanup): Use iclose instead of close.
(finish_up): Process signals first.
(skip, dd_copy, main): Use ifstat instead of fstat.
(dd_copy): Use ifdatasync and ifsync instead of fdatasync and fsync.
Problem reported by Jakub Kulik (Bug#35713).
* NEWS: Mention this.
* configure.ac (DEV_FD_MIGHT_BE_CHR): New macro.
* src/copy.c (DEV_FD_MIGHT_BE_CHR): Default to false.
(follow_fstatat): New function.
(copy_internal): Use it.
* src/copy.h (XSTAT): Remove; no longer used.
The wording of the dd --help text suggests that output will be skipped
for sparse *input* blocks (i.e. that NUL-checking is done on input
blocks) while the code actually checks/skips all-NUL *output* blocks.[1]
* src/dd.c (usage): Update the --help text to clarify the above.
* tests/dd/sparse.sh: Ensure sparseness is controlled with obs.
[1]: https://superuser.com/a/1136358
Its support for the -include option is flaky. Problem reported by
Michael Osipov (Bug#35650). Plus, we could run into other
compilers that don’t support any option like -include. Change the
code so that -include is not needed. Although this causes us to
depart from the upstream version, we’re already doing that for
other reasons.
* configure.ac (USE_XLC_INCLUDE): Remove, as there’s no
guarantee a compiler will support something like -include.
* src/blake2/b2sum.c [HAVE_CONFIG_H]: Include <config.h>.
* src/local.mk (src_b2sum_CPPFLAGS): Add -DHAVE_CONFIG_H.
Do not use -include or a substitute.
Problem reported by Michael Osipov (Bug#35650).
* configure.ac: Use AC_LANG_WERROR to pay attention to compiler
and linker warnings when testing whether stdbuf will work.
* src/blake2/blake2.h (BLAKE2_PACKED):
Don’t assume __attribute__ ((packed)) works on non-Microsoft
compilers. Instead, assume it works only if we have good
reason to assume so, and fall back on Microsoft (or not packing)
otherwise. In practice, not packing is good enough and the
BLAKE2_PACKED macro is mostly just for documentation.
* src/basenc.c: Various minor style cleanups.
(struct base_decode_context): Do not use anonymous unions, as
they’re not in C99. Use a named union instead. All uses changed.
* src/system.h (X2NREALLOC, X2REALLOC, DECIMAL_DIGIT_ACCUMULATE):
Use verify_expr instead of verify_true, which has been removed.
(DECIMAL_DIGIT_ACCUMULATE): Remove unnecessary size check.
Problem reported by John Marino (Bug#34894).
* src/ln.c (main): Port ln -s to Solaris symlink function,
where symlink ("x", ".") fails with errno == EINVAL.
Some files are physically copied from gnulib, and should get sync'ed
after each update to latest gnulib. This was forgotten during recent
updates.
* COPYING: Merge from gnulib/doc/COPYINGv3.
* tests/init.sh: Merge from gnulib/tests/init.sh.
* tests/misc/test-N.sh: The subsecond values for atime and mtime
were potentially seen to differ on newlyl created files.
So we include the subsecond portion when comparing stat values.
* tests/misc/wc-nbsp.sh: Add gating checks for all characters,
as there are disparate classifications on various systems:
SunOS 5.10 treats \u202F, \u2060 as !iswprint()
SunOS 5.10 treats \u00A0, \u2007 as iswspace()
AIX 7.2, Darwin 17.4.0, NetBSD 7.1 treat \u2060 as !iswprint()
* tests/id/zero.sh: sed on OSX will output a \n even
if the input doesn't have a \n on the last "line".
So ensure we always have a trailing '\n' to avoid the disparity.
Testing by Assaf Gordon on OSX showed the atime wasn't
being updated when explicitly set back in time.
Also Debian 8.11 / mips64 was seen to not update the
mtime when truncating an empty file.
* tests/misc/test-N.sh: Isolate from different timestamping
behaviors of various (file) systems, by correlating
the timestamps with stat(1) before using `test -N`.
Very old makeinfo-4.13 fails with:
./doc/coreutils.texi:2286: Unknown command `hashchar'.
./doc/coreutils.texi:2286: Misplaced {.
./doc/coreutils.texi:2286: Misplaced }.
Reported Bernhard Voelker in
https://lists.gnu.org/r/coreutils/2019-03/msg00016.html .
* doc/coreutils.texi (basenc invocation): Replace @hashchar{} with
actual hash character. The special syntax is only required
when referring to #line directives.
* tests/misc/wc-nbsp.sh: FreeBSD and OS X don't
treat non breaking space as printable characters.
So use wc -L to determine printability before
testing non breaking space functionality.
* src/env.c (initialize_signals): A new function to initialize
the signals array on the heap, to avoid a build failure on
opensolaris, where SIGNUM_BOUND is not a constant.
* man/local.mk: commit f114495e added an extra check to ensure
a binary was working before using it to generate the man page.
However this was not working for the false(1) command,
and also one can generally specify that one should not
be using generated commands on the current system by passing
'cross_compiling=yes' to the configure invocation.
* src/env.c (main): Output blocked or ignored signals
before a command is executed.
* doc/coreutils.texi (env invocation): Add the option.
* tests/misc/env-signal-handler.sh: Add a test case.
* NEWS: Mention the new feature.
New options to set signal handlers for the command being executed.
--block-signal suggested by Paul Eggert in http://bugs.gnu.org/34488#71
--default-signal is useful to overcome the POSIX limitation that shell
must not override inherited signal state, e.g. the second 'trap' here is
a no-op:
trap '' PIPE && sh -c 'trap - PIPE ; seq inf | head -n1'
Instead use:
trap '' PIPE && sh -c 'env --default-signal=PIPE seq inf | head -n1'
Similarly, the following will prevent CTRL-C from terminating the
program:
env --ignore-signal=INT seq inf > /dev/null
See https://bugs.gnu.org/34488#8
* NEWS: Mention new options.
* doc/coreutils.texi (env invocation): Document new options.
* man/env.x: Add example of --default-signal=SIG usage.
(SEE ALSO): Mention sigprocmask.
* src/env.c (signals): New global variable.
(longopts): Add new options.
(usage): Print new options.
(parse_signal_params): Parse comma-separated list of signals, store in
signals variable.
(reset_signal_handlers): Set each signal to SIG_DFL/SIG_IGN.
(parse_block_signal_params): Parse command-line options.
(set_signal_proc_mask): Call sigprocmask to block/unblock signals.
(main): Process new options.
* src/local.mk (src_env_SOURCES): Add operand2sig.c.
* tests/misc/env-signal-handler.sh: New test.
* tests/local.mk (all_tests): Add new test.
* configure.ac: Check for statx(), available on glibc >= 2.28.
* src/stat.c (get_birthtime): Call statx() when available.
* NEWS: Mention the improvement.
* src/df.c (replace_problematic_chars): A new wrapper to be
more conservative in our replacement when not connected to a tty.
* tests/df/problematic-chars.sh: Add a test case.
* doc/coreutils.texi (node seq invocation): Clarify to use the tool
'yes'; otherwise the reader may interpret the sentence as if one
could pass 'yes' as the INCREMENT value.
* src/wc.c (iswnbspace): A new function to match
characters in this class.
(isnbspace): Likewise for single byte charsets.
(main): Initialize posixly_correct from the environment,
to allow disabling honoring NBSP in non C locales.
(wc): Call is[w]nbspace() along with is[w]space.
* bootstrap.conf: Ensure btowc is available.
* tests/misc/wc-nbsp.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the change in behavior.
The recent Gnulib update fixed Bug#34608; document and test this.
* NEWS: Mention the change.
* doc/coreutils.texi (Padding and other flags):
Update doc to cover new flag and other POSIX.1-2017 changes.
* tests/misc/date.pl (date-century-plus): New test.
For select programs which accept only --help and --version options
(in addition to non-option arguments), process these options before
any other options.
Before:
$ dd bs=1 --help
dd: unrecognized option '--help'
Try 'dd --help' for more information.
$ yes me --help
me --help
me --help
...
After:
Any occurrence of '--help' in the arguments (prior to '--') will
show the help screen.
Discussed in https://bugs.gnu.org/33468 .
* NEWS: Mention change.
* src/cksum.c, src/dd.c, src/hostid.c, src/hostname.c, src/link.c,
src/logname.c, src/nohup.c, src/sleep.c, src/tsort.c, src/unlink.c,
src/uptime.c, src/users.c, src/whoami.c, src/yes.c (main): Replace
parse_long_options() + getopt_long() calls with
parse_gnu_standard_options_only(); Remove <getopt.h> inclusion;
Remove empty 'struct long_options' variable;
* tests/misc/help-version-getopt.sh: Add test.
* tests/local.mk (all_tests): Reference it.
* src/sort.c (main): Adjust the debug info regarding locales,
to clarify that only textual comparisons are affected.
* tests/misc/sort-debug-warn.sh: Adjust accordingly.
Fixes https://bugs.gnu.org/34490
* src/comm.c (main): Output a warning right before exit,
in case previous errors have scrolled from view.
* src/join.c (main): Likewise.
* tests/misc/comm.pl: Addjust accordingly.
* tests/misc/join.pl: Likewise.
Fixes https://bugs.gnu.org/34347
* gnulib: Update to make the new strtold module available.
* bootstrap.conf: strtod is now a dependency of c-strtod,
which in turn is a dependency of cl-strtod. This treats
strtold and strtod similarly.
* gl/lib/cl-strtod.c: Adjust to assume strtold is available.
* tests/misc/sort-float.sh: Likewise.
* src/sort.c: Likewise.
(nan_compare): Adjust comment to indicate
we still have to init padding bits as per
https://sourceware.org/bugzilla/show_bug.cgi?id=13246
* src/seq.c (print_numbers): Only reset the locale if it
was successfully set originally.
* tests/misc/seq-locale.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the fix.
bootstrap.conf: Explicitly depend on select, rather than transitively.
* src/tail.c: Unconditionally include select.h as we use select()
outside inotify contexts now.
* src/extract-magic: Treat android like linux,
which fixes the build by ensuring the constants are defined.
* src/stat.c: Support all constants on android, including
the android specific "sdcardfs".
* src/tail.c: Fix inclusion of statfs headers to be independent
of inotify availability, as fremote() is used on linux even
if inotify has been disabled. Also enable fremote() on android.
* NEWS: Mention the improvment.
Fixes https://bugs.gnu.org/34239
These commands now accept floating-point numbers in the
current locale, as well as in the C locale.
Compatibility problem reported by Robert Elz.
* NEWS: Document this.
* bootstrap.conf (gnulib_modules): Add cl-strtod, cl-strtold.
Remove c-strtold.
* doc/coreutils.texi (Floating point, tail invocation)
(printf invocation, timeout invocation, sleep invocation)
(seq invocation): Document this.
* gl/lib/cl-strtod.c, gl/lib/cl-strtod.h, gl/lib/cl-strtold.c:
* gl/modules/cl-strtod, gl/modules/cl-strtold: New files.
* src/printf.c, src/seq.c, src/sleep.c, src/tail.c, src/timeout.c:
Include cl-strtod.h instead of c-strtod.
* src/printf.c (vstrtold):
* src/seq.c (scan_arg, print_numbers):
* src/sleep.c (main):
* src/tail.c (parse_options):
* src/timeout.c (parse_duration):
Use cl_strtold instead of c_strtold.
Problem reported by Robert Elz.
* doc/coreutils.texi (sleep invocation):
Say that arguments must be non-negative, which means they cannot
be arbitrary floating-point numbers. Mention POSIX, not
“historical implementations” that are no longer of practical
interest. List the extensions to POSIX.
* src/sleep.c (usage): Omit needless words, removing dubious
commentary about “most implementations” and incorrect commentary
about “arbitrary”. Details about exactly which numbers are
allowed can be found in the documentation.
* init.cfg (trap_sigpipe_or_skip_): A new function refactored from...
* tests/misc/printf-surprise.sh: ...here.
* tests/misc/seq-epipe.sh. Likewise.
* src/tail.c (die_pipe): Ensure we exit upon sending SIGPIPE.
* tests/tail-2/pipe-f.sh: Ensure we exit even if SIGPIPE is ignored.
* NEWS: Mention the bug fix.
Fail developer builds if VLAs are used,
as there are portability concerns to consider with them.
* configure.ac: Enable -Wvla which is implicit in the full list added.
* m4/jm-macros.m4: Define GNULIB_NO_VLA which disables use of
VLAs within gnulib code.
When building against an incompatible GLIBC version compared to that
on the build host, then running the just-built binary might fail
although it is the same platform - thus CROSS_COMPILING is false.
As a result, generating the man pages fails.
* man/local.mk (.x.1): Add a check to verify that running the utility
with --help succeeds, otherwise falling back to using 'dummy-man'.
* src/ls.c (is_linked_directory): A new function to
also consider symlinked directories.
(main): Rename check_symlink_color to check_symlink_mode,
and enable that with --group-directories-first.
(DIRFIRST_CHECK): Adjust to use is_linked_directory,
rather than just is_directory.
(gobble_file): Simplify to always update f->linkmode
if the stat() succeeds.
* tests/ls/group-dirs.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the change in behavior.
Suggested by Amin Bandali in
https://lists.gnu.org/r/coreutils/2018-12/msg00017.html
* src/tail.c: Fix the check_output_available check on AIX.
Note we don't use poll for all systems as the overhead
of adding the gnulib poll module wouldn't be worth it
just for this single use.
* tests/tail-2/pipe-f.sh: Fix the test which always passed
due to only the exit code of sleep being checked.
* NEWS: Mention the bug fix and rearrange alphabetically.
Fixes http://bugs.gnu.org/33946
Allocate the encoding/decoding buffers dynamically on the heap instead
of using variable-length-array (VLA) on the stack.
Discussed in https://lists.gnu.org/r/coreutils/2019-01/msg00004.html .
* src/basenc.c (do_encode,do_decode): Allocate inbuf/outbuf using
xmalloc, and free if using LINT.
* src/system.h: Adjust lines containing URLs so that
they don't wrap on 80 column terminals. One could also
use .UR macros, but these aren't universally available.
Note the adjustments here need to be compatible with
the pattern matching done in help2man.
Addresses https://bugs.gnu.org/33914
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
In the following invocation, 'a' is the input file, and 'b' is the extra
operand:
$ base64 a b
Report 'b' in the error message instead of 'a':
$ base64 a b
base64: extra operand 'b'
Discussed in https://lists.gnu.org/r/coreutils/2018-12/msg00008.html .
* src/basenc.c (main): If there is more than one non-option operand,
report the second one (assuming the first is a the input file name).
* tests/misc/base64.pl: Add tests.
* tests/misc/basenc.pl: Adjust expectedc error message in tests.
* NEWS: Mention bugfix.
Encodes/decodes data in various common formats:
base64,base64url,base32,base32,base16,base2,z85.
Discussed here:
https://lists.gnu.org/r/coreutils/2018-11/msg00014.htmlhttps://lists.gnu.org/r/coreutils/2018-12/msg00019.html
* AUTHORS: Add basenc.
* README: Reference the new program.
* NEWS: Mention the new program.
* build-aux/gen-lists-of-programs.sh: Add basenc.
* doc/coreutils.texi: (basenc invocation): Document the new command.
* man/.gitignore: Ignore the generated man page.
* man/basenc.x: A new template, with few examples.
* man/local.mk: Reference the new man page.
* scripts/git-hooks/commit-msg: Allow basenc as program prefix.
* src/.gitignore: Ignore the new binary.
* src/basenc.c:
(usage): Mention new options.
(main): Handle new options.
(isbase*, base*_length, base*_encode, base*_decode_ctx): Implement new
encoding/decoding formats.
* src/local.mk: Add new program.
* tests/local.mk: Add new test.
* tests/misc/basenc.pl: New tests.
* tests/misc/help-version.sh (basenc_setup): use '--version' for default
invocation (basenc errors with no parameters).
Problem reported for split by Scott Worley (Bug#33761):
* src/shred.c (do_wipefd):
Also report an error if ftruncate fails on a shared memory object.
* src/sort.c (get_outstatus): New function.
(stream_open, avoid_trashing_input): Use it.
* src/sort.c (stream_open):
* src/split.c (create):
If ftruncate fails, do not report an error
unless it is a regular file or a shared memory object.
* NEWS (Bug fixes): Mention the fix in commit 94d364f157.
While at it, remove duplicate "Changes in behavior" heading.
* tests/misc/sync.sh: Add a test with a write-only file for the fix.
Open a target directory and use its file descriptor in linkat,
symlinkat, etc. syscalls, instead of constructing long file names
by concatenating the target directory name to a basename.
This avoids O(N²) behavior with ‘ln F1 F2 ... Fn DIR’ when DIR is
a long file name with many slashes. It also avoids some races if
DIR is renamed while ln is running.
* bootstrap.conf (gnulib_modules): Add openat-safer.
* src/ln.c: Include fcntl-safer.h.
(O_PATHSEARCH): New constant.
(errno_nonexisting, target_directory_operand): Remove; no longer used.
(atomic_link, do_link): New arg DESTDIR_FD. All uses changed.
(do_link): New arg DEST_BASE. All uses changed.
(main): Open target directory and use its file descriptor
as DESTDIR_FD.
* src/echo.c (usage): Assert that STATUS is always EXIT_SUCCESS.
* tests/misc/echo.sh: Add further tests for all hex and escape and
escape characters.
To get coverage statistics, run:
make coverage -j 4 TESTS=tests/misc/echo.sh SUBDIRS=.
xdg-open doc/coverage/src/echo.c.gcov.frameset.html
* src/echo.c (main): Always enable backslash processing if
POSIXLY_CORRECT is set.
* tests/misc/echo.sh: Add (the first) test for the echo command.
* tests/local.mk: Reference the new test.
* tests/misc/printf.sh: Update a stale comment.
* doc/coreutils.texi (echo invocation). Mention that POSIXLY_CORRECT
now always enables backslash processing.
* NEWS: Mention the change in behavior.
Fixes https://bugs.gnu.org/32703
Issue identified by Eric Blake.
Bash knows 'test -N FILE'. Add it to GNU 'test' as well.
* src/test.c (unary_operator): Add a case for 'N'.
(usage): Document it.
* doc/coreutils.texi (node File characteristic tests): Likewise.
* NEWS (New features): Likewise.
* tests/misc/test-N.sh: Add a test.
* tests/local.mk (all_tests): Reference it.
Remove the function 'test_unop', as the cases therein are redundant to
those handled by 'unary_operator'; exception: the cases 'o' and 'N':
they had been present in test_unop and handling the commands
test -N STR
test -o STR
and
test x = x -a -N STR
test x = x -a -o STR
which ran into an error later on anyway.
With this commit, the error diagnostic will change from ...
$ /usr/bin/test -N STR
/usr/bin/test: extra argument '-N'
$ /usr/bin/test -o STR
/usr/bin/test: extra argument '-o'
... to ...
$ src/test -N STR
src/test: '-N': unary operator expected
$ src/test -o STR
src/test: '-o': unary operator expected
* src/test.c (test_unop): Remove.
(unary_operator): Fail with test_syntax_error in the default case.
(term): Directly call unary_operator.
(two_arguments): Likewise.
* tests/misc/test-diag.pl: Adjust error diagnostic.
* src/test.c (unary_operator): Remove case 'a'.
(test_unop): Likewise.
* NEWS (Changes in behavior): Document the change.
Discussed at https://bugs.gnu.org/33097
On my openSUSE:Tumbleweed system, I get a false positive test failure
in the above 'check-root' test because the group lists inside and
outside the chroot have a different order:
++ chroot --userspec=berny / id -G
++ id -G berny
+ test '100 454 457 480 492' = '100 480 492 457 454'
+ fail=1
* tests/misc/chroot-credentials.sh (num_sort): Add function to sort
group lists, and use it in the test cases which test multiple groups.
Previously, 'ln A B' did 'stat("B"), lstat("A"), link("A","B")'
where the stat and lstat were necessary to avoid hard-linking
directories on systems that can hard-link directories.
Now, in situations that prohibit hard links to directories,
'ln A B' merely does 'link("A","B")'. The new behavior
avoids some races and should be more efficient.
This patch was inspired by Bug#10020, which was about 'ln'.
* bootstrap.conf (gnulib_modules): Add unlinkdir.
* src/force-link.c (force_linkat, force_symlinkat): New arg for
error number of previous try. Return error number, 0, or -1 if
error, success, or success after removal. All callers changed.
* src/ln.c: Include priv-set.h, unlinkdir.h.
(beware_hard_dir_link): New static var.
(errnoize, atomic_link): New functions.
(target_directory_operand): Use errnoize for simplicity.
(do_link): New arg for error number of previous try. All callers
changed. Do each link atomically if possible.
(main): Do -r check earlier. Remove linkdir privileges so we can
use a single linkat/symlinkat instead of a racy substitute for the
common case of 'ln A B' and 'ln -s A B'. Set beware_hard_dir_link
to disable this optimization.
$ id root nobody
uid=0(root) gid=0(root) groups=0(root)
uid=99(nobody) gid=99(nobody) groups=99(nobody)
* src/id.c (main): Make variables opt_zero, just_group_list,
just_group, use_real, just_user global to be used in a new
function.
(print_stuff): New function that will print user and group
information for the specified USER.
When using -G option delimit each record with two NULs.
Restructure the code in the file to have global variables
followed by functions.
* tests/id/zero.sh: Add test cases to check the usage
of -z option with multiple users.
* tests/id/uid.sh: Add a test case to ensure all users
are queried in the presence of errors.
* doc/coreutils.texi: Document the interface changes.
* NEWS: Mention the new feature.
* doc/coreutils.texi (csplit invocation): Detail the behavior
with regexp patterns and negative offsets, which differs from
line number patterns, to avoid looping on the input. For example:
$ seq 50 | csplit -s - /15/-5 /12/
csplit: ‘/12/’: match not found
* doc/coreutils.texi (csplit invocation): Clarify that
portions of the input may be skipped and thus the input
may not be reproducible by just concatenating the output files.
Fixes https://bugs.gnu.org/32317
This improves on the earlier fix for the problem reported by
Chih-Hsuan Yen (Bug#32236), by also looking for other control
characters and for encoding errors.
* src/df.c: Include wchar.h and wctype.h instead of c-ctype.h.
(hide_problematic_chars): Process the string as multibyte.
Use iswcntrl, not c_iscntrl.
* src/df.c (hide_problematic_chars): Use c_iscntrl() as
passing 8 bit characters to iscntrl() is not supported on macOS.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/32236
* tests/rm/rm-readdir-fail.sh: Skip the test entirely on 32 bit,
so we avoid conflating the 32bit and 64 bit types, as that
triggers alignment issues (SIGBUS) on Gentoo sparc.
Fixes https://bugs.gnu.org/29886
* tests/cp/cp-a-selinux.sh: Use 'skip_' rather than the probably
undefined 'skip'.
* tests/du/2g.sh: Likewise.
* tests/install/install-Z-selinux.sh: Likewise.
* tests/misc/chcon.sh: Likewise.
* tests/misc/selinux.sh: Likewise.
* tests/mkdir/restorecon.sh: Likewise.
* cfg.mk (sc_prohibit-skip): A new syntax check to catch the issue.
* init.cfg (require_membership_in_two_groups_): This fixes a bug
introduced by me in v8.15-8-gdd0e4c562. Luckily, the consequence
of low-probability triggering the bug was the mere added backslash
in the diagnostic: "...but running id -G\ either...". It would be
triggered in a test failure for one who is a member of only one or
fewer groups.
* src/copy.c (copy_internal): Use the lint protected src_mode,
rather than accessing the src_sb again. Also unconditionally
populate src_sb when !x->move_mode and in lint mode.
Reported by Kamil Dudka with coverity and clang analyzer.
* src/env.c (main): Don't process '-' specially since
that causes an issue on the openbsd getopt implementation
where a lone '-' is now processed as an option, and anyway
it doesn't particuarly help diagnosing common shebang
usage issues. Also don't restrict the extra diagnostics
for shebang usage to the case with 3 arguments, as
further arguments can be passed to a script.
* tests/misc/env-S.pl: Adjust accordingly.
On OpenBSD 6.2, invalid single options produce error messages
without single quotes:
$ ./src/chroot -/
chroot: unknown option -- /
As opposed to other systems:
./src/chroot: invalid option -- '/'
Modify the grep search to accept this.
* tests/misc/usage_vs_getopt.sh (checkprg): Change the grep pattern
to accomodate no-single-quotes cases.
The module is not needed anymore (was used during development).
Despite being a Perl core module, platforms like CentOS don't install
it by default. Reported by Bruno Haible at
https://lists.gnu.org/r/coreutils/2018-06/msg00093.html.
* tests/misc/csplit-suppress-matched.pl: Remove Data::Dumper.
Add attribute 'malloc' to mpz_get_str to prevent
the following on GCC 8.1.1
src/expr.c:117:1: error: function might be candidate for attribute
'malloc' if it is known to return normally
[-Werror=suggest-attribute=malloc]
mpz_get_str (char const *str, int base, mpz_t z)
^~~~~~~~~~~
cc1: all warnings being treated as errors
* src/expr.c (mpz_get_str): Add _GL_ATTRIBUTE_MALLOC.
* gnulib: Update to latest.
* .gitignore: Add new entries.
* bootstrap.conf: Enable wchar-single, which will enable more
efficient replacements of wcwidth and mbrtowc, as we indicate
that the charset will no change between invocations of these functions.
* src/local.mk (fs_normalize_perl_subst): `make src/fs-magic-compare`
was reporting incorrectly that AFS was not being handled.
Add a mapping to our KAFS identifier.
* .gitignore: Add intermediate files from `make src/fs-magic-compare`
With an uninitialized variable 'fail', the unquoted use like
test $fail = 1
lead to the shell error
"unary operator expected".
The uninitialized 'fail' variable was a side effect of
https://git.sv.gnu.org/cgit/gnulib.git/commit/?id=e91c0d4f9
which was pulled into coreutils-v8.26 with
https://git.sv.gnu.org/cgit/coreutils.git/commit/?id=ef9650170
Coreutils test code relied and relies on 'fail' to be initialized,
so initialize that variable here.
* tests/local.mk (TESTS_ENVIRONMENT): Initialize fail=0.
* src/remove.c (rm_fts): With the --preserve-root=all extension,
reject command line arguments that are mount points.
* src/remove.h (rm_options): Add preserve_all_root to store config.
* src/mv.c (rm_option_init): Init preserve_all_root to false.
* src/rm.c (main): Init preserve_all_root as per option.
(usage): Describe the new option.
* src/remove.c (rm_fts): Lookup the parent device id,
and reject the cli argument if a separate file system.
* tests/rm/one-file-system.sh: Add a test case.
* NEWS: Mention the new feature.
This mode is currently the default, but most if not all users of
reflink-capable filesystems want --reflink=auto, which is often
encapsulated into an alias. Adding --reflink=never allows overriding
such an alias.
* doc/coreutils.texi (cp invocation): Describe the new option.
* src/cp.c: Support --reflink=never.
* tests/cp/reflink-auto.sh: Add a test case.
* NEWS: Mention the new feature.
Adopted from FreeBSD's env(1), useful for specifing multiple
parameters on a shebang (#!) script line, e.g:
#!/usr/bin/env -S perl -w -T
Discussed in https://lists.gnu.org/r/coreutils/2018-04/msg00011.html
* src/env.c (valid_escape_sequence,escape_char,scan_varname,
extract_varname,validate_split_str,build_argv,
parse_split_string): New functions.
(main): Process new option and call parse_split_string.
(usage): Mention new option.
* tests/misc/env-S.pl: Test new option from the command line.
* tests/misc/env-S-script.sh: Test new option from shebang scripts.
* tests/local.mk (all_tests): Add new tests.
* man/env.x (OPTIONS): Show a brief example of -S usage and point to
the full documentation for more information.
* doc/coreutils.texi (env invocation): Detail usage of -S/--split-string
option.
* NEWS: Mention new option.
Keep unset envvars (-uFOO) in an array for later deletion,
instead of reiterating over argv. Done in preparation for
'-S string' feature. Related to '-u' discussion in
https://lists.gnu.org/r/coreutils/2018-04/msg00013.html
* src/env.c (append_unset_var,unset_envvars): New functions.
(main): Use new functions.
* src/ls.c (get_color_indicator): s/STREQ_LEN/c_strncasecmp/
* src/dircolors.hin: Remove a now redundant entry.
* tests/ls/color-ext.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the change in behavior.
* doc/coreutils.texi (md5sum invocation): Describe the new option,
and how it's not supported by --check, and how it disables escaping.
* src/md5sum.c (delim): A new global to parmeterize the out delimiter.
(main): Don't enable file name escaping with -z, and output '\0'.
* tests/misc/md5sum-newline.pl: Add a test case.
* NEWS: Mention the new feature.
===== Benchmark setup (on GNU/Linux) ====
$ yes áááááááááááááááááááá | head -n100000 > mbc.txt
$ yes 12345678901234567890 | head -n100000 > num.txt
===== Before ====
$ time src/wc -Lm < mbc.txt
real 0m0.186s
$ time src/wc -m < mbc.txt
real 0m0.186s
$ time src/wc -Lm < num.txt
real 0m0.055s
$ time src/wc -m < num.txt
real 0m0.056s
==== After ====
$ time src/wc -Lm < mbc.txt
real 0m0.196s
$ time src/wc -m < mbc.txt
real 0m0.173s
$ time src/wc -Lm < num.txt
real 0m0.031s
$ time src/wc -m < num.txt
real 0m0.028s
* src/wc.c (wc): Only call wide variant functions like
iswprint() and wcwidth() for non is_basic() characters.
I.E. non ISO C "basic character set" characters.
This is especially significant on OSX where wcwidth()
is very expensive (about 10x in tests).
* NEWS: Mention the improvement.
Suggested by Eric Fischer.
Problem reported by Eric S. Raymond (Bug#31803).
* man/test.x: Add SYNOPSIS section, since help2man
understandably gets confused by the square brackets.
* src/ln.c (usage): Omit parenthetical "(Nth form)" in usage,
as it confuses doclifter.
This issue was introduced in commit v8.19-145-g24ebca6
* src/copy.c (copy_internal): With --no-preserve=mode,
only reset permissions for newly created files.
(copy_reg): Likewise.
* NEWS: Mention the fix.
* tests/cp/preserve-mode.sh: Add a test case.
Fixes https://bugs.gnu.org/31675
* tests/ls/abmon-align.sh: Base relative month adjustment
from the middle of the month, to avoid failures due
to months being repeated.
Fixes https://bugs.gnu.org/31644
Avoid warnings from: groff -b -e -mandoc -T utf8 -rF0 -t -w w -z
* man/du.x: Change ".BR" to ".B" if there is only one argument.
Protect an end-of-sentence indicator (.?!) with '\&'
if it does not mean an end of a sentence.
Change '--' to '\-\-' if it indicates an option.
* man/rm.x: Change '\=' to '='.
* src/copy.c (copy_internal): Don't fail immediately upon
getting ELOOP when running stat() on the destination,
rather proceeding if -f specified, allowing the link
to be removed. If the loop is not in the final component
of the destination path, we still fail but at the
subsequent unlink() stage.
* doc/coreutils.texi (cp invocation): Adjust wording to say
that --force doesn't work with dangling links, rather than
all links that can't be traversed.
* tests/cp/thru-dangling.sh: Add a test case.
* NEWS: Mention the change in behavior.
Discussed in https://bugs.gnu.org/31335
Ensure this _does_ recreate the symlink
Given "path1" and "path2" are on different devices.
$ touch "path1/file"
$ cd path2/; ln -s path1/file
$ cp -dsf path1/file .
Ensure this does _not_ overwrite file
$ touch file
$ ln -s file l1
$ cp -sf l1 file
* src/copy.c (same_file_ok): Remove device ids from consideration,
instead deferring to future EXDEV with --link or allowing
the first case above to work.
Also ensure that we do not exist this function too early,
when the destination file is not a symlink, which protects
against the second case.
* tests/cp/cross-dev-symlink.sh: Add a test for the first case.
* tests/cp/same-file.sh: Add a test for the second case above.
* NEWS: Mention the bug fixes.
* THANKS.in: Mention the reporters who also analyzed the code.
Fixes https://bugs.gnu.org/31364
* src/cp.c (target_directory_operand): Allow through inaccessible
arguments with -f or --remove.
* doc/coreutils.texi (cp invocation): Clarify that -f doesn't directly
impact the removal of non-traversable symlinks.
* tests/cp/dir-rm-dest.sh: Test the new behavior.
* tests/cp/thru-dangling.sh: Enforce -f behavior wrt symlinks.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/31335
* src/chmod.c: Deallocate the mode change array in dev mode.
* src/chown.c: Make chopt_free() actually deallocate, but
only call in dev mode.
* src/chgrp.c: Likewise.
* src/chown-core.h (chopt_free, gid_to_name, uid_to_name):
No longer const.
* src/make-prime-list.c (xalloc): Add malloc attribute.
* src/who.c (make_id_equals_comment): Work around GCC bug 85602
by using mempcpy rather than strncat. Although the old code
was correct, strncat raises so many hackles that it’s not
worth maintaining its use here.
* NEWS: Expand on the 8.28 description of how tail more
responsively reacts to closed output, and move from "Improvements"
to "Changed behavior".
* cfg.mk (old_NEWS_hash): Regenerate.
Fixes https://bugs.gnu.org/31225
* src/timeout.c (usage): Mention that a duration of 0 disables
the associated timeout, which is both concise info and useful
functionality as timeouts are frequently configured.
Increase max range from SIZE_MAX to UINTMAX_MAX, which will
allow cut to support line lengths up to the max file size
on all systems. The inherent SIZE_MAX limitation in cut was
removed with the enhancements in https://bugs.gnu.org/13127.
Also numfmt gets similarly increased --field ranges due to
shared code.
* src/cut.c: s/size_t/uintmax_t/.
* src/numfmt.c: Likewise.
* src/set-fields.c: Likewise.
* src/set-fields.h: Likewise.
* tests/misc/cut-huge-range.sh: Adjust accordingly.
* tests/misc/numfmt.pl: Likewise.
* NEWS: Mention the improvement.
Problem reported by Karl Berry (Bug#30963).
* NEWS: Mention this.
* src/ls.c (decode_switches): Implement this.
* tests/ls/a-option.sh: New file.
* tests/local.mk (all_tests): Add it.
This will impact relatively few languages,
and will make Arabic or Catalan etc.
output unambiguous abbreviated month names.
* src/ls.c (MAX_MON_WIDTH): Increase from 5 to 12.
* NEWS: Mention the bug fix.
* tests/ls/abmon-align.sh: Augment to check for ambiguous output.
Fixes https://bugs.gnu.org/30814
Enhanced XFS (EXFS) is a version of XFS maintained by HPE.
EXFS uses a unique magic number to allow the use of community
XFS, and EXFS filesystems at the same time.
* src/stat.c (human_fstype): Add file system ID definition,
and use "exfs" as the name.
* NEWS: Mention the Improvement.
Problem reported by John Wiersba (Bug#30718)
* src/stat.c (human_time): Avoid giving an integer constant
expression a name, as it runs afoul of a bug in IBM XL C/C++ for
AIX 12.01.0000.0002.
This issue was introduced in commit v8.19-145-g24ebca6
* src/copy.c (copy_internal): When setting default permissions
to use with --no-preserve=mode, only set executable bits for
directories or sockets.
* NEWS: Mention the fix.
* tests/cp/preserve-mode.sh: Add a test case.
Fixes https://bugs.gnu.org/30534
* doc/coreutils.texi: Use @example consistently
as we don't need the smaller or fixed width representation.
This is especially true for the synopsis of commands.
@smallexample is rendered left aligned for HTML
which is awkward to read with the center aligned main content.
This builds on a previous patch for mv atomicity (Bug#29961).
It merely improves performance; it does not fix bugs.
* src/copy.h (struct cp_options): New members last_file, rename_errno.
* src/copy.c (copy_internal): Support new rename_errno member
for the copy options. Avoid calling stat when new members
suggest it’s not needed.
(cp_options_default): Initialize new members.
* src/mv.c: Include renameat2.h.
(main): With two arguments, first call ‘renamat2 (AT_FDCWD, "a",
AT_FDCWD, "b", RENAME_NOREPLACE)’. Use its results to skip
remaining processing if possible; for example, if it succeeds
there is no need to stat either "a" or "b". Also, set
x.last_file when it is the last file to rename.
Problem reported by Kamil Dudka (Bug#29961).
* NEWS: Mention this.
* src/copy.c: Include renameat2.h.
(copy_internal): If mv, try renameat2 first thing, with
RENAME_NOREPLACE. If this works, skip most of the remaining code.
Also, fail quickly if it fails with EEXIST, and we are using -n.
In both chown and chgrp (which shares its code with chown), operating
on symlinks recursively has a window of vulnerability where the
destination user or group can change the target of the operation.
Warn about combining the --dereference, --recursive, and -L flags.
* doc/coreutils.texi (warnOptDerefWithRec): Add macro.
(node chown invocation): Add it to --dereference and -L.
(node chgrp invocation): Likewise.
See also: CVE-2017-18018
* NEWS: Mention these fixes.
* doc/coreutils.texi (cp invocation, mv invocation):
Mention that -n is silent, and that it overrides -u.
* src/cp.c, src/mv.c (main): -n overrides -u.
* doc/coreutils.texi: the documentation for the --dereference
flag of chown/chgrp states that it is the default mode of
operation. Document that this is only the case when operating
non-recursively.
* tests/misc/shred-remove.sh: AIX xargs defaults to using
'_' to indicate end of input, thus ignoring it.
Rather than specifying -E to avoid this behavior, simplify
by removing sed and xargs usage.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
* tests/tail-2/pipe-f.sh: Close stdout in a subshell
to ensure the current shell isn't impacted. Subsequent
piped commands like `echo foo | blah` were seen to fail
due to the previous closing of stdout.
Reported by Assaf Gordon.
* README: Document the new handling of 32 bit time_t,
with examples of how to build in 64 bit mode on AIX.
Also mention that GNU make is desired on AIX
due to its mishandling of the "[" target.
Suggested by Assaf Gordon.
* tests/misc/ptx.pl: Escape the '^' character which is
otherwise considered as a line continuation character.
* tests/misc/shred-remove.sh: sed doesn't support \n.
* tests/misc/timeout.sh: dash outputs the "Killed"
message to stderr rather than the terminal.
* tests/misc/usage_vs_getopt.sh: dash doesn't yet
support the POSIX proposed $'...' shell quoting syntax.
* src/ptx.c (fix_output_parameters): GCC 6.3.1 with
./configure --enable-single-binary would give:
error: assuming signed overflow does not occur
when simplifying conditional to constant [-Werror=strict-overflow]
if (file_index > 0)
So change the type of file_index to signed (size_t).
* src/tail.c (tail_bytes): On systems were blksize_t is unsigned
and the same size or wider than off_t (android for example),
our initialized (off_t) -1 would be promoted to unsigned before
comparison, and thus fail to follow the appropriate path.
* tests/tail-2/tail-c.sh: Add a test case.
* NEWS: Mention the fix.
This issue was introduced in commit v8.23-47-g2662702
Reported at https://github.com/termux/termux-app/issues/233
* m4/jm-macros.m4: Check for the header.
* src/dd.c: Avoid the workaround where the header
is not available (on non glibc systems).
* src/shred.c: Likewise.
* src/ls.c (usage): Clarify -k only applies to -s usage
and directory 'total' lines. Move the description
of TIME_STYLE out of the option section as it was awkward
to read and write there within 80 columns.
* man/chmod.x: Update the information to state one can
clear the setuid and setgid bits for directories numerically
using an additional leading '0' or a leading '='.
That has been supported since v8.15-64-g8931cdb.
Fixes https://bugs.gnu.org/29390
* doc/coreutils.texi (cp invocation): The language used
to describe recreating the file was a little confusing
as it mentioned opening a removed file.
Fixes https://bugs.gnu.org/29315
Verify that all options mentioned in usage are actually recognized
by the program.
* tests/misc/usage_vs_getopt.sh: Add test.
* tests/local.mk (all_tests): Reference it.
Co-authored-by: Pádraig Brady <P@draigBrady.com>
* tests/cp/preserve-mode.sh: This was the only use of awk,
which may not be available on the system resulting
in an ineffective test. Also the permissions bits for
directories were not being checked at all.
* src/timeout.c (main): Add short option character 'v' to getopt_long
call.
* tests/misc/timeout.sh: Run the test both for the long and the short
option.
* src/dd.c (iread): Handle read error with a non-aligned
file offset in the O_DIRECT case. This is not an issue
on XFS at least, but on EXT4 the final read will return
EINVAL rather than the expected 0 to indicate EOF.
* tests/dd/direct.sh: Test the iflag=direct case also.
* NEWS: Mention the improvement.
This is useful as handling in shell is complicated
with the varying exit status in the --kill-after case.
* src/timeout.c (main): Handle '-v' and store
COMMAND for the diagnostic.
(cleanup): Diagnose the signal name before sending.
(usage): Document -v, --verbose.
* doc/coreutils.texi (timeout invocation): Likewise.
* tests/misc/timeout.sh: Add a test case.
* NEWS: Mention the new feature
Fixes https://bugs.gnu.org/21760
* src/tail.c (tail_bytes): Try lseek(..., SEEK_END) when
we can't determine the file size.
* tests/tail-2/end-of-device.sh: Add a new root only test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
Paul Eggert suggested using lseek() (rather than ioctl(BLKGETSIZE64)).
Fixes https://bugs.gnu.org/29259
These templates instruct contributors not to use github, and instead
use the upstream GNU development resources. Discussed in
http://lists.gnu.org/archive/html/coreutils/2017-11/msg00007.html .
* .github/ISSUE_TEMPLATE.txt,
.github/PULL_REQUEST_TEMPLATE.txt: New files.
Older versions of 'makeinfo' choke on a missing reference:
./doc/coreutils.texi:14177: `Realpath usage examples' has no Up field\
(perhaps incorrect sectioning?).
makeinfo: Removing output file `doc/coreutils.info' due to errors; \
use --force to preserve.
* doc/coreutils.texi (realpath invocation): Add a menu referencing
the usage examples - introduced in v8.27-91-g7449f0d.
Suggested by L A Walsh in https://bugs.gnu.org/28763 .
* src/stat.c (fmt_terse_fs): Define format for --terse -f here.
(fmt_terse_regular): Define format for --terse here.
(fmt_terse_selinux): Likewise for when SELinux is enabled.
(default_format): Use the above constants.
(usage): Output the formats for the terse modes.
* src/df.c (main): stat() before open(), and avoid
the optional open when given a fifo argument.
* tests/df/unreadable.sh: Add a test case.
* NEWS: Mention the fix.
Fixes https://bugs.gnu.org/29038
* src/ls.c (DEFINE_SORT_FUNCTIONS): Apply _GL_ATTRIBUTE_PURE
to each strcmp-derived function definition, since GCC8 with
-Wsuggest-attribute=pure now warns it is needed.
* man/stat.x (SEE ALSO): Mention statfs(2) in addition to stat(2).
Note statfs() is generally used rather than statvfs(),
so we'll defer that reference to the SEE ALSO section of statfs(2).
Fixes https://bugs.gnu.org/28989
* tests/dd/nocache_eof.sh: Only run the O_DIRECT tests
when 512 byte alignment is supported. Otherwise with older
XFS on systems with > 1MiB pages, or on file systems not
supporting O_DIRECT, there would have been false failures.
* tests/dd/direct.sh: Clarify the skip message.
Previously with oflag=direct the call to invalidate_cache()
was not passed to the kernel, as it was less than a page size,
and a subsequent call was not made to invalidate the pending space.
Similarly with oflag=nocache the pending space at EOF was
not invalidated. Even though these amount to only a single page
in the page cache it can be significant. For example on
XFS before kernel patch v4.9-rc1-4-g0ee7a3f, O_DIRECT files
would have been read inefficiently if any pages were cached,
even if they were already synced to storage.
* src/dd.c (i_nocache_eof, o_nocache_eof): New bools used
to control when we want invalidate_cache(,0) to clear to EOF.
(cache_round): Use IO_BUFSIZE (currently 132KiB) to minimize
calls to the relatively expensive advise function, rather
than page_size. This also makes it clear that while the
kernel function operates on pages, this size is chosen for
performance reasons.
(invalidate_cache): Refactor to share more code between
input and output paths. Use i_nocache_eof and o_nocache_eof
rather than proxying off max_records. Ensure we
invalidate full pages when clearing to EOF as the kernel
will ignore any non complete pages. Fix the offset used
for the output path.
(dd_copy): Invalidate the cache of the input after the
offset is updated, for consistency and so we don't try to
invalidate before the start of the file. When we read
EOF on input, set flags so that we invalidate to EOF.
(main): Invalidate to EOF in more cases, by depending
on the i_nocache_eof and o_nocache_eof flags.
* doc/coreutils.texi (dd invocation): Clarify the alignment
and persisted caveats on the example applying "nocache"
to part of a file.
* tests/dd/nocache_eof.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Issue reported by Eric Bergen.
This should have been part of commit v8.28-17-gf926f7c
* src/stty.c (check_argument): Align line continuation chars,
and ensure the function macro is immune to usage with if/else.
Suggested by Jim Meyering and Paul Eggert.
This was a latent issue that became significant with
the addition of the -F option in FILEUTILS-3_16n-56-ge46a424
* src/stty.c (apply_settings): Refactor argument checking
to a function macro. Augment the argument check to ignore
NULLed out arguments (already processed -F).
* NEWS: Mention the fix.
* tests/misc/stty-invalid.sh: Add a test case.
Fixes https://bugs.gnu.org/28859
This fixes a regression from commit v8.26-39-g2f69dba
* src/timeout.c (block_cleanup_and_chld): Rename from block_cleanup
to indicate we also block SIGCHLD to avoid the race where SIGCHLD
fires between waitpid() polling and sigsuspend() waiting for a signal.
* NEWS: Mention the fix.
We inherit the signal mask from our parent process,
therefore ensure SIGCHLD is not blocked.
If SIGCHLD is blocked, sigsuspend() won't be interrupted
when the child process exits and we hang until the timeout (SIGALRM).
This fixes a regression from commit v8.26-39-g2f69dba
* src/timeout.c (install_sigchld): Ensure SIGCHLD is unblocked.
* NEWS: Mention the issue.
man pages change little between systems,
so falling back to distributed pages make sense
when cross compiling or lacking perl.
* man/local.mk: Add all man pages to EXTRA_DIST
so that they're distributed in the generated tarball.
Use the dummy-man page generator if cross compiling.
Set TZ to avoid a distcheck failure where man pages
used a diffent month than those rebuilt (with a .timestamp).
* man/dummy-man: Only fall back to generating a stub
if copying an existing man page fails.
* man/help2man: Sync portable TZ=UTC0 specification
from upstream help2man.
* NEWS: Mention the build-related change.
Fixes https://bugs.gnu.org/28574
* tests/split/filter.sh: Due to an invalid 'FILE = zero.in'
construct trying to initialize a FILE variable, it would
instead try to run the FILE command which is present on
macOS 10.13 with APFS.
We also remove a redundant duplicate test clause introduced
during a rebase, and simplify the piped timeout command,
to avoid requiring a subshell and associated quoting.
* THANKS.in: Add the reporter Jack Howarth.
Fixes https://bugs.gnu.org/28506
* tests/misc/expr.pl: Skip the quote varying tests in
the multi-byte locales as these tests aren't that interesting
in those locales. Also ERR_SUBST is already defined for
some tests so awkward to redefine to munge UTF8 quotes to ASCII.
Show offending argument instead of a generic 'syntax error' message.
Suggested by Bernhard Voelker in https://bugs.gnu.org/28461#13 .
* src/expr.c (syntax_error): Remove.
(required_more_args): New function.
(eval7, main): Replace syntax_error call with detailed die message.
* tests/misc/expr.pl: Add tests for new messages.
* doc/coreutils.texi: Remove doubled word.
* src/targetdir.c: Explicitly mark exported function.
* tests/local.mk: This is not a root only test.
* tests/mv/vulnerable-target.sh: Use returns_.
Introduced in v8.28-3-g44ccd1c
This was unintentionally removed in v8.27-60-g2ae1460
* src/shred.c (wipename): Interate through all name lengths.
* tests/misc/shred-remove.sh: Add test cases.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/28507
* NEWS, doc/coreutils.texi (Target directory): Document this.
* src/cp.c, src/install.c, src/ln.c, src/mv.c: Include targetdir.h.
(target_directory_operand): Use the new targetdir_operand_type
function to check for vulnerable target directories.
* src/cp.c (stat_target_operand): New function.
(target_directory_operand, do_copy): Use it.
* src/local.mk (noinst_HEADERS): Add src/targetdir.h.
(src_ginstall_SOURCES, src_cp_SOURCES, src_ln_SOURCES)
(src_mv_SOURCES): Add src/targetdir.c.
* src/targetdir.c, src/targetdir.h: New files.
* tests/mv/vulnerable-target.sh: New test.
* tests/local.mk (all_root_tests): Add it.
* src/ptx.c (find_occurs_in_text): Die with an appropriate error
diagnostic when the given regular expression returns a match of
length 0.
* tests/misc/ptx.pl (S-infloop): Add a test.
* NEWS (Bug fixes): Mention the fix.
Fixes https://bugs.gnu.org/28417 which was detected using
Symbolic Execution techniques developed in the course of the
SYMBIOSYS research project at COMSYS, RWTH Aachen University.
* tests/ls/hyperlink.sh: If the hostname or any part of
the absolute path would be changed due to URL encoding,
the test would fail. Therefore simplify to remove
these components of the URL from consideration.
* gnulib: The only change in this gnulib update
is the tagging of the fts-tests module as longrunning,
which gnulib-tool currently implicitly excludes.
This test was seen to take about 20s and 285MB.
Reported by Assaf Gordon on space restricted VMs.
* src/tty.c (main): Don't distinguish ENOTTY from other errors,
because isatty() doesn't portably distinguish errors.
Solaris returns ENOENT for all input errors for example.
Musl also returns ENOENT, and ENODEV may be returned as disscussed at:
http://openwall.com/lists/musl/2017/04/06/6
* tests/misc/tty.sh: Adjust accordingly.
On some setups the root:object_r:tmp_t context is invalid.
This does indicate a limitation in the test framework,
but for now we'll relax this to skipping the tests.
The tests still run on a Fedora 25 system for example.
* tests/cp/cp-a-selinux.sh: Upon chcon error, skip rather than ERROR.
* tests/install/install-Z-selinux.sh: Likewise.
* tests/misc/chcon.sh: Likewise.
* tests/misc/runcon-no-reorder.sh: Likewise.
* tests/misc/selinux.sh: Likewise.
* tests/mkdir/restorecon.sh: Likewise.
* src/tty.c (main): All systems mention that isatty()
man return EINVAL as well as (the POSIX compliant) ENOTTY.
Also Centos 6 was seen to return EINVAL from ttyname().
* tests/misc/tty.sh: Fix a test issue where we assume
standard input is always a valid tty.
Reported by Assaf Gordon on OpenSolaris 5.10 and 5.11,
and Centos 6.5
This reverts commit v8.27-97-g8cb06d4 because
the setsid() fallback was not implemented correctly
and disabling the ioctl was not a complete solution
to the security issue of the child being passed
the tty of the parent.
Given runcon is not really a sandbox command,
the advice is to use `runcon ... setsid ...`
to avoid this particular issue.
We only use one of statfs or statvfs for `stat -f`
and on the BSDs we use statfs which doesn't have the
f_namelen member. However on OpenBSD and later FreeBSD
systems statfs does provide f_namemax, so use that.
* NEWS: Mention the improvement for OpenBSD and FreeBSD.
* m4/stat-prog.m4: Check for f_namemax in the statfs struct.
* src/stat.c: Return '?' rather than '*' when we can't
determine the max length of the file system.
* tests/ln/sf-1.sh: This test was failing on all BSDs
due to '*' being returned for the max length which
caused the test to attempt to create 1Mi+1 names.
The test now uses a short name when we can't determine
the max name length to use.
Reported by Assaf Gordon on various BSD based systems.
Similar to the issue with SELinux sandbox (CVE-2016-7545),
children of runcon can inject arbitrary input to the terminal
that would be run at the originating terminal privileges.
The new libseccomp dependency is widely available and used
on modern SELinux systems, but is not available by default
on older systems like RHEL6 etc.
* m4/jm-macros.m4: Check for libseccomp and
warn if unavailable on selinux supporting systems.
* src/local.mk: Link runcon with -lseccomp.
* src/runcon.c (disable_tty_inject): A new function to
disable use of the TIOCSTI using libseccomp, or with setsid()
where libseccomp is unavailable.
* tests/misc/runcon-no-inject.sh: A new test that uses
python to make the TIOCSTI call, and ensure that doesn't succeed.
* tests/local.mk: Reference the new test
* NEWS: Mention the fix.
Addresses http://bugs.gnu.org/24541
Terminals such as iTerm2 and VTE based terminals
(as of version 0.49.1), support hyperlinks when
passed terminals codes as described at:
https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda
* src/ls.c (gobble_file): Allocate an absolute file name to output.
(quote_name): Output the absolute name with the appropriate codes.
(file_escape): A new function to encode file names as per rfc8089.
(main): Handle the new option and call the file_escape_init() helper.
Disable --dired when --hyperlink is specified.
(print_dir): Get the absolute file name here too, so that the
directory name can be linkified.
* NEWS: Mention the new feature.
* tests/ls/hyperlink.sh: Add a new test.
* tests/local.mk: Reference the new test.
* doc/coreutils.texi (ls invocation): Describe --hyperlink.
This saves about 0.5MB uncompressed from the tarball.
* Makefile.am: Following on from v8.26-34-g2c64bc8
update the oldest documented version to 8.18 which
is now about 5 years old. Also remove older ChangeLogs
that were previously thought to be for changes not
in the git history, but are adequately recorded upon review.
* build-aux/ChangeLog-2007: Remove file.
* lib/ChangeLog-2007: Likewise.
* m4/ChangeLog-2007: Likewise.
This is useful when chaining with other commands that run commands in a
different context, while avoiding using the shell to cd, and thus
having to consider shell quoting the chained command.
* NEWS (New features): Document the new option.
* doc/coreutils.texi (env invocation): Likewise.
* src/env.c (usage): Likewise.
(main): Implement the new option.
* tests/misc/env.sh: Test the new option.
When systemd is configured to automount a remote file system - see
'man systemd.automount(5)', then the mount point is initially
mounted by systemd with the file system type "autofs".
When the resource is used later on, then the wanted file system is
mounted over that mount point on demand.
'df -l' triggered systemd to mount the file system because it called
stat() on the mount point.
Instead of single-casing "autofs" targets, we can avoid stat()ing
all dummy file systems (which includes "autofs"), because those are
skipped later on in get_dev() anyway.
*src/df.c (filter_mount_list): Also skip dummy file systems unless
the -a option or a specific target are given.
* NEWS: Mention the fix.
Co-authored-by: Bernhard Voelker <mail@bernhard-voelker.de>
Fixes http://bugzilla.suse.com/show_bug.cgi?id=1043059
The 'readlink' node has '@findex realpath' in it. This results in
info doc/coreutils.info realpath
incorrectly jumping to the 'readlink' node (instead of the 'realpath'
node). Change it to @cindex instead.
* doc/coreutils.texi (readlink): Change '@findex realpath' to @cindex.
* src/realpath.c (usage): Explicitly say 'DIR' instead of 'FILE' for
--relative-{to,base} parameters, to avoid giving the impression
that regular files can be used as relative base.
* doc/coreutils.texi (realpath): Same.
* src/ls.c (gobble_file): Disable the optimization to avoid quoting
if the symlink target itself needs quoting. This was introduced
with the quoting alignment adjustments in v8.25-106-g01971c0
* tests/ls/symlink-quote.sh: Add a test.
* tests/local.mk: Reference the test.
* NEWS: Mention the fix.
commit v8.27-44-g18f6b22 was too aggressive in
only allowing inotify use with regular files. This will
support responsive processing of `tail -f fifo | ...`
* src/tail.c (any_non_regular): Adjust to allow FIFOs
since inotify supports these well.
* tests/tail-2/inotify-only-regular.sh: Adjust comment.
* tests/misc/seq-epipe.sh: Remove stale comment.
* tests/misc/sort-debug-warn.sh: musl doesn't indicate a set_locale()
failure with missing locales, so avoid a test portion in that case.
* tests/misc/wc-files0.sh: Avoid a bug on older ash implementations.
Addresses http://bugs.gnu.org/28054
Problem reported by Lukas Zachar at:
http://bugzilla.redhat.com/1482445
* src/ptx.c (line_width, gap_size, maximum_word_length)
(reference_max_width, half_line_width, before_max_width)
(keyafter_max_width, truncation_string_length, compare_words)
(compare_occurs, search_table, find_occurs_in_text, print_spaces)
(fix_output_parameters, define_all_fields):
Use ptrdiff_t, not int, for object offsets and sizes.
(WORD, OCCURS): Use ptrdiff_t, not short int.
(WORD_TABLE, number_of_occurs, generate_all_output):
Prefer ptrdiff_t to size_t where either will do.
(total_line_count, file_line_count, OCCURS, fix_output_parameters)
(define_all_fields):
Use intmax_t, not int, for line counts.
(DELTA): Remove. All uses changed.
(OCCURS, find_occurs_in_text, fix_output_parameters):
Use int, not size_t, for file indexes.
(tail_truncation, before_truncation, keyafter_truncation)
(head_truncation, search_table, define_all_fields)
(generate_all_output):
Use bool for booleans.
(digest_word_file, find_occurs_in_text):
Use x2nrealloc instead of checking for overflow by hand.
(find_occurs_in_text, fix_output_parameters, define_all_fields):
Omit unnecessary cast.
(fix_output_parameters): Don’t assume integers fit in 11 digits.
(fix_output_parameters, define_all_fields):
Use sprintf return value rather than calling strlen.
(define_all_fields): Do not rely on sprintf to generate a string
that may contain more than INT_MAX bytes.
(main): Use xstrtoimax, not xstrtoul.
Use xnmalloc to catch integer overflow.
POSIX says sigprocmask has unspecified behavior in a multithreaded
program like ‘sort’.
* src/sort.c (pthread_sigmask) [GNULIB_defined_pthread_functions]:
New macro, for use when ‘sort’ is not multithreaded.
(cs_enter, cs_leave): Use it. Pass address, not value, as
this is typically a tad faster. All callers changed.
Use O_CLOEXEC when creating file descriptors, so that subsidiary
processes do not inherit file descriptors that they do not need.
This is helpful for ‘sort’, as it is a multithreaded program that
forks and execs.
* bootstrap.conf (gnulib_modules): Add mkostemp, open, pipe2.
* src/sort.c (create_temp_file): Open temporary file with O_CLOEXEC.
(stream_open): Open the stream with O_CLOEXEC.
(pipe_fork): Create the pipe with O_CLOEXEC.
(check_output): Open the output file with O_CLOEXEC.
(main): Use xfopen/xfclose to handle --files0-from, so that
O_CLOEXEC is used properly. This is simpler anyway.
* tests/misc/sort-files0-from.pl: Adjust to change in diagnostic
wording.
* src/operand2sig.c (operand2sig): AIX uses a different bit pattern
in the returned status from the wait() functions and from shells.
Therefore hardcode the selection of the lower bits of the number.
* NEWS: Mention the fix.
* configure.ac: Set USE_XLC_INCLUDE when __xlc__ is defined.
* src/local.mk: Use it to select the appropriate include option.
Reported by Michael Felt.
* tests/ln/sf-1.sh: Limit the symlink size to 1MiB
to avoid memory exhaustion seen on NFS on AIX, giving:
+ printf '%0*d' 4294967296 0
+ ./tests/ln/sf-1.sh: line 38: printf: warning: 0: Result too large
* tests/id/setgid.sh: Skip the test when the adjusted gid
would equal 4294967295, as that's reserved on AIX.
Reported by Michael Felt.
* src/sort.c (main): Don't assume hard_LC_COLLATE implies
a successful setting of the locale as musl defaults to
UTF8 when failing to set the specified locale.
* tests/misc/sort-debug-warn.sh: Adjust for the now
separated locale debug info and map the musl specific
message back to the common case.
Addresses https://bugs.gnu.org/28054
* src/seq.c (io_error): Use the same error message as would
be generated at exit time when closing the stdout stream.
The inconsistency was added with commit v8.25-26-gc92585b.
This was noticed due to an inconsistency in the expected
error message generated by seq on musl libc.
Addresses https://bugs.gnu.org/28054
* tests/misc/printf-surprise.sh: With musl libc the
large printf format does succeed, outputting data.
To avoid SIGPIPE being generated we ignore that signal
and then handle the subsequent EPIPE error.
Addresses https://bugs.gnu.org/28054
* configure.ac: Disable some new warnings to avoid false positives.
Building with warnings enabled and latest gcc would evoke build
failure without these changes. Disable the following in coreutils
proper: -Wformat-overflow=2 -Wformat-truncation=2, and
disable these for gnulib: -Wformat-truncation=2 -Wduplicated-branches
* src/shred.c (wipename): As per the comment, the arguments
to error() are sufficiently quoted, so split the call over
multiple lines to avoid the syntax-check.
Suggested by Kamil Dudka in:
http://lists.gnu.org/archive/html/coreutils/2017-07/msg00072.html
* NEWS: Document the changed nature of the fix.
* doc/coreutils.texi, tests/cp/backup-is-src.sh:
* tests/mv/backup-is-src.sh: Revert previous change.
* src/copy.c (source_is_dst_backup): New function.
(copy_internal): Use it. Fail instead of falling back on numbered
backups when it looks like the backup will overwrite the source.
Although this reintroduces a race, it's more compatible with
previous behavior.
* NEWS, doc/coreutils.texi (Backup options): Document the change.
* bootstrap.conf (gnulib_modules): Add backup-rename.
* src/copy.c (copy_internal): Silently switch to numbered backups
if a simple backup might lose data. Use backup_file_rename
to avoid races with numbered backups.
* tests/cp/backup-is-src.sh, tests/mv/backup-is-src.sh:
Adjust to match new behavior.
Use renameat2 to avoid a rename race condition, on recent-enough
GNU/Linux.
* bootstrap.conf (gnulib_modules): Add renameat2.
* src/shred.c: Include renameat2.h.
(wipename): Use renameat2 instead of rename.
* src/shred.c (dopass): shred used to read the input file,
and so needed to ensure internal memory was cleared.
This is no longer the case since SH-UTILS-1_16f-260-gf381610
so avoid this redundant clearing.
(do_wipefd): Likewise.
* NEWS: Remove the recent mention of this issue.
* tests/misc/sync.sh: Normalize the error messages
when syncing a non read/write directory, as AIX
gives the "Is a directory" error.
Also ensure that sync(1) returns an error for this
case on all systems.
Most programs take care to operate on all command-line-specified
operands before exiting. That is an important feature that allows
to identify all problems with the first run. However, groups would
exit upon the first problematic user name.
Bug introduced via commit v6.10-56-g167b8025ac.
* src/groups.c (main): Do not exit immediately upon error.
* tests/misc/groups-process-all.sh: New file. Test for this.
* tests/local.mk (all_tests): Add it.
* NEWS (Bug fixes): Mention this.
* tests/misc/groups-dash.sh: Avoid false failure on a system for which
"none" is a valid user name. The first invocation would succeed, and
the second would fail with "groups: ‘--’: no such user".
Use a user name that cannot exist.
Discussed in https://bugs.gnu.org/26779 .
* NEWS: Mention the improvement.
* bootstrap.conf: Add gnulib modules mbslen,mbschr.
* src/expr.c (mbs_logical_substr): New function to return a substring
based on logical character positions (instead of bytes).
(mbs_logical_cspn): Similar to strcspn/mbscspn, but returns number of
logical characters instead of byte offset.
(mbs_offset_to_chars): New function to return number of logical
characters fitting in a given byte offset.
(docolon): Report matched logical characters instead of bytes.
(eval6): For length/substr/index operations, use logical characters
instead of bytes by calling the above new functions.
* tests/misc/expr.pl: Repeat all tests with non-C locale to detect any
regressions.
* tests/misc/expr-multibyte.pl: New tests with multibyte input.
* tests/local.mk: Add new test file.
When the generated file, doc/constants.texi, happens to be older than
doc/coreutils.info, it will not be updated until/unless its generated
contents change. This is due to way that rule is careful to update
the file, to avoid provoking a pointless rerunning of makeinfo.
Note that this does not happen when one first runs "make distclean",
as recommended in README-release. However, I sometimes run it as
a more-rigorous "make check", and shouldn't have to manually run
"make distclean" first, in that case.
Before this change, one could reproduce the failure by running
`touch -dyesterday doc/constants.texi && make distcheck`. It would
fail with "makeinfo: could not open ../../doc/coreutils.info-t
for writing: Permission denied"
* Makefile.am (dist-hook): Touch the two generated files, so that
they cannot be out of date wrt doc/coreutils.texi.
* src/tail.c (any_non_regular): A new function to check passed files.
(main): Use the above to skip inotify if any non regular files passed
like /dev/tty or /dev/ttyUSB0 etc.
* tests/tail-2/inotify-only-regular.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/21265 and http://bugs.gnu.org/27368
* src/tail.c: (main): Only issue the warning about -f being
ineffective when we're not going into simple blocking mode.
* tests/tail-2/follow-stdin.sh: Ensure the warning is output correctly.
Fixes http://bugs.gnu.org/27368
This will support use cases like:
tail -f file.log | grep -q trigger &&
process_immediately
* src/tail.c (check_output_alive): A new function that
uses select on fifos or pipes to detect if they're broken.
(tail_forever): Call check_output_alive() periodically.
(tail_forever_inotify): Merge the select() call from
check_output_alive() into the select() originally present
for the --pid case, and adjust accordingly.
* tests/tail-2/pipe-f.sh: Add test cases.
* NEWS: Mention the improvement.
* NEWS: Mention the bug fix.
* src/tail.c (tail_forever_inotify): With --pid, avoid waiting
for new events if there are still events to process.
* tests/tail-2/inotify-dir-recreate.sh: Adjust to trigger.
* src/copy.c (copy_internal): Honor the x->require_preserve flag
for symlinks as we do for ordinary files, so we don't exit with
failure upon failure to chown a symbolic link.
* NEWS: Mention the bug fix.
* src/uptime.c (main): 00-23 was always used for the hour component
of the current time, so remove the AM/PM output (which was only
present in some locales anyway). Also add seconds to the time
to be more consistent with the usual procps-ng uptime implementation
on GNU/Linux.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/26783
* HACKING: Change from explicit instructions to using gnulib
provided coverage testing targets. Also include instructions
for adding root only tests to the report.
Fixes http://bugs.gnu.org/26709
Add a test for CVE-2017-7476 which was fixed in gnulib at:
http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=commitdiff;h=94e01571
* tests/misc/date-tz.sh: Add a new test which overwrites enough
of the heap to trigger a segfault, even without ASAN enabled.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Problem reported by Benno Schulenberg (Bug#26621).
* NEWS: Document this.
* src/dd.c (print_xfer_stats): With status=progress,
format times with %.0f rather than %g. Improve
translator comments.
* doc/coreutils.texi (Options for date): Capitalize a sentence.
* tests/misc/date-debug.sh: Adjust --debug output to match
recent changes to Gnulib’s parse-datetime module.
* src/shred.c (fillpattern): Fix the "off by one" issue when
testing whether we have enough space to copy the already
written portion of the buffer to the remainder of the buffer.
Specifically for buffer sizes that are (3*(2^x))+1, i.e. 7,13,...
we both use an uninitialized byte and invoke undefined
behavior in memcpy() operation on overlapping memory regions.
* tests/misc/shred-passes.sh: Add an invocation that will
trigger either valgrind UMR, or ASAN like:
ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges
#1 0x403065 in fillpattern src/shred.c:293
A direct test is awkward due to the random writes surrounding
the problematic pattern writes.
Fixes http://bugs.gnu.org/26545
* src/tail.c (tail_forever_inotify): Add the IN_DELETE_SELF flag when
creating watch for the parent directory. After the parent directory
is removed, an event is caught and then we switch from inotify to
polling mode. Till now, inotify has always frozen because it waited for
an event from a watched dir, which has been already deleted and was not
added again.
* tests/tail-2/inotify-dir-recreate.sh: Add a test case.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/26363
Reported at https://bugzilla.redhat.com/1283760
On some platforms, isatty succeeds but ttyname fails.
POSIX does not seem to allow this, but there it is.
Problem reported by Christian Brauner (Bug#26371).
While we’re at it, check for errors more carefully and return a
new exit status 4 if stdin is closed or a similar error occurs.
* doc/coreutils.texi (tty invocation): Document new behavior.
* init.cfg (stderr_fileno_):
Don't assume have_input_tty is not in the environment.
* src/tty.c (TTY_STDIN_ERROR): New constant.
(main): Exit with nonzero status if there is a usage error,
like other coreutils programs.
Check for error in getting stdin type.
* tests/misc/tty.sh: New file.
* tests/local.mk (all_tests): Add it.
* src/expand-common.c (emit_tab_list_info): A new function to
output the extended info on --tab=LIST, including the new
'+' and '/' prefixes.
* src/expand-common.h: Declare the above.
* src/expand.c (usage:): Call emit_tab_list_info and
match alignment with that used in unexpand --help.
* src/unexpand.c (usage): Likewise.
Support --tabs="1,+8" which is equivalent to --tabs="1,9,17,..."
useful for viewing unified diff output with its 1 character
gutter for example.
* doc/coreutils.texi ({expand,unexpand} invocation): Document,
using diff processing as the example.
* src/expand-common.c (set_increment_size): Update the new
increment_size global.
(parse_tab_stops): Handle the new '+' prefix.
(finalize_tab_stops): Verify both '+' and '/' prefixes
are not used together.
* tests/misc/expand.pl: Add test cases.
* NEWS: Mention the new feature.
* tests/misc/cut-huge-range.sh: On some systems returns_ may
use more memory, so incorporate that in the determination
of the ulimit value to use. Noticed on ARMv7 with bash-4.4.12,
and x86_64 with bash-4.2.37.
Fixes http://bugs.gnu.org/26253
* doc/coreutils.texi (split invocation): Document the new option.
* src/split.c (usage): Likewise.
(main): Process the new option much like --numeric-suffixes,
but with an adjusted alphabet.
* tests/split/numeric.sh: Refactor to support --hex mode.
* NEWS: Mention the new feature.
* src/md5sum.c (split_3): Verify hex digits internally before
triggering the global bsd_reversed mode flag.
(bsd_split_3): Likewise.
* tests/misc/md5sum-bsd.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/26263
* src/df.c (filter_mount_list): Avoid stat() on
explicitly excluded file systems, which is especially
significant in cases like `-x nfs` which may hang.
* NEWS: Mention the bug fix.
* src/expand-common.c (next_file): We're dependent on calling
this function with NULL to initialize things appropriately.
So enforce this with assert(), which avoids a warning from
clang-anaylzer.
* src/split.c (bytes_split): Don't write to an existing filter
if it has exited. When filters exit early, skip input data if
possible. Refactor out 2 redundant variables.
* tests/split/filter.sh: Improve test coverage given the
new more efficient processing. Also use a 10TB file to
expand the file systems tested on.
commit v8.25-4-g62e7af0 introduced the issue as it
broke out of the processing loop irrespective of
the value of new_file_flag which was used to indicate
a finite number of filters or not.
For example, this ran forever (as it should):
$ yes | split --filter="head -c1 >/dev/null" -b 1000
However this exited immediately due to EPIPE being propagated
back through cwrite and the loop not considering new filters:
$ yes | split --filter="head -c1 >/dev/null" -b 100000
Similarly processing would exit early for a bounded number of
output files, resulting in empty data sent to all but the first:
$ truncate -s10T big.in
$ split --filter='head -c1 >$FILE' -n 2 big.in
$ echo $(stat -c%s x??)
1 0
I was alerted to this code by clang-analyzer,
which indicated dead assigments, which is often
an indication of code that hasn't considered all cases.
* src/split.c (bytes_split): Change the last condition in
the processing loop to also consider the number of files
before breaking out of the processing loop.
* tests/split/filter.sh: Add a test case.
* NEWS: Mention the bug fix.
* tests/misc/sort-debug-keys.sh: Disparate LC_CTYPE and LC_MESSAGES
are not supported, with the result LC_MESSAGES=C is used throughout.
Therefore just set LC_ALL in the test, and normalize the message
variants with sed.
Reported and tested by J Rogowsky.
* src/local.mk [check-duplicate-no-install]: Depend on the
single-binary tr, or the system tr, as the edge case
where these are not available only result in the sanity
check being effectively ignored.
Switching to non-recursive makefiles broke the 'install-html' target:
The gettext plumbing requires an 'install-html' target in po/Makefile.
This was fixed in gettext v0.19.8.1-41-ge5a008a, but packages using
older gettext need to manually patch po/Makefile.in.in.
Reported (for 'sed') and suggested fix by Eric Blake in
https://bugs.gnu.org/25690 .
* bootstrap.conf (bootstrap_epilogue): Add 'install-{html,pdf,dvi,ps}'
targets to po/Makefile.in.in (if needed).
* bootstrap.conf: s/4.13/6.1/ as versions previous to that
generated invalid html with interspersed <span> tags that
were visible to the user. Version 6.1 is available for a
year now, and is available in most distros.
makeinfo issues the following:
doc/coreutils.texi:6568: warning: @sc argument all uppercase,\
thus no effect.
* doc/coreutils.texi (join invocation): Remove the @sc macro around
the all uppercase "GNU".
* src/timeout.c (install_sigchld): A new function to
install the SIGCHLD handler using sigaction() rather
than signal(), because with the latter on solaris
the signal handler is reset to default and thus
sigsuspend() only returns for the first finished child.
Reported by Assaf Gordon.
* src/libstdbuf.c: undef malloc so as libstdbuf is
not linked with gnulib, and anyway the replacement is
never needed since we never malloc(0).
Reported by Assaf Gordon.
* man/md5sum.x: Give a more direct warning againt the use
of this hash algorithm for security purposes.
* man/sha1sum.x: Likewise.
Suggested by Jim Meyering.
* src/expand-common (parse-tab-stops): Exit earlier upon overflow
so another warning isn't issued (on 32 bit) in add_tab_stop().
Flagged in https://hydra.nixos.org/build/49499970
* doc/coreutils.texi (sha1sum invocation): Given that a SHA-1
preimage attack has occurred as documented at http://shattered.io/,
document sha1sum as having the same limitations as md5sum.
(md5sum): Parameterize the warning for use in both cases.
* man/md5sum.x: Mention b2sum(1) as a more secure alternative.
* man/sha1sum.x: Give the same warning as done for md5sum(1).
* doc/coreutils.texi (expand invocation): Document the feature.
(unexpand invocation): Likewise.
* src/expand-common.c (extend_size): A new global to use
when the last tab stop is prefixed by '/'.
(set_extend_size): A new function to validate and set
the new extend_size global.
(parse_tab_stops): Call set_extend_size() for '/' prefixes.
(finalize_tab_stops): Ensure a single specified '/' is
treated like a standard tabsize, but also ensure that
when '/' is specified with a single other entry that
we process as a list rather than a tab size.
(get_next_tab_stop): Use the tab size if set,
for items after the user specified tab position list.
* tests/misc/expand.pl: Add test cases
* NEWS: Mention the new feature.
Fixes http://bugs.gnu.org/25540
This comes from the latest gnulib.
Also handling of OMP_NUM_THREADS has been adjusted
to support comma separated values indicating a nesting level,
in which case the first value is taken. Also OMP_NUM_THREADS=0
is now ignored instead of being treated as 1, to match
the behavior of libgomp.
* NEWS: Mention the OMP_THREAD_LIMIT improvement,
and OMP_NUM_THREADS now handling nested values.
* doc/coreutils.texi (nproc invocation): Describe OMP_THREAD_LIMIT
as a way to set the max value, with OMP_THREAD_LIMIT setting the min.
* tests/misc/nproc-override.sh: A new test to exercise the
updated gnulib code with all combinations of these OMP variables.
* tests/local.mk: Reference the new test.
* src/copy.c (set_process_security_ctx, set_file_security_ctx):
Export for use in cp.c.
* src/copy.h: Likewise.
* src/cp.c (make_dir_parents_private): Call the exported functions
to set the security context for new and updated directories.
* tests/cp/cp-a-selinux.sh: Add a test case.
Fixes http://bugs.gnu.org/25378
* bootstrap.conf, src/base64.c, src/cat.c, src/cksum.c:
* src/head.c, src/md5sum.c, src/od.c, src/split.c, src/sum.c:
* src/tac.c, src/tail.c, src/tee.c, src/tr.c, src/wc.c:
Adjust to renaming of the xsetmode module to xbinary-io,
and of the xsetmode function to xset_binary_mode.
This fixes a bug noted by Eric Blake. Code was using xfreopen to
change files to binary mode, but this fails for stdout when in
append mode. Such code should use xsetmode instead. This affects
only the port on platforms like MS-Windows which distiguish text
from binary I/O.
* bootstrap.conf (gnulib_modules):
Remove xfreopen and add xsetmode. Sort.
* src/base64.c (main):
* src/cat.c (main):
* src/cksum.c (cksum):
* src/head.c (head_file, main):
* src/md5sum.c (digest_file):
* src/od.c (open_next_file):
* src/split.c (main):
* src/sum.c (bsd_sum_file, sysv_sum_file):
* src/tac.c (tac_file, main):
* src/tail.c (tail_file):
* src/tee.c (tee_files):
* src/tr.c (main):
* src/wc.c (wc_file): Use xsetmode, not xfreopen.
* src/force-link.h: Don't include headers already included by system.h
* src/force-link.c: Likewise. Also include system.h and
explicitly mark extern functions as such.
If the file B already exists, commands like 'ln -f A B' and
'cp -fl A B' no longer remove B before creating the new link.
Instead, they arrange for the new link to replace B atomically.
This should fix a race condition reported by Mike Crowe (Bug#25680).
* NEWS, doc/coreutils.texi (cp invocation, ln invocation):
Document this.
* bootstrap.conf (gnulib_modules): Add symlinkat.
* src/copy.c, src/ln.c: Include force-link.h.
* src/copy.c (same_file_ok): It's also OK to remove a destination
symlink when creating symbolic links, or when the source and
destination are on the same file system and when creating hard links.
* src/copy.c (create_hard_link, copy_internal):
* src/ln.c (do_link):
Rewrite using force_linkat and force_symlinkat, to close a window
where the destination temporarily does not exist.
* src/cp.c (main): Do not set x.unlink_dest_before_opening
merely because we are in link-creation mode.
* src/force-link.c, src/force-link.h: New files.
* src/local.mk (copy_sources, src_ln_SOURCES): Add them.
* tests/cp/same-file.sh: Adjust test case to match fixed behavior.
The race is unlikely, as timeout(1) needs to receive a signal
in the few operations between waitpid() returning and exit().
Also the system needs to have reallocated the just released pid
in this time window.
Previously we never disabled the signal handler that sent
the termination signal to the "child" pid. However once waitpid()
has reaped the child, the system is free to allocate that pid,
so we must ensure we don't process any further signals.
* build-aux/gen-lists-of-programs.sh: Build timeout(1) optionally...
* configure.ac: ...predicated on sigsuspend() being available.
* src/timeout.c (block_cleanup): A new function to ensure the
cleanup() handler is disabled after waitpid has returned.
(main): Use sigsuspend() to wait with cleanup() enabled but
disabled once it returns, and thus disabled for the waitpid() call.
(monitored_pid): Change to the more accurate pid_t.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/25624
* doc/coreutils.texi (realpath invocation): Mention that realpath
is the preferred command for canonicalization.
(readlink invocation): Likewise.
* man/readlink.x: Likewise.
* src/tail.c (check_fspec): Only enable printing of the file header
if we've actually read some data and this is a new file. Also
move printing of the file header to...
(dump_remainder): ...here, to allow printing only when data read.
* tests/tail-2/overlay-headers.sh: A new test for suspension
and resumption of tail.
* tests/local.mk: Reference the new test.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/23539
* tests/tail-2/retry.sh: The replacement of the "missing" directory
is not atomic, and therefore tail(1) can take a different path,
especially if there is a delay between the rmdir(2) and creat(2).
This is noticeable for example with `make coverage` because in
that case the coverage files written by rmdir(1) on exit,
induce a significant delay thus triggering the issue.
* src/tail.c (recheck): Set f->ignore before we
use it to show the appropriate error.
* tests/tail-2/retry.sh: Ensure the "giving up" message
is not presented.
Remove old log files that have corresponding entries
in the source code repository.
This saves about 2.5MB uncompressed, 0.5M compressed.
* Makefile.am (gen-ChangeLog): Adjust to taking all
logs since a particular version (8.15 in this case).
Also mention in the truncated log where to get older entries.
(changelog_etc): Remove the no longer distributed files.
* build-aux/git-log-fix: Remove now unused entries.
* ChangeLog-200[5-8]: Delete.
* doc/ChangeLog-2007: Likewise.
* po/ChangeLog-2007: Likewise.
* old/*: Likewise.
* src/copy.c (punch_hole): Work around an empty definition
of HAVE_FALLOCATE which leads to a build error of:
"error: #if with no expression"
That was triggered by the inclusion of <linux/fs.h> in
commit v8.25-68-g89e1fef with kernel-headers-2.6.18.
Reported by Nelson H. F. Beebe
Problem reported by Paul Wise for Debian, in:
https://bugs.debian.org/851934
This is fallout from the fix for GNU Bug#23035.
* src/date.c (batch_convert): New args TZ and TZSTRING.
All uses changed.
(batch_convert, main): Adjust to parse_datetime2 API change.
(main): Allocate time zone object.
* tests/misc/date-debug.sh: Fix incorrect test case,
caught by the fix.
* tests/misc/date.pl: Test the fix.
* NEWS: Document this behavior, which comes with recent Gnulib.
* doc/coreutils.texi (Formatting file timestamps, du invocation)
(Time conversion specifiers, Setting the time, Options for date):
Mention when -00 is output for numeric time zones.
Be more careful about Internet RFC numbers, ISO 8601, etc.
A lot of this is converting http: to https:.
Also, gmane went away, so remove URLs that no longer work and
are not easy to figure out what they were.
Some of this stuff is so old that it no longer matters anyway.
The above new section looked a bit odd as the only general documentation
in between the utility chapters.
* doc/coreutils.texi (File timestamps): Move to a separate chapter.
Commit 4f650aad was incomplete; it changed NEWS but not the hash,
and introduced a grammar error.
* cfg.mk (old_NEWS_hash): Update via 'make update-NEWS-hash'.
* doc/coreutils.texi (File timestamps): Fix doubled word.
Prompted by a bug report from Scott Deerwester (Bug#25407).
* doc/coreutils.texi (File timestamps): New section.
Revamp other sections to use this new section, and
use more-consistent terminology.
* src/stty.c (apply_settings): A new function refactored
from main() that is used to both check and apply options.
(main): Call apply_settings before we open the device,
so all validation is done before interacting with a device.
* NEWS: Mention the improvement.
* tests/misc/stty.sh: Add a test case.
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
* tests/chgrp/basic.sh: On some NFS setups a user is
not allowed to set a group on a file even if a member
of that group. Therefore skip this test on remote file systems.
* tests/chgrp/default-no-deref.sh: Likewise.
* tests/chgrp/no-x.sh: Likewise.
* tests/chgrp/posix-H.sh: Likewise.
* tests/chgrp/recurse.sh: Likewise.
* tests/tail-2/inotify-rotate-resources.sh: Change to skipping
on remote file systems in the standard way.
* src/wc.c (wc): Avoid reading the end of the file
when the size is not a multiple of PAGE_SIZE,
as the special case handling for files in /proc and /sys
is only required when st_size is 0 or a multiple of PAGE_SIZE.
* tests/misc/wc-proc.sh: Add a test case.
While st_size would have been incorrect for subsequent
files since v7.1, it was only used since v8.24.
* tests/misc/wc-files0.sh: s/7.1/8.24/
* NEWS: Likewise.
Reported by Bernhard Voelker
* src/wc.c (main): Reset fstatus[0].failed between files when reusing
the fstatus[0] entry in --files-from0 mode. This ensures a stat() is
done for each file, avoiding incorrect counts and redundant reading.
* NEWS: Mention the bug fix.
* tests/misc/wc-files0.sh: Add a test case.
Fixes http://bugs.gnu.org/23073
When running "make check" on a Linux system running SELinux with a
non-MLS policy, tests/mkdir/restorecon.sh test fails with:
chcon: invalid context: root:object_r:tmp_t:s0: Invalid argument
Indeed in such a configuration, contexts cannot have ":s0" suffix.
* init.cfg (get_selinux_type): Refactor this function to here
from various tests. Update to work with a non-MLS policy.
(mls_enabled_): A new function to detect if MLS is enabled.
(skip_if_mcstransd_is_running_): Update to not skip when
MLS is not enabled.
* tests/mkdir/restorecon.sh: Use a valid non-MLS context when needed.
* tests/install/install-Z-selinux.sh: Likewise.
* tests/cp/cp-a-selinux.sh: Likewise.
* tests/misc/selinux.sh: Likewise.
* tests/misc/chcon.sh: Skip if non-MLS as --range used throughout.
Fixes http://bugs.gnu.org/22631
* tests/misc/ls-time.sh: Skip the test rather than ERROR
when `touch -m -d ...` fails (Hurd).
* tests/tail-2/follow-stdin.sh: Avoid false FAILs by ignoring
the variances in sterror output.
* tests/rm/rm-readdir-fail.sh: Likewise. Also avoid ERRORs
on systems that don't define _D_EXACT_NAMELEN.
* src/head.c (elide_tail_bytes_file): Ensure we don't use
st_size unless we've previously used seek() to determine
the CURRENT_POS in the seekable file.
This was seen to cause issue on FreeBSD 11 when the pipe
buffer was filled with `yes | head --lines=-0`, in which
case st_size was 64KiB while ST_BLKSIZE() was 4KiB.
Reported by Assaf Gordon.
... in the case where two or more directories nested in each other are
created and each of them defaults to a different SELinux context.
* src/install.c (make_ancestor): When calling defaultcon(), give it the
same path that is given to mkdir(). The other path is not always valid
wrt. current working directory.
* src/mkdir.c (make_ancestor): Likewise.
* NEWS: Mention the bug fix.
Reported at https://bugzilla.redhat.com/1398913
This was detected with ASAN, but can also be seen without ASAN with:
$ tac - - <&-
tac: standard input: read error: Bad file descriptor
*** Error in `tac': malloc(): memory corruption: 0x...
* src/tac.c (copy_to_temp): Don't close our output stream on
(possibly transient) output error, or on input error.
(temp_stream): clearerr() on the stream about to be reused,
to ensure future stream use is not impacted by transient errors.
* tests/misc/tac-2-nonseekable.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/25041
Reproduced under UBSAN with `tail -f <&-` giving:
tail.c:2220:18: runtime error: load of value 190,
which is not a valid value for type ‘_Bool'
* src/tail.c (tail_file): Ensure f->ignore is initialized
in all cases where we can't tail the specified file.
* tests/tail-2/follow-stdin.sh: Add a test case which
checks stderr has no UBSAN warnings.
Fixes http://bugs.gnu.org/25041
* NEWS: Mention in improvements about the workaround for
the glibc issue with closed stdin, and the new supported file systems.
* tests/misc/b2sum.sh: Spelling fix.
* src/longlong.h: Sync from gmp repo incorporating:
Protect umul_ppmm with do ... while (0)
Replace obsolete ARC asm 'J' constraints with 'Cal'
Provide umul_ppmm for riscv64
glibc has changed the public define
from _STRING_ARCH_unaligned to _STRING_INLINE_unaligned as per
https://sourceware.org/bugzilla/show_bug.cgi?id=19462
* gl/lib/rand-isaac.c: Cater for both defines.
* gl/lib/randread.c: Likewise.
* src/system.h: Update commented out code.
* tests/ls/quote-align.sh: Remove "total" line
which can vary per file system depending on allocation.
Reported by Assaf Gordon on OpenSolaris (5.11/5.10).
This was detected with:
echo a > a; pr "-S$(printf "\t\t\t")" a -m a > /dev/null
Resulting in ASAN triggering:
====================================================
ERROR: AddressSanitizer: global-buffer-overflow
READ of size 1 at 0x00000041b622 thread T0
#0 0x40506a in print_sep_string ../src/pr.c:2241
#1 0x407ec4 in read_line ../src/pr.c:2493
#2 0x40985c in print_page ../src/pr.c:1802
#3 0x40985c in print_files ../src/pr.c:1618
#4 0x4036e0 in main ../src/pr.c:1136
* src/pr.c (init_parameters): Ensure we only override the
specified separator when it's a single tab, thus matching
the calculated separator length.
* tests/pr/pr-tests.pl: Add a test case.
* NEWS: Mention the fix.
* src/ptx.c (fix_output_parameters): Ensure line_width doesn't
go negative, which can happen when the --width is less
than the --gap-size.
* tests/misc/ptx-overrun.sh: Add a test case that triggers
with ASAN. (Note the longer filename is needed to trigger).
Fixes http://bugs.gnu.org/25011
* tests/rm/rm-readdir-fail.sh: ASAN correctly indicated
that fts was writing to freed memory. This was because
we reused a single dirent in our readdir() test wrapper.
Since fts was deallocating those dirents, we now get
a new dirent for each call to our readdir wrapper.
ASAN reported this error for: split -n2/3 /dev/null
ERROR: AddressSanitizer: negative-size-param: (size=-1)
#0 0x7f0d4c36951d in __asan_memmove (/lib64/libasan.so.2+0x8d51d)
#1 0x404e06 in memmove /usr/include/bits/string3.h:59
#2 0x404e06 in bytes_chunk_extract src/split.c:988
#3 0x404e06 in main src/split.c:1626
Specifically there would be invalid memory access
and subsequent processing if the chunk to be extracted
was beyond the initial amount read from file (which is
currently capped at 128KiB). This issue is not in a
released version, only being introduced in commit v8.25-4-g62e7af0
* src/split.c (bytes_chunk_extract): The initial_read != SIZE_MAX
should have been combined with && rather than ||, but also this
condition is always true in this function so remove entirely.
* tests/split/b-chunk.sh: Add a test case.
Fixes http://bugs.gnu.org/25003
In recent commit v8.25-93-g7fc7206 we used the f->remote flag
which wasn't set in all cases. This was detected with
ASAN giving this error when reading f->remote;
runtime error: load of value 190,
which is not a valid value for type '_Bool'
* src/tail.c (fremote): Query the system even without inotify.
(recheck): Always set f->fremote for valid files.
Installing with -j2 or greater could result in a failure like
/bin/sh: line 29: /P/bin/install: Permission denied
when /P/bin/install specifies your PATH-selected install program.
This would arise because we're using "install" to install all
man/*.1 files, and that command would run concurrently with the one
that installs "/P/bin/install" itself. We would run this command:
"src/ginstall src/ginstall /P/bin/install", and it would result
in intervals during which the destination file does not exist,
is empty or incomplete and not executable. We addressed this
problem long ago for installation of actual binaries by telling the
installation rules to use our just-built bin/ginstall (only when not
cross-compiling) rather than the PATH-resolved "install" program.
This change is to do the same for those .1 files.
* src/local.mk (INSTALL): Override automake's default of something
like "INSTALL = /P/bin/install -c".
(INSTALL_PROGRAM): Now that we set INSTALL, there is no longer any
need to set this derived variable. Its default definition,
"INSTALL_PROGRAM = ${INSTALL}" does what we require.
Improved by Eric Blake.
* src/comm.c (total_option): Add bool variable for the new option.
(TOTAL_OPTION): Add enum value.
(long_options): Add array element for the new option.
(usage): Document the new option here.
(compare_files): Count the lines in total[3], and output the summary at
the end.
(main): Accept the new option.
* doc/coreutils.texi (comm invocation): Document it.
* tests/misc/comm.pl: Test it. While at it, improve the test data
to have 1 unique line in the first file, 2 unique lines in the second
file, and 3 common lines.
* NEWS (New Features): Mention the new option.
Fixes http://bugs.gnu.org/24929
* tests/misc/printf-quote.sh: FreeBSD 11 was seen to treat
\u0378 as a printable character. Therefore change to
using the \u0081 C1 control character. We use the UTF-8
representation because our printf implementation explicitly
disallows \u0081 as input.
Reported by Assaf Gordon
On BSD /bin/sh it was seen that unexported env vars passed to
returns_() would not be propagated to the wrapped command.
* cfg.mk (sc_prohibit_env_returns): Add a syntax check to disallow.
* tests/misc/csplit-io-err.sh: Rearrange to export vars in a subshell.
* tests/rm/rm-readdir-fail.sh: Likewise.
* tests/misc/nohup.sh: Export and unset vars around returns_.
* tests/misc/printenv.sh: Likewise.
Reported by Assaf Gordon
This provides better alignment when some names are quoted,
which also provides better indication that quotes are not
part of the name.
* src/ls.c (align_variable_outer_quotes): A new variable
set when ls is aligning columns (not using -m, non-zero -w),
and has a variable quoting style (shell, shell-escape, c-maybe).
(quote_name_buf): Writes to buffer rather than FILE,
taking care to avoid data copying if possible. Refactored from...
(quote_name): ...here. This now manages the buffer passed
to quote_name_buf() and outputs the padding, colors and name
in the appropriate order, while managing the --dired offsets.
(get_color_indicator): A new function to return the color sequence,
refactored from...
(print_color_indicator): ...here. This now simply outputs.
(print_dir): Refactor common parts to quote_name().
(clear_files): Reset the flag indicating at least one
file is quoted in the current directory.
(needs_quoting): A new function to indicate at the scan stage
whether a name needs quoting. Called from...
(gobble_file): ...here, until we find the first quoted file.
(print_name_with_quoting): Mostly refactored to quote_name().
* tests/ls/quote-align.sh: A new test for various output formats.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
Note we don't support the --algorithm option of the
b2sum command in the external BLAKE2 project, as that
was deemed too confusing for users.
"BLAKE2b" was chosen as the default algorithm to use,
which is single threaded but performs well on 64 bit.
* src/blake2: CC0 source copied from external project.
* cfg.mk[VC_LIST_ALWAYS_EXCLUDE_REGEX]: Exclude blake2/
from syntax checks, make update-copyright, etc.
* src/local.mk: Reference the sources for b2sum,
and set the compilation flags.
* doc/coreutils.texi (b2sum invocation): Reference the
md5sum invocation node, and add descriptions of -l.
* tests/misc/b2sum.sh: Add new test.
* tests/local.mk: Reference new test.
* AUTHORS: Add new binary.
* README: Likewise.
* build-aux/gen-lists-of-programs.sh: Likewise.
* man/.gitignore: Likewise.
* scripts/git-hooks/commit-msg: Likewise.
* man/b2sum.x: New man page template.
* man/local.mk: Reference new template.
* src/.gitignore: Ignore new binaries.
* src/blake2/.gitignore: Ignore new build atrifacts.
* src/md5sum.c (usage): Describe the new -l option.
* NEWS: Mention the new program.
* Makefile.am (my-distcheck): Remove all .deps directories before the
recursive diff that searches for left-behind files. Otherwise, with
automake master (some time after v1.15), "make distcheck" would fail
due to those directories being left behind after "make distclean".
* src/copy.h (cp_options): Add a new flag for install(1).
* src/copy.c (copy_internal): For cp, also output remediation
advice which also indicates why directories aren't copied by default.
The message is unchanged for install(1).
* src/cp.c (cp_option_init): Init install_mode to false.
* src/mv.c (cp_option_init): Likewise.
* src/install.c (cp_option_init): Init install_mode to true.
* tests/install/basic-1.sh: Add a test case.
* tests/cp/link-deref.sh: Adjust test case.
Fixes http://bugs.gnu.org/24958
* tests/misc/sort-h-thousands-sep.sh: The sv_SE locale
may be available while sv_SE.utf8 is not. Therefore
test and use the same locale, now definfed in a variable.
Reported by Jim Meyering.
The previous "returns_"-using change failed to convert many
uses of "$?". Convert all but two of the remaining ones.
* tests/ls/stat-vs-dirent.sh: Likewise.
* tests/misc/head-write-error.sh: Likewise.
* tests/misc/nice.sh: Likewise.
* tests/misc/nohup.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/misc/sync.sh: Likewise.
* tests/tail-2/pid.sh: Likewise.
* tests/tail-2/wait.sh: Likewise.
Thanks to Bernhard Volker for spotting this.
This reduces a standard coreutils install size by about 160K.
* src/cat.c: Change to proper_name() which removes about 18K text.
* src/cp.c: Likewise.
* src/df.c: Likewise.
* src/du.c: Likewise.
* src/getlimits.c: Likewise.
* src/realpath.c: Likewise.
* src/split.c: Likewise.
* src/stdbuf.c: Likewise.
* src/timeout.c: Likewise.
* src/truncate.c: Likewise.
* src/local.mk: Remove -llibiconv from the above programs.
* cfg.mk (sc_check-AUTHORS): Adjust to use factor(1).
* AUTHORS: Adjust to use ASCII to satisfy sc_check-AUTHORS.
The previous commit v8.25-96-g22063c8 lets 'make syntax-check' fail,
because the above check falsely detects a case where 'returns_' is
already in use.
* cfg.mk (sc_prohibit_and_fail_1): Add 'returns_' to exclude list.
* src/tail.c (tail_file): On failure to open a file,
set ignore=true when --retry is not specified.
* tests/tail-2/assert-2.sh: Adjust to the new behavior.
* tests/tail-2/retry.sh: Add a test case. Also change
from `tail ... && fail=1` to the more robust `returns_ 1 ...`
construct which detects segfaults etc.
* NEWS: Document the fix.
* src/tail.c (tail_forever): The BLOCKING optimization is only
enabled for non regular files (which can't be truncated), so ensure
we don't enable that unless we've a valid file descriptor.
* tests/tail-2/retry.sh: Add a test case.
* NEWS: Mention the bug fix.
* src/tail.c (tail_forever): Only read up to st_size on network
file systems to avoid the issue with a stale attribute cache
returning a smaller st_size than we have already read().
The was seen with glusterfs at least and caused the complete
file to be repeatedly output due to assuming the file was
truncated in this case.
* NEWS: Mention the fix.
* src/tail.c (ignore_pipe_or_fifo): Mark the descriptor as -1
for pipes so that any_live_files() detects correctly that
the entry is no longer live.
* tests/tail-2/pipe-f.sh: Add a test case.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/24903 which was detected
using Symbolic Execution techniques developed in
the course of the SYMBIOSYS research project at
COMSYS, RWTH Aachen University.
* doc/coreutils.texi(cut invocation): Give a more accurate description
of cut field handling limitations.
* tests/misc/stat-fmt.sh: Fix the test header to be more general.
* tests/tail-2/retry.sh: Spellings.
* src/system.h (emit_backup_suffix_note): A new function to
output the backup suffix info. The strings are unchanged,
so translations are not impacted.
* src/cp.c (usage): Use the new function.
* src/ln.c (usage): Likewise.
* src/mv.c (usage): Likewise.
* src/install.c (usage): Likewise.
* src/cp.c (main): Avoid the getenv("SIMPLE_BACKUP_SUFFIX") call,
which is now done if needed in the gnulib backupfile module.
Also avoid the redundant strdup, as we don't modify this suffix.
* src/install.c (main): Likewise.
* src/ln.c (main): Likewise.
* src/mv.c (main): Likewise.
Fixes http://bugs.gnu.org/23153
Incorporating these coreutils specific changes:
backupfile: initialize default suffix within the implementation
strftime,strptime: support %q to represent the quarter
Also sync these copies with gnulib:
* bootstrap: Now supports AIX.
* gnulib: Update to the latest.
* tests/init.sh: dash(1) is now disabled.
* doc/coreutils.texi (stat invocation): Describe the
QUOTING_STYLE values now supported.
* src/stat.c (getenv_quoting_style): A new function called
from main, that sets the default quoting style for quotearg.
(main): Call getenv_quoting_style() when %N specified.
* tests/misc/stat-fmt.sh: Add a test case.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/23422
* src/dd.c (parse_integer): Suggest to use "00x" instead of "0x",
which is significant for the "count", "seek", and "skip" operands.
* tests/dd/misc.sh: Add a test case.
Fixes http://bugs.gnu.org/24874
* NEWS: Mention the fix.
* src/md5sum.c (digest_file): Add a new MISSING parameter to
return whether the file was missing, separately from the digest.
* tests/misc/md5sum.pl: Add a test case.
Fixes http://bugs.gnu.org/24795
* src/expr.c (integer_overflow): Remove an abort-after-die.
* src/paste.c (write_error): Likewise.
* src/sort.c (badfieldspec): Likewise.
* src/tr.c (string2_extend): Likewise. Also remove a few
break statements, each after an abort.
die() has the advantage of being apparent to the compiler
that it doesn't return, which will avoid warnings in some cases,
and possibly generate better code.
* cfg.mk (sc_die_EXIT_FAILURE): A new syntax check rule to
catch any new uses of error (CONSTANT, ...);
* src/die.h (die): New file/function from grep.
Note: we expect this file to migrate to gnulib.
* src/csplit.c: Include die.h.
(check_format_conv_type): Use die in place of error-nonzero;break;
* src/install.c (strip): Likewise.
* src/nl.c (proc_text): Likewise. This also suppresses a new warning
from GCC 7's -Werror=strict-overflow.
* src/tail.c (parse_options): Likewise.
* src/basename.c (main): Adjust "fall through" comment
so that GCC 7's -Wimplicit-fallthrough honors it.
* src/cp.c (main): Add a "fall through" comment.
* src/ls.c (gobble_file): Likewise.
(get_funky_string): Adjust a "fall through" comment so it is
recognized.
* cfg.mk (exclude_file_name_regexp--sc_system_h_headers): Add die.h
to this list of exempt src/*.h files.
* src/pr.c (main): Avoid this warning from GCC 7:
src/pr.c:1119:6: error: assuming signed overflow does not occur when \
simplifying conditional to constant [-Werror=strict-overflow]
if (n_files == 0)
* src/rm.c (main): Ensure the full --no-preserve-root
option is specified, rather than allowing --n etc.
* tests/rm/r-root.sh: Add a test case.
* NEWS: Mention the change in behavior.
Improved by Jim Meyering.
Fixes http://bugs.gnu.org/24604
which was not the case when inotify was not available.
* src/tail.c (any_live_files): Simplify, since the IGNORE
flag is now only set when a file should be ignored indefinitely.
(recheck): Only output the "giving up on name" message
when that's actually the case. Only set the IGNORE flag
when ignoring a file indefinitely.
(tail_file): Likewise.
* tests/tail-2/retry.sh: Add a test case. Also run
all existing test cases with and without inotify.
NEWS: Mention the fix.
THANKS.in: Add the reporter.
Fixes http://bugs.gnu.org/24495 which was detected
using Symbolic Execution techniques developed in
the course of the SYMBIOSYS research project at
COMSYS, RWTH Aachen University.
* src/who.c (idle_string): This function would fail to compile
with -Werror and today's built-from-git gcc due to this warning:
src/who.c: In function 'print_user':
src/who.c:201:36: error: may write format character ':' at offset 4 \
past the end of the destination [-Werror=format-length=]
sprintf (idle_hhmm, "%02d:%02d",
^~~~~
The fix is to use an assertion to inform gcc of the existing
invariant that guarantees the number of hours is less than 24.
The old code mishandled --time-spec='+%%b', as it misinterpreted
the '%b' as being the month abbreviation. Also, it mishandled
the extremely unlikely case of a month abbreviation containing '%'.
The performance part of this patch sped up 'ls' by about 1% on my
little benchmark of 'ls -lR' on the source directory in the
en_US.UTF-8 locale (Fedora 24 x86-64).
* NEWS: Document the bug fix.
* src/ls.c (first_percent_b, abformat_init): New static functions.
(ABFORMAT_SIZE): New constant.
(use_abformat): New static var.
(abmon, required_mon_width): Remove these static vars.
(abmon_init): Now accepts a pointer to abmon, and returns a boolean.
All callers changed. Reject month abbrs containing '%', as these
would mess up strftime. Simplify mbsalign result checking,
since (size_t) -1 exceeds ABFORMAT_SIZE.
(abformat_init, align_nstrftime): Precompute all 24 formats at
startup, rather than computing a format for each time stamp.
(decode_switches): Call abformat_init instead of abmon_init.
(align_nstrftime): Accept recentness bool instead of format.
All callers changed.
* tests/misc/time-style.sh: Test for format with '%%b'.
Given that `ttyname` is already conditionally declared in src/system.h,
other declarations are redundant and problematic for example in ChromeOS
which has a new FORTIFY implementation that, as an artifact of how it's
implemented, causes the compiler to complain if certain standard library
functions are redeclared without special compiler-specific attributes.
* src/pinky.c: Remove declaration (which was unused anyway).
* src/who.c: Remove declaration.
* src/system.h (getlogin, getuid, geteuid, getgrgid, getpwuid, ttyname):
Add the parameter type to the declarations to avoid warnings when these
backup declarations are used.
This doesn't affect the generated code on my system; it's merely
a cleaner way to use the recently-introduced Linux-specific API.
* m4/jm-macros.m4 (gl_CHECK_ALL_HEADERS): Check for linux/fs.h.
* src/copy.c: Include <linux.fs.h> if available.
(FICLONE) [__linux__]: Define if not already defined.
(clone_file): Use FICLONE instead of the older BTRFS_IOC_CLONE,
as this ioctl is no longer btrfs-specific.
* bootstrap.conf (gnulib_modules): Add flexmember.
* src/sort.c: Include flexmember.h.
(struct tempnode): Make the last member flexible.
(create_temp_file): Port to strict C11/C99 rules for
allocation alignment with flexible array members.
* src/factor.c (lbuf_putc): Only buffer more than a line
when not using the tool interactively.
* NEWS: Mention the bug fix.
Fixes http://pad.lv/1620139
Postpone installation of signal handlers until they're needed.
That is right before the first escape sequence is printed.
* src/ls.c (signal_setup): A new function refactored from main()
to set and restore signal handlers.
(main): Move signal handler setup to put_indicator()
so that the default signal handling is untouched as long as possible.
Adjusted condition for restoring signal handlers to reflect the change.
(put_indicator): Install signal handlers if called for the very first
time. It uses the same code that was in main() prior to this commit.
* NEWS: Mention the improvement.
See https://bugzilla.redhat.com/1365933
Fixes http://bugs.gnu.org/24232
* src/tr.c (unquote): Use indices of type "unsigned int", rather
than a mix or "size_t" and "int", presuming that no command-line-
specified string will have length longer than UINT_MAX.
* src/ptx.c (fix_output_parameters): Switch to an unsigned type that
matches the OCCURS.file_index type. This avoids the following error
from gcc-7.0.0 20160829 (experimental):
src/ptx.c:1220:14: error: assuming signed overflow does not occur \
when simplifying conditional to constant [-Werror=strict-overflow]
if (file_index > 0)
* src/cp.c (make_dir_parents_private): Use default permissions for
created directories when --no-preserve=mode is specified.
* tests/cp/cp-parents.sh: Add a test case.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/24251
* src/expand.c, src/unexpand.c: Move global variables from here...
* src/expand-common.h, src/expand-common.c: ... to here.
* src/expand.c, src/unexpand.c: (parse_tab_stops, validate_tab_stops,
next_file): Move identical functions to new module.
(add_tab_stop): Move to new module, including additional code from
'unexpand' (keeping max_column_width) which will have no effect in when
used in 'expand'. Refactor common next-column calculation code
into a new function 'get_next_tab_column'.
* src/local.mk: (src_expand_SOURCES, src_unexpand_SOURCES): Add
'expand-common.c'; (noinst_HEADERS): Add 'expand-common.h'.
* po/POTFILES.in: Add 'expand-common.c'.
* tests/misc/expand.pl: Add more tests.
* tests/misc/unexpand.pl: Likewise.
* TODO: Move conclusions to above test after investigation.
Use hash table for seaching in filter_mount_list() and get_dev()
This improves performance for 20K mount entries from:
real 0m1.731s
user 0m0.532s
sys 0m1.188s
to:
real 0m1.066s
user 0m0.028s
sys 0m1.032s
* src/df.c (devlist_table): Define hash table.
(devlist_hash): Add hash function.
(devlist_compare): Add hash comparison function.
(devlist_for_dev): Add lookup function.
(devlist_free): Add cleanup function.
(filter_mount_list): Use the above hash table.
While at it, rename the variable 'devlist' to 'seen_dev' for
better readability.
(me_for_dev): Use the above lookup function.
NEWS: Mention the improvement.
THANKS.in: Remove the committer; add original submitter Josef Cejka.
* src/date.c (usage): This was thought to introduce other ambiguities,
and was inconsistent with the format presented in the touch(1) docs.
* doc/coreutils.texi (date invocation): Likewise.
See http://bugs.gnu.org/24077
* cfg.mk (sc_THANKS_in_sorted): This check would fail on systems
for which "." is not ignored. Add a quick sort-based check for
that error, and skip the check on any broken system.
* src/date.c (usage): Make the characters used to summarize
the input format, match the output +FORMAT characters.
* doc/coreutils.texi (date invocation): Likewise.
Fixes http://bugs.gnu.org/24077
The following is a reproducer for the wrong dependency:
$ ./configure --enable-install-program=arch
$ make
$ rm -f src/arch man/arch.1
$ make man/arch.1
GEN man/arch.1
help2man: can't get `--help' info from man/arch.td/arch
Try `--no-discard-stderr' if option outputs to stderr
Makefile:14378: recipe for target 'man/arch.1' failed
make: *** [man/arch.1] Error 127
* man/local.mk (man/arch.1): Change to depend on src/arch rather than
src/uname: while the arch binary depends on uname.c and uname-arch.c,
its man page depends on the arch binary.
Reported downstream by Rodrigues Goldwyn <rgoldwyn@suse.com> in
https://build.opensuse.org/request/show/415172
* src/sort.c (traverse_raw_number): Accept thousands separator only
if it is immediately followed by a digit.
* tests/misc/sort-h-thousands-sep.sh: Cover the fix for this bug.
Suggested by Pádraig Brady in http://bugs.gnu.org/24015
* src/sort.c (traverse_raw_number): Allow to skip only one occurrence
of thousands_sep to avoid finding the unit in the next column in case
thousands_sep matches as blank and is used as column delimiter.
* tests/misc/sort-h-thousands-sep.sh: Add regression test for this bug.
* tests/local.mk: Reference the test.
* NEWS: Mention the bug fix.
Reported at https://bugzilla.redhat.com/1355780
Fixes http://bugs.gnu.org/24015
* src/sort.c (traverse_raw_number): New function for traversing numbers.
(find_unit_order): Use traverse_raw_number() instead of open-coding it.
(debug_key): Likewise.
* doc/coreutils.texi (install invocation): Update -Z documentation.
* src/install.c (make_ancestor): Set default security context before
calling mkdir() if the -Z option is given.
(process_dir): Call restorecon() on the destination directory if the
-Z option is given.
(usage): Update -Z documentation.
* tests/install/install-Z-selinux.sh: A new test for 'install -Z -D'
and 'install -Z -d' based on tests/mkdir/restorecon.sh.
* tests/local.mk: Reference the test.
* NEWS: Mention the improvement.
Reported at https://bugzilla.redhat.com/1339135
Fixes http://bugs.gnu.org/23868
* tests/rm/rm-readdir-fail.sh: Since we use the returned dirent
from the readdir wrapper it must be the correct type and not
just cast. Therefore setup so that we only have to define a
wrapper for readdir(), which works appropriately on 32 and 64 bit.
Issue reported by Bernhard Voelker, where rm was seen to invoke
rmdir() on invalid file names.
Follow-up to commit c92585b1: improve epipe test script, with
suggestions from Bernhard Voelker and Pádraig Brady.
see: http://lists.gnu.org/archive/html/coreutils/2016-04/msg00067.html .
* tests/misc/seq-epipe.sh: Avoid too many subshells,
stricter validation of errors and output from seq,
skip if trapping SIGPIPE is not supported.
* src/stty.c: Disable setting of "swtch" to ^z as that
conflicts with and disables using ^z as "susp".
* NEWS: Mention the bug fix.
Reported and tested by Rich Burridge at:
http://bugs.gnu.org/23866
* tests/rm/rm-readdir-fail.sh: A new test to simulate readdir()
failing immediately or after returning a few entries, and verifying
that rm does the appropriate thing.
This was initially reported at:
http://bugzilla.opensuse.org/show_bug.cgi?id=984910
where it was mentioned that readdir() may fail
when an NFS server has a poor readdir cookie implementation.
* gnulib: Update to latest.
* NEWS: Specifically mention the fts readdir() fix
and reindent to standard indentation.
* tests/init.sh: Update from gnulib.
* src/yes.c: Include full-write.h.
(main): Use full_write, not write, to simplify handling of partial
writes. Don't bother using stdio to output data; just use
full_write with a buffer as large as needed. Reuse operand
strings if possible, and if the buffer would otherwise be large.
* src/yes.c (main): Loop over the write buffer to
handle the case where write may write less than requested.
* NEWS: Mention the bug fix.
Reported by Paul Eggert.
GCC 7 warned about undefined behavior in this unlikely case.
Problem reported by Jim Meyering in: http://bugs.gnu.org/23825
* src/md5sum.c (main):
* src/paste.c (main):
* src/yes.c (main):
Avoid undefined behavior when argc == INT_MAX.
* doc/coreutils.texi (sort invocation): Mention in the summary
dicussion that --key is used to specify fields. Give a summary
in the --key description, of the most common use case of specifying
a field, and that by default those fields include the blank separators
at the start of each field in the comparisons.
Print warning regardless of locale, avoid warning if key is zero width.
Problem reported by Karl Berry in http://bugs.gnu.org/23665 .
* src/sort.c: (key_warnings): change conditions for 'leading spaces'
warning.
* tests/misc/sort-debug-warn.sh: adjust tests accordingly.
Easier troubleshooting of individual 'sort --debug' messages.
Discussed in http://bugs.gnu.org/23665 .
* tests/misc/sort-debug-warn.sh: add progress number before each sort
invocation.
* src/dircolors.hin: Reduce the list by replacing all specific
mentions of "color" with the pattern "*color*". This will also
cater for other entries like "konsole-256color".
Fixes http://bugs.gnu.org/23542
* stc/stat.c (human_fstype): Add file system ID definition,
and use "smb2" as the name.
* NEWS (Improvements): Mention the change.
Fixes http://bugs.gnu.org/23516
* src/id.c (main): When configured with --enable-gcc-warnings and using
the very latest gcc built from git, building would fail with this:
src/id.c:200:8: error: assuming signed overflow does not occur when \
simplifying conditional to constant [-Werror=strict-overflow]
bool default_format = (just_user + just_group + just_group_list
^~~~~~~~~~~~~~
Rewrite to use bool-appropriate operators.
Ensure I/O errors are detected (and terminate seq), preventing seq
from infloop (or running for long time with a large
range) upon write errors or ignored SIGPIPE. Examples:
seq 1 inf > /dev/full (seq_fast)
seq 1.1 0.1 inf >/dev/full (print_numbers)
* src/seq.c (io_error): A new function to diagnose appropriate
stdio errors and exit the program with failure status.
(seq_fast, print_numbers): Explicitly check for write errors
and terminate the program with diagnostic.
* tests/misc/seq-io-errors.sh: Test error detection with /dev/full.
* tests/misc/seq-epipe.sh: Test error detection with broken pipes.
* tests/local.mk: Add new tests.
* NEWS: Mention the fix.
* src/stat.c (human_fstype): Add file system ID definition,
and use "wslfs" as the name.
* NEWS (Improvements): Mention the change.
Fixes http://bugs.gnu.org/23273
* src/sleep.c (main): Allow ERANGE since we allow "inf" values.
* src/timeout.c (parse_duration): Likewise.
* tests/misc/sleep.sh: New file. Tests for sleep(1).
* tests/misc/timeout-parameters.sh: Add case for newly allowed
$LDBL_MAX. Also use returns_ throughout the file. Also avoid
small timeout values which might give false failures under load.
* cfg.mk (sc_prohibit_expr_unsigned): expr uses intmax_t internally
when GMP is not available, so flag any uses of unsigned limits.
* tests/misc/timeout-parameters.sh: Remove the overflow related
to UINT_MAX as it's handled by the following case. Change the
following case from 99... to $TIME_T_OFLOW to be more expressive
in what it's actually testing. Directly check that commands succeed,
rather than inspecting $? afterwards.
* tests/dd/skip-seek-past-dev.sh: Guard against large device sizes.
* tests/id/setgid.sh: Protect: Guard against large group IDs.
* tests/misc/cut-huge-range.sh: Tweak comment to avoid syntax check.
It's tempting to simplify to just skip the test if expr fails,
but SIZE_MAX > INTMAX_MAX is the common case.
* tests/cp/parent-perm-race.sh: This new race introduced in
commit v8.25-5-g632eda5 is quite hard to hit, but is due to
`ls > fifo` doing write()/close()/exit() once `cp` has
open() the source fifo. Then the subsequent comparison of the
destination file may fail due to the file being missing or empty.
Previously `ls` generated output that was independent of `cp`.
Now we must wait for `cp` to finish before inspecting the
destination file that it wrote.
* src/seq.c (isnan): Define macro.
(scan_arg): Add check if the given argument is NaN, and exit with
a proper error diagnostic in such a case.
(usage): Document it.
* tests/misc/seq.pl: Add tests.
* doc/coreutils.texi (seq invocation): Document the change.
* NEWS (Changes in behavior): Mention the change.
* src/seq.c (main): Exit with an error diagnostic when the given
step value is Zero.
(usage): Document it.
* doc/coreutils.texi (seq invocation): Likewise.
* tests/misc/seq.pl: Add tests.
* NEWS (Changes in behavior): Mention the change.
Reported by Маренков Евгений in:
http://bugs.gnu.org/23110
* src/stat.c (human_fstype): Add file system ID definition.
* NEWS (Improvements): Mention the change, moving the previously added
"prl_fs" change note from "Changes in behavior" to here.
Fixes http://bugs.gnu.org/23283
* src/stat.c (human_fstype): Identify the parallels file system.
Also tag as remote so that tail(1) doesn't use inotify, which
fails to detect changes made outside a VM.
Fixes http://bugs.gnu.org/23143
The problematic code computed a struct tm in one time zone, and
then printed it or converted it to a string in another. To be
portable the same time zone needs to be used for both operations.
On GNU platforms this is not an issue, but incorrect output can be
generated on System V style platforms like AIX where time zone
abbreviations are available only in the 'tzname' global variable.
Problem reported by Assaf Gordon in: http://bugs.gnu.org/23035
* NEWS: Document the bug.
* src/date.c (show_date):
* src/ls.c (long_time_expected_width, print_long_format):
* src/pr.c (init_header):
* src/stat.c (human_time): Use localtime_rz instead of localtime,
so that the time zone information is consistent for both localtime
and time-formatting functions like fprintftime and nstrftime. For
'stat' this change is mostly just a code cleanup but it also
causes stat to also print nanoseconds when printing time stamps
that are out of localtime range, as this is more consistent with
what other programs do. For programs other than 'stat' this fixes
bugs with time zone formats that use %Z.
* src/du.c, src/pr.c (localtz): New static var.
(main): Initialize it.
* src/du.c (show_date): New time zone argument, so that localtime
and fprintftime use the same time zone information. All callers
changed.
* tests/misc/time-style.sh: New file.
* tests/local.mk (all_tests): Add it.
* tests/misc/date.pl: Test alphabetic time zone abbreviations.
* gl/lib/regcomp.c.diff: Remove file, now that gnulib's
regcomp.c compiles regex.c with -Wno-unused-parameter.
* gl/lib/regex_internal.h.diff: Likewise.
* gl/lib/regex_internal.c.diff: This file induced a change to ensure
that the "Idx" type was unsigned and to remove a few "VAR < 0"
comparisons. These days, it is probably fine to stay in sync with
gnulib/glibc's copies
of these files, so remove these patches, too.
* gl/lib/regexec.c.diff: Likewise.
Prompted by a report by Assaf Gordon and a suggestion from Paul Eggert.
* tests/split/filter.sh: Use OFF_T_MAX-1 rather than OFF_T_MAX
as the size of a test file, to avoid false failure on an XFS file
system (or any file system permitting a file of size OFF_T_MAX).
Reported as http://bugs.gnu.org/22931
POSIX recommends avoiding -a and -o, for good reason.
src/test.c (usage): Mention that inherent ambiguities exist with
binary -a and -o.
Problem reported by Martin Gebert in: http://bugs.gnu.org/22909
sort, tail, and uniq now support traditional usage like 'sort +2'
and 'tail +10' on systems conforming to POSIX 1003.1-2008 and later.
* NEWS: Document this.
* doc/coreutils.texi (Standards conformance, tail invocation)
(sort invocation, uniq invocation, touch invocation):
Document new behavior, or behavior's dependence on POSIX 1003.1-2001.
* src/sort.c (struct keyfield.traditional_used):
Rename from obsolete_used, since implementations are now allowed
to support it. All uses changed.
(main): Allow traditional usage if _POSIX2_VERSION is 200809.
* src/tail.c (parse_obsolete_option): Distinguish between
traditional usage (which POSIX 2008 and later allows) and obsolete
(which it still does not).
* src/uniq.c (strict_posix2): New function.
(main): Allow traditional usage if _POSIX2_VERSION is 200809.
* tests/misc/tail.pl: Test for new behavior.
Problem reported by Yanyan Jiang 蒋炎岩 in: http://bugs.gnu.org/22769
* doc/coreutils.texi (sort invocation):
Mention possibility of 'sort -o F F' data loss during crashes.
* src/split.c (lines_rr): Reinstate the conditional
setting of the WROTE boolean, as otherwise split -n r/1 would
consume all input when all --filter commands are stopped.
There was a test in place to check for this, but
it was incorrect as detailed below.
(input_file_size): Immediately disallow --number with
non seekable inputs, as such an invocation is not currently
generally supported and will fail as the data overflows
the internal buffer.
* tests/split/l-chunk.sh: Adjust to again disallow -n /dev/zero.
Also change all '&& fail=1' checks to use the 'returns_ 1' form.
* tests/split/filter.sh: Change the no longer supported /dev/zero
case to a regular $OFF_T_MAX file (supported on XFS for example).
Also fix the timeout(1) commands so they're not subject to
pipefail issues.
* src/stty.c (usage): Remove an erroneous call to translate an
empty string, added in commit v8.23-112-g564f84a, which results
in the gettext header being printed for translated languages.
* THANKS.in: Remove the now committer.
* NEWS: Mention the bug fix.
* tests/cp/parent-perm-race.sh: Add timeouts so that the test does
not wait forever on GNU/Hurd. This does not fix the underlying
bug but at least lets the tests make progress.
Problem reported by Nelson H.F. Beebe in: http://bugs.gnu.org/22624
Also, check that 'cp' actually copies the data; this is a better
test anyway, and simplifies the test code.
Also, avoid unlikely race if 'pid' is set in the environment.
Problem reported by Nelson H.F. Beebe in: http://bugs.gnu.org/22624
Other problems also fixed: basically, the code got confused because
GNU/Linux reports that /dev/zero has size zero.
* src/split.c (input_file_size): Now takes struct stat *, not just
size. Always store the first buffer. All callers changed. Treat
/dev/zero as an infinitely-large file, both on GNU/Linux where
fstat and lseek say its size is zero, and on GNU/Hurd where they
say the size is OFF_T_MAX.
(cwrite): Return true on success.
(bytes_split): Don't try to read past EOF, and stop if a write fails.
(lines_rr): Omit stray check for ignorable errno.
(main): Get file size only when n_units > 1, since that's the only
time it is needed. Defer most of the work to input_file_size.
* tests/split/l-chunk.sh: Adjust tests to match new behavior
on oddball inputs.
* src/date.c (usage): The colon is used in the timezone offset
since commit v8.24-64-g17bbf6c.
* THANKS.in: Remove committer.
Fixes http://bugs.gnu.org/22491
Solaris Studio 12 on sparc (not x86) will not remove
unused functions, thus leaving a reference to an
undefined program_name symbol from emit_try_help().
* src/system.h (emit_try_help): Change from an inline function
to a macro, so that the inline function is not actually
defined in libstdbuf.c.
Fixes http://bugs.gnu.org/22430
* tests/tail-2/wait.sh: Restrict inotify specific test portion
to local file systems and also not with ---disable-inotify specified.
Failure noticed on NFS.
* tests/rm/rm1.sh: Also remove the group write bit which
was required on one NFS setup at least. Note u-w was
enough to deny file creation, g-w was also required to
deny file removal.
* tests/rm/cycle.sh: Likewise.
* tests/mv/perm-1.sh: Likewise.
* tests/cp/preserve-slink-time.sh: Add a delay between the
ln and the cp so that there is enough difference between
the timestamps so GPFS won't discard the update.
Reported by Assaf Gordon.
* tests/misc/shred-passes.sh: Specify an exact amount to shred,
to avoid running out of simulated random data on file systems
with a large st_blksize like GPFS for example.
Reported by Assaf Gordon.
tests/tail-2/F-headers.sh and test/tail-2/retry.sh fail on
on remote file systems due to tail going into inotify mode
due to not being able to determine the remoteness of the
non existent files.
* src/tail.c (any_non_remote_file): A new function used
to disable inotify when there are no open files, as
we can't determine remoteness in that case.
* NEWS: Mention the bug fix.
* gnulib: Update to latest where the only change is to
not unconditionally enable leaf optimization for fts on NFS,
as it was seen to abort() with some NFS servers as per:
https://bugzilla.redhat.com/1299169
This affects utilities that traverse directories like
cp, rm, chmod etc.
* NEWS: Adjust the improvement message to leave only XFS.
* tests/rm/dangling-symlink.sh: The effective delay was only
1.5s. So delay before the operation, and increase the iteration
count by 1 to increase the delay to 6.3s. This failure was
noticed once on a FreeBSD 10.2 x86_64 virtual machine.
* tests/dd/stats.sh: Remove quotes on sleep argument
to ensure arguments are accumulated appropriately.
* tests/du/move-dir-while-traversing.sh: sleep before operation,
to increase the effective delay from 1.6s to 3.2s.
* tests/tail-2/flush-initial.sh: Likewise.
* tests/tail-2/pipe-f2.sh: Sleep first so that the effective delay
matches the commented value.
* tests/rm/r-root.sh: This test was seen to ERROR due to
sed input not having a terminating '\n'. Therefore just skip
with diagnostics upon failure to adjust the error output with sed.
Reported by Assaf Gordon.
* init.cfg (dump_mount_list_): A new function to output the
system mount list.
* tests/df/df-symlink.sh: Call dump_mount_list_ upon failure.
* tests/df/over-mount-device.sh: Likewise.
* tests/df/problematic-chars.sh: Likewise.
* tests/df/skip-rootfs.sh: Likewise.
* tests/df/total-verify.sh: Likewise.
* tests/df/unreadable.sh: Likewise.
* tests/df/df-symlink.sh: Only check the symmetry of
the source <-> target lookup, when the source is only mounted once,
which isn't the case if '.' is a BTRFS subvolume for example.
Reported by Assaf Gordon on a default OpenSUSE 42.1 install.
* src/copy.c (copy internal): Remember directories irrespective
of their link count, because on some file systems like BTRFS,
directories always have a link count of 1.
* tests/misc/date.pl: This test was dependent on perl
unconditionally calling the system localtime() as date(1) does.
However perl has its own configure checks in place to avoid buggy
localtime implementations. Therefore comment out this test
as a FIXME for now. This test was seen to fail on FreeBSD 10
where localtime() returns nonsense values instead of failing
for arguments >= 2^56.
* tests/misc/head-c.sh: Further increase the limit, to account
for the fact that head(1) needs at least 1.576MB over the base test
command on FreeBSD 10. 4MB should be enough headroom while
still providing over allocation protection.
Reported by Assaf Gordon.
* src/operand2sig.c (operand2sig): Add a detailed comment explaining
why we validate even very large shell exit status values.
* tests/misc/kill.sh: Add a test case for the ksh scheme.
Simplify the INVALID signal number determination which also avoids
a false failure on systems like FreeBSD 10 with incomplete
signal list (caused by inaccurate NSIG).
* tests/cp/fiemap-extents.sh: Support RHEL6 fallocate
which doesn't support IEC suffixes like "MiB" on numbers.
Also add some extra framework_failure_ protections.
Reported by Assaf Gordon.
Locale categories are not equivalent on OpenBSD,
and LC_COLLATE only supports "C" for example.
Now LC_ALL is supported to set multiple other categories
on OpenBSD, so setlocale(LC_ALL, "") returns a string
indicating which categories were updated and which ignored.
Therefore...
* src/sort.c (main): ...Call setlocale(LC_COLLATE, "")
to explicitly check whether a specified LC_ALL or
LC_COLLATE environment variable value is supported
for the LC_COLLATE category. Also use !! to explicitly
convert to bool to support c89 systems where bool is an int,
and thus would get values > 1.
Reported by Assaf Gordon.
doc/coreutils.texi (dd invocation): Add oflag=sync to the streaming
example. Also reference the "direct" flag. Mention this is only
a request to the system.
* src/dd.c (usage): Mention the "sync" flag along with "nocache".
Also mention that it's only a request to drop the cache.
* THANKS.in: Add reporter Francois Rigault.
* src/stat.c (usage): Mention the '#' and '0' flags are useful with %a.
* doc/coreutils.texi (stat invocation): Likewise. Also give an example
printing unambiguous octal output.
Reported at http://bugs.debian.org/810539
commit v8.23-31-g90aa291 failed to consider this case,
where the previous rename has failed, thus causing the
following to remove the specified directory:
mv dir dir dir
* src/copy.c (copy_internal): Assume this rename attempt has
succeeded, as a previous failure will already have been handled,
and we don't want to remove the source directory in this case.
* tests/cp/duplicate-sources.sh: Consolidate this test file to...
* tests/mv/dup-source.sh: ...here. Add test cases for same
source and dest.
* tests/local.mk: Remove the consolidated test.
* NEWS: Mention the bug fix.
Reported at https://bugzilla.redhat.com/1297464
* src/wc.c (write_counts): Shell escape the file name
if it contains '\n' so only a single line per file is output.
* tests/misc/wc-files0.sh: Add a test case.
* NEWS: Mention the improvement.
* doc/coreutils.texi (numfmt invocation): Reference the description.
* src/numfmt.c: Parameterize '\n' references.
* tests/misc/numfmt.pl: Add tests for character and field processing.
* NEWS: Mention the new feature.
* doc/coreutils.texi (paste invocation): Reference -z description.
* src/paste.c (main): Parameterize the use of '\n'.
* tests/misc/paste.pl: Add test cases.
* NEWS: Mention the new feature.
* src/comm.c (main): Track the output delimiter length,
so that it can be adjusted to 1 for the NUL delimiter.
Also rename the global variable from "delimiter" to
"col_sep" so its use is more obvious, and to distinguish
from the recently added "delim" global variable.
* tests/misc/comm.pl: Adjust accordingly.
* doc/coreutils.texi (comm invocation): Reference option description.
* src/comm.c (main): Use readlinebuffer_delim() to support
a parameterized delimiter.
* tests/misc/comm.pl: Add test cases.
* NEWS: Mention the new feature.
* doc/coreutils.texi (tac invocation): Mention the
NUL delineation with an empty --separator.
* src/tac.c (main): Allow an empty separator when -r not specified.
* tests/misc/tac.pl: Add test cases.
* NEWS: Mention the new feature.
Fixes http://bugs.gnu.org/8103
* doc/coreutils.texi (cut invocation): Reference the description.
* src/cut.c: Parameterize '\n' references.
* tests/misc/cut.pl: Add tests for character and field processing.
* NEWS: Mention the new feature.
* NEWS: Mention the change in behavior.
* doc/coreutils.texi (newlineFieldSeparator): A new description,
referenced from ({join,sort,uniq} invocation).
* src/system.h (field_sep): A new inline function to determine
if a character is a field separator.
* src/join.c (usage): s/whitespace/blank/ to be more accurate
wrt which characters are field separators.
(xfields): s/isblank/field_sep/.
* src/sort.c (inittables): Likewise.
* src/uniq.c (find_field): Likewise.
* tests/misc/join.pl: Adjust -z test, and add a test/example
for processing the whole record with field processing.
* tests/misc/sort.pl: Add -z test cases, including case with '\n'.
* tests/misc/uniq.pl: Add -z -f test case with \n.
At least the false positive in tail-2/follow-stdin.sh could be seen
on a 4-core i5 system with -j8. Fix similar cases. too.
* tests/tail-2/follow-stdin.sh: Empty the 'out' file in each iteration.
Otherwise, under heavy system load, 'check_tail_output' would see the
expected output from the previous round before tail would have the
chance to come up in the background.
While at it, move the creation of the 'exp' file out of the loop.
* tests/dd/stats.sh: Empty the output file of the background process
here, too.
* tests/misc/cat-buf.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/tail-2/wait.sh: Likewise.
This is used to discard terminal output on FreeBSD based systems.
* src/stty.c (mode_info): Add FLUSHO where supported.
(usage): Document for "local" and "combination" settings.
* doc/coreutils.texi (local stty settings): Document,
mentioning that this option is supported but ignored on GNU/Linux.
Instead of commit v8.24-132-g5171bef which only provides
control to disable this behavior (with -I), provide
the symmetrical "[-]drain" special setting.
* src/stty.c (main): Parse the [-]drain setting instead of -I,
and treat like a global option.
(usage): Adjust accordingly.
* tests/misc/stty.sh: Test "drain" with and without options.
* NEWS: Mention the new feature.
In some cases an initial drain may block indefinitely as discussed at:
http://lists.gnu.org/archive/html/coreutils/2016-01/msg00007.html
* src/stty.c (main): Use TCSANOW rather than TCSADRAIN if -I specified.
(usage): Document the new option.
* doc/coreutils.texi (stty invocation): Likewise.
* tests/misc/stty.sh: Ensure -I is supported.
* NEWS: Mention the new feature.
* doc/coreutils.texi (coreutils): Remove this direntry
so that `info coreutils` will display the table of contents,
rather than the unlikely intended coreutils Multi-call invocation.
This also supports lookups like `info coreutils readlink`,
but note that only works with the standard info viewer,
and not with pinfo for example.
* src/install.c (usage): Mention this commonly required functionality
in the -D option description.
* doc/coreutils.texi (install invocation): Likewise for the
--target-directory description.
* src/install.c (main): As an optimization, when calling
install_file_in_dir() for each file, only attempt to create
the target directory once, as this is invariant over the loop.
* src/install.c (mkancesdirs_safe_wd): Unconditionally
restore the current working directory when possibly called
multiple times (from install_file_in_dir()).
* tests/install/create-leading.sh: Add a test case.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/21497
Since commit v8.23-60-g414a8cf, the above check failed to find sources
where the length of the longest line is a 3 or more digit number.
* cfg.mk (sc_long_lines): Remove the '\' escape character before '{'
and '}', because this would mean literal '{' and '}' characters in
an extended regular expression in sed(1).
Run "make update-copyright" and then...
* gnulib: Update to latest with copyright year adjusted.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
Problem noted by Pádraig Brady in: http://bugs.gnu.org/22277#8
Also, make the output a bit more precise while we're at it.
* NEWS: Document this.
* src/dd.c (previous_time): Remove, replacing with ...
(next_time): New var. All uses changed.
This avoids some rounding errors, and should be a bit faster.
(newline_pending): Remove, replacing with ...
(progress_len): New var. All uses changed.
This lets us keep track of how many trailing spaces to append.
(print_xfer_stats): Get the time first thing, so that it's
closer to being correct. Count the bytes output, and append
trailing spaces if needed. Add remarks to translators about
translation lengths.
Problem reported by Linda Walsh in: http://bugs.gnu.org/17505
* NEWS: Document this.
* doc/coreutils.texi (dd invocation): Use a simpler script.
Adjust output example to match new behavior.
* src/dd.c (human_size): Remove.
All uses changed to use human_readable and ...
(human_opts): ... this new constant.
(abbreviation_lacks_prefix): New function.
(print_xfer_stats): Use it. Output both --si and --human-readable
summaries, but only if they have prefixes.
* tests/dd/reblock.sh, tests/dd/stats.sh: Test new behavior.
Mainly for these changes:
- freadptr: fix to work with ungetc on all uClibc configs
- fts: enable leaf optimization for XFS
- fts: ensure leaf optimization used for NFS
- strftime-tests: avoid false failure on OS X
- intprops-tests: avoid warnings (causing CI failures)
* NEWS: Update with items from above that are
significant from the previous coreutils release.
* doc/coreutils.texi (Connectives for test): Add notes
on precedence and associativity. Also mention the
portability caveats with these operators.
* cfg.mk: Avoid sc_prohibit_test_minus_ao for coreutils.texi.
Fixes http://bugs.gnu.org/22216
Commit v8.24-116-g896006b changed the file permissions for the tarball
using ug+rw. Better to let the umask decide whether the file should
be group-writeable or not.
* Makefile.am (dist-hook): Change chmod call from 'ug+rw' to '+rw'.
* init.cfg (get_min_ulimit_v_): Increase the determined
virtual memory limit for the command by a page size
to avoid alignment issues causing false failures for
subsequent runs.
* tests/misc/cut-huge-range.sh: Be defensive and match
the cut invocations under test, more closely with the
form used to determine the VM memory limit.
This was noticed on i686 linux at:
http://hydra.nixos.org/build/28990456
* Makefile.am (dist-hook): Ensure all files are writable
so that extracted tarballs are easy to remove, without
getting prompts about removing read-only files.
Reported by Benno Schulenberg.
We were erroneously skipping blanks in the marked comparison
_after_ the key start offset was applied.
* src/sort.c (debug_keys): Don't skip starting blanks
if already handled by begfield().
* tests/misc/sort-debug-keys.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/22155
* src/stat.c (human_fstype): Add "overlayfs", and tag it as "remote"
to ensure that tail continues to use the more conservative polling mode.
* README-release: Remove a stale comment about updating fremote()
in tail.c. Also give a link to *_SUPER_MAGIC definitions
not in the standard linux/magic.h location.
* NEWS: Mention that this file system is recognized.
procps, bash >= 4.4, and util-linux >= 2.23
all have the -L option to output a table.
* src/kill.c (main): Have -L synonymous with -t.
Luckily there is no signal that begins with 'L'.
* tests/misc/kill.sh: Update the test.
The character 'K' representing numeric user input can be confusing,
especially in the context where it can be suffixed by the multipliers
K, M, G, etc.
Use NUM instead.
* doc/coreutils.texi (head invocation): Replace @var{k} by @var{num}.
Also change @var{count} in the paragraph about the obsolete option
syntax.
(tail invocation): Likewise. Furthermore, adjust the option order to
alphabetical sorting.
* src/head.c (usage): s/K/NUM/ in as placeholder in the -c and -n
options. Furthermore, print the actual DEFAULT_NUMBER instead of the
hard-coded 10.
* src/tail.c (usage): s/K/NUM/ in as placeholder in the -c and -n
options. Add "[+]" to clarify that NUM may optionally be preceded by
this sign. Remove the redundant explanation of the optional '+' prefix
for NUM, as that is now clear from the option description above.
Suggested by Dan Jacobson in http://bugs.gnu.org/22042
* man/head.x: Add "SEE ALSO" section referencing tail(1).
* man/tail.x: Add "SEE ALSO" section referencing head(1).
Suggested by Dan Jacobson in http://bugs.gnu.org/22041
Use a consistent date in the generated man pages
even if SOURCE_DATE_EPOCH is not set.
* Makefile.am: Generate .timestamp in the tarball.
* man/local.mk: Pass .timestamp if available to help2man.
* man/dummy-man: Don't bother with the year in the stub.
Reported in http://bugs.debian.org/806321
Sync with version 1.47.3 which provides these significant changes:
- support for reproducible builds by using $SOURCE_DATE_EPOCH.
- Adjust spacing of italic text at roman/italic boundaries.
* doc/local.mk (constants.texi): Avoid calling makeinfo unless
the constants derived from tail and shred have changed.
This avoids a dependence on makeinfo from patched sources,
and avoids an expensive makeinfo call for developers.
* doc/coreutils.texi (uniq invocation): Fix the description of
problematic input to say "blank lines" rather than "two or more
blank lines".
(tr invocation): Clarify that -s only works with SET1 when
not translating (when SET2 not specified). Also explicitly
state in examples where blank lines are deleted.
Also add "deleting" to the menu item.
* src/tr.c (usage): Improve the -s summary to say it always
operates on the last specified SET.
* doc/coreutils.texi (tail invocation): Provide an example using awk
to convert tail ==> file <== headers to file: prefixes on each line.
Suggested by Stephen Shirley.
fallocate can allocate extents beyond EOF via FALLOC_FL_KEEP_SIZE.
Where there is a gap (hole) between the extents, and EOF is within
that gap, the final hole wasn't reproduced, resulting in silent
data corruption in the copied file (size too small).
* src/copy.c (extent_copy): Ensure we don't process extents
beyond the apparent file size, since processing and allocating
those is not currently supported.
* tests/cp/fiemap-extents.sh: Renamed from tests/cp/fiemap-empty.sh
and re-enable parts checking the extents at and beyond EOF.
* tests/local.mk: Reference the renamed test.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/21790
* src/md5sum.c (digest_check): Update a matched_checksums bool upon
matched checksum, and fail (loudly unless --status is specified)
if there were no matches. Also change properly_formatted_lines
to a bool while at it since we don't need to track the plurality.
* tests/misc/md5sum.pl: Add a test case.
Suggested by Jim Meyering.
* doc/coreutils.texi (md5sum invocation): Document the new option.
* src/md5sum.c (digest_file): Return an empty digest to indicate
a missing file.
(digest_check): Don't fail or output status given an empty checksum.
(usage): Document the new option.
(main): Process and validate the new option.
* tests/misc/md5sum.pl: Add new test cases.
* NEWS: Mention the new feature.
Fixes http://bugs.gnu.org/15604
This can be useful if you want to further process data
from process substitutions. For example:
datagen | tee >(md5sum --tag) > >(sha256sum --tag) | sort
* doc/coreutils.texi (tee invocation): Mention that -p is
useful with pipes that may not consume all data.
Add an example, similar to the one above.
* THANKS.in: Add Jirka Hladky.
* tests/ls/stat-failed.sh: Skip the test if 'd' is returned as the type,
and document where this was seen. Also flag failure to write small
temp files during the test as an error rather than a failure.
Fixes http://bugs.gnu.org/21130
* src/local.mk: Run `mkdir -p src` in all our explicit rules,
as in a VPATH build the src/ dir is only created at configure time
as a side effect of dependency tracking generation.
discussed in:
http://lists.gnu.org/archive/html/coreutils/2015-10/msg00091.html
* src/csplit.c: (save_line_to_file): check fwrite failures, report
and exit immediately instead of deferring to 'close_output'.
* tests/misc/csplit-io-err.sh: test fwrite failure using LD_PRELOAD.
* tests/local.mk: add new test.
programs may not be built due to missing system dependencies,
or any program can be excluded at configure time with
--enable-no-install-program. So ensure we're not testing the
system versions in these cases.
* init.cfg (print_ver_): Call require_built_ first.
* tests/misc/tty-eof.pl: Skip programs not built.
* tests/Coreutils.pm (run_tests): Likewise.
* tests/misc/ls-misc.pl: Use 'env test' rather than abs path.
* tests/misc/test-diag.pl: Likewise.
* tests/local.mk: Adjust include order for dependencies.
* tests/misc/arch.sh: Remove redundant calls to require_built_.
* tests/misc/chroot-fail.sh: Likewise.
* tests/misc/groups-dash.sh: Likewise.
* tests/misc/groups-version.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/cp/acl.sh: Remove problematic call to print_ver_ [gs]etfacl.
* tests/mv/acl.sh: Likewise.
* cfg.mk (sc_env_test_dependencies): A new syntax check to enforce
specifying dependencies with print_ver_ for programs
specified through the env command.
* du/bigtime.sh: Add new print_ver_ dependencies.
* du/max-depth.sh: Likewise.
* dd/ascii.sh: Likewise.
* tests/ls/capability.sh: Likewise.
* tests/ls/root-rel-symlink-color.sh: Likewise.
* tests/misc/chroot-fail.sh: Likewise.
* tests/misc/readlink-fp-loop.sh: Likewise.
* tests/misc/sort-debug-keys.sh: Likewise.
* tests/readlink/can-e.sh: Likewise.
* tests/readlink/can-f.sh: Likewise.
* tests/readlink/can-m.sh: Likewise.
* tests/tail-2/inotify-race.sh: Likewise.
* tests/tail-2/inotify-race2.sh: Likewise.
* tests/touch/no-create-missing.sh: Likewise.
* tests/touch/no-dereference.sh: Likewise.
* tests/misc/printenv.sh: Tweak to avoid syntax check trigger.
* tests/misc/help-version.sh: Likewise.
* tests/misc/yes.sh: Likewise.
* tests/misc/printf-quote.sh: Use previously unused $prog.
* configure.ac (EXTRA_MANS): Add $gl_no_install_prog to the list
so that check-x-vs-1 syntax check is satisfied.
A side effect of this cleanup is we no longer
depend on our own kill command being built.
* init.cfg (require_trap_signame_): A new function to verify
that the shell supports specifying traps by signal name.
(require_kill_group_): A new function to ensure the shell
supports sending a signal to a group.
* tests/du/move-dir-while-traversing.sh: Ensure trap supports
signal names.
* tests/misc/stty-invalid.sh: Likewise.
* tests/misc/stty-pairs.sh: Likewise.
* tests/misc/stty-row-col.sh: Likewise.
* tests/misc/stty.sh: Likewise.
* tests/misc/sort-compress.sh: Likewise. Also simplify trap call.
* tests/install/trap.sh: Likewise.
* tests/misc/timeout.sh: Likewise.
* tests/dd/stats.sh: Likewise. Also use default kill command.
* tests/misc/timeout-group.sh: Likewise.
* init.cfg (require_dirent_d_type_): Don't use df -x
to exclude XFS, since this depends on a correct mtab
which is brittle and often not correct within chroots.
* tests/d_type-check: Check also the d_type of files,
which excludes XFS appropriately. Specify all argument
and return types to avoid truncated pointers being passed,
which skipped the test due to crashes on x86_64 at least.
Simplify the C library lookup by reusing the interpreter's.
chroot issue reported at https://bugzilla.redhat.com/1263341
* src/paste.c (main): Use our styled wrapper for quotearg_colon().
* cfg.mk (sc_prohibit-quotearg): A new syntax check to avoid
future uses of unstyled quotearg to one of the internal slots,
and thus destined for diagnostic output.
* src/paste.c (main): Setting the quoting style to "escape"
went against the intent of the comment about presenting
doubled backslashes to the user. Instead use "c-maybe"
which is the only mode which avoids doubled backslashes,
and provides protection against arbitrary control characters.
* tests/misc/paste.pl: Adjust accordingly.
* src/ls.c (decode_switches): Set "shell-escape" if isatty().
* doc/coreutils.texi (ls invocation): Update the defaults description.
* NEWS: Mention the change in behavior. It should not have
backwards compat issues, but mentioning here just in case.
* src/printf.c (usage): Mention the new format.
(print_formatted): Handle the quoting by calling
out to the quotearg module with "shell-escape" mode.
* doc/coreutils.texi (printf invocation): Document %q.
* tests/misc/printf-quote.sh: New test.
* tests/local.mk: Reference new test.
* NEWS: Mention the new feature.
This is mainly noticeable when the multi-byte code
within ls.c is triggered by multi-byte quotes.
$ seq 200000 | xargs touch
$ time ls-old -U --quoting=locale --hide-control-chars >/dev/null
real 0m0.483s
$ time ls-new -U --quoting=locale --hide-control-chars >/dev/null
real 0m0.430s
* src/ls.c (quote_name): Avoid rescanning the output looking for
unprintable chars when we know the quoting mode already escapes them.
* tests/misc/ls-misc.pl: Add tests for all quoting modes, with and
without -q, to verify this assumption.
* doc/coreutils.texi (ls invocation): Describe the new
'shell-escape' and 'shell-escape-always' quoting options.
* src/ls.c (usage): Mention the new quoting options.
* tests/misc/ls-misc.pl: Add a test for 'shell-escape'
Quote file names using the "shell-escape" or "shell-escape-always"
methods, which quote as appropriate for most shells,
and better support copy and paste of presented names.
The "always" variant is used when the file name is
embedded in an error message with surrounding spaces.
* cfg.mk (sc_error_shell_quotes): A new syntax check rule
to suggest quotef() where appropriate.
(sc_error_shell_always_quotes): Likewise for quoteaf().
* src/system.h (quotef): A new define to apply shell quoting
when needed. I.E. when shell character or ':' is present.
(quoteaf): Likewise, but always quote.
* src/*.c: Use quotef() and quoteaf() rather than quote()
where appropriate.
* tests/: Adjust accordingly.
* src/md5sum.c: Use the same file name escaping method used
when generating and checking checksums. I.E. ensure a single line
per file by starting the line with '\' for any file name containing '\n'
and replacing those with "\\n".
* NEWS: Move the item from changes in behavior to improvements,
since this is no longer a backwards incompat change when
processing stdout status messages.
* tests/misc/md5sum.pl: Remove quotes from expected status output.
* tests/misc/sha1sum.pl: Likewise.
Related to commit v8.24-61-g6796698 this provides
more consistent quoting, as quotearg_colon() defaults
to "literal" quoting by default, while quote()
provides appropriate quoting for diagnostics by default.
* gl/modules/randread: Depend on quote module rather than quotearg.
* gl/lib/randread.c: Used quote() not quotearg_colon().
* src/: Likewise.
* src/shred.c: Likewise. Also avoid unnecessary quoting
introducing overhead when wiping names.
* cfg.mk: Relax the matching expression to allow
"qname" variables as used in shred.c to satisfy the check.
* tests/: Adjust accordingly.
* doc/coreutils.texi: (tail invocation): Add missing -s,
along with the existing --sleep-interval description.
(csplit invocation): s/--suffix/--suffix-format/.
(head invocation): Use same variable (COUNT) for -n and --head-count.
(seq invocation): Add opindex items for all options.
(ptx invocation): Likewise.
Fix typo s/--flac-truncation/--flag-truncation/.
(touch invocation): State explicitly that -d takes a parameter,
which also indicates that an '=' is not to be used
for the short option syntax.
(ls invocation): Likewise for the -w option.
Fixes http://bugs.gnu.org/21809
* doc/coreutils.texi (du invocation): Remove 'like' from
mentions of ISO-8601 as the components are individually conformant.
(ls invocation): Likewise, except for --time-style=iso for recent
files, where the MM-DD component is not a valid ISO-8601 timestamp.
* src/date.c (main): Use %:z rather than %z with --iso-8601
as the standard states to consistently use extended format.
Note either format can be parsed by date.
* tests/misc/date.pl: Adjust accordingly.
* doc/coreutils.texi (du invocation): Clarify that "iso"
time styles are only similar to ISO-8601.
(ls invocation): Likewise.
(date invocation): Adjust the comment stating
that only --rfc-3339 output can be parsed by date(1).
* NEWS: Mention the change in behavior.
Reported at http://bugs.debian.org/799479
* src/factor.c: Move LIKELY() definition to...
* src/system.h: ...here.
(is_nul): Reimplement with a version that doesn't
require a sentinel after the buffer, and which calls
down to (the system optimized) memcmp.
Performance analyzed at http://rusty.ozlabs.org/?p=560
* src/dd.c (alloc_obuf): Simplify the is_nul() call by
not needing to write the sentinel.
* src/copy.c (sparse_copy): Likewise.
(copy_reg): Simplify the buffer allocation by avoiding
consideration of the sentinel in the buffer size calculation.
These strings are often file names or other user specified
parameters, which can give confusing errors in
the presence of unexpected characters for example.
* cfg.mk (sc_error_quotes): A new syntax check rule.
* src/*.c: Wrap error() string arguments with quote().
* tests/: Adjust accordingly.
* NEWS: Mention the improvement.
This is especially significant when using --check
with files generated on a windows system, where the \r
characters produce corrupted and confusing error messages.
This also ensures status messages are output on a single line.
* src/md5sum.c: Use quote() for printed file names.
* tests/misc/md5sum.pl: Adjust accordingly.
* NEWS: Mention the change in behavior.
Fixes http://bugs.gnu.org/21757
* src/ls.c (main): Account for the first column not including
a separator when calculating max_idx.
* tests/ls/w-option.sh: Add a test case.
* NEWS: Mention the bug fix.
* src/ls.c (print_with_separator): Renamed from print_with_commas,
and parameterized to accept the separator to print.
Also fix an edge case where '\n' not printed when
the POS variable overflows SIZE_MAX.
(print_current_files): Degenerate -x and -C to using the
cheaper print_with_separator() in the -w0 case.
* doc/coreutils.texi (ls invocation): Document the new feature.
* tests/ls/w-option.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the improvement.
Fixes http://bugs.gnu.org/21325
* src/ls.c (know_term_type): Corresponding to commit v8.24-48-gc249a5a,
use fnmatch to inspect the dircolors database. Noticed due to
failing tests/ls/color-{dtype-dir,term}.sh tests.
The gnulib provided sc_tight_scope target was ineffective,
as it was checking against an invalid blank regular expression,
and thus ignoring any extern function issues. This is now
fixed up in gnulib, and so we need to fix our scoping issues
before the next gnulib update.
* cfg.mk: Setup and document the tight_scope config variables
appropriately.
* src/selinux.h: Since declared in *_SOURCES, use the two line
form for the extern function declarations.
* src/set-fields.h: Add the extern declarations, and since declared
in noinst_HEADERS use the single line form.
It was a little confusing as to whether the SQUFOF algorithm was
enabled, and in fact there were no options available to enable it.
Therefore clarify the 3 configurable behaviors for the code to
3 defines at the top of the program, and only include the SQUFOF
code if enabled at compile time.
$ size src/factor-before
text data bss
93997 1412 2504
$ size src/factor-after
text data bss
87885 1404 2504
* src/factor.c: Only include the SQUFOF factor code
when enabled via the USE_SQUFOF define.
* doc/coreutils.texi (factor invocation): Update note about
factor limits, as we can factor 128 bit numbers without GMP.
* tests/local.mk (all_tests): Remove the tests 'tests/id/setgid.sh' and
'tests/mkdir/smack-root.sh' because they are mentioned in the
'all_root_tests' list; these tests are skipped anyway during a non-root
run because flagged with 'require_root_'.
sha512sum can be faster than sha256sum.
E.g., ‘dd if=/dev/zero bs=1024k count=1024 | time sha256sum’
reports 8.16 user CPU seconds on my host, whereas sha512sum
consumes 5.45 seconds (Fedora x86-64 on an AMD Phenom II X4 910e).
Although sha512sum is still considerably slower on x86, a good
chunk of uses are on 64-bit hosts and anyway there’s little point
to scaring people away from sha512sum nowadays.
* doc/coreutils.texi (sha2 utilities): Remove obsolete comment.
* src/uniq.c (main): Make the type of "nfiles" unsigned,
to avoid a brand new warning from a gcc I built from today's
sources (gcc version 6.0.0 20151015 (experimental) (GCC)):
src/uniq.c:523:14: error: assuming signed overflow does not occur \
when simplifying conditional to constant [-Werror=strict-overflow]
if (nfiles == 2)
^
* src/dircolors.c (dc_parse_stream): Support globbing of
TERM entries, to allow entries like "TERM *256color*" for example.
* src/dircolors.hin: Reduce the internal list with globbing.
* tests/misc/dircolors.pl: New test cases.
* NEWS: Mention the improvement.
* tests/rm/r-root.sh: Skip the test if there are gdb warnings
that will impact further stderr checks. For example some
buggy gdb versions may report "Got object file from memory
but can't read symbols: File truncated". Also fix an incorrect
stderr check from the previous change.
Reported by Bernhard Voelker.
* tests/rm/r-root.sh: Use gdb rather than timeout(1) as the
last resort protection against unlinkat() calls. The timeout
of 2s was susceptible to false positives under load, and
gdb is stronger protection in any case. We remove the
"expensive" tag on this test also since it should be robust.
Reported by Jim Meyering.
* tests/tail-2/follow-stdin.sh: Use the standard tail
testing framework to avoid the race seen under very high load,
and also test the non inotify case.
Reported by Jim Meyering
To reproduce:
setfacl -dm group::rx .
setfacl -dm other::rx .
make check
* init.cfg (require_no_default_acl_): A new function to skip
when default ACLs are detected, or if the getfacl utility is
not available then skip if any non LSM ACLs detected.
* tests/cp/existing-perm-race.sh: Call require_no_default_acl_.
* tests/mkdir/parents.sh: Likewise.
* tests/mkdir/perm.sh: Likewise.
* src/tail.c (fremote): No longer prompt the user to email
with the unrecognized file system constant, since we have
process in place to sync periodically with the latest Linux
constants, and the fall back polling mode is still fully functional.
* tests/cp/cp-a-selinux.sh: Ensure we skip the portion of the test
depending on restorecon to be effective. I.E. also skip when restorecon
warns, as it doesn't exit with error status when matchpathcon fails to
find a match for a file. This is the case in /tmp on Fedora for
example, in which case the new destination that cp creates will have the
default security context of the process, rather than the explicit
context we set on the source file.
Details at: http://bugzilla.redhat.com/1247641
* tests/misc/sort-compress-hang.sh: Use --foreground with the
timeout(1) command (noting the caveats), to run the sort command
in the foreground program group, and thus be responsive to Ctrl-C.
This very_expensive_ test takes over a minute on a i3-2310M,
with RAM backed /tmp.
* tests/dd/no-allocate.sh: Account for timeout(1) when
determining the required mem, as timeout has additional shared libs.
This avoids the need for the hardcoded 4M addition to the limit.
* tests/misc/head-c.sh: Increase the base limit, to account for
the fact that head(1) will allocate some additional mem in this case.
* tests/misc/cut-huge-range.sh: Remove mention of specific limits.
* tests/misc/printf-surprise.sh: Likewise.
Reported by Dmitry V. Levin
* src/tee.c (tee_files): Last arg is now char ** instead of char
const **, as that is a bit simpler. All callers changed. Modify
files[-1], not files[nfiles], as that is a bit faster and simpler.
Latter problem pointed out by Rainer Deyke in:
http://bugs.gnu.org/21611
This pacifies GCC 5 in a better way, without disabling diagnostics.
* src/df.c (main): Tell compiler that optind is positive.
* src/shred.c (known): New function.
(dopass): Go back to off_t for file sizes.
Avoid integer overflow if we run off the end of the file.
Tell compiler that a write cannot write more bytes than requested.
Avoid the intermittent loss of "... has become inaccessible" messages.
That would cause tests/tail-2/assert.sh to fail sometimes,
mainly on uniprocessor systems.
* src/tail.c (tail_forever_inotify): Also monitor IN_DELETE
events on the directory, to avoid a dentry unlink()..open() race,
where the open() on the deleted file was seen to succeed after an,
unlink() and a subsequent IN_ATTRIB, was sent to tail. Note an
IN_ATTRIB is sent on the monitored file to indicate the change in
number of links, and we can't just use a decrease in the number of
links to determine the file being unlinked, due to the possibility
of the file having multiple links.
Reported by Assaf Gordon and Ludovic Courtès.
Fixes http://bugs.gnu.org/21460
* src/shred.c (dopass): With -O2, GCC 5.1 gives "assuming signed
overflow does not occur when simplifying conditional to constant"
warnings, in regard to the signed (off_t) variables. Therefore
use unsigned (uintmax_t) instead, and a separate boolean to cater
for the special meaning of the negative part of the integer range.
Noticed at http://hydra.nixos.org/build/24983447
Includes a change to xalloc.h to avoid -Wstrict-overflow warnings
with GCC 5.1 on 32 bit with optimization enabled. A subsequent
commit will fix similar issues in shred.
* src/dircolors.hin: Add "xterm-termite" as this VTE based terminal
emulator is quite different from xterm, despite the name.
For example "Termite supports italic text and it won't work if TERM
is set to xterm. Even the backspace key won't work properly anymore
for applications relying on terminfo".
Reported also by Lukas Sabota and Sven-Hendrik Haase.
In the presence of bind mounts of a device, the 4th "mount root" field
from /proc/self/mountinfo is now considered, so as to prefer mount
points closer to the root of the device. Note on older systems with
an /etc/mtab file, the source device was listed as the originating
directory, and so this was not an issue.
Details at http://pad.lv/1432871
* src/df.c (filter_mount_list): When deduplicating mount entries,
only prefer sources nearer or at the root of the device, when the
target is nearer the root of the device.
* NEWS: Mention the change in behavior.
This includes a change to propagate the 4th "mount root"
field from /proc/self/mountinfo from the mountlist module,
which is needed in a subsequent commit in df.
* gl/lib/regcomp.c.diff: Regenerate against latest gnulib.
* gl/lib/regex_internal.c.diff: Likewise.
* gl/lib/regex_internal.h.diff: Likewise.
* cfg.mk: Exclude diffs from trailing whitespace check,
which is generally correct, and now needed.
When configured with either 'symlinks' or 'shebangs' as value for
the --enable-single-binary option, tests based on `ulimit -v` are
skipped. The reason is that the multicall 'coreutils' binary requires
much more memory due to shared libraries being loaded, and the size of
the 'date' binary (~290KiB) compared to the multicall binary (~5MiB),
of course. Finally, in the case of 'shebangs', the starting shell
requires more memory, too
Instead of using hard-coded values for the memory limit, use an
adaptive approach: first determine the amount of memory for a similar,
yet more trivial invocation of the command, and then do the real test
run using that limit (plus some buffer in some cases).
* init.cfg (require_ulimit_v_): Remove function.
(get_min_ulimit_v_): Add function to determine the minimum memory limit
required for a given command in an adaptive way.
* cfg.mk (sc_prohibit_test_ulimit_without_require_): Change the name
of the above function in the syntax-check rule.
* tests/cp/link-heap.sh: Use the above function to determine the
minimum memory required to run a command simpler than in the real test
run. Use that limit plus a buffer there. While at it, change to list
of commands in the subshell to fail also if the beginning `ulimit -v`
fails.
* tests/dd/no-allocate.sh: Likewise.
* tests/misc/csplit-heap.sh: Likewise.
* tests/misc/cut-huge-range.sh: Likewise.
* tests/misc/head-c.sh: Likewise.
* tests/misc/printf-surprise.sh: Likewise.
* tests/split/line-bytes.sh: Likewise.
* tests/rm/many-dir-entries-vs-OOM.sh: Likewise - doing it separately
for each program under test.
* src/runcon.c (main): As per the compile time warning from
libselinux-2.4-3, lookup the class with string_to_security_class(),
rather than using defines from flask.h.
* src/sort.c (main): With --debug, warn upon setlocale() failure,
which can happen due to incorrectly specified environment variables,
or due to memory exhaustion (simulated with ulimit -v), etc.
* tests/misc/sort-debug-warn.sh: Add a test case.
See also http://savannah.gnu.org/bugs/11004
du calls stat for each mount point at startup. This would block or
even make du fail if stat for an unrelated mount point hangs.
The result is not needed in the normal case anyway and therefore
should be avoided. Issue introduced in commit v8.19-2-gcf7e1b5.
* src/du.c (fill_mount_table): Move function up as it's not used ...
(mount_point_in_fts_cycle): ... here, i.e., the DI_MNT set is
initialized and filled only iff FTS has detected a directory cycle.
(main): Remove the initialization and filling of the DI_MNT set here,
and free the DI_MNT set only if it was used.
The -NUMBER option was removed from ‘shred’ in 1999, but the
manual wasn’t updated to match. Problem reported by Nick Rose in:
http://bugs.gnu.org/21502
* doc/coreutils.texi (shred invocation):
Remove documentation for -NUMBER option.
numfmt --field=LIST can accept the same options as cut.
* bootstrap.conf: remove xlist, linked-list
* src/local.mk: link numfmt with set-fields
* src/numfmt.c: use set-fields.c instead of custom field parsing code.
(include_field): adapt to new code.
* tests/misc/numfmt.pl: add new tests, adapt current tests to new
error message wording from set-fields.c
Extract the functionality of parsing --field=LIST into a separate
module, to be used by other programs.
* src/cut.c: move field parsing code from here ...
* src/set-fields.{c,h}: ... to here.
(set_fields): generalize by supporting multiple parsing/reporting
options.
(struct range_pair): rename to field_range_pair.
* src/local.mk: link cut with set-field.
* po/POTFILES.in: add set-field.c
* tests/misc/cut.pl: update wording of error messages
* src/sort.c (main): Ensure we don't free() and invalid
pointer when reading implicit stdin. Also avoid
"definitely lost" valgrind warnings in the --files0-from case.
Since commit v8.23-19-g8defcee, main() will return,
rather than call exit(), this inducing "definitely lost"
warnings in valgrind's leak checker. That precludes using
the following to flag memory leaks:
valgrind --leak-check=full --error-exitcode=1 \
--errors-for-leak-kinds=definite
* src/pr.c (main): In dev builds, explicitly free memory allocated.
* src/sort.c (main): Likewise.
* src/tail.c (main): Likewise.
* src/tsort.c (tsort): Likewise.
* src/ls.c (getenv_quoting_style, decode_switches, parse_ls_color):
Use quote() rather than quotearg(), as the latter defaults to
outputting the input unquoted.
* src/ptx.c (main): Likewise.
* src/base64.c (main): Support decimal numbers with leading zeros,
by disabling the auto detection of octal and hex. It's not
envisaged that base conversion is needed for --wrap parameters,
and in the edge case it is, $((0x0)) shell constructs can be used.
* tests/misc/base64.pl: Adjust accordingly.
* NEWS: Mention the change in behavior.
* src/base64.c (main): Use the higher level xnumtoumax()
rather than xstrtoumax(), which is simpler and improves
validation of input. Also pass the _empty_ rather than NULL
string as the suffixes parameter so that invalid trailing
characters are not allowed. For example -w08 is now
flagged as an error, rather than being interpreted as 0.
A subsequent commit will further improve verification
of numbers with leading zeros by dropping backwards compatibility
wrt auto parsing oct and hex numbers.
* tests/misc/base64.pl: Add tests for invalid wrap values.
Suggested in https://bugzilla.redhat.com/1250113
* AUTHORS: Add base32.
* THANKS.in: Add suggester.
* README: Reference the new program.
* NEWS: Mention the new program.
* src/.gitignore: Ignore the new binary.
* bootstrap.conf: Reference the gnulib base32 module.
* build-aux/gen-lists-of-programs.sh: Add base32.
* man/base32.x: A new template.
* man/.gitignore: Ignore the new man page.
* man/local.mk: Reference the new man page.
* doc/coreutils.texi (base32 invocation): Document the new command.
* src/local.mk: Adjust to build base32 based on base64.c.
* src/base64.c: Parameterize to use the correct headers,
functions and buffer sizes, depending on which binary
is being built.
* tests/misc/base64.pl: Adjust to test both base32 and base64.
* tests/misc/tty-eof.pl: Add base32 as a program that
accepts input on stdin without any options specified.
* scripts/git-hooks/commit-msg: Add base32 to the template.
* src/shred.c (usage): For -u, separate the decscription
of the short and long option, to clarify that the short option
takes no parameter.
* src/split.c (usage): Likewise for -d.
* src/tee.c (usage): Likewise for -p.
* src/uniq.c (usage): Likewise for -D.
Suggested by Stephane Chazelas
Since glibc-2.22, specifically commit [0], the opendir() implementation
implicitly makes an additional stat call thus leading to a FP.
Seen on openSUSE:Tumbleweed since snapshot 20150821.
[0]
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=46f894d8c60a
* tests/ls/stat-free-color.sh: Change the test to verify that ls(1)
needs the same number of stat-like calls for a single, empty directory
argument as for one with a few directory entries (sub-directory,
regular file, symlink, etc.).
* doc/coreutils.texi (split invocation): Clarify that -d takes no param.
(uniq invocation): Likewise for -D.
(shred invocation): Likewise for -u.
(tee invocation): Likewise for -p.
* cfg.mk (sc_man_check_x_copyright): Add rule to ensure that non-trivial
.x files in the 'man/' subdirectory, i.e., files exceeding a line count
of 20 or a byte count of 1000, contain a proper Copyright notice.
* bootstrap.conf (gnulib_modules): Add time_rz,
since the main source code now uses timezone_t.
* src/date.c (batch_convert, main, show_date):
* src/ls.c (align_nstrftime, long_time_expected_width)
(print_long_format):
* src/stat.c (human_time):
Use timezone_t rather than boolean to specify which time zone
is wanted.
* src/ls.c (localtz): New static var.
(main): Initialize it.
* src/uname.c (usage): State that the non POSIX -i and -p options
are non-portable.
* doc/coreutils.texi (uname invocation): Mention the discrepancies
even across GNU/Linux distros, and that the results should
be used as informational only, rather than impacting any
logic decisions.
Fixes http://bugs.gnu.org/13001
* src/realpath.c (usage): Mention 'directory' in the --help
output, so that ENOTDIR errors may be more easily investigated,
by inspecting the man page.
Reported at http://pad.lv/1474519
This was detected in about 25% of runs with gcc -fsanitize=address
ERROR: AddressSanitizer: global-buffer-overflow on address ...
READ of size 4 at 0x000000416628 thread T0
#0 0x40479f in genpattern src/shred.c:782
#1 0x4050d9 in do_wipefd src/shred.c:921
#2 0x406203 in wipefile src/shred.c:1175
#3 0x406b84 in main src/shred.c:1316
#4 0x7f3454a1ef9f in __libc_start_main (/lib64/libc.so.6+0x1ff9f)
#5 0x4025d8 (/tmp/coreutils-8.23/src/shred+0x4025d8)
0x000000416628 is located 56 bytes to the left of
global variable '*.LC49' from 'src/shred.c' (0x416660) of size 17
0x000000416628 is located 12 bytes to the right of
global variable 'patterns' from 'src/shred.c' (0x416540) of size 220
SUMMARY: AddressSanitizer: global-buffer-overflow src/shred.c:782
* src/shred.c (gen_patterns): Restrict pattern selection
to the K available, which regressed due to v5.92-1462-g65533e1.
* tests/misc/shred-passes.sh: Add a deterministic test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/20998
'\" Copyright (C) 2018-2021 Free Software Foundation, Inc.
'\"
'\" This is free software. You may redistribute copies of it under the terms
'\" of the GNU General Public License <https://www.gnu.org/licenses/gpl.html>.
'\" There is NO WARRANTY, to the extent permitted by law.
[NAME]
basenc \- Encode/decode data and print to standard output
[DESCRIPTION]
.\" Add any additional description here
[ENCODINGS EXAMPLES]
.PP
.nf
.RS
$ printf '\\376\\117\\202' | basenc \-\-base64
/k+C
$ printf '\\376\\117\\202' | basenc \-\-base64url
_k-C
$ printf '\\376\\117\\202' | basenc \-\-base32
7ZHYE===
$ printf '\\376\\117\\202' | basenc \-\-base32hex
VP7O4===
$ printf '\\376\\117\\202' | basenc \-\-base16
FE4F82
$ printf '\\376\\117\\202' | basenc \-\-base2lsbf
011111111111001001000001
$ printf '\\376\\117\\202' | basenc \-\-base2msbf
111111100100111110000010
$ printf '\\376\\117\\202\\000' | basenc \-\-z85
@.FaC
.RE
.fi
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
