Runtime unformatted I/O conversion between big and little endian formats

[d-parks]

Flang supports both compile time and runtime methods of specifying the endianness of unformatted files, via the OPEN statement’s CONVERT= specifier in the source code or the FORT_CONVERT environment variable at runtime.

Flang’s implementation of specifying a file’s endianness at runtime via FORT_CONVERT differs from both GNU, NVIDIA, and Intel’s implementations. Specifically, flang will only honour the FORT_CONVERT setting iff the CONVERT= specifier not specified on the OPEN statement.

From flang-rt/lib/runtime/external-unit.cpp:

bool ExternalFileUnit::OpenUnit(common::optional<OpenStatus> status,
    common::optional<Action> action, Position position,
    OwningPtr<char> &&newPath, std::size_t newPathLength, Convert convert,
    IoErrorHandler &handler) {
  if (convert == Convert::Unknown) {
    convert = executionEnvironment.conversion;
  }

Does anyone know if this behaviour is intentional or a simple oversight?

GNU states their GFORTRAN_CONVERT_UNIT runtime behaviour is (https://gcc.gnu.org/onlinedocs/gfortran/GFORTRAN_005fCONVERT_005fUNIT.html:

Note that the values specified via the GFORTRAN_CONVERT_UNIT environment variable override the CONVERT specifier in the OPEN statement. This is to give control over data formats to users who do not have the source code of their program available.

NVIDIA’s and Intel’s implementations and documentation express similar behaviour.

[kiranchandramohan]

FYI. Have not checked whether this is relevant for your question but there were some discussions when adding the fconvert option. Please check the following github issue and the discussion and links there.

[Flang] Support the `-fconvert` option for switching endiannes

[szakharin]

It looks like this behavior was introduced in [flang] Give explicit convert= specifiers precedence over FORT_CONVERT · llvm/llvm-project@bfcd536 · GitHub, whereas before that the ENV variable had a higher priority.

The author says:

In discussing the appropriate priority for the fconvert option,
convert specifiers were decided to take highest priority.

I am not sure what those discussions were, but it looks like this change made Flang diverge from other compilers’ behavior. I would prefer to follow what gfortran is doing.

[kiranchandramohan]

Some of the discussion is in the github issue i gave above. Some more might have happened during the Flang calls.

[d-parks]

My inquiry originated because I’m close to sending out a pull request for review that adds a new runtime environment variable FORT_CONVERT_UNIT which mimicks GNU’s GFORTRAN_CONVERT_UNIT behaviour, but suddenly realized that flang’s FORT_CONVERT differed. Flang’s implementation of FORT_CONVERT is an outlier among other Fortran compilers.

[szakharin]

After reading Peter’s comment at [Flang] Support the `-fconvert` option for switching endiannes · Issue #55961 · llvm/llvm-project · GitHub, I am not 100% sure any more

Maybe we should indeed have two ENV variables: one for overriding every open statement, and another for setting the convert for open statements that do not specify it (neither explicitly nor via compilation option).

[sscalpone]

Hi Dave! (And everyone else too)

We could offer both global a default configuration and hard per-unit overrides by establishing a precedence between FORT_CONVERT, FORT_CONVERT_UNIT, and in-code OPEN(CONVERT=...) specifiers.

Here is a proposed behavior and precedence hierarchy:

• FORT_CONVERT: Sets the global default conversion for all units, but defers to explicit CONVERT= specifiers on an OPEN statement in the source code.

• FORT_CONVERT_UNIT: Sets the global default conversion and overrides explicit CONVERT= specifiers in the source code.

• Per-Unit Settings (both variables): Both FORT_CONVERT and FORT_CONVERT_UNIT should accept per-unit configuration strings (e.g., specifying SWAP for specific unit numbers). Per-unit settings always override explicit in-code CONVERT= specifiers, regardless of which environment variable defines them.

Precedence Summary

1. Highest Priority: Per-unit environment settings (from either FORT_CONVERT or FORT_CONVERT_UNIT).

2. Second Priority: In-code OPEN(CONVERT=...) specifiers.

3. Third Priority: Global default settings from FORT_CONVERT_UNIT (overrides in-code specifiers when no per-unit rule applies).

4. Lowest Priority: Global default settings from FORT_CONVERT (yields to in-code specifiers).

This approach provides a flexible mechanism: developers can set global defaults without breaking explicit code logic, while site administrators and script runners retain full per-unit control when overriding binaries.

If the community is open to this design, I’d love to hear your thoughts on implementing FORT_CONVERT_UNIT and extending FORT_CONVERT’s syntax to support per-unit rules.

[d-parks]

Hi Steve!

In my sandbox, I’ve got the FORT_CONVERT_UNIT implemented with the semantics of NVIDIA’s FORT_CONVERT_UNIT and GNU’s GFORTRAN_CONVERT_UNIT environment variables along with changing the precedence of the CONVERT= specifier (this needs changing).

I’d like to explore the need for both FORT_CONVERT and FORT_CONVERT_UNIT supporting per unit endianness. Do you think supporting unit specific settings for both environment variables could be confusing? Additionally, could we explore the possibility that the global setting from FORT_CONVERT_UNIT overrides FORT_CONVERT when both environment variables are present (and correct)?

[d-parks]

Taking @sscalpone 's suggestions into account, but simplifying what environment variables can have optional unit numbers, here’s my flang/docs/RuntimeEnvironment.md write up for FORT_CONVERT and FORT_CONVERT_UNIT:

FORT_CONVERT

Determines data conversions applied to all unformatted I/O units that do not have an explicit CONVERT= specification on OPEN statements.
FORT_CONVERT is an alias for the FORT_CONVERT_UNIT environment variable without any exceptions.

FORT_CONVERT=mode

• mode:
• NATIVE: no conversions (default)
• LITTLE_ENDIAN: assume input is little-endian; emit little-endian output
• BIG_ENDIAN: assume input is big-endian; emit big-endian output
• SWAP: reverse endianness (always convert)

FORT_CONVERT_UNIT

Determines data conversion applied to specific unformatted I/O units.

FORT_CONVERT_UNIT= mode | mode ';' exception | exception ;
mode: 'NATIVE' | 'LITTLE_ENDIAN' | 'BIT_ENDIAN' | 'SWAP';
exception: mode ':' unit_list | unit_list ;
unit_list: unit_spec | unit_list ',' unit_spec ;
unit_spec: integer | integer '-' integer ;
integer: [0-9]+ ;

Endianness of unformatted files is determined in ascending order:

1. Global setting from FORT_CONVERT=mode or FORT_CONVERT_UNIT=mode
   environment variables. Contradictory mode settings between FORT_CONVERT and
   FORT_CONVERT_UNIT result in FORT_CONVERT_UNIT taking priority.
2. Explicit setting of CONVERT= specifier on OPEN statement.
3. Exception setting for individual unit or range of units from the
   FORT_CONVERT_UNIT environment variable.

Examples

• Environment variables FORT_CONVERT,FORT_CONVERT_UNIT, and OPEN statement CONVERT= specifier all missing: no data conversion for unformatted I/O (uses host processor encoding).
• Environment variable FORT_CONVERT=BIG_ENDIAN and CONVERT= specifier not present on OPEN statement: assume input is big-endian; emit big-endian output
• Little endain host with environment variable FORT_CONVERT=SWAP and OPEN statement CONVERT=LITTLE_ENDIAN specified: OPEN statement takes precedence: assume input is little-endian; emit little-endian output
• Opening unit 10 on a little endian host with environment variable FORT_CONVERT=SWAP, FORT_CONVERT_UNIT=BIG_ENDIAN:10, and OPEN statement CONVERT=LITTLE_ENDIAN specified: exception BIG_ENDIAN:10 takes precedence: assume input is big-endian; emit big-endian output.

This approach addresses Peter’s concerns from 2022 that CONVERT= specifiers on OPEN statements should have some priority over a global setting. CONVERT= specifiers are now overridden with exception.

Thoughts?