Proposal To Extend .atf Files - Revision history

Sudleyplace: /* A Proposal */

2008-10-21T18:09:05Z

A Proposal

← Older revision		Revision as of 14:09, 21 October 2008
Line 11:		Line 11:

	Extend the file format to [http://en.wikipedia.org/wiki/UTF-16 UTF-16] (16-bit words with variable width characters, one or two words per character) possibly with a leading [http://en.wikipedia.org/wiki/Byte_Order_Mark BOM] (Byte Order Mark). This would also eliminate the need to translate to/from the APL2 character set as everything would be in a common character set, i.e., UTF-16. Note that the existing .atf file format is already considered a binary file because it may contain bytes beyond 0x7F, so switching to UTF-16 wouldn't change that.		Extend the file format to [http://en.wikipedia.org/wiki/UTF-16 UTF-16] (16-bit words with variable width characters, one or two words per character) possibly with a leading [http://en.wikipedia.org/wiki/Byte_Order_Mark BOM] (Byte Order Mark). This would also eliminate the need to translate to/from the APL2 character set as everything would be in a common character set, i.e., UTF-16. Note that the existing .atf file format is already considered a binary file because it may contain bytes beyond 0x7F, so switching to UTF-16 wouldn't change that.

	~~Also, there would be no need to use <apll>⎕AF</apll> or <apll>⎕UCS</apll> in the file.~~

	== Byte Order Mark ==		== Byte Order Mark ==

Sudleyplace: /* A Proposal */

2008-10-21T17:55:20Z

A Proposal

← Older revision		Revision as of 13:55, 21 October 2008
Line 10:		Line 10:
	Given that more and more APL systems support Unicode, that suggests a possible solution:		Given that more and more APL systems support Unicode, that suggests a possible solution:

	Extend the file format to [http://en.wikipedia.org/wiki/UTF-16 UTF-16] (words with variable width characters, one or two words per character) possibly with a leading [http://en.wikipedia.org/wiki/Byte_Order_Mark BOM] (Byte Order Mark). This would also eliminate the need to translate to/from the APL2 character set as everything would be in a common character set, i.e., UTF-16. Note that the existing .atf file format is already considered a binary file because it may contain bytes beyond 0x7F, so switching to UTF-16 wouldn't change that.		Extend the file format to [http://en.wikipedia.org/wiki/UTF-16 UTF-16] (16-bit words with variable width characters, one or two words per character) possibly with a leading [http://en.wikipedia.org/wiki/Byte_Order_Mark BOM] (Byte Order Mark). This would also eliminate the need to translate to/from the APL2 character set as everything would be in a common character set, i.e., UTF-16. Note that the existing .atf file format is already considered a binary file because it may contain bytes beyond 0x7F, so switching to UTF-16 wouldn't change that.

			Also, there would be no need to use <apll>⎕AF</apll> or <apll>⎕UCS</apll> in the file.

	== Byte Order Mark ==		== Byte Order Mark ==

WikiSysop: /* Transfer Form System Function */

2008-07-08T14:10:43Z

Transfer Form System Function

← Older revision		Revision as of 10:10, 8 July 2008
Line 36:		Line 36:
	== Transfer Form System Function ==		== Transfer Form System Function ==

	Just as <apll>)OUT</apll> needs to be told which format to use as output, the system function <apll>⎕TF</apll> needs the same information. For the purposes of testing this idea, the NARS implementation of <apll>⎕TF</apll> uses left ~~arguments~~ of <apll>¯1</apll> and <apll>¯2</apll> to generate Type 1 and 2 forms in UTF-16 format (really UCS-2).		Just as <apll>)OUT</apll> needs to be told which format to use as output and what format is used as input, the system function <apll>⎕TF</apll> needs the same information. For the purposes of testing this idea, the NARS implementation of <apll>⎕TF</apll> uses a left argument of <apll>¯1</apll> and <apll>¯2</apll> to generate or accept Type 1 and 2 transfer forms in UTF-16 format (really UCS-2).

WikiSysop: /* File Extension */

2008-07-08T14:07:03Z

File Extension

← Older revision		Revision as of 10:07, 8 July 2008
Line 20:		Line 20:
	== File Extension ==		== File Extension ==

	Depending upon how robust the existing code for )IN is in each of our systems (that is, how well it handles an unexpected leading BOM and/or UTF-16), we might not even need a new file extension, although the file extension .utf has a certain appeal.		Depending upon how robust the existing code for <apll>)IN</apll> is in each of our systems (that is, how well it handles an unexpected leading BOM and/or UTF-16), we might not even need a new file extension, although the file extension .utf has a certain appeal.

	== UCS-2 Systems ==		== UCS-2 Systems ==

WikiSysop at 14:05, 8 July 2008

2008-07-08T14:05:13Z

← Older revision		Revision as of 10:05, 8 July 2008
Line 4:		Line 4:

	== Current File Format ==		== Current File Format ==
	The current file format is essentially UCS-1 (bytes with fixed width characters, one byte per character). The byte values may range from 0x00 to 0xFF.		The current file format is essentially UCS-1 (bytes with fixed width characters, one byte per character), using the APL2 character set. The byte values may range from 0x00 to 0xFF.

	== A Proposal ==		== A Proposal ==
Line 28:		Line 28:
	== Importing Data ==		== Importing Data ==

	The import mechanism — e.g., system command )IN — needs to be enhanced to recognize files in the new format and import them appropriately.		The import mechanism — e.g., system command <apll>)IN</apll> — needs to be enhanced to recognize files in the new format and import them appropriately.

	== Exporting Data ==		== Exporting Data ==

	The export mechanism — e.g., system command )OUT — needs to be augmented, perhaps with a switch on the command or a new command altogether so as to be told which format (UCS-1 vs. UTF-16) to use when writing out the workspace contents.		The export mechanism — e.g., system command <apll>)OUT</apll> — needs to be augmented, perhaps with a switch on the command or a new command altogether so as to be told which format (APL2 UCS-1 vs. UTF-16) to use when writing out the workspace contents.

			== Transfer Form System Function ==

			Just as <apll>)OUT</apll> needs to be told which format to use as output, the system function <apll>⎕TF</apll> needs the same information. For the purposes of testing this idea, the NARS implementation of <apll>⎕TF</apll> uses left arguments of <apll>¯1</apll> and <apll>¯2</apll> to generate Type 1 and 2 forms in UTF-16 format (really UCS-2).

WikiSysop: /* Byte Order Mark */

2008-07-08T13:52:20Z

Byte Order Mark

← Older revision		Revision as of 09:52, 8 July 2008
Line 14:		Line 14:
	== Byte Order Mark ==		== Byte Order Mark ==

	The BOM is not strictly necessary as a UTF-16 format .atf file could be recognized by noting that the second byte is zero for little-endian systems and the first byte is zero for big-endian systems. However, rather than rely upon the currently limited range of the initial character in a .atf file (which is already used to distinguish between EBCDIC and ASCII), a BOM may be preferred. Alternatively, we can settle on either big-endian (UTF-16BE) or little-endian (UTF-16LE) as the file format and not need a BOM.		The BOM is not strictly necessary because a UTF-16 format .atf file could be recognized by noting that the second byte is zero for little-endian systems and the first byte is zero for big-endian systems. However, rather than rely upon the currently limited range of the initial character in a .atf file (which is already used to distinguish between EBCDIC and ASCII), a BOM may be preferred. Alternatively, we can settle on either big-endian (UTF-16BE) or little-endian (UTF-16LE) as the file format and not need a BOM.

	~~The~~ BOM is U+FEFF — in UTF-16, on big-endian systems, it's represented as (0xFE, 0xFF), and on little-endian systems as (0xFF, 0xFE).		For reference, the BOM is U+FEFF — in UTF-16, on big-endian systems, it's represented as (0xFE, 0xFF), and on little-endian systems as (0xFF, 0xFE).

	== File Extension ==		== File Extension ==

Sudleyplace: /* Byte Order Mark */

2008-06-29T14:03:57Z

Byte Order Mark

← Older revision		Revision as of 10:03, 29 June 2008
Line 16:		Line 16:
	The BOM is not strictly necessary as a UTF-16 format .atf file could be recognized by noting that the second byte is zero for little-endian systems and the first byte is zero for big-endian systems. However, rather than rely upon the currently limited range of the initial character in a .atf file (which is already used to distinguish between EBCDIC and ASCII), a BOM may be preferred. Alternatively, we can settle on either big-endian (UTF-16BE) or little-endian (UTF-16LE) as the file format and not need a BOM.		The BOM is not strictly necessary as a UTF-16 format .atf file could be recognized by noting that the second byte is zero for little-endian systems and the first byte is zero for big-endian systems. However, rather than rely upon the currently limited range of the initial character in a .atf file (which is already used to distinguish between EBCDIC and ASCII), a BOM may be preferred. Alternatively, we can settle on either big-endian (UTF-16BE) or little-endian (UTF-16LE) as the file format and not need a BOM.

	In UTF-16, ~~the BOM is U+FFFE~~ (0xFE, 0xFF) ~~for big~~-endian~~, and U+FEFF~~ (0xFF, 0xFE) ~~for little-endian~~.		The BOM is U+FEFF — in UTF-16, on big-endian systems, it's represented as (0xFE, 0xFF), and on little-endian systems as (0xFF, 0xFE).

	== File Extension ==		== File Extension ==

Sudleyplace: /* A Proposal */

2008-06-27T03:08:13Z

A Proposal

← Older revision		Revision as of 23:08, 26 June 2008
Line 10:		Line 10:
	Given that more and more APL systems support Unicode, that suggests a possible solution:		Given that more and more APL systems support Unicode, that suggests a possible solution:

	Extend the file format to [http://en.wikipedia.org/wiki/UTF-16 UTF-16] (words with variable width characters, one or two words per character) possibly with a leading [http://en.wikipedia.org/wiki/Byte_Order_Mark BOM] (Byte Order Mark). This would also eliminate the need to translate to/from the APL2 character set as everything would be in a common character set, i.e., UTF-16. Note that the existing .atf file format is already considered a binary file because it ~~contains~~ bytes beyond 0x7F, so switching to UTF-16 wouldn't change that.		Extend the file format to [http://en.wikipedia.org/wiki/UTF-16 UTF-16] (words with variable width characters, one or two words per character) possibly with a leading [http://en.wikipedia.org/wiki/Byte_Order_Mark BOM] (Byte Order Mark). This would also eliminate the need to translate to/from the APL2 character set as everything would be in a common character set, i.e., UTF-16. Note that the existing .atf file format is already considered a binary file because it may contain bytes beyond 0x7F, so switching to UTF-16 wouldn't change that.

	== Byte Order Mark ==		== Byte Order Mark ==

Sudleyplace: /* A Proposal */

2008-06-27T03:07:16Z

A Proposal

← Older revision		Revision as of 23:07, 26 June 2008
Line 10:		Line 10:
	Given that more and more APL systems support Unicode, that suggests a possible solution:		Given that more and more APL systems support Unicode, that suggests a possible solution:

	Extend the file format to [http://en.wikipedia.org/wiki/UTF-16 UTF-16] (words with variable width characters, one or two words per ~~characters~~) possibly with a leading [http://en.wikipedia.org/wiki/Byte_Order_Mark BOM] (Byte Order Mark). This would also eliminate the need to translate to/from the APL2 character set as everything would be in a common character set, i.e., UTF-16. Note that the existing .atf file format is already considered a binary file because it contains bytes beyond 0x7F, so switching to UTF-16 wouldn't change that.		Extend the file format to [http://en.wikipedia.org/wiki/UTF-16 UTF-16] (words with variable width characters, one or two words per character) possibly with a leading [http://en.wikipedia.org/wiki/Byte_Order_Mark BOM] (Byte Order Mark). This would also eliminate the need to translate to/from the APL2 character set as everything would be in a common character set, i.e., UTF-16. Note that the existing .atf file format is already considered a binary file because it contains bytes beyond 0x7F, so switching to UTF-16 wouldn't change that.

	== Byte Order Mark ==		== Byte Order Mark ==

Sudleyplace: /* File Extension */

2008-06-27T03:03:20Z

File Extension

← Older revision		Revision as of 23:03, 26 June 2008
Line 20:		Line 20:
	== File Extension ==		== File Extension ==

	Depending upon how robust the existing code for )IN is in each of our systems (that is, how well it handles an unexpected leading BOM), we might not even need a new file extension, although the file extension .utf has a certain appeal.		Depending upon how robust the existing code for )IN is in each of our systems (that is, how well it handles an unexpected leading BOM and/or UTF-16), we might not even need a new file extension, although the file extension .utf has a certain appeal.

	== UCS-2 Systems ==		== UCS-2 Systems ==