• Transcoding of characters is performed to/from the internal UTF-32 format when reading/writing to disk.

• The field length represents the number of characters that can be stored in the alpha field.

APPX Data Dictionary Considerations: (untested)

• All existing alpha fields will have a default single-byte encoding type of RAW.
• DD length is the number of 8-bit bytes that the field can store.
• Internal format in the engine is 8-bit bytes.
• May contain binary data.
• May contain 8-bit characters (uses 8859-15 encoding when interpreted as character data).
• Stored in APPXIO records as 8-bit bytes.
• No transcoding is performed when reading/writing a RAW field in APPXIO.

UNICODE Encoding:

• DD length is the number of 4-byte UTF-32 characters that the field can store.
• Internal format in the engine is UTF-32 (4 bytes per character).
• Used to store Unicode characters in a UTF-32 encoding.
• Stored in APPXIO as 4-byte UTF-32 characters.
• No transcoding is performed when reading/writing a UNICODE field in APPXIO.

NATIONAL Encoding:

• DD length is the number of 8-bit characters that the field can store.
• Internal format in the engine is UTF-32 (4 bytes per character).
• Used to store Unicode characters using a specified 8-bit encoding method.
• The encoding method is specified in System Admin (SYSPARM, SYSDFLT, and SYSDIR).
• Stored in APPXIO records as 1-byte characters.
• Transcoding of characters is performed to/from the internal UTF-32 format when reading/writing an NATIONAL field in APPXIO.

APPX Data Dictionary Considerations:

• The field length represents the number of characters that can be stored in the alpha field

PDF Fields That Are Now UNICODE:

• All TEMP fields are now Unicode.
• --- COMPARE CURRENT VALUE
• --- COMPARE OTHER VALUE
• --- STD REPORT TITLE
• --- STD SEP CHAR
• --- STD PAGE HEADING
• --- STD COLUMN HEADING
• --- STD SEPARATOR HEADING
• --- STD DATE
• --- STD PAGE DESC

FMS Considerations: (initial implementation is for Appx-IO and Oracle only)

• Appx-IO
  • Supports all 3 Encoding types
  • Transcodes all NATIONAL encoded fields to/from our internal UTF-32 encoding.
• Oracle
  • Transcodes all alpha fields to/from Oracle DB encoding and our UTF-32 encoding
  • File creates should now use
    • Long data types for large fields
    • Appropriate column types for UNICODE and NATIONAL types.

Restructure Considerations:

Client Considerations:

Test Plan:

APPX Data Dictionary Considerations:

• Does the DD allow encoding type for all alpha field types to be specified?
• Does the DD compiler support 1-byte or 4-byte characters in alpha fields?

ILF Statements Effected:

• APPEND
• BEG AT/END AT
• BROWSE
• CALL
• CNV BIN
• CNV PORT – 0DB Export/Import routines
• CNV TEXT – Used throughout 0-Apps
• COMPUTE/TOTAL (hash totals)
• DEFINE/UNDEFINE
• IF/AND/OR
• IF DIFF - Sets fields values into COMPARE file
• PACK/UNPACK
• PASS/RECEIVE/LOCAL/GOSUB
• PRINT - Only used in STD Report Heading routines
  • References WORK fields
  • References --- STD fields
  • Is not used with any TEMP fields
• PUSH/POP
• RUN
• SCAN
• SET
• SET DAY/SET MNTH
• SET TEMP
• STORE/RESTORE
• VERIFY
• WARNING/ERROR/CANCEL/MESSAGE/REFRESH

Process Specific Considerations:

• Does the QUERY processor use correct collating sequence when sorting all UNICODE alpha field types?

Unicode Encoding Types

Add new encoding types to alpha fields, RAW, NATIONAL, and UNICODE.

Overview

Alpha fields will have three choices for Encoding.

• RAW
• NATIONAL
• UNICODE

Field Type considered Alpha are:

• ALPHA
• DATE with ALPHA Storage Type
• FORMAT
• TEXT

RAW Encoding:

Allow large alpha fields to be defined in the APPX DD (up to 4GB)

Note: Even thought the theoretical maximum size of an alpha field is 4GB, we have imposed edit checks to limit the maximum size of an alpha field to 1MB (1048576) in characters.

Note: Large alpha field implementation in the initial release is only implemented for Permanent disk files stored in Oracle.

This change effects...

• Permanent (Oracle) disk files
• Memory Files
• Working Storage files
• Work fields

Exceptions...

• Consecutive disk files (no initial support for large alpha fields)
• Temporary disk files (no initial support for large alpha fields)
• One-Record disk files (no initial support for large alpha fields)
• Non-Oracle Permanent Disk Files (no initial support for large alpha fields)

Note: Large Field support on Data entry is limited to the Java Client at this time. The Character mode and Win32 client do not support large fields.

Test Plan:

DD Edit Checks: (untested)

Test Requirements:

Note: Test these with key and non-key fields

Does the Data Dictionary Editor enforce a 1M field size limit for...

• Permanent disk files?
• Memory Files?
• Working Storage files?
• Work fields?

Does the Data Dictionary Editor enforce the original 32K limit for...

• Consecutive disk files?
• Temporary disk files?
• One-Record disk files?

Test Results and Notes:

Untested

Storage and Processing Capabilities: (untested)

Test Requirements:

Note: Test there with key and non-key fields

Note: Perform the following tests for the following...

• Permanent disk files
• Memory Files
• Working Storage files
• Work fields

Note: The following FMS interfaces will not initially support large fields...

• Appx-IO
• DB2
• ODBC
• SQL Server
• C-ISAM
• Vision

Does Restructure handle the large field correctly?

Does the Oracle interface support large fields correctly?

Do you get a reasonable error message when attempting a non-supported FMS type?

Using a Subroutine process, can you store and retrieve 1M character in an alpha field?

Using an Input process can you store and retrieve 1M character in an alpha field using the Java Client?

Using an Output process can you store and retrieve 1M characters in an alpha field for the following as Text and PDF...
(I'm not sure this is a doable test, is there a way to print 1M character in an Appx report from a single field?)

Using a Query process can you correctly Sort and Select on 1M alpha fields?

Using Child Constraints can you correctly access parent/child data with 1M alpha fields?

Does TECHDOC print the field sizes correctly up to the max of 1M?

Test Results and Notes:

untested

ILF Statements: (untested)

Test Requirements:

Do the following ILF Statements handle 1M alpha fields correctly?

• APPEND
• BEG AT / END AT
• IF / AND / OR - All relational operators
• IF DIFF
• PASS / RECEIVE
• SET
• STORE / RESTORE

Test Results and Notes:

untested

-- PeteBrower - 2011-03-04