[haiku-bugs] Re: [Haiku] #8473: Concerns about ISO 9660 implementation

#8473: Concerns about ISO 9660 implementation
-------------------------------------+-----------------------
   Reporter:  scdbackup              |      Owner:  nobody
       Type:  enhancement            |     Status:  new
   Priority:  normal                 |  Milestone:  R1
  Component:  File Systems/ISO 9660  |    Version:  R1/alpha3
 Resolution:                         |   Keywords:
 Blocked By:                         |   Blocking:
Has a Patch:  0                      |   Platform:  All
-------------------------------------+-----------------------

Old description:

> On occasion of ticket #8460 i noticed two potential problems of the
> ISO 9660 filesystem read-only implementation:
>
> It looks like data files of 4 GiB or larger are not supported.
> Even if it is not intended to augment Haiku for reading large data files,
> it should at least be tested whether it tolerates such files gracefully.
>
> It has not been challenged whether the block-to-byte address computations
> of Directory Entry addresses are safe against byte address of 4 GiB or
> larger.
> Only the address computations for data file content have been fixed by
> #8450.
>
> I am not a user of Haiku, but a producer of ISO 9660 filesystems.
> In that role i would be glad to help with better readability of ISO 9660.
>
> ------------------------------------------------------------------------
>
> Large data files:
>
> {{{struct iso9660_inode}}} in
> http://cgit.haiku-os.org/haiku/tree/src/add-
> ons/kernel/file_systems/iso9660/iso9660.h
> only has provisions for memorizing a single data extent:
> {{{
>         uint32          startLBN[2];                    // Logical block
> # of start of file/directory data
>         uint32          dataLen[2];                             // Length
> of file section in bytes
> }}}
>
> ECMA-119 (the public specification of ISO 9660) says:
>
>   6.5.1 Relation to File Sections
> "Each file shall consist of one or more File Sections. Each File Section
> of a file shall be identified by a record in the same directory. The
> sequence of the File Sections of a file shall be identified by the order
> of the corresponding records in the directory.
> A File Section may be part of more than one file and may occur more than
> once in the same file. A File Section may be identified by more than one
> record in the same or a different directory."
>
> So if {{{struct iso9660_inode}}} is indeed the 1:1 representation of a
> filesystem inode, then it should be in a 1:n relation to ISO 9660
> Directory
> Entries.
> (The array size [2] is not a 1:2 relation but rather mirroring the fact
> that ISO 9660 Directory Entries bear address and length in both byte
> orderings.)
>
> More challenging will be to collect all ISO 9660 Directory Entries which
> bear identical names in the same directory.
>
> But as said, at least it should be made sure that two ISO 9660 Directory
> Entries with the same name do not confuse Haiku beyond the scope of that
> single data file.
>
> Proposal for producing a challenging ISO image:
>
> Create a file with at least 4 GiB of size. Plus some small files with
> names alphabetically lower and higher than the big one's ("aaa", "zzz").
>
> Put them into an ISO 9660 image e.g. by my program
> {{{
> xorriso -outdev my_image.iso -add aaa file_of_4gb zzz --
> }}}
> or use mkisofs
> {{{
> mkisofs -o my_image.iso -R -iso-level 3 aaa file_of_4gb zzz
> }}}
> (xorriso can operate directly on DVD burners on systems with Linux
> kernel,
> FreeBSD, or Solaris. Like -outdev /dev/sr0 , -outdev /dev/cd0, ...)
>
> Mount the image and check whether all three files behave nice.
> E.g. whether file_of_4gb does not appear twice in a directory listing,
> whether at least the small files show correct content, ...
>
> ------------------------------------------------------------------------
>
> Directory entries in high block numbers:
>
> Normally the Directory Entries are stored in a few hundred blocks
> shortly after block 16. So there occur no large byte addresses.
> But in the case of multi-session, the new directory tree gets written
> after the data blocks of the previous session.
>
> So if the first session is 4 GiB or larger, then the address computations
> for ISO 9660 Directory can produce byte addresses which may get spoiled
> by 32-bit bottlenecks like the one of #8460.
>
> Proposal for producing a challenging ISO image:
>
> Produce an ISO image which contains two data files of exactly 2 GiB each:
> {{{
> xorriso -outdev my_image.iso -add file1 file2 --
> }}}
> (Note that other than mkisofs -o, this will refuse to overwrite an
> existing
> image.)
>
> Add a session to the image by adding a small marker file
> "this_is_session_2"
> {{{
> touch this_is_session_2
> xorriso -dev my_image.iso -add this_is_session_2 --
> }}}
> (Note that the second run uses option -dev rather than -outdev. By this
> it
> loads the tree of the first session and adds the new file to it.)
>
> The resulting file "my_image.iso" will bear two sessions. The PVD at
> block 16 will direct the reader to a root directory above 4 GiB.
> The image will still fit on a single-layer DVD.

New description:

 On occasion of ticket #8460 i noticed two potential problems of the
 ISO 9660 filesystem read-only implementation:

 It looks like data files of 4 GiB or larger are not supported.
 Even if it is not intended to augment Haiku for reading large data files,
 it should at least be tested whether it tolerates such files gracefully.

 It has not been challenged whether the block-to-byte address computations
 of Directory Entry addresses are safe against byte address of 4 GiB or
 larger.
 Only the address computations for data file content have been fixed by
 #8460.

 I am not a user of Haiku, but a producer of ISO 9660 filesystems.
 In that role i would be glad to help with better readability of ISO 9660.

 ------------------------------------------------------------------------

 Large data files:

 {{{struct iso9660_inode}}} in
 http://cgit.haiku-os.org/haiku/tree/src/add-
 ons/kernel/file_systems/iso9660/iso9660.h
 only has provisions for memorizing a single data extent:
 {{{
         uint32          startLBN[2];                    // Logical block #
 of start of file/directory data
         uint32          dataLen[2];                             // Length
 of file section in bytes
 }}}

 ECMA-119 (the public specification of ISO 9660) says:

   6.5.1 Relation to File Sections
 "Each file shall consist of one or more File Sections. Each File Section
 of a file shall be identified by a record in the same directory. The
 sequence of the File Sections of a file shall be identified by the order
 of the corresponding records in the directory.
 A File Section may be part of more than one file and may occur more than
 once in the same file. A File Section may be identified by more than one
 record in the same or a different directory."

 So if {{{struct iso9660_inode}}} is indeed the 1:1 representation of a
 filesystem inode, then it should be in a 1:n relation to ISO 9660
 Directory
 Entries.
 (The array size [2] is not a 1:2 relation but rather mirroring the fact
 that ISO 9660 Directory Entries bear address and length in both byte
 orderings.)

 More challenging will be to collect all ISO 9660 Directory Entries which
 bear identical names in the same directory.

 But as said, at least it should be made sure that two ISO 9660 Directory
 Entries with the same name do not confuse Haiku beyond the scope of that
 single data file.

 Proposal for producing a challenging ISO image:

 Create a file with at least 4 GiB of size. Plus some small files with
 names alphabetically lower and higher than the big one's ("aaa", "zzz").

 Put them into an ISO 9660 image e.g. by my program
 {{{
 xorriso -outdev my_image.iso -add aaa file_of_4gb zzz --
 }}}
 or use mkisofs
 {{{
 mkisofs -o my_image.iso -R -iso-level 3 aaa file_of_4gb zzz
 }}}
 (xorriso can operate directly on DVD burners on systems with Linux kernel,
 FreeBSD, or Solaris. Like -outdev /dev/sr0 , -outdev /dev/cd0, ...)

 Mount the image and check whether all three files behave nice.
 E.g. whether file_of_4gb does not appear twice in a directory listing,
 whether at least the small files show correct content, ...

 ------------------------------------------------------------------------

 Directory entries in high block numbers:

 Normally the Directory Entries are stored in a few hundred blocks
 shortly after block 16. So there occur no large byte addresses.
 But in the case of multi-session, the new directory tree gets written
 after the data blocks of the previous session.

 So if the first session is 4 GiB or larger, then the address computations
 for ISO 9660 Directory can produce byte addresses which may get spoiled
 by 32-bit bottlenecks like the one of #8460.

 Proposal for producing a challenging ISO image:

 Produce an ISO image which contains two data files of exactly 2 GiB each:
 {{{
 xorriso -outdev my_image.iso -add file1 file2 --
 }}}
 (Note that other than mkisofs -o, this will refuse to overwrite an
 existing
 image.)

 Add a session to the image by adding a small marker file
 "this_is_session_2"
 {{{
 touch this_is_session_2
 xorriso -dev my_image.iso -add this_is_session_2 --
 }}}
 (Note that the second run uses option -dev rather than -outdev. By this it
 loads the tree of the first session and adds the new file to it.)

 The resulting file "my_image.iso" will bear two sessions. The PVD at
 block 16 will direct the reader to a root directory above 4 GiB.
 The image will still fit on a single-layer DVD.

--

Comment (by umccullough):

 Fix ticket description per comment:1

-- 
Ticket URL: <http://dev.haiku-os.org/ticket/8473#comment:3>
Haiku <http://dev.haiku-os.org>
Haiku - the operating system.

Other related posts: