XISF support in 2.5?

Hello, I’ll formally put the request in for XISF support into v2.5?

Not sure. I could be wrong, but I don’t see a lot of traction here on the capture side (I know that we could help to influence this, but… I am referring to wants and needs). I’ll keep this here for others to comment on and we can prioritize based on that.

Well, writing as an SGP and Pixinsight user, I’m unsure how I would use it. I don’t find the lack of .xisf support to be any hindrance in my workflow as I don’t have occasion to open a PI-processed file in SGP.

My primary reason would be file conversion. Most of us are PI users and I don’t see the point in maintaining FITS files when my processing software prefers a new format.

True… and I’m not saying I’m opposed to it, but (devil’s advocate):

  • FITS is a universal standard that extends far beyond PI
  • PI will not drop support for FITS

So… because this will take time away from the implementation of other features, it needs to be clear to us what advantages, from the capture and storage perspective, are exclusive to the XISF format? Can cameras even take advantage of these standards? You will always be able to open and pre-process FITS in PI and then save them as any format you’d like. It’s just not immediately clear to me what advantages are present from raw data capture perspective.

Well in my case it would be a detriment. I often use the fits files in ccdstack as there are certain things I can do there easier than PI. I would prefer the greater flexibility myself.

1 Like

Stu,

If implemented, this would certainly be optional. More just a question of where we spend our time…

Sure. I get that FITS has support for a wider audience. I get the feeling most of us aren’t doing those tasks and those people aren’t using SGP.

For the most of us, we’re exclusively PI.

I don’t really see the benefit of it at this time. No matter what, the data is stored in a 16bit grayscale format. This is the raw format as the data comes off of the camera so putting it in a different “box” isn’t going to change the data or make it better.

I’m not sure what you mean by “file conversion” is the main reason to want xisf. Can you add some more detail here?

I’m not sure what this means. Ken and I both process in PI so if we felt that there was some benefit to XISF for our workflow we would be quick to jump on it…we just don’t see it. But maybe I’m missing something?

Thanks,
Jared

Ditto Jared’s comments. I use SGP and PI exclusively, and see no benefit for SGP using XISF. PI processes my FITS files the same as it would if they were XISF files, so why bother?

Isn’t it legitimately just something I could have a check box for? My understanding is it’s a wrapper just like FITS.

It is indeed… but my question still stands. Nobody seems to have a good reason why we should do this. It will take time away from other things and I have not seen a compelling reason to do that.

Again, I’m not opposed to it and think the guys at PI do a great job, I just need a compelling reason.

Lack of formal resources to define how image data have to be interpreted and represented. For example, application A stores floating point images in the range from -12345.123 to +6789.001, but this range is not declared publicly. No problem, as this can be done legally with FITS. Application B uses the range from 0 to 123456, also undeclared. B has no way to know how to interpret floating point pixel data written as FITS files by A, and vice-versa, so despite the fact that both applications implement the FITS standard, they have a serious interoperability problem. The developers of B are smart and, after wasting precious time that they should invest in creative development tasks, manage to discover by reverse engineering the black and white points of images written by A. That’s great because now B can read FITS files written by A so B users using A data are happy. One day, the developers of A decide, for strictly technical reasons, to change image ranges in their next version, and again they don’t publicize them. Suddenly, all B users who need to import images written by A are lost. This cannot happen—and will never happen—with XISF.

  • Undefined physical disposition of pixel data. For example, a two-dimensional RGB color image can be stored as a three-dimensional image in a single header-data unit (HDU), with three contiguous blocks to store the red, green and blue channels. It can also be stored as three one-dimensional images in three HDUs, one for each channel. Or as four HDUs, one with all the metadata and no data and three with minimal metadata and one-dimensional channel data. It is unclear (to me at least) if it can also be stored as a one-dimensional sequence of groups of contiguous red, green and blue pixel samples in a single HDU. This “flexibility” can be nice, but FITS has no formal resources to define the organization of multichannel or vector-valued images. In XISF we have two pixel storage models (planar and normal models) formally defined without ambiguity.

  • No color spaces. If an application stores a three-dimensional image (also known as a data cube), what is it? An RGB color image? A grayscale image with two alpha channels? HSV? CIE XYZ? CIE Lab*? We need colorimetrically defined color spaces to perform brightness/chromaticity separations and other essential tasks. The word “color” appears only once in the FITS Standard version 3.0 document to say this (section G.2.1):

FITS data arrays contain elements which typically represent the values of a physical quantity at some coordinate location. Consequently they need not contain any pixel rendering information in the form of transfer functions, and there is no mechanism for color look-up tables.

  • Lack of essential metadata and auxiliary data structures such as ICC color profiles, CFA patterns, image resolution parameters, image thumbnails, optimal visualization parameters, etc.

  • Obsolete metadata and lack of Unicode support. Punched cards are cool, but as of 2015, we really need more than 7-bit ASCII, 8-character property names, and 80-byte metadata rows. We need full Unicode support, structured property identifiers, unlimited length names and values, and much more fundamental data types and data structures than ‘logical’, ‘numeric’, and ‘character’. For example, if my name were “Iván Martínez Añejo”, I would have to store it as something like this in FITS:

    name=“AUTHOR” value=“‘Ivan Martinez Anyejo’” comment=“Not my real name!”

while in XISF:

Iván Martínez Añejo
  • Rigid header-data sequential organization. Magnetic tapes also are very cool, but definitely not contemporary devices. For example, imagine a FITS file storing three images as three consecutive HDUs, as required by the FITS standard. If the file is available on a local hard disk this is not a practical problem: to access the headers of the second and third HDUs we need two file seek operations, but hard disks are very fast so we don’t notice. However, what happens if the file is in a remote server, for example, being accessed as “http://www.example.com/foobar.fits” ? We have to download the first HDU, including the whole first image (which we perhaps don’t need), before starting to download the header of the second HDU, and the same for the third one. Special network interface applications can be created to improve this situation, but this can’t be an optimal solution. With a monolithic XISF file, an application can download the entire XISF header, which contains all of the metadata including all image and property descriptions, in a single operation, without needing to read a single pixel. With a distributed XISF unit, we can download the header and then download just the required image data.

  • Lack of a distributed storage model. Distributed XISF units store the header (metadata) and the data as separate files, including local and remote resources. This allows for flexible storage configurations that are not possible with FITS. Furthermore, an XISF data blocks file allows indexed access to images and data objects through symbolic identifiers. This means that XISF data blocks files can be reorganized and extended freely without invalidating existing XISF units that depend on them. Distributed XISF units are also relocatable, so one can transport them between file systems and machines.

  • No effective data integrity and authentication protection. Digital signatures based on XML signatures, X.509 certificates and cryptographic checksums are formalized in the XISF specification. A digitally signed XISF unit is effectively sealed and cannot be modified (There is a registered FITS convention for checksum keywords, but it does not protect authenticity because checksums can be tampered).

Quote
3. should I change (I am a real newcomer, remember)

If you are a PixInsight user, you’ll be using XISF to store your images in a natural way because it is PixInsight’s native file format. You should always keep your original XISF files because XISF provides resources that don’t exist in other formats. Unfortunately, you’ll have to export them in different formats (FITS, TIFF, etc.) to use them with other applications. Hopefully, more applications will adopt XISF in the near future, and we’ll work hard for this to happen.

Quote
4. when I go to save a file there are seven options of file format to choose from (from 8-bit unsigned integer to 64-bit IEEE 754 floating point). I always choose the default (32-bit IEEE 754 floating point). Is this the correct one to use and if so why all the other options?

The 32-bit floating point format is the most logical option in most cases. The 32-bit unsigned integer and 64-bit floating point formats can be necessary to encode images with very large dynamic ranges, such as very deep linear HDR images generated with our HDRComposition tool. For the rest of images, 32-bit floating point should normally be used.
« Last Edit: 2015 February 17 14:24:12 by Juan Conejero »

Here is the info I found on PI’s forums as to why their developer thinks it’s important. I’m not a tech guy so I don’t understand most of it. Anyways, I asked Juan if he’d come over here and post. Guess we’ll see.

Chris

Right… I’ve read all this stuff. None of it is very compelling when we are talking about storage of 16-bit unsigned integers. I can see how it would be very useful once the processing steps come into play. Some of the storage stuff seems pretty cool, but not really necessary at this juncture.

Again, I would like to have this feature… just don’t think I willing to prioritize it over anything else we have slated for 2.5.

Oh yea, I don’t disagree with any of that! I wouldn’t make it mission #1, but if you guys get bored.

I haven’t seen a huge tally of additional items people want added this time around. Observatory support has grown significantly, weather support works, and the new plate solving feature fixes all the Elbrus issues SGP was plagued with in the past.

Sounds like multi-camera support will be a good add on for those lucky people :).

Here is the current 2.5 list (not in priority order):

I’ve looked at reducing my file library size and i found XISF has support for pretty good file compression.
File compression seems to be the biggest reason to add XISF support to SGP

For FITS files a compression of 35-60% is normal for mye files depending on the compression algorithm so there’s a good amount of space saved, especially for people accesing their remote observatories over the internet on slower connections it would be a hugh benefit.

1 Like

I would like to take up this thread again because i found another reason to use XISF.

I found there is a speed boost in starting out with XISF compared to FITS in PI, here’s the explanation by Juan Conejero at Pixinsight.

If yes to both, then I guess I know where the problem is. As you probably know we use NASA’s CFITSIO library as the back-end for our FITS support module. The Windows implementation of CFITSIO is not thread-safe for disk I/O operations (we have had many stability issues in the past because of this problem), and hence we have disabled parallel file accesses for the Windows build of our FITS module. The Linux, FreeBSD and macOS versions don’t have this problem.
This means that although the calibration process is fully parallelized on all platforms, loading a FITS file is a thread-locking task on Windows. This problem does not happen during the rest of the preprocessing pipeline because the XISF format implementation is of course fully thread-safe in every aspect.

So speed boost and a possibilty to use compression for archiving purposes are for me good reasons for me to want to get XISF directly from SGP.

Seems like like a good point in time to add it to SGP too before you start on 4.0.

1 Like