Into the Dark

I propose to use none of the usual methods to find rocks.  I propose to use a method that requires such ridiculously vast amounts of compute power that it's never been tried before -- because the compute power just wasn't available.

We will be looking in our images for streaks so faint that no amount of contrast enhancement can make them visible.  So faint that they can only be detected by some combination of statistics and machine vision.

But to do that -- we will need to understand the dark between the stars.



So first, let's go find some.

Here is the image I showed a while ago, that contains the Hubble Deep Field in its center.

OK, so I don't want to use the center, because I actually was able to see some galaxies in there.  I want to find a rectangle that has no discernible stars even after serious contrast enhancement.  And I want the rectangle to be as big as practical, because we are going to do statistics on its pixels.

After some searching around, I find this nice little spot right here:

Let's look a little closer.

Okay!  No stars that I can see, after doing the best contrast enhancement I could get out of the Gimp.

( Note!  In all of the work I have done or will do on this blog, I use only free public websites, free open source software, or software that I have written myself. )


So, we found a dark place.  What do we want to know about it?

What I hope to see is that those dark pixels, statistically, have a nice normal distribution, and spatially, are nice and smooth, with no discernible clumps.

It's easy to get the statistics from that region.  Here they are:
  
          count: 71416   
     min:       0.000 
     max:    1513.000 
     mean:    972.074 
     sigma:    37.493

The images I get from these telescopes are 16-bit grayscale, so the total range of pixel values is from 0 to 65535.   So an average brightness of 972 is pretty darn dark, which is why that rectangle looks plain black in the original image.



Okay, the stats will be useful, but they  don't tell the whole story.  What do those pixels actually look like?  I can't see them now because display technology can't show 16 bits of grayscale, and human eyes wouldn't be able to see it if they did.

So here's what we can do.  The pixels in that rectangle go from 0 to 1513.  So divide them all by 5.9, and that will make them all be in the range 0..255, and then we will able to see them properly in a normal image!

Here's what that rectangle looks like after this treatment:

A few little smudges in there, but basically nice and smooth!

And finally, I would like to know if those pixels are more or less normally distributed.  Because a lot of the reasoning I will do will depend on that.

After going through all kinds of pain to try and find a test I could use to determine whether or not a bunch of numbers are indeed normally distributed, I have finally settled on the simple expedient of graphing the darn things and looking at the graph.  Just print out the values, pipe them through "sort | uniq -c", and use a nice little gadget called gnuplot.

Here's the result:

That's glorious.  That curve looks like a textbook illustration of a normal curve.  On the Mick's Arbitrary Graphical Expedient Test of Normalcy, that curve gets a score of 0.995, which means "Heck Yes That Is Normal!"


This all means that the Dark Between the Stars is just how I hoped it would be.  It will make a perfect hunting ground in which to seek the faint tracks of our prey.