Tantalus 2

So I'm claiming that sliced images is the best thing since sliced bread.  Well if I recall correctly, the image I used as an example in yesterday's post actually has an asteroid streak in it.  (In fact, it has the only decent streak I have yet acquired, but we'll get to that later.)

So -- what effect did this slicing process have on that streak.

Let's zoom in.

The streak is there, but the central part of it is too bright for this image, and has been blacked out.

That's good news!  The brightness of Tantalus when I took that image -- the 'apparent magnitude' -- was 17.62.  I want to be able to see streaks that are much dimmer than that, and this result suggests that dimmer streaks might work very well.

And there's a way we can estimate how dim we might have been able to go with this telescope, on that night.  We can simulate our own streak, in this very image.

The Streak Simulator

I haven't told you about how I do this software yet, and -- I probably won't.  It's not terrifically interesting, compared to pictures of rocks in the sky. 

OK, I'll make it quick.  I write my image processing software in C, on a Fedora 20 system with a GCC compiler.  I write it all from the ground up, using no ancillary image processing libraries.  I like it that way.  Programming the bare metal.

I use the tool-building philosophy of all intelligent programmers: make simple tools that do a single job well, and combine them together with a scripting language.  Bash, actually.  (Until recently I used csh, which means that I got started doing this stuff when the universe was quite a bit less red-shifted than it is now.)

So -- the streak simulator is a program I wrote recently.  You tell it how much total energy you want it to deposit on the image, where the x,y start point is, what the direction of travel is, how far it should go, and how many seconds that should take. 

The program then moves its idea of where the asteroid is in tenth-second increments, at every moment doling out its increment of energy in a randomly-chosen direction, and at a random (normally-distributed) distance from the asteroid's 'true' position.

It's not perfect, but it's pretty close.  Here is what it did when I used it to try emulating the real streak that Tantalus made in my image.  (The simulated streak is just to the right of the real streak, and I made the simulated one perfectly vertical.

That looks pretty good!  Except I had to put 45,000 grayvalues of brightness into it, when the real streak only used 31,000.  Hmm.   And it's still a little scrawnier-looking than the real one.   Hmm.  

So, I don't know how good a model this really is, but just in case it is predictive, here's what it predicts:

If that is really what a mag 19 streak looks like, I think I can detect that like falling off a log.  And this is with a 20" telescope!  Through iTelescope.org, I have access to a 27" that collects twice as much light.  With that, I might hope for mag 20, or better!

But!  Simulation is one thing.  Ground-truthed (so to speak) data is another.

What I really need now is more images of known rocks, at known brightnesses.

Out of the Wilderness

I've wandered far since my last post, lost in the dark places of the southern sky and lost in the myriad possibilities of the things that can be done, trying to distinguish them from the things that should be done.

I think I understand better now, and hopefully even well enough to explain.



A hunter does not simply load his rifle and go running into the brush waving it about.  A hunter makes a plan before even choosing the weapon, and the first part of the plan must be: What am I looking for?


Well, what I am looking for is very faint streaks -- streaks that are just barely above the background noise.

OK, good.  Now we're getting somewhere.  This clear statement of purpose immediately suggests a question.  What is the background noise?  What does it mean to be 'just barely above it'?


To answer that, let's look at my raw image again.

Let's see.  How will we determine what the background of this image is?  Using my years of training in machine vision, together with an innate talent for noticing facts that are glaringly obvious, I soon determine -- that this image is all background.

All we need to do is look at a histogram of the image, select the most popular value, and we will have found the mean of the background.

So, let's get the histogram.

The red parts are the plotted points, showing how many pixels were found in the image at a given gray value.  That spike to the far left is so large that it makes the rest of the plot look flat.  That's what I mean by an image that is "all background".


Let's zoom in on the part of the histogram that only shows the darker values that are in the background.

Here is a graph of just the darkest 4% or so of possible grayvalues.

Now you can see some detail.  The background, that simply looks black in the image, actually makes what looks like a nice, normal distribution whose mean is just below grayvalue 1000.  Practically all of the pixels in the image are represented by the spike we are seeing here.  This is the distribution of the background.

Let's look a little closer yet.

That's about as nice of a normal distribution as you will ever see.  Its mean looks like it's at about 940, and the point at which it falls to half that value looks to be about 900 on the left and 1000 on the right, which means that the "Full Width at Half Maximum" of this distribution is 100, which means that the standard deviation is about 42.    (The FWHM of a normal curve is always 2.35 sigma.)

But what does this tell us about how we can view the pixels we care about?



What it says is that we can do something beautifully simple.  Make an eight-bit image with its paltry 256 gray values simply centered on the highest point of that histogram.  That will show us pixels that are 3 standard deviations above and below the mean, which should be plenty.

I call this a 'slice' image, because I am slicing out 256 gray values from the 65,536 possible from the 16-bit original.

Here's how we do it.  Look at the peak of that histogram at 940.  Go 128 below that, to grayvalue 812, and make that our zero point.

Now make an 8-bit image as large as the original.  Go through and subtract 812 from every value in the original.  Any pixel that would be below zero, we just set to 0 in our new image.  And any pixel that would be above 255, we also set to 0.

Because we don't care about those pixels.  They are not close enough to the background to be part of the really faint streaks we want to find.

So what does the result look like?
Let's see.

We just got rid of all the stars.  They have turned black because they are too bright to be in our new 8-bit image.  What you are seeing now is a really nice smooth picture of just the background, against which we will be able much more easily to find our faint streaks.

SInce May, I have been wondering how in the heck I could make the stars go away, and here it is in a simple thresholding operatrion that took all of 134 milliseconds of a simple processor on my laptop -- and that was for the full sized original image that is 3056x3056.  That is a dirt-cheap operation.

This job just got a lot easier.

Tantalus, my First Rock

My first rock, 1.5 miles across, 85 million miles away, images from the iTelescope T24 in Auberry, California about 2 hours ago.

The image is processed down to 8 bit gray with my own software -- in the original you cannot see the streak.

These images are zoomed way in.  They are both about 250x250, while the originals are 3056x3056.


At 0200 Auberry time, 0500 Michigan time, 0900 GMT -- a ten minute exposure:

And at 0211 Auberry time, 0511 Michigan time, 0911 GMT:

Si muove!


Oh, we are going to have a lot of fun with these two images.

By the way, here is what the images looked like before being 8-bitified so you could see faint gray values better.  You can see the brightest few stars, and that's about it.

And, say hello to my little friend, who took the picture for me, made by PlaneWave Instruments, owned, adapted for the internet, and very well managed by iTelescope.net, the lovely and talented T24, of Auberry, California:

Into the Dark

I propose to use none of the usual methods to find rocks.  I propose to use a method that requires such ridiculously vast amounts of compute power that it's never been tried before -- because the compute power just wasn't available.

We will be looking in our images for streaks so faint that no amount of contrast enhancement can make them visible.  So faint that they can only be detected by some combination of statistics and machine vision.

But to do that -- we will need to understand the dark between the stars.



So first, let's go find some.

Here is the image I showed a while ago, that contains the Hubble Deep Field in its center.

OK, so I don't want to use the center, because I actually was able to see some galaxies in there.  I want to find a rectangle that has no discernible stars even after serious contrast enhancement.  And I want the rectangle to be as big as practical, because we are going to do statistics on its pixels.

After some searching around, I find this nice little spot right here:

Let's look a little closer.

Okay!  No stars that I can see, after doing the best contrast enhancement I could get out of the Gimp.

( Note!  In all of the work I have done or will do on this blog, I use only free public websites, free open source software, or software that I have written myself. )


So, we found a dark place.  What do we want to know about it?

What I hope to see is that those dark pixels, statistically, have a nice normal distribution, and spatially, are nice and smooth, with no discernible clumps.

It's easy to get the statistics from that region.  Here they are:
  
          count: 71416   
     min:       0.000 
     max:    1513.000 
     mean:    972.074 
     sigma:    37.493

The images I get from these telescopes are 16-bit grayscale, so the total range of pixel values is from 0 to 65535.   So an average brightness of 972 is pretty darn dark, which is why that rectangle looks plain black in the original image.



Okay, the stats will be useful, but they  don't tell the whole story.  What do those pixels actually look like?  I can't see them now because display technology can't show 16 bits of grayscale, and human eyes wouldn't be able to see it if they did.

So here's what we can do.  The pixels in that rectangle go from 0 to 1513.  So divide them all by 5.9, and that will make them all be in the range 0..255, and then we will able to see them properly in a normal image!

Here's what that rectangle looks like after this treatment:

A few little smudges in there, but basically nice and smooth!

And finally, I would like to know if those pixels are more or less normally distributed.  Because a lot of the reasoning I will do will depend on that.

After going through all kinds of pain to try and find a test I could use to determine whether or not a bunch of numbers are indeed normally distributed, I have finally settled on the simple expedient of graphing the darn things and looking at the graph.  Just print out the values, pipe them through "sort | uniq -c", and use a nice little gadget called gnuplot.

Here's the result:

That's glorious.  That curve looks like a textbook illustration of a normal curve.  On the Mick's Arbitrary Graphical Expedient Test of Normalcy, that curve gets a score of 0.995, which means "Heck Yes That Is Normal!"


This all means that the Dark Between the Stars is just how I hoped it would be.  It will make a perfect hunting ground in which to seek the faint tracks of our prey.

Rockhounds

How do people find asteroids now?  I mean, other than by having one detonate over your city, or extinguish your species?

As far as I can tell, the methods used today are identical in spirit to what was done in the days of photographic plates, except that digital cameras and computers make all the steps a lot easier and less expensive.


The first method is blinking.  You take several exposures, pausing for some time after each one, and then play them all like a movie.  All the stars stay still because the telescope is following their motion.  But if you are lucky enough to have a sufficiently bright asteroid in the field of view, it will show up as a little moving dot. 

A related method is to take just three exposures, using first a red filter, then green, then blue.  Combine all three to make a color image.  The stars will appear white (more or less) while anything moving will appear as a sequence of three dots: red, green, and blue.

Finally, you could just take an exposure long enough that the asteroid makes a nice streak. 

The streak method usually happens by accident, while someone is taking a long exposure for other purposes.  It's not a great way to find new asteroids because:

• An asteroid has to be pretty bright to leave a beautiful streak like the one I have simulated here.
• You can't tell which way it was moving.  ( Although you can probably make a high-probability guess. )

I don't want to do any of these things.

What I want to do is use image processing and machine vision techniques to allow us to find streaks that are really, really faint.  So faint that they are right down in the noise.

This will allow us to do two cool things:

• Find rocks that are much fainter than the other methods can.
• Find them several times faster than either of the multi-exposure methods.

The concept is -- compute power is getting exponentially cheaper.  Since I left grad school, compute power has gotten cheaper by a factor of 10,000.  Telescopes have gotten more expensive.  Moreover, compute power is continuing to get cheaper by a factor of two every couple years, while telescopes are continuing to get more expensive.  If we can use compute power to make telescopes more effective at finding asteroids -- that would be a Big Deal.

But to do that, we will need to go to a cold and lonely place.  The dark between the stars.

First Light

A few nights ago it was cool and clear in Mayhill, New Mexico, and I got my first image of a special part of the sky: a place just above the two back stars in the cup of the Big Dipper: Phecda and Megrez.

It's a special part of the sky because there's a little area in which there are no 'local' stars at all.  No stars from our galaxy. It's the place where the Hubble pointed when it took the exposures that were put together to make the Hubble Deep Field image, almost twenty years ago.

My exposure lasted 600 seconds.  Here it is, though greatly reduced for your viewing convenience.

I have also greatly messed with the image's grayscale curve, so that you can see more stars in it.  The original is 16 bits deep -- each pixel is a number between 0 and 65535 -- which your monitor cannot display and your eyes cannot discern.

Now let's zoom in by a factor of two on the center....

Please bear in mind that the Hubble exposure lasted over 100 hours -- about 600 times longer than mine, and its mirror has 23 times the light-collecting area that my beloved T11 has, and, oh just by the way, that it's above the freaking atmosphere! 
So my picture won't look like theirs.

But it's mine.


So...now let's zoom in by one more factor of two.

And here is your window.

The stars that are linked with the pretty cyan lines are local.  Everything else inside that box, all of those dim lights, are distant galaxies.


Wanna take a ride?

Today

Today, in my part of Michigan, six inches of new snow fell and I had to go outside twice on my tractor to blow the driveway clean.  Now it's drifted over again.

Today there are Russian troops gathering near the eastern Ukrainian border, and an American general said that the U.S. is ready to enter the Ukraine with military force.

Today the United State government went further into debt at the rate of one billion dollars per hour, when measured with Generally Accepted Accounting Principles -- which they do not use.

Today a Facebook executive was in the news because she has recruited a number of celebrities in her campaign to ban the word "bossy" as applied to girls and women.

Today a Chinese satellite may have spotted pieces of a Malaysian jetliner that disappeared four days ago, with two hundred and thirty-nine people on board.

Today Edward Snowden, recently of the NSA, said that he had no regrets about revealing the fact that the US government has engaged in massive spying upon its own citizens, to better protect their freedoms.  Senator Dianne Feinstein, who recently defended the government's right to collect such data, expressed indignation at discovering that it has been spying on her, too.

Today a building in Harlem, on Manhattan Island, exploded because of a gas leak, killing between three and twelve people, injuring many others, and causing the collapse of two nearby building.

Today the Dow Jones Industrial Average was almost unchanged at 16,340 points.  The donut company Krispy Kreme rose sharply in after-hours trading.  More than 90% of all trades were executed by high-frequency trading computers.  The average period of ownership for a given share of stock dropped to less than 1 second.

And today, somewhere in the two hundred thousand trillion trillion cubic miles that constitute our solar system, there is a rock drifting in space that will intersect our orbit, enter our atmosphere, and strike our planet sometime between next week and a million years from now. 

Well, of course there's more than one.  Those like the one I saw long ago, or like the one that exploded over Chelyabinsk, Siberia early last year -- those are exciting enough.  But the one that worries me is about five miles across, will strike with the force of about 40 teratons of TNT, will raise a tsunami about 100 feet high that strikes the entire pacific rim about 8 hours after impact, and will end up killing somewhere between one half and two-thirds of the human population with prompt plus delayed effects in the two or three years after impact.  Assuming the wave doesn't bust open any nuclear plants.  Oops.

I like galaxies, and planets, and nebulae, and all those things.  I really like all the stuff that people look at through telescopes.  But myself, what I want to find is that rock.  As far in advance as possible.

In this blog, I will be talking about some ideas -- that I think might be original -- for finding asteroids with telescopes.  I don't want to do it the way we used to do with film, or the way I see people describing on the internet (which is pretty much the same as what we used to do with film).

I want to be able to find the tiniest little streak of an asteroid, so dim that it's right down in the background noise, so dim that an astronomer who knows what he's doing will tell me "You can't detect an object of that brightness with an instrument of that size."

Doesn't that sound like fun?

Sailing Home

I had a nice long Christmas vacation, and right at the start of it I decided to spend most of it on fantasizing.

I decided that this was a mid-life crisis thing, which I've been in the midst of for a couple years now. I am at the point in life where I finally come up for air after thirty years or so of straight work and wonder How Did I Get Here and What the Heck Am I Doing?

Men do all kinds of crazy things when they reach their mid-life crisis. They buy speed-boats or fast cars. They leave their wives and find new wives. They get hair implants.

What are we looking for when we do stuff like that? I think we're saying I'm scared of the dark. I'm pretty sure that my death is closer to me in the future than my high school graduation is in the past, and I want to stay young.

With those examples in mind, I decided it wouldn't be so bad if I allowed myself to fantasize about any damn thing I wanted, no matter how silly or meaningless. I theorized that it was like a one-person version of the exercise of "brainstorming" that I've done once or twice in corporate settings. Don't suppress any ideas, no matter how apparently bizarre, because you never know what might come from one of them.

So at one point, of course, I found myself daydreaming about telescopes. What else?

"Honey?", I called into the living room at one point, "Can I spend two hundred thousand dollars on a telescope?"

I only call her "honey" when I am about to talk nonsense.

"Sure!" she called back. "No problem!"

The telescope of my fantasy was the 1-meter monster from Jim's Mobile. I do not know the actual price, I was just estimating based on the cost of smaller scopes, and throwing in some extra for a nice, permanent dome.

It did not upset my fantasy at all that my financial net worth is probably negative just now, and I doubt that I could borrow money for a new car let alone two-hundred large for a telescope. So what? we're fantasizing, right? That's what fantasy is for!

But, alas.  Upon some rocks even the gossamer ships of Fantasie must founder, spilling their cargoes of hopes and dreams, desires and schemes into the cold salt water of reality.

The problem is, I live in Michigan.

Let's say I get two hundred large to spend.  In fact, why not make it four hundred!  There' s no limit on fantasy-money, is there?  (Apparently not.  Just ask the Fed.)

Let's say I buy a beautiful enormous telescope, and put it in a beautiful dome.  In fact, let's put that dome up on top of a tower, and I will wear white and silver robes like frikking Gandalf every time I go up there!

Fine.

You want to see what my first few nights of observing would look like?

Night one.  Clouds.

Night two.  Clouds.

Night three.  Starting to see a pattern here.

The Great State of Michigan, the largest state east of the Mississippi, the state with the longest coastline in the continental United States, once the most prosperous state per capita in the union  -- Michigan is not on anybody's top ten list of places to put telescopes.

I checked.  It truly is not.

Now, I'm sure that Jim would say "Yeah, that's why I made it mobile!"

Okay.  But where will I go pulling a monster like that?  Frikking Arizona?  And then who will take care of my ducks, while I am wearing my Gandalf robes and enjoying the crystalline skies of far-off countries?

Okay -- but why not just imagine being somewhere else, then?  If you're already imagining impossible things, why not one more?

Well, um, shucks.  I think it's because my land, where I have built a house and planted oak trees, where I have planted American Chestnuts that were almost extinct -- I think that my land has extended its own roots down even to the level of my dreams.  I can imagine having lots of money to spend.  I cannot imagine leaving this place.


The Ship of Fantasy, the Ship of Dreams, broken on the rocks, its cargo of glowing hopes floating into the dark waves.  Flickering.  Darkening.


And that was when my Christmas Miracle happened.

I kept believing.   Wiping the bitter tears from my eyes (more salt water!), I climbed back on to that foundering vessel Fantasie, wrestled her battered wheel around and launched off once again upon the darkling waves of the Internet.  We smote the sounding furrows!  Our purpose held!

If I could not imagine owning a telescope in Michigan, then I would instead imagine the great instruments to come!  I would read about instruments with names out of legend:  the Giant Magellan Telescope.  The Large Synoptic Survey Telescope.  The European Very Large Telescope.  The European Extremely Large Telescope.  (I guess the Europeans must have shot all their poets quite some time ago.  Seriously guys.  Could you use some help naming these things?)

And it was while I was looking at the Giant Magellan with its seven great mirrors, and the Europäische Unglaublich Riesige Teleskop, with its seven hundred and ninety-eight hexagons -- that's when my dreams looked back at me out of the internet, and smiled.

I found a page and thought, "Huh.  What's that?"

It was a place called iTelescope, and they had telescopes for rent.  By the minute.


In retrospect, it seems obvious.  All astronomical photography is digital, right?  Digital stuff travels over wires, ja?  For years we have had "GoTo" telescopes that, once oriented, can find any star, any point in the sky you want.  So why not robotic telescopes, operable over the internet, that you can use for a few bucks a minute, and then download the picture you took?

Why not indeed?

Except I didn't know about it, and would have never have found it, if not for the Good Ship Fantasie.

So, would you like to see my new telescope?
It's big, it's beautiful.  It is of a quality in its optics, its mount, and its camera that I did not even know enough to dream about.  And it's real.

It's 20 inches across.  Even larger in metric!

Finally, it is in the distant land of Mayhill, New Mexico, at an altitude of 7300 feet, where the skies are not cloudy all day.

My ship has come in.

The Visitor

In the summer of 1967, as near as I can judge after nearly half a century, I was standing on the small front porch of my family's house near Jackson, Michigan, hoping to see a meteor.

I'm pretty sure it was the Perseid shower; I remember a warm summer evening. We knew about the upcoming shower because the local newspaper, the Jackson Citizen Patriot, had published a small article about it. My father had gone so far as to purchase a small Tasco telescope -- one of those tiny 2.5" or 3" white refractors standing on a tripod that I expect that you could bend if you leaned on it.

Even at the age of nine, I knew that a telescope would be useless for seeing shooting stars. I guess I must have seen some before, because I knew perfectly well that they would make such brief streaks in the sky that you wouldn't be able to follow them by turning your head, let alone by looking through a little telescope and swinging it in a wild arc.

I believe the time was late evening, with daylight still in the sky. We were just getting set up, and I believe that my mother and younger brother had not yet come outside.

There may have been other people on the street. It was the kind of suburban neighborhood that I expect doesn't exist anymore. It had been rural land just a few years before, the people who bought houses there were all middle-class-aspiring-to-be-upper-middle-class thirty-somethings, and they all had children in a pretty small age range. At Halloween there, I remember swarms of children rushing back and forth across the street like piles of many-colored autumn leaves blown by the wind, in a world before rumors of poisoned candy, or razor blades embedded in apples.

But on that summer evening I was waiting for the sky to get full-dark, and wondering when the meteors would show up. I turned my head toward the east, and I saw the one that showed up.

Many years later, I learned that it was a rock the size of a small house. It struck the Earth's atmosphere at a shallow enough angle to skip off at a high altitude rather than plunging to its destruction -- and maybe our as well. I believe I saw an estimate in later years that if it had impacted the planet it would have struck with a force of 10 or 15 kilotons. The size of a Hiroshima bomb. And it would have struck the heartland of the United States. Were we sophisticated enough in 1967 to know that it was not a nuke? Probably. Maybe. As it was, the rock's closest approach to the Earth's surface was fifty miles high, and that was over the Rocky Mountains, thirteen hundred miles west of my house. Which would have happened less than two minutes after I first saw it.

The fireball was as big as my thumbnail at arm's length -- like the full moon -- so big that you could see fiery structure in it. It was not just a featureless blob of light. There was also obvious, curdling structure in the tail which, as the bolide continued on its course toward us, soon covered a third of the sky.

I think my father and I just stared at it. I recall clearly that I was too dumbstruck to even think of calling to my mother and brother in the house.

I have always felt that I could actually hear the object, but that can't be right, can it? It would take forever for the sound to propagate down from fifty miles high, and what the heck kind of sound would propagate that far, especially coming down from the incredibly thin air that high? The voice of Doomsday? I'm sure that my memories must be wrong about the sound. It just looked like it should have a sound. Like it should have a very loud sound. My first sight of the meteor was so far to the east that it actually appeared to rise in the sky as it approached us. It passed almost straight overhead trailing its slow-motion smoke and fire, and then it continued into the west straight toward the sunset.

The entire apparition lasted twenty or thirty seconds, which seemed like forever. In that time it must have traveled four or five hundred miles.

It's kind of hard to deal with something like that when it simply flies away into the sunset and leaves you behind. I think I wanted to catch up with it and ask questions. Or have it come back by sometime. Or fly away with it.

Many years later when I was going to college in Ann Arbor, one of my friends asked me what I thought would be a good way to die. We were not very many years past the days of Viet Nam, Hippies, and the Weather Underground, and we used to have some pretty philosophical discussions. (Do people still do that?)

Without thinking, I told my friend that I wanted to die during re-entry into the atmosphere, making a fiery streak that a little boy would see.

It was not long after that earlier day in 1967 that I started using our little Tasco telescope to look at things, mostly by taking it to the golf course where my father would practice sometimes until the late evening. Mostly I looked at the moon, because that was the only thing I could actually find, besides random stars like tiny jewels.

I think it only took until the end of that summer to realize that I was going to want a much bigger telescope.