PAN010: Build thread for UofA

[atrodack]

Hi all,

We have been continuing our work up on the roof of Steward Observatory with trying to track stars and schedule a target for observing a transit. We have run in to some things that we were hoping we could get some advice on. I will attach some pictures below to illustrate.

Our Polar Alignment to start our night of observing:

1. After some experimenting, we figured out we can change the exposure time and number of exposures run_pocs uses by adding keywords into the targets yaml file in $POCS/resources/targets/. Is there a good rule of thumb for how to set these numbers for a given target beyond making sure you cover the full transit (ie exposure time * min number of exposures >= total time of transit), and don’t over-expose the images?

2. Is there a way to set the start time of the observation in the configuration so that run_pocs can be left running waiting for the transit to start, and automatically start observing rather than us having to start run_pocs manually at the correct time?

3. When we pick our own target, we follow the formatting in simple.yaml to create our target list that has 1 target star that we found has a transit. We then point pocs_local.yaml to look at that file, and run the pocs shell. From there we “run setup_pocs night weather” and then “run_pocs”. Once it starts observing our target, it doesn’t appear to track the star while exposing the cameras (see 120s exposure below that we did to make clear if we were tracking or not, so it is a little overexposed). How do we make sure that the mount tracks the star? Is there something we need to include in our target or pocs_local config? We tried having the pointing auto-correct both True and False and that didn’t seem to matter for this.
   

   

4. It is our goal to track a transit and get the data collected so we can ask for help reducing it to verify we see an accurate light curve. Obviously we need to solve the previous questions to get to that point, but we would like to try doing the data collection by the end of this upcoming weekend. @wtgee can you help us pick a quality target and understand how to track it (mostly just questions 1-3 above), and then maybe help us do the data reduction following if our data collection goes well?

[atrodack]

Here is a picture when it did appear to be tracking at the correct speed

[atrodack]

We are using the version of POCS with this commit key: 4a3d30545fd4608d58b806c8f256a09f2933b169

[jmk1729]

@james.synge, @wtgee, I can’t upload the log files directly, so I’ve hyperlinked to them from a google drive folder. The links are labeled the same as their corresponding images Alex posted above (post 34 and post 35).

20190408T043207.jpg - this is a transit we were actually trying to track. We picked it using the transit calculator.

20190407T073437.jpg - this is a transit selected from simple.yaml.

Clearly the tracking is off when trying to set our own observation target. Both observations have a 120 second exposure time, but the image from 04/07/2019 uses f/5.6 while the one from 04/08/2019 uses f/2.8.

[jlumbres]

The UA team has been pretty busy these past several weeks, especially since Olivier was visiting UA for 5 days (a new record in over a year!). Most of the content has been mentioned in the weekly meetings and on slack, but I’ll reiterate it for the log.

Observation Attempt (Sunday 2019/04/21):

• Olivier was there to join us for most of our time on the roof! We did a polar alignment and took some pictures of M51.
  – I’m told “pictures or it didn’t happen” and given Olivier’s wavefunction equation, I present the obligatory embarrassing adviser photo: Olivier using the trackpad on my laptop wrangling ds9
  

  

• There wasn’t a good transit for the night, so we decided that we’ll do an overnight test to check that the PANOPTES unit will shut itself off correctly. Past tests have shown to us that POCS checks the time and horizon status to determine shut down. Only the weather was simulated in POCS.

• The next morning when I came in to check on the unit, something went astray and it could not park itself correctly. The good news is that it didn’t point east towards the sunrise.
  

  
  
  I manually moved the unit back to park, collected the data and logs, then shut down the unit.
  
  

• We shared the logs with @wtgee. It seems like there wasn’t any error, just the serial device stopped communicating. It is possible that we bumped a cable while maneuvering the night before in the darkness and packing it up for overnight observations. So that is something we should be careful about in future testing.

Observation Attempt (Monday 2019/04/22):

• We discussed the night before to keep the entire PANOPTES unit on the roof for the duration of the week. The forecast predicted cloudy skies for Monday night, so we did not cover the unit. Given last night’s observation results, we decided to keep the unit off until the next observation attempt.
• However, the forecast proved to be slightly untrue and there was a bit of rain. I got ahold of last minute roof access to cover the unit with tarp:
  
  

Observation Attempt (Tuesday 2019/04/23)

• Very cloudy day, but forecast predicted the clouds would clear out late in the evening. The original plan was to inspect the unit, polar align it again, and leave it alone to track a transit. Olivier suggested that we’ll call the decision at 10 PM since the transit was to begin at 12:30 AM. When Justin and I went to the roof, the seeing was very bad (the stars twinkled so strongly, Sirius was pretty much a strobe light) and not suitable for photometry. Test run cancelled for this night.

Wednesday’s observation attempt (2019/04/24):

• We had weird mounting issues, where it behaved like it was unbalanced? (It’s always something new on the roof.)
  – The mount made a loud groaning gear noise, probably a load noise.
  – When we went from unpark, park, unpark, go_home, sometimes only one axis on the mount would move correctly and the other not so much, if at all.
  – The mount worked well with the controller, but it was not with POCS. We contacted Wilfred about it and dragged Olivier up to the roof from the lab.
  – We can probably contact iOptron about this? This is the second mount issue we’ve come across.

• After the mount started to act like it was working, we started tracking a transit even if late. However, the transit we picked that night required a meridian flip, so POCS vetoed it. We overran it by adjusting some parameters in the yaml file.

• We analyzed our previous 120 second exposure data of M51 to study the star magnitude limits we could handle observing from the Steward Observatory roof. We used ds9 to check the star intensity counts and Stellarium to determine the star mapping and magnitude. A magnitude 11 star was about 100 counts above the noise floor. Therefore, any stars for observing transits from the Steward Observatory roof will have to be no fainter than magnitude 10.

• We left the unit operating overnight. Justin came in the morning to check on it, it parked itself correctly.
  

  

• Justin also pulled the data from the harddrive. Unfortunately, we couldn’t do any data reduction because of the meridian flip.

Friday (2019/04/26):

• We pulled the unit down from the roof. We are going to do some indoor work on it (see below). The unit is currently living in Justin/Olivier’s shared office. We are aiming to be ready for observing HD 189733b on May 23 at the roof of Steward. We will be leaving the unit on the roof for a bit and have it run autonomously for several days (weather pending) to ensure confidence at leaving it up on Mt Lemmon.

Wednesday (2019/05/01):

• Olivier and I had a telecon with Alan Strauss (Mt Lemmon Sky Center director) and Don McCarthy (SO faculty) for discussing how to deploy our unit onto Mt Lemmon.
  – We ultimately chose Mt Lemmon over Kitt Peak because Mt Lemmon Sky Center is maintained by Steward Observatory, which will make the internet/power costs, accessibility, and approval much easier. Kitt Peak is a multi-university area similar to Mt Wilson, so despite darker skies, it will be more difficult at this stage of PAN010.
• The mountain ops head couldn’t make it to the meeting, but Alan asked some basic questions about PANOPTES.
• Alan has started the email chain involving the mountain operations group and other key people (e.g. Buell Januzzi, Director of Steward Observatory). Not sure how long it’ll take for this to get approved though.

What’s up next?

• Figure out what’s wrong with the mount and if we need to contact iOptron
• Weather proofing
  – Protecting the mount
  – Acquiring USB and electrical cables to handle outdoors
  – Soldering down cable wire to the connectors (they’re currently only screwed in)
• Updating POCS codes
• Getting weather sensor to work
• Image coadding to make pretty pictures

[wtgee]

Thanks @jlumbres for the write-up, very helpful!.

Regarding the mount gear noise, was this during a polar alignment test or during normal operations? If you have time in the lab and the unit is set up it would be nice to see if it is still making that noise.

@james.synge is also setting up PAN005 for cloud access so I’ve asked him to check on PAN010 permissions at the same time. We can test the cloud access at any time and should do it before putting it up for next observation.

What’s the status of the weather station? Is that just something that needs to be done or were there problems with it?

[jmk1729]

The mount struggled, or outright failed, at moving the RA axis in two situations. The first is moving from park to home, then back to park. In this case it failed consistently returning to park from home for a block of time.

The second situation is during polar alignment, typically when capturing an image while rotating from east to west. This is when the gear noise happened as I recall.

We can play around with the mount when we find some time. Same goes for setting up for the google cloud storage.

The weather station needs to be integrated into the system, meaning we need to understand what it’s doing and how its sensors are being read from PEAS/POCS.

[wtgee]

Does sound like a balance issue so that’s the first thing I would check. Make sure all the wires are plugged in when doing that as it can add just enough weight.

[jlumbres]

UA PAN010 Update from our work today:

• We connected to the Google Cloud server.

• We’re in the process of getting ssh access for roof testing. It’ll be good for us when we’re ready to go to park it on the roof.

• We uploaded the telemetry and camera boards arduino scripts directly using @atrodack’s laptop and the most recent arduino gui version.
  – We tried in the past using the cloud version, but it didn’t work.
  – On the camera board, the accelerometer and humidity sensors worked in both the arduino gui and PEAS. However, the humidity sensor was finicky and moving something caused NaN values to output. We’re not sure what is going on there, so any suggestions welcomed.
  – For the telemetry board, the humidity sensor worked in both the arduino gui and PEAS. However, we’re not sure what’s going on with the temperature sensors (they output -127.0 values). I suspect it could be a funny way I wired the temperature sensors on the telemetry board. I’ll go investigate it.

• We managed to get the cloud sensor to be detected in PEAS (took a while, though…)
  – We found that having the arduino scripts uploaded gets PEAS to see the cloud sensor.
  – We hung out the cloud sensor outside the window in Justin/Olivier’s office. The weather stayed at “Very Cloudy” (it is indeed very cloudy today here in Tucson) and the sky temperature increased. We suspect that some of our results are skewed because it’s detecting parts of the building. We’ll need to test this on the roof.

• We set the mount’s meridian position limit to 25 degrees using the controller.

• We’re in the process for setting up the weatherproofing.
  – We have access to a 3D resin printer (through CAAO/LBTI) that we’re going to use for the camera box seals (featured from WISRD). We’re going to go through some test versions on another 3D printer to make sure we get the fit correctly before going through the resin printing. We can’t use a lot of the resin, so we decided to use it for the camera box seals.
  – We’re making a list of equipment we need to procure.

[nem]

Great to see the UofA unit in action.

In regards to setting the exposure time, we did this for HD189733 by taking an exposure, and checking the counts (in ds9 I believe). Aru showed me that you can type the name of a target into Ds9 and it will find it in the field so you dont need to look for it manually. From last Augusts efforts we worked out that the camera has a 2048 count pedestal (you cant get less counts than this) and a maximum of 13???. So you want the signal from the target to be between these two values. About 9000-10000 counts would be perfect but it will vary as the star moves across the pixels and has a different count in each color channel. So we adjusted things until we go 8000-10000 counts in the brightest channel and took a few images to see how much it fluctuated as it walked across pixels. I think in the end we went with a 35 second exposure time with the SL1 for this target and an open aperture. So this should work for you too.

Good luck

[lucboucher]

Hey PAN010 team,
did you ever solve the issue on downloading the script to the arduinos from the NUC?? if so I’m really interested. I tried this but it did not help…
The workaround you mention (downloading from another computer) means that the issue is just downloading and that the NUC is able to read the serial monitor from the arduinos to get the telemetry and allow you to switch stuffs on/off right?
Thanks,

[lucboucher]

Ok answering myself as I was able to test stuffs earlier than I thought.
YES even if we cannot download the script to the Arduino with the NUC we can actually read the serial monitor telemetry and switch things off/on.

I am still interested in knowing if you solved your script download issue!

Cheers,

[atrodack]

Hi Luc,

I do not believe we were ever able to solve the uploading problem. We were going to look into our circuit boards again and try to figure out if we had a grounding loop problem or something else strange that could be causing the problem interacting with the NUC for upload from there. You are correct that we could read off the serial monitor on the NUC still. We have tested reading out the temperature and humidity sensors from the NUC and been able to without issue if the script is uploaded from a separate computer first.

[lucboucher]

thanks @atrodack!
I have more a feeling of software issue here but I might be biased…
I have tried various USB cables, with and without USB hubs and still never works.
I am also seeing different error messages depending on the execution of the permission stuff…
I’ll copy them and upload here when I got back to this again just to check if they are similar to what you have.
Thanks and good luck!

[jmk1729]

tl;dr: after a long layoff, we are putting the finishing touches on weatherproofing PAN010, and are now in the process of installing it at a permanent site on Mt. Lemmon.

Weatherproofing the head unit

We took a page out of PAN015’s playbook and 3D printed each camera lens cap. We will attach or link to the openSCAD file we made when we get permission to upload from someone (@wtgee, @nem?).
EDIT: Here is the link.
The resulting lens caps are shown below in Fig. 1 and 2.

Figure 1: Lens caps in the head unit. Each cap is made up of the same UV cured resin material; they have been spray painted white to protect them once they are outdoors.

Figure 2: Each cap is attached with sugru to its respective lens baffle. There is a notch in one of the caps to avoid hitting the latch on the head unit while rotating the baffle into the lock position on the camera lens.

Referencing Fig. 1, we inlet the USB and power cables through the small hole between the cameras. We have cut a hole into some screen door mesh to avoid bugs making their way into the head unit. Minus some minor adjustments, this completes the weatherproofing on the head unit.

Weatherproofing the mount

Using instructions from @tmcook along with experience @jlumbres gained from her trip to Bhutan, we have finished putting together the mount sheathing. This process is detailed in the figures below.

Figure 3: A montage of Jhen cutting, shaping, and gluing (with silicone) the aluminum sheeting to the mount.

One concern we have is that the folded aluminum panel covering the right ascension motor will collide with the DEC clutch nut at the corners of the folds during certain motions. One collision region is the corner of the panel marked with sharpie in the top-right corner image of Fig. 4. While the mount may rarely, if ever, rotate in such a way as to cause a collision between the aluminum and the DEC clutch nut, we may prevent this by cutting or bending the aluminum corners, or restrict such mount movements using the POCS software.

Figure 4: PAN010 suited up from four different angles. We still have some gaps to patch up with sugru or silicone and some edges to file down.

Next steps

• We will go back to testing our unit outdoors, ensuring that it can operate completely autonomously before we leave it up on a mountain.

• There are a number of items to purchase and set up during the installation process on Mt. Lemmon. These include equipment for securing PAN010 onto a rooftop, isolating the hardware from the mountain operations network (in case of lightning strikes), and lengthening the cables from the electronics control box to the head unit.

We will keep the thread updated with our progress.

Justin

[nem]

Great work guys.

Not sure why you wouldn’t be able to put things on the drive. Ill send you a link on slack.

The plate covering the electronics is pushed too far up. It should not cover the black box fully. It should stop roughly 5-10 mm from the top. Either way, make mods as you need. We tested ours and dont have any collisions in any orientation.

[jmk1729]

We have successfully installed POCS using the docker containers. Thanks @wtgee and @tmcook for the on-going instructions. Our unit is soon to be deployed on the Steward Observatory roof in preparation for the Mercury transit now that the weatherproofing is complete. We have uncovered some errors from playing around before taking it up though:

Moving the mount from POCS

When calling go_home or park from the POCS shell, we occasionally encounter the following error which stops the mount in its tracks and freezes POCS.

Traceback (most recent call last):
File "/var/panoptes/POCS/scripts/pocs-shell.py",  line 850, in <module>
PocsShell().cmdloop()
File "/opt/conda/lib/python3.7/cmd.py", line 138, in cmdloop
stop = self.onecmd(line)
File "/opt/conda/lib/python3.7/cmd.py", line 217, in onecmd
return func(arg)
File "/var/panoptes/POCS/scripts/pocs-shell.py", line 376, in do_polar_alignment_test
mount.slew_to_home(blocking=True)
File "/var/panoptes/POCS/pocs/mount/mount.py", line 591, in slew_to_home
while self.is_home is False:
File "/var/panoptes/POCS/pocs/mount/ioptron.py", line 98, in is_home
self._is_home = 'Stopped - Zero Position' in self.status().get('state', '')
File "/var/panoptes/POCS/pocs/mount/mount.py", line 122, in status
status.update(self._update_status())
File "/var/panoptes/POCS/pocs/mount/serial.py", line 262, in _update_status
self.ra_guide_rate = int(guide_rate[0:2]) / 100
ValueError: invalid literal for int() with base 10: ''

It seems that using run_pocs encounters no such errors so far.

Running PAWS

We try python app.py in $PANDIR/PAWS as we used in the previous (non-docker) version of POCS and see the following error:

Traceback (most recent call last):
File "app.py", line 4, in <module>
import tornado.escape
ImportError: No module named tornado.escape

Is this how we’re supposed to be running PAWS in the docker POCS? We do see the paws container service start up and run with bin/pocs up -d.

Weather sensor

We can run PEAS, but we can’t load the weather sensor (aag). The camera and control boards load without issue and read out sensible values. This is the error we see which kills the PEAS shell:

Loading weather reader endpoint
Traceback (most recent call last):
File "/var/panoptes/POCS/scripts/peas-shell.py", line 469, in <module>
PanSensorShell().cmdloop()
File "/opt/conda/lib/python3.7/cmd.py", line 138, in cmdloop
stop = self.onecmd(line)
File "/opt/conda/lib/python3.7/cmd.py", line 217, in onecmd
return func(arg)
File "/var/panoptes/POCS/scripts/peas-shell.py", line 224, in do_load_weather
db_type=get_config('db.type', default='file')
File "/var/panoptes/POCS/peas/remote_sensors.py", line 39, in __init__
raise error.PanError(f'No endpoint_url for {sensor_name}')
panoptes.utils.error.PanError: PanError: No endpoint_url for weather

Is there a config file somewhere we need to edit this url other than what’s in pocs.yaml and pocs_local.yaml? We’ll continue to familiarize ourselves with this new version of POCS, but suggestions are welcome. This is a pressing issue that we need to solve to have our unit outside autonomously.

Power

When running POCS, we seem to have to include power as simulated in setup_pocs for the unit to work, even though we have an AC sensor on our mains power line. This is what is printed in the pocs-all log:

core.py:438] Checking for AC power
__init__.py:060] No record found for power
core.py:452] No mains "power" reading found in database.
core.py:469] No record found in DB: 'NoneType' object is not subscriptable
core.py:478] AC power not detected.

Is this indicative of a problem with our AC sensor, or is this expected behavior for running POCS?

[zacharyt]

@jmk1729 Over at PAN015 Wilfred told us to start pocs in the foreground by emitting the -d in bin/pocs up -d. On our unit, this gave us much more feed back and revealed issues/errors we hadn’t seen in the POCS shell. When running in the foreground we had many communication errors between docker containers. If this is an issue with the docker version of the software it could explain why your AC power is not being detected (Assuming the pocs-shell container imports information from other containers).

[wtgee]

Is there a config file somewhere we need to edit this url other than what’s in pocs.yaml and pocs_local.yaml ? We’ll continue to familiarize ourselves with this new version of POCS, but suggestions are welcome. This is a pressing issue that we need to solve to have our unit outside autonomously.

Sorry, just answered this over in PAN015: Wildwood Institute for STEM Research and Development, WISRD (Wildwood School)

[atrodack]

We want to provide an update on our progress here at PAN010.

The good stuff

With the help of @wtgee, we were able to debug most of our remaining issues, including resolving the endpoint_url for the AAG weather sensor, and the issues with our instance of the pocs shell failing power checks even though the arduino was reporting correctly that mains power was present.

To fix the WS, the last two lines were added to the pocs_local.yaml file:

environment:
  auto_detect: false
  camera_board:
    serial_port: /dev/ttyACM1
  control_board:
    serial_port: /dev/ttyACM0
  weather:
    url: http://127.0.0.1:5000/latest.json

To fix the power issue, we edited $POCS/peas/sensors.py, line 146:

Old:
# Make a separate power entry
   if 'power' in sensor_data:
       self.db.insert_current('power', data['power'])

Changed to:
# Make a separate power entry
   if 'power' in data:
       self.db.insert_current('power', data['power'])

We also purchased a new USB-to-RJ9 cable to connect our mount to the NUC as we were starting to see frequent issues with communication that we pinned down to being our original RJ9 cable, which was fairly short and often abused some when leveling/moving around the mount. Since changing the cable, we have not experienced any communication issues with the mount.

Now on to the not so good stuff.

During some testing of run_pocs and its ability to respond to changing parameters (‘cloudy’ or loss of WS input, loss of mains power), we have uncovered (or perhaps reuncovered) an issue. With the NUC we have (if memory serves, we purchased a slightly more powerful, in terms of CPU, NUC than recommended because the recommended one was not available), we had to adjust the voltage coming to it from mains to 13.6V in order for it to boot/run safely without attempting to draw too much current and causing an automatic shutdown with a thermal trip. It turns out, with our 12V backup battery taking over when mains power is lost, our NUC again suffers this fate, and shuts off before POCS can operate successfully to park and power off the unit. We had just enough time to query the control_board in peas-shell to see that our AC sensor correctly identified that mains power was lost before the NUC failed and shut itself down.
We are open to any suggestions on how to fix this. Our list of possible solutions includes: 1) Getting a larger voltage backup battery 2) Adding a second, smaller voltage backup battery in series with the first 3) Trying to further adjust the NUC settings to try to reduce the power requirement (running without graphics, etc). Maybe @wtgee @james.synge or @nem have something better in mind.

PAWS is still not working well, but that isn’t a critical issue as it doesn’t effect our ability to run the unit.

Things to do

We have our AAG WS operating to try to test that it correctly identifies when clouds are present. @wtgee suggested we may need to update the coefficients using the Windows software to make sure this is accurate. Unfortunately, it has not been cloudy for the last 3 days while we want to test this. Fortunately, starting tomorrow we have several cloudy/rainy days forecasted.

We want to have out unit out in this rain to convince ourselves that our weatherproofing is complete.

We need to verify one of cameras, which has been acting up since we did the test removing mains power and had the NUC shut down mid run_pocs. We pulled the camera out and tested it with a camera battery, and it seems to behave well, but we haven’t yet tested in place in the unit again with the power coming from the control box to see if we damaged the power cable, or it just needed to be powered off to clean up.