PAN010: Build thread for UofA

Does sound like a balance issue so that’s the first thing I would check. Make sure all the wires are plugged in when doing that as it can add just enough weight.

UA PAN010 Update from our work today:

  • We connected to the Google Cloud server.

  • We’re in the process of getting ssh access for roof testing. It’ll be good for us when we’re ready to go to park it on the roof.

  • We uploaded the telemetry and camera boards arduino scripts directly using @atrodack’s laptop and the most recent arduino gui version.
    – We tried in the past using the cloud version, but it didn’t work.
    – On the camera board, the accelerometer and humidity sensors worked in both the arduino gui and PEAS. However, the humidity sensor was finicky and moving something caused NaN values to output. We’re not sure what is going on there, so any suggestions welcomed.
    – For the telemetry board, the humidity sensor worked in both the arduino gui and PEAS. However, we’re not sure what’s going on with the temperature sensors (they output -127.0 values). I suspect it could be a funny way I wired the temperature sensors on the telemetry board. I’ll go investigate it.

  • We managed to get the cloud sensor to be detected in PEAS (took a while, though…)
    – We found that having the arduino scripts uploaded gets PEAS to see the cloud sensor.
    – We hung out the cloud sensor outside the window in Justin/Olivier’s office. The weather stayed at “Very Cloudy” (it is indeed very cloudy today here in Tucson) and the sky temperature increased. We suspect that some of our results are skewed because it’s detecting parts of the building. We’ll need to test this on the roof.

  • We set the mount’s meridian position limit to 25 degrees using the controller.

  • We’re in the process for setting up the weatherproofing.
    – We have access to a 3D resin printer (through CAAO/LBTI) that we’re going to use for the camera box seals (featured from WISRD). We’re going to go through some test versions on another 3D printer to make sure we get the fit correctly before going through the resin printing. We can’t use a lot of the resin, so we decided to use it for the camera box seals.
    – We’re making a list of equipment we need to procure.

1 Like

Great to see the UofA unit in action.

In regards to setting the exposure time, we did this for HD189733 by taking an exposure, and checking the counts (in ds9 I believe). Aru showed me that you can type the name of a target into Ds9 and it will find it in the field so you dont need to look for it manually. From last Augusts efforts we worked out that the camera has a 2048 count pedestal (you cant get less counts than this) and a maximum of 13???. So you want the signal from the target to be between these two values. About 9000-10000 counts would be perfect but it will vary as the star moves across the pixels and has a different count in each color channel. So we adjusted things until we go 8000-10000 counts in the brightest channel and took a few images to see how much it fluctuated as it walked across pixels. I think in the end we went with a 35 second exposure time with the SL1 for this target and an open aperture. So this should work for you too.

Good luck

Hey PAN010 team,
did you ever solve the issue on downloading the script to the arduinos from the NUC?? if so I’m really interested. I tried this but it did not help…
The workaround you mention (downloading from another computer) means that the issue is just downloading and that the NUC is able to read the serial monitor from the arduinos to get the telemetry and allow you to switch stuffs on/off right?
Thanks,

Ok answering myself as I was able to test stuffs earlier than I thought.
YES even if we cannot download the script to the Arduino with the NUC we can actually read the serial monitor telemetry and switch things off/on.

I am still interested in knowing if you solved your script download issue!

Cheers,

Hi Luc,

I do not believe we were ever able to solve the uploading problem. We were going to look into our circuit boards again and try to figure out if we had a grounding loop problem or something else strange that could be causing the problem interacting with the NUC for upload from there. You are correct that we could read off the serial monitor on the NUC still. We have tested reading out the temperature and humidity sensors from the NUC and been able to without issue if the script is uploaded from a separate computer first.

thanks @atrodack!
I have more a feeling of software issue here but I might be biased…
I have tried various USB cables, with and without USB hubs and still never works.
I am also seeing different error messages depending on the execution of the permission stuff…
I’ll copy them and upload here when I got back to this again just to check if they are similar to what you have.
Thanks and good luck!

tl;dr: after a long layoff, we are putting the finishing touches on weatherproofing PAN010, and are now in the process of installing it at a permanent site on Mt. Lemmon.

Weatherproofing the head unit

We took a page out of PAN015’s playbook and 3D printed each camera lens cap. We will attach or link to the openSCAD file we made when we get permission to upload from someone (@wtgee, @nem?).
EDIT: Here is the link.
The resulting lens caps are shown below in Fig. 1 and 2.

LensCaps_HeadUnit
Figure 1: Lens caps in the head unit. Each cap is made up of the same UV cured resin material; they have been spray painted white to protect them once they are outdoors.

LensCaps
Figure 2: Each cap is attached with sugru to its respective lens baffle. There is a notch in one of the caps to avoid hitting the latch on the head unit while rotating the baffle into the lock position on the camera lens.

Referencing Fig. 1, we inlet the USB and power cables through the small hole between the cameras. We have cut a hole into some screen door mesh to avoid bugs making their way into the head unit. Minus some minor adjustments, this completes the weatherproofing on the head unit.

Weatherproofing the mount

Using instructions from @tmcook along with experience @jlumbres gained from her trip to Bhutan, we have finished putting together the mount sheathing. This process is detailed in the figures below.


Figure 3: A montage of Jhen cutting, shaping, and gluing (with silicone) the aluminum sheeting to the mount.

One concern we have is that the folded aluminum panel covering the right ascension motor will collide with the DEC clutch nut at the corners of the folds during certain motions. One collision region is the corner of the panel marked with sharpie in the top-right corner image of Fig. 4. While the mount may rarely, if ever, rotate in such a way as to cause a collision between the aluminum and the DEC clutch nut, we may prevent this by cutting or bending the aluminum corners, or restrict such mount movements using the POCS software.


Figure 4: PAN010 suited up from four different angles. We still have some gaps to patch up with sugru or silicone and some edges to file down.

Next steps

  • We will go back to testing our unit outdoors, ensuring that it can operate completely autonomously before we leave it up on a mountain.

  • There are a number of items to purchase and set up during the installation process on Mt. Lemmon. These include equipment for securing PAN010 onto a rooftop, isolating the hardware from the mountain operations network (in case of lightning strikes), and lengthening the cables from the electronics control box to the head unit.

We will keep the thread updated with our progress.

Justin

Great work guys.

Not sure why you wouldn’t be able to put things on the drive. Ill send you a link on slack.

The plate covering the electronics is pushed too far up. It should not cover the black box fully. It should stop roughly 5-10 mm from the top. Either way, make mods as you need. We tested ours and dont have any collisions in any orientation.

We have successfully installed POCS using the docker containers. Thanks @wtgee and @tmcook for the on-going instructions. Our unit is soon to be deployed on the Steward Observatory roof in preparation for the Mercury transit now that the weatherproofing is complete. We have uncovered some errors from playing around before taking it up though:

Moving the mount from POCS

When calling go_home or park from the POCS shell, we occasionally encounter the following error which stops the mount in its tracks and freezes POCS.

Traceback (most recent call last):
File "/var/panoptes/POCS/scripts/pocs-shell.py",  line 850, in <module>
PocsShell().cmdloop()
File "/opt/conda/lib/python3.7/cmd.py", line 138, in cmdloop
stop = self.onecmd(line)
File "/opt/conda/lib/python3.7/cmd.py", line 217, in onecmd
return func(arg)
File "/var/panoptes/POCS/scripts/pocs-shell.py", line 376, in do_polar_alignment_test
mount.slew_to_home(blocking=True)
File "/var/panoptes/POCS/pocs/mount/mount.py", line 591, in slew_to_home
while self.is_home is False:
File "/var/panoptes/POCS/pocs/mount/ioptron.py", line 98, in is_home
self._is_home = 'Stopped - Zero Position' in self.status().get('state', '')
File "/var/panoptes/POCS/pocs/mount/mount.py", line 122, in status
status.update(self._update_status())
File "/var/panoptes/POCS/pocs/mount/serial.py", line 262, in _update_status
self.ra_guide_rate = int(guide_rate[0:2]) / 100
ValueError: invalid literal for int() with base 10: ''

It seems that using run_pocs encounters no such errors so far.

Running PAWS

We try python app.py in $PANDIR/PAWS as we used in the previous (non-docker) version of POCS and see the following error:

Traceback (most recent call last):
File "app.py", line 4, in <module>
import tornado.escape
ImportError: No module named tornado.escape

Is this how we’re supposed to be running PAWS in the docker POCS? We do see the paws container service start up and run with bin/pocs up -d.

Weather sensor

We can run PEAS, but we can’t load the weather sensor (aag). The camera and control boards load without issue and read out sensible values. This is the error we see which kills the PEAS shell:

Loading weather reader endpoint
Traceback (most recent call last):
File "/var/panoptes/POCS/scripts/peas-shell.py", line 469, in <module>
PanSensorShell().cmdloop()
File "/opt/conda/lib/python3.7/cmd.py", line 138, in cmdloop
stop = self.onecmd(line)
File "/opt/conda/lib/python3.7/cmd.py", line 217, in onecmd
return func(arg)
File "/var/panoptes/POCS/scripts/peas-shell.py", line 224, in do_load_weather
db_type=get_config('db.type', default='file')
File "/var/panoptes/POCS/peas/remote_sensors.py", line 39, in __init__
raise error.PanError(f'No endpoint_url for {sensor_name}')
panoptes.utils.error.PanError: PanError: No endpoint_url for weather

Is there a config file somewhere we need to edit this url other than what’s in pocs.yaml and pocs_local.yaml? We’ll continue to familiarize ourselves with this new version of POCS, but suggestions are welcome. This is a pressing issue that we need to solve to have our unit outside autonomously.

Power

When running POCS, we seem to have to include power as simulated in setup_pocs for the unit to work, even though we have an AC sensor on our mains power line. This is what is printed in the pocs-all log:

core.py:438] Checking for AC power
__init__.py:060] No record found for power
core.py:452] No mains "power" reading found in database.
core.py:469] No record found in DB: 'NoneType' object is not subscriptable
core.py:478] AC power not detected.

Is this indicative of a problem with our AC sensor, or is this expected behavior for running POCS?

@jmk1729 Over at PAN015 Wilfred told us to start pocs in the foreground by emitting the -d in bin/pocs up -d. On our unit, this gave us much more feed back and revealed issues/errors we hadn’t seen in the POCS shell. When running in the foreground we had many communication errors between docker containers. If this is an issue with the docker version of the software it could explain why your AC power is not being detected (Assuming the pocs-shell container imports information from other containers).

Sorry, just answered this over in PAN015: Wildwood Institute for STEM Research and Development, WISRD (Wildwood School)

We want to provide an update on our progress here at PAN010.

The good stuff

With the help of @wtgee, we were able to debug most of our remaining issues, including resolving the endpoint_url for the AAG weather sensor, and the issues with our instance of the pocs shell failing power checks even though the arduino was reporting correctly that mains power was present.

To fix the WS, the last two lines were added to the pocs_local.yaml file:

environment:
  auto_detect: false
  camera_board:
    serial_port: /dev/ttyACM1
  control_board:
    serial_port: /dev/ttyACM0
  weather:
    url: http://127.0.0.1:5000/latest.json

To fix the power issue, we edited $POCS/peas/sensors.py, line 146:

Old:
# Make a separate power entry
   if 'power' in sensor_data:
       self.db.insert_current('power', data['power'])

Changed to:
# Make a separate power entry
   if 'power' in data:
       self.db.insert_current('power', data['power'])

We also purchased a new USB-to-RJ9 cable to connect our mount to the NUC as we were starting to see frequent issues with communication that we pinned down to being our original RJ9 cable, which was fairly short and often abused some when leveling/moving around the mount. Since changing the cable, we have not experienced any communication issues with the mount.

Now on to the not so good stuff.

During some testing of run_pocs and its ability to respond to changing parameters (‘cloudy’ or loss of WS input, loss of mains power), we have uncovered (or perhaps reuncovered) an issue. With the NUC we have (if memory serves, we purchased a slightly more powerful, in terms of CPU, NUC than recommended because the recommended one was not available), we had to adjust the voltage coming to it from mains to 13.6V in order for it to boot/run safely without attempting to draw too much current and causing an automatic shutdown with a thermal trip. It turns out, with our 12V backup battery taking over when mains power is lost, our NUC again suffers this fate, and shuts off before POCS can operate successfully to park and power off the unit. We had just enough time to query the control_board in peas-shell to see that our AC sensor correctly identified that mains power was lost before the NUC failed and shut itself down.
We are open to any suggestions on how to fix this. Our list of possible solutions includes: 1) Getting a larger voltage backup battery 2) Adding a second, smaller voltage backup battery in series with the first 3) Trying to further adjust the NUC settings to try to reduce the power requirement (running without graphics, etc). Maybe @wtgee @james.synge or @nem have something better in mind.

PAWS is still not working well, but that isn’t a critical issue as it doesn’t effect our ability to run the unit.

Things to do

We have our AAG WS operating to try to test that it correctly identifies when clouds are present. @wtgee suggested we may need to update the coefficients using the Windows software to make sure this is accurate. Unfortunately, it has not been cloudy for the last 3 days while we want to test this. Fortunately, starting tomorrow we have several cloudy/rainy days forecasted.

We want to have out unit out in this rain to convince ourselves that our weatherproofing is complete.

We need to verify one of cameras, which has been acting up since we did the test removing mains power and had the NUC shut down mid run_pocs. We pulled the camera out and tested it with a camera battery, and it seems to behave well, but we haven’t yet tested in place in the unit again with the power coming from the control box to see if we damaged the power cable, or it just needed to be powered off to clean up.