PAN015: Wildwood Institute for STEM Research and Development, WISRD (Wildwood School)

Hmm, I thought the install script was properly doing that but it might not be. I’ll double-check. But ultimately yes, you do need a $PANDIR/.env file that contains some of the variables.

I usually don’t use the -d option and it keeps everything in the foreground. That will show some error messages from startup in a clean fashion. You would then need to open another terminal to do anything. It’s mostly personal preference for how you would want set up.

Docker works like a virtual machine, so all of the required software and files are loaded inside the docker image. When you run a copy of the docker image it is referred to as a docker container. In this way you can have many instances (i.e. docker containers) of the same virtual machine-like setup (i.e. the docker image).

So here what you are seeing are errors that are happening inside the virtual container. This container contains its own small version of linux and filesystem, so the /opt errors you are seeing are from inside the container. If using the docker setup you shouldn’t need to install any additional software such as anaconda.

Does the main computer have a panoptes user? The best would be to grant the panoptes user as the owner and group of $PANDIR, i.e.:

$ sudo chown -R panoptes:panoptes $PANDIR

There might be some misplaced owner/permissions from a previous setup so the above command should fix all of that.

The errors with connecting to the arduino might just be port issues (as listed in the config file). I think @tmcook had made some notes in the document about trying to determine the usb ports. It mostly has to do with the order in which they were plugged in but sometimes takes a bit of troubleshooting.

See if any of the above helps with the permission issues and then maybe try to unplug the usb ports and only plug in one at a time to test and verify things. i.e. just plug in the weather station, figure out what port it is, try to load it via peas_shell, etc.

Thanks for the help @wtgee . I made the panoptes user the owner and group of $PANDIR, and then tried bin/pocs up without -d and saw hundreds of error messages. The ones I looked at appeared to be caused by the AAG cloud sensor not being found, which makes sense since we don’t have one. (We were going to deploy on our school’s roof and use a weather station we built a few years ago, but we may be deploying at Mount Wilson along with PAN012) Is there a way to pass an argument to this command to have it load specific docker containers? Can the sensor be simulated the way it can be in the pocs shell or can its absence be stated in pocs_local.yaml?

UPDATE:
We purchased the AAG Cloud sensor, connected it, and tested the docker software today. We are getting significantly less error messages when running bin/pocs up, and the AAG weather sensor errors are completely gone. Output of bin/pocs up is:

Starting config-server      ... done
Starting aag-weather-server ... done
Starting aag-weather-reader ... done
Starting messaging-hub      ... done
Starting paws               ... done
Starting peas-shell         ... done
Starting pocs-shell         ... done
Attaching to config-server, aag-weather-server, aag-weather-reader, messaging-hub, paws, peas-shell, pocs-shell
config-server         | Starting with UID : 9001
config-server         | useradd: user 'panoptes' already exists
aag-weather-server    |  * Serving Flask app "scripts/serve-aag"
aag-weather-server    |  * Environment: production
aag-weather-server    |    WARNING: This is a development server. Do not use it in a production deployment.
aag-weather-server    |    Use a production WSGI server instead.
aag-weather-server    |  * Debug mode: off
messaging-hub         | Starting with UID : 9001
messaging-hub         | useradd: user 'panoptes' already exists
paws                  | Starting with UID : 9001
paws                  | useradd: user 'panoptes' already exists
peas-shell            | Starting with UID : 9001
peas-shell            | useradd: user 'panoptes' already exists
pocs-shell            | Starting with UID : 9001
pocs-shell            | useradd: user 'panoptes' already exists
aag-weather-server    |  * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
paws                  | ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /var/panoptes/PAWS
paws                  | Traceback (most recent call last):
paws                  |   File "app.py", line 67, in <module>
paws                  |     http_server = tornado.httpserver.HTTPServer(WebAdmin(get_config(host='0.0.0.0')))
paws                  |   File "app.py", line 34, in __init__
paws                  |     name = config.setdefault('name', 'PAWS')
paws                  | AttributeError: 'NoneType' object has no attribute 'setdefault'
paws exited with code 1
messaging-hub         | wait-for-it.sh: waiting 15 seconds for localhost:6563
config-server         |  * Serving Flask app "panoptes.utils.config.server" (lazy loading)
config-server         |  * Environment: production
config-server         |    WARNING: This is a development server. Do not use it in a production deployment.
config-server         |    Use a production WSGI server instead.
config-server         |  * Debug mode: off
messaging-hub         | wait-for-it.sh: localhost:6563 is available after 2 seconds
peas-shell            | wait-for-it.sh: waiting 15 seconds for localhost:6563
peas-shell            | wait-for-it.sh: localhost:6563 is available after 0 seconds
pocs-shell            | wait-for-it.sh: waiting 15 seconds for localhost:6563
pocs-shell            | wait-for-it.sh: localhost:6563 is available after 0 seconds

The error:

paws                  | ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /var/panoptes/PAWS
paws                  | Traceback (most recent call last):
paws                  |   File "app.py", line 67, in <module>
paws                  |     http_server = tornado.httpserver.HTTPServer(WebAdmin(get_config(host='0.0.0.0')))
paws                  |   File "app.py", line 34, in __init__
paws                  |     name = config.setdefault('name', 'PAWS')
paws                  | AttributeError: 'NoneType' object has no attribute 'setdefault'

After this, the script hangs. I believe that this script is supposed to hang and then the user can access peas and pocs shells from other terminal windows, but I would expect there to be some sort of message to the user saying that everything is complete. @wtgee Does pocs up have a message like this and is our script hanging to early?

I’m a little confused on why useradd: user 'panoptes' already exists is appearing. Is this adding panoptes as a user/owner/group to the docker containers or is it trying to create a new user outside of the containers?

Note: We learned the hard way that bin/pocs up does not check to see if docker containers are already started, and creates many errors.

Also, while waiting for the weather sensor to arrive I created a python script using the reset_usb_device.py script that automatically finds the dev path of a usb port (/dev/tty*). If you would be interested in putting it on github let me know.

HI @zacharyt,

Yes this hangs by default. Some logs messages from the various containers will be output here as well. You can run bin/pocs up -d (for daemon) if you want it to not hang. The log messages it shows aren’t critical although the messages shown during startup can be helpful.

It looks like your /var/panoptes/.env is missing a LOCAL_USER_ID=1000, which might help with this issue. However it is most likely a result of restarting existing containers, which might be what you are referring to by saying you learned the hard way?

You are right that it should check to see if the containers are already running. I will make an issue on github to fix it (unless you want to? :grin:).

Yes, I’m curious about this. It automatically finds it how? Auto-assigning the usb ports in the config would be very helpful but has proven to be more difficult than one would imagine. Not a ton of time has been spent on this though but it’s starting to to be more necessary as more people bring their units online and have confusion about the ports. Please make a PR on github if you would like to share.

As an FYI, PAWS is still pretty broken with docker setup so I wouldn’t worry about those errors.

@wtgee I added LOCAL_USER_ID=1000 to .env. I created the .env by coping and pasting the contents of env_file into a new file and renaming it .env. What should be contained in .env? Our current .env only has:

PANDIR=/var/panoptes
POCS=/var/panoptes/POCS
PANLOG=/var/panoptes/logs
LOCAL_USER_ID=1000

The addition of LOCAL_USER_ID fixed the errors in the peas and pocs shells! :raised_hands:

We ran into some small errors that are likely not related to the software.
pocs-shell:
We cannot use setup_pocs, the command exits with something along the lines of cameras are not available. This makes sense since we have not killed the processes on the camera’s after we moved our unit downstairs for poster presentation night, where we unplugged USB’s and plugged them back in. We have been killing the processes by using kill [PID], and have been getting the PID from lsof | grep /dev/bus/usb/*** (where *** is our camera usb number given by gphoto auto detect). It appears that docker stops lsof from working, and we cant find the PID to kill to free up the camera USBs.

peas-shell
We cannot load the weather sensor. The error in the peas-shell is:

PEAS > load_weather
Loading weather reader endpoint
Traceback (most recent call last):
  File "/var/panoptes/POCS/scripts/peas-shell.py", line 469, in <module>
    PanSensorShell().cmdloop()
  File "/opt/conda/lib/python3.7/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/opt/conda/lib/python3.7/cmd.py", line 217, in onecmd
    return func(arg)
  File "/var/panoptes/POCS/scripts/peas-shell.py", line 224, in do_load_weather
    db_type=get_config('db.type', default='file')
  File "/var/panoptes/POCS/peas/remote_sensors.py", line 39, in __init__
    raise error.PanError(f'No endpoint_url for {sensor_name}')
panoptes.utils.error.PanError: PanError: No endpoint_url for weather

Note: I setup the weather sensor today, and was a little crunched for time, so if the AAG cloud sensor has a power button that explains the error…

We can load the camera and control boards although there seems to be parsing errors when trying to view their data using last_reading. I know we modified the camera board’s arduino script to work with an Uno instead of a Micro, and I’m guessing we might have commented out some print functions. Tomorrow I’ll update our fix for the Uno on the docker arduino file’s and see if the errors go away (and turn on the cloud sensor if it has a power switch).

I would be happy to put the usb port script up on github, but it simply parses information from the bash command: ls -lR /dev/serial/by-id. The script doesn’t properly name devices that are connected by an adapter to the NUC (Namely the mount and the camera). Also, it is my first time doing text parsing, and the script is based on a very brute force approach. I’ll try implement the checking for open containers on github, but like the usb script I’m mostly relying on my knowledge of bash commands and parsing their output, so it likely won’t be pretty or easy on the processor.

This is probably just a config issue. The PAN010 team (@jlumbres, @atrodack, @jmk1729) are having the same problem right now.

I think you just need this in the config:

environment:
    weather:
        url: https://127.0.0.1:5000/latest.json

It’s helpful to have httpie installed on the NUC (not in the containers). This can be installed via apt. You can then do simple calls on the command line to verify it’s working:

(panoptes-env) ➜  ~ http :5000/latest.json

HTTP/1.0 200 OK
Content-Length: 518
Content-Type: application/json
Date: Thu, 14 Nov 2019 02:37:26 GMT
Server: Werkzeug/0.16.0 Python/3.7.4

[
    {
        "ambient_temp_C": 14.13,
        "date": "2019-11-14T02:36:44.576Z",
        "errors": "error_1=0 error_2=0 error_3=0 error_4=0",
        "gust_condition": "Calm",
        "internal_voltage_V": 5.9019230769,
        "ldr_resistance_Ohm": 0.4970414201,
        "pwm_value": 9.9706744868,
        "rain_condition": "Dry",
        "rain_frequency": 2652.0,
        "rain_sensor_temp_C": 19.0,
        "safe": false,
        "sky_condition": "Very Cloudy",
        "sky_temp_C": 6.43,
        "weather_sensor_firmware_version": 5.51,
        "weather_sensor_name": "CloudWatcher",
        "weather_sensor_serial_number": 821,
        "wind_condition": "Calm",
        "wind_speed_KPH": 0.0
    }
]

Note that the weather url config does go under environment and not under the weather entry.

Basically the weather station has been decoupled from POCS and PEAS so can operate separately. For PAN008 (and a few others) the AAG is housed at the observatory site but is separate from PANOPTES so is not available on a USB port or anything. The AAG Weather Repo is in charge of the weather and sets up a little web server that serves JSON entries. PEAS then treats this as a RemoteSensor, which just listens on any url for some JSON text (we call those “endpoints” in the web development world). The aag-weather-server container just starts up that repo and serves the web page.

@wtgee is the end point url put before or after the camera_board and control_board information? Does that even matter? Is the config file parsed for info and put into a python dict? If I get it working I’ll push to github.

We have been having this issue as well at PAN010. I added that bit after the camera/control board lines, and ran the test that @wtgee suggested and it printed the same result as above. We will be testing with our AAG plugged in momentarily.

So our pocs_local.yaml environment section looks like this:

environment:
  auto_detect: false
  camera_board:
    serial_port: /dev/ttyACM1
  control_board:
    serial_port: /dev/ttyACM0
  weather:
    url: http://127.0.0.1:5000/latest.json

@atrodack

I think he meant to put it under the environment section where the arduinos are. I’m testing as well and will let you know if I get it working. I would also expect that we have to download the AAG Weather Repo Wilfred mentioned.

He meant to not put it in this section:

weather:
  aag_cloud:
    serial_port: /dev/ttyUSB1
    threshold_cloudy: -25
    threshold_very_cloudy: -15.
    threshold_windy: 50.
    threshold_very_windy: 75.
    threshold_gusty: 100.
    threshold_very_gusty: 125.
    threshold_wet: 2200.
    threshold_rainy: 1800.
    safety_delay: 15     ## minutes
    heater:
      low_temp: 0       ## deg C
      low_delta: 6       ## deg C
      high_temp: 20       ## deg C
      high_delta: 4       ## deg C
      min_power: 10       ## percent
      impulse_temp: 10       ## deg C
      impulse_duration: 60       ## seconds
      impulse_cycle: 600       ## seconds
  plot:
    amb_temp_limits: [-5, 35]
    cloudiness_limits: [-45, 5]
    wind_limits: [0, 75]
    rain_limits: [700, 3200]
    pwm_limits: [-5, 105]

I clarified this point with him, so it should look like what my first post was. It is meant to place a weather subcategory in the environment section, and to not add the url to the weather section printed here

1 Like

Thanks for the clarification. After downloading the aag-weather repo to $PANDIR the peas-shell no longer exits and gives me what looks like a parsing/data storage issue with JSON server. The error in the peas shell is:

Problem storing captured data: SSLError(MaxRetryError('HTTPSConnectionPool(host=\'127.0.0.1\', port=5000): Max retries exceeded with url: /latest.json (Caused by SSLError(SSLError("bad handshake: Error([(\'SSL routines\', \'ssl3_get_record\', \'wrong version number\')])")))'))

Note: This could be happening because I was unable to run the setup.py file in the aag-weather repo as I was stopped by an error. I’m going to look at the README.md file inside the software to confirm I did the install properly.

I am also seeing that error printed in the PEAS-shell

@zacharyt replace https with http in the config to fix that issue. I updated the post above to reflect the change
@wtgee told us to try that and our AAG is now successfully reading the weather in PEAS-shell

Awesome! I’m trying it now. I will definitely be making a lot of documentation in the coming days…

Edit: AAG just started working for me. I’m going to do a PR on github to fix the config file.

Thanks @atrodackand @zacharyt, glad some things are working.

@zacharyt, you shouldn’t have had to manually download the aag-weather repo so it’s odd that it started working after. But I’ll check and see how it is set up.

@wtgee I downloaded the docker version of the software about a month ago, so I am probably behind on a lot of changes. Can the docker version be updated with git pull like the previous version?

Also, did you see my PR on github? I can’t seem to get it to appear in the open PR tab.

Hi @wtgee,
The issues I am having with the PEAS shell have continued despite using the most recent arduino files. In the peas-shell I am seeing:

Welcome to PEAS Shell! Type ? for help
PEAS > load_weather
Loading weather reader endpoint
PEAS > load_camera_board
Loading camera board sensor
PEAS > load_control_board
Loading control board sensor
PEAS > start
Starting sensors
PEAS > last_reading weather
********************************************************************************
WEATHER:
{'_id': '5e8cced1-b11d-4201-bc34-abe9f39e8ced',
 'data': {'ambient_temp_C': 27.39,
          'date': '20191114T180055',
          'errors': 'error_1=0 error_2=0 error_3=0 error_4=0',
          'gust_condition': 'Calm',
          'internal_voltage_V': 6.0176470588,
          'ldr_resistance_Ohm': 1.9251769464,
          'pwm_value': 9.9706744868,
          'rain_condition': 'Dry',
          'rain_frequency': 3969.0,
          'rain_sensor_temp_C': 34.51,
          'safe': False,
          'sky_condition': 'Very Cloudy',
          'sky_temp_C': 23.15,
          'weather_sensor_firmware_version': 5.54,
          'weather_sensor_name': 'CloudWatcher',
          'weather_sensor_serial_number': 1533,
          'wind_condition': 'Calm',
          'wind_speed_KPH': 0.0},
 'date': datetime.datetime(2019, 11, 14, 18, 0, 55, 3000),
 'type': 'weather'}
********************************************************************************
4.5 seconds old
PEAS > last_reading camera_board
/var/panoptes/panoptes-utils/panoptes/utils/database/__init__.py:62: UserWarning: No record found for camera_board
  warn(*args)
No reading found for 'camera_board'
PEAS > last_reading control_board
/var/panoptes/panoptes-utils/panoptes/utils/database/__init__.py:62: UserWarning: No record found for control_board
  warn(*args)
No reading found for 'control_board'
PEAS > exit

In the peas-shell log these errors are repeated:

E1114 18:02:23.811                  error.py:013] InvalidDeserialization: Error: JSONDecodeError('Expecting value: line 1 column 1 (char 0)') Message: '],"temps":[nan,-127.00,-127.00,-127.00], "humidity":nan,"name":"power_board"}\r\n'
E1114 18:02:25.736                  error.py:013] InvalidDeserialization: Error: JSONDecodeError('Expecting value: line 1 column 1 (char 0)') Message: 'Accelerometer not ready, or not present.\r\n'
E1114 18:02:25.738                  error.py:013] InvalidDeserialization: Error: JSONDecodeError('Expecting value: line 1 column 1 (char 0)') Message: 'LED blink complete\r\n'
E1114 18:02:25.741                  error.py:013] InvalidDeserialization: Error: JSONDecodeError('Expecting value: line 1 column 1 (char 0)') Message: 'Accelerometer not ready, or not present.\r\n'
E1114 18:02:25.742                  error.py:013] InvalidDeserialization: Error: JSONDecodeError('Expecting value: line 1 column 1 (char 0)') Message: 'EXIT setup()\r\n'
E1114 18:02:26.745                  error.py:013] InvalidDeserialization: Error: JSONDecodeError('Expecting value: line 1 column 1 (char 0)') Message: 'Accelerometer not ready, or not present.\r\n'
E1114 18:02:26.751                  error.py:013] InvalidDeserialization: Error: JSONDecodeError('Expecting value: line 1 column 109 (char 108)') Message: '{"name":"camera_board", "count":316, "num":1, "inputs_received":0, "camera_00":1, "camera_01":0, "humidity":nan, "temp_00":nan}\r\n'

Is this a parsing error on the arduino side or POCS side? Where are the files responsible for parsing arduino serial output? Could it just be that the error line “Accelerometer not found” is what’s causing these issues?

Hi @zacharyt. Yes, it looks like the board is giving odd results in itself. Did you ever look at that output via the Arduino IDE or anything like that? Just curious if the output looked ok at that time.

Also I would try to load and test only one board at a time via peas-shell. Looks like the weather is working okay. Try just the control_board and test to see if it gives okay results and then try just the camera_board. It seems like this error is just with the camera_board so it’s odd that last_reading control_board isn’t working.

Thanks for the advice @wtgee
I tried running the peas-shell twice, once with just the camera board and once with the control board and got the same error. While I was exploring the peas-shell help arguments, I found load_control_env_board and load_camera_env_board. I tried loading each env_board independently(in different peas-shells not at the same time) after loading the corresponding control/camera board. I got an error that is almost identical to the one last night for the AAG cloud sensor. Its asking for an endpoint url, do I need to specify that in pocs_local.yaml? If so whats the syntax?