PAN015: Wildwood Institute for STEM Research and Development, WISRD (Wildwood School)

We are getting the same errors before and after the software update. Also, if it would be helpful to use something like TeamViewer to give you remote access to try and fix the issue we can set that up as well. Thanks for helping us out.

Thanks @zacharyt and @Brown_Taalib-Din. Was there an error when you did the git pull? That specific fix is about four months old so the code should definitely be there. It was merged in

What time of day are you usually working on the unit? Happy to do some kind of remote desktop, but I’m down in Australia so we would just need to coordinate the times.

Hi Wilfred!

I, along with the rest of my team, are trying to get onto the Slack platform, but I need the invitation in order to get onto it. How can I get this invitation or is there another way to get onto the platform? Please let me know. Thanks!

Also @wtgee
We are available to do remote desktop access on Wednesday at 12:55 PST(GMT -7), 3:45 on Thursday, and at 2:10 on Friday. Let us know if any of these times work for you and if not when your available. Also, we need to install a remote access program, so if there is a program that is easier for you let us know. We also have access to slack.

We would love to chat and clear up some coding issues. We will meet up at your convenience. We have SSH set up so you can poke around and help with the problem. I am on Skype with a handle of Coolharleydude.

We hope to hear from you soon. We are -7 he GET until Nov 3 when daylight saving time kicks in. My text # is 310 985 0277 and email is



We have made some progress on getting setup_pocs to run, but still need some help. I don’t have very much experience with bash or python, so let me known if I’ve done anything wrong. Also, I’ve noticed @wtgee’s new docker installation method, and we are happy to switch if that method is more time efficient than having to trouble shoot our current version.

I found that some of the issues we were having were similar to the ones experienced by PAN005 and PAN012, so we decided to reinstall the software with the instructions Aru outlines here. The install process worked, skipped previous errors we encounter with astroscrappy, and downloaded files it hadn’t in the previous install. I created pocs_local.yaml and added information to it before moving on to setting up the pocs_shell.

I successfully entered the pocs_shell, however I encountered the same error as before when running setup_pocs. After much unsuccessful trouble shooting, I accidentally unplugged the USB hub in the head box. I then plugged it back into a different port, located on the front of the intel NUC. Later, I tried running setup_pocs in the pocs_shell again, and found that this change magically allowed the unit to connect with the cameras and create them as objects. Sadly, the setup_pocs command failed later on.

By entering ls -la /dev/bus/usb/001/*, I noticed that the camera USB ports had new permissions. I then added user panoptes to dialout, and did sudo setfacl -m u:panoptes:rwx /dev/bus/usb/001/* and sudo setfacl -m u:panoptes:rwx /dev/bus/usb/002/*, to make sure USB permissions would not be an issue(I think).

I’ve split this post into two segments because the log file is incredibly long.

The pocs_shell-all.log’s new output is:

D1012 03:47:13.978     ] matplotlib data path: /var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/matplotlib/mpl-data
D1012 03:47:13.980     ] loaded rc file /var/panoptes/POCS/matplotlibrc
D1012 03:47:13.986     ] matplotlib version 3.1.1
D1012 03:47:13.986     ] interactive is False
D1012 03:47:13.987     ] platform is linux
D1012 03:47:13.987     ] loaded modules: ['builtins', 'sys', '_frozen_importlib', '_imp', '_warnings', '_thread', '_weakref', '_frozen_importlib_external', '_io', 'marshal', 'posix', 'zipimport', 'encodings', 'codecs', '_codecs', 'encodings.aliases', 'encodings.utf_8', '_signal', '__main__', 'encodings.latin_1', 'io', 'abc', '_weakrefset', 'site', 'os', 'errno', 'stat', '_stat', 'posixpath', 'genericpath', 'os.path', '_collections_abc', '_sitebuiltins', 'sysconfig', '_sysconfigdata_m_linux_x86_64-linux-gnu', '_bootlocale', '_locale', 'types', 'functools', '_functools', 'collections', 'operator', '_operator', 'keyword', 'heapq', '_heapq', 'itertools', 'reprlib', '_collections', 'weakref', '', 'importlib', 'importlib._bootstrap', 'importlib._bootstrap_external', 'warnings', 'importlib.util', '', 'importlib.machinery', 'contextlib', 'google', '', 'google.logging', 'mpl_toolkits', 'ruamel', 'encodings.cp437', 'readline', 'time', 'zmq', 'ctypes', '_ctypes', 'struct', '_struct', 'ctypes._endian', 'platform', 're', 'enum', 'sre_compile', '_sre', 'sre_parse', 'sre_constants', 'copyreg', 'subprocess', 'signal', '_posixsubprocess', 'select', 'selectors', 'math', 'threading', 'traceback', 'linecache', 'tokenize', 'token', 'zmq.backend', '', 'zmq.backend.cython', 'zmq.backend.cython.constants', 'cython_runtime', 'zmq.backend.cython.error', '_cython_0_29_5', 'zmq.backend.cython.message', 'zmq.error', 'zmq.backend.cython.context', 'zmq.backend.cython.socket', 'copy', 'random', 'hashlib', '_hashlib', '_blake2', '_sha3', 'bisect', '_bisect', '_random', 'pickle', '_compat_pickle', '_pickle', 'zmq.backend.cython.utils', 'zmq.backend.cython._poll', 'zmq.backend.cython._version', 'zmq.backend.cython._device', 'zmq.sugar', 'zmq.sugar.constants', 'zmq.utils', 'zmq.utils.constant_names', 'zmq.sugar.context', 'atexit', 'zmq.sugar.attrsettr', 'zmq.sugar.socket', 'zmq.sugar.poll', 'zmq.utils.jsonapi', 'zmq.utils.strtypes', 'json', 'json.decoder', 'json.scanner', '_json', 'json.encoder', 'zmq.sugar.frame', 'zmq.sugar.tracker', 'zmq.sugar.version', 'zmq.sugar.stopwatch', 'cmd', 'string', '_string', 'pprint', 'astropy', 'astropy.version', '__future__', 'datetime', '_datetime', 'numpy', 'numpy._globals', 'numpy.__config__', 'numpy.version', 'numpy._distributor_init', 'mkl', 'mkl._mklinit', 'mkl._py_mkl_service', 'six', 'numpy.core', '', 'numpy.core.multiarray', 'numpy.core.overrides', 'textwrap', 'numpy.core._multiarray_umath', 'numpy.compat', 'numpy.compat._inspect', 'numpy.compat.py3k', 'pathlib', 'fnmatch', 'ntpath', 'urllib', 'urllib.parse', 'numpy.core.umath', 'numpy.core.numerictypes', 'numbers', 'numpy.core._string_helpers', 'numpy.core._type_aliases', 'numpy.core._dtype', 'numpy.core.numeric', 'numpy.core._exceptions', 'numpy.core._asarray', 'numpy.core._ufunc_config', 'numpy.core.fromnumeric', 'numpy.core._methods', 'numpy.core.arrayprint', 'numpy.core.defchararray', 'numpy.core.records', 'numpy.core.memmap', 'numpy.core.function_base', 'numpy.core.machar', 'numpy.core.getlimits', 'numpy.core.shape_base', 'numpy.core.einsumfunc', 'numpy.core._add_newdocs', 'numpy.core._multiarray_tests', 'numpy.core._dtype_ctypes', 'numpy.core._internal', 'numpy._pytesttester', 'numpy.lib', '', 'numpy.lib.type_check', 'numpy.lib.ufunclike', 'numpy.lib.index_tricks', 'numpy.matrixlib', 'numpy.matrixlib.defmatrix', 'ast', '_ast', 'numpy.linalg', '', 'numpy.linalg.linalg', 'numpy.lib.twodim_base', 'numpy.linalg.lapack_lite', 'numpy.linalg._umath_linalg', 'numpy.lib.function_base', 'numpy.lib.histograms', 'numpy.lib.stride_tricks', 'numpy.lib.mixins', 'numpy.lib.nanfunctions', 'numpy.lib.shape_base', 'numpy.lib.scimath', 'numpy.lib.polynomial', 'numpy.lib.utils', 'numpy.lib.arraysetops', 'numpy.lib.npyio', 'numpy.lib.format', 'numpy.lib._datasource', 'shutil', 'zlib', 'bz2', '_compression', '_bz2', 'lzma', '_lzma', 'pwd', 'grp', 'numpy.lib._iotools', '', 'decimal', '_decimal', 'numpy.lib.arrayterator', 'numpy.lib.arraypad', 'numpy.lib._version', 'numpy.fft', '', 'numpy.fft.pocketfft', 'numpy.fft.pocketfft_internal', 'numpy.fft.helper', 'mkl_fft', 'mkl_fft._pydfti', '_cython_0_29_12', 'mkl_fft._version', 'mkl_fft._numpy_fft', 'mkl_fft._float_utils', 'numpy.polynomial', 'numpy.polynomial.polynomial', 'numpy.polynomial.polyutils', 'numpy.polynomial._polybase', 'numpy.polynomial.chebyshev', 'numpy.polynomial.legendre', 'numpy.polynomial.hermite', 'numpy.polynomial.hermite_e', 'numpy.polynomial.laguerre', 'numpy.random', 'numpy.random._pickle', 'numpy.random.mtrand', 'numpy.random.common', 'numpy.random.bounded_integers', 'numpy.random.mt19937', 'numpy.random.bit_generator', '_cython_0_29_13', 'secrets', 'base64', 'binascii', 'hmac', 'numpy.random.entropy', 'numpy.random.philox', 'numpy.random.pcg64', 'numpy.random.sfc64', 'numpy.random.generator', 'numpy.ctypeslib', '', '', '', 'numpy.testing', 'unittest', 'unittest.result', 'unittest.util', '', 'difflib', 'logging', 'unittest.suite', 'unittest.loader', 'unittest.main', 'argparse', 'gettext', 'locale', 'unittest.runner', 'unittest.signals', 'numpy.testing._private', 'numpy.testing._private.utils', 'gc', 'tempfile', 'numpy.testing._private.decorators', 'numpy.testing._private.nosetester', 'astropy.utils', 'astropy.utils.codegen', 'inspect', 'dis', 'opcode', '_opcode', 'astropy.utils.introspection', 'distutils', 'distutils.version', 'astropy.utils.decorators', 'astropy.utils.exceptions', 'astropy.utils.misc', 'unicodedata', 'astropy.config', 'astropy.config.paths', 'astropy.config.configuration', 'astropy.extern', 'astropy.extern.configobj', 'astropy.extern.configobj.configobj', 'astropy.extern.configobj.validate', 'astropy.config.affiliated', 'astropy.tests', 'astropy.tests.runner', 'glob', 'shlex', 'astropy.logger', 'astropy.utils._compiler', 'astropy.units', 'astropy.units.core', 'astropy.units.utils', 'fractions', 'astropy.units.format', 'astropy.units.format.base', 'astropy.units.format.generic', 'astropy.units.format.utils', 'astropy.units.format.cds', 'astropy.units.format.console', 'astropy.units.format.fits', 'astropy.units.format.latex', 'astropy.units.format.ogip', 'astropy.units.format.unicode_format', 'astropy.units.format.vounit', 'astropy.units.quantity', 'astropy.utils.compat', 'astropy.utils.compat.misc', 'astropy.utils.compat.numpycompat', 'astropy.utils.data_info', 'astropy.utils.metadata', 'astropy.units.quantity_helper', 'astropy.units.quantity_helper.converters', 'astropy.units.quantity_helper.helpers', 'astropy.units.quantity_helper.scipy_special', 'astropy.units.quantity_helper.erfa', '', 'astropy.constants', 'astropy.constants.constant', '', 'astropy.constants.codata2014', 'astropy.constants.iau2015', 'astropy.constants.cgs', 'astropy.constants.utils', 'astropy.units.cgs', 'astropy.units.astrophys', 'astropy.extern.ply', 'astropy.extern.ply.yacc', 'astropy.units.format.generic_parsetab', 'astropy.extern.ply.lex', 'astropy.units.format.generic_lextab', 'astropy.units.photometric', 'astropy.units.function', 'astropy.units.function.core', 'astropy.units.function.logarithmic', 'astropy.units.function.units', 'astropy.units.function.mixin', 'astropy.units.physical', 'astropy.units.imperial', 'astropy.units.equivalencies', 'astropy.units.function.magnitude_zero_points', 'astropy.units.decorators', 'astropy.coordinates', 'astropy.coordinates.errors', 'astropy.coordinates.angles', 'astropy.coordinates.angle_utilities', 'astropy.coordinates.baseframe', 'astropy.coordinates.transformations', 'astropy.coordinates.representation', 'astropy.coordinates.distances', 'astropy._erfa', 'astropy._erfa.core', 'astropy._erfa.ufunc', 'astropy.coordinates.matrix_utilities', 'astropy.coordinates.attributes', '', 'socket', '_socket', 'urllib.request', 'email', 'http', 'http.client', 'email.parser', 'email.feedparser', 'email.errors', 'email._policybase', 'email.header', 'email.quoprimime', 'email.base64mime', 'email.charset', 'email.encoders', 'quopri', 'email.utils', 'email._parseaddr', 'calendar', 'email.message', 'uu', 'email._encoded_words', 'email.iterators', 'ssl', 'ipaddress', '_ssl', 'urllib.error', 'urllib.response', '', 'shelve', 'astropy.coordinates.sites', 'astropy.coordinates.builtin_frames', 'astropy.coordinates.builtin_frames.baseradec', 'astropy.coordinates.builtin_frames.icrs', 'astropy.coordinates.builtin_frames.fk5', 'astropy.coordinates.earth_orientation', 'astropy.time', 'astropy.time.formats', 'astropy.time.utils', 'astropy.time.core', 'astropy.extern._strptime', '_strptime', 'astropy.coordinates.builtin_frames.utils', 'astropy.utils.iers', 'astropy.utils.iers.iers', 'astropy.table', 'astropy.table.column', 'astropy.utils.console', 'multiprocessing', 'multiprocessing.context', 'multiprocessing.process', 'multiprocessing.reduction', 'array', '__mp_main__', 'fcntl', 'termios', 'astropy.table.groups', 'astropy.table.index', 'astropy.table.bst', 'astropy.table.sorted_array', 'astropy.table.pprint', 'astropy.table.np_utils', 'astropy.table._np_utils', 'numpy.lib.recfunctions', '', 'astropy.table._column_mixins', 'astropy.table.table', '', '', 'astropy.table.row', '', 'astropy.table.connect', 'astropy.table.operations', 'astropy.table.soco', 'astropy.table.serialize', 'astropy.table.jsviewer', '', '', 'csv', '_csv', '', '', '', '', '', '_cython_0_29_10', 'mmap', 'queue', '', '', '', 'astropy.table.meta', '', '', 'astropy.utils.xml', 'astropy.utils.xml.writer', 'astropy.utils.xml._iterparser', '', '', '', '', '', '', 'yaml', 'yaml.error', 'yaml.tokens', '', 'yaml.nodes', 'yaml.loader', 'yaml.reader', 'yaml.scanner', 'yaml.parser', 'yaml.composer', 'yaml.constructor', 'yaml.resolver', 'yaml.dumper', 'yaml.emitter', 'yaml.serializer', 'yaml.representer', '', '', '', 'gzip', '', '', '', '', '', '', 'zipfile', '', '', '', '', '', '', '', '', '', '', '', '', 'astropy.utils.diff', '', '', '', '', '', '', '', '', '', 'astropy.utils.collections', '', '', '', '', 'astropy.utils.xml.check', 'astropy.utils.xml.validate', '', 'astropy.utils.xml.iterparser', '', '', '', '', '', 'astropy.coordinates.builtin_frames.fk4', 'astropy.coordinates.builtin_frames.galactic', 'astropy.coordinates.builtin_frames.galactocentric', 'astropy.coordinates.builtin_frames.lsr', 'astropy.coordinates.builtin_frames.supergalactic', 'astropy.coordinates.builtin_frames.altaz', 'astropy.coordinates.builtin_frames.gcrs', 'astropy.coordinates.builtin_frames.cirs', 'astropy.coordinates.builtin_frames.itrs', 'astropy.coordinates.builtin_frames.hcrs', 'astropy.coordinates.builtin_frames.ecliptic', 'astropy.coordinates.builtin_frames.skyoffset', 'astropy.coordinates.builtin_frames.icrs_fk5_transforms', 'astropy.coordinates.builtin_frames.fk4_fk5_transforms', 'astropy.coordinates.builtin_frames.galactic_transforms', 'astropy.coordinates.builtin_frames.supergalactic_transforms', 'astropy.coordinates.builtin_frames.icrs_cirs_transforms', 'astropy.coordinates.builtin_frames.cirs_observed_transforms', 'astropy.coordinates.builtin_frames.intermediate_rotation_transforms', 'astropy.coordinates.builtin_frames.ecliptic_transforms', 'astropy.coordinates.name_resolve', 'astropy.coordinates.sky_coordinate', 'astropy.wcs', 'astropy.wcs.wcs', 'astropy.wcs.docstrings', 'astropy.wcs._wcs', 'astropy.wcs.wcsapi', 'astropy.wcs.wcsapi.low_level_api', 'astropy.wcs.wcsapi.high_level_api', 'astropy.wcs.wcsapi.utils', 'astropy.wcs.wcsapi.high_level_wcs_wrapper', 'astropy.wcs.wcsapi.sliced_low_level_wcs', 'astropy.wcs.wcsapi.fitswcs', 'astropy.wcs.utils', 'astropy.coordinates.sky_coordinate_parsers', 'astropy.utils.state', 'astropy.coordinates.matching', 'astropy.coordinates.funcs', 'astropy.coordinates.calculation', 'astropy.coordinates.solar_system', 'astropy.coordinates.orbital_elements', 'pocs', 'pocs.version', 'pocs.hardware', 'pocs.core', 'pocs.base', 'pocs.utils', 'pocs.utils.config', 'pocs.utils.database', 'pymongo', 'pymongo.collection', 'bson', 'uuid', 'ctypes.util', 'bson.binary', 'bson.py3compat', 'bson.code', 'bson.codec_options', 'bson.dbref', 'bson.son', 'bson.decimal128', 'bson.errors', 'bson.int64', 'bson.max_key', 'bson.min_key', 'bson.objectid', 'bson.tz_util', 'bson.regex', 'bson.timestamp', 'bson._cbson', 'bson.raw_bson', 'pymongo.common', 'pymongo.auth', 'pymongo.errors', 'pymongo.saslprep', 'stringprep', 'pymongo.compression_support', 'pymongo.monitoring', 'pymongo.helpers', 'pymongo.driver_info', 'pymongo.encryption_options', 'pymongo.read_concern', 'pymongo.read_preferences', 'pymongo.max_staleness_selectors', 'pymongo.server_type', 'pymongo.server_selectors', 'pymongo.ssl_support', 'certifi', 'certifi.core', 'pymongo.write_concern', 'pymongo.message', 'pymongo._cmessage', 'pymongo.aggregation', 'pymongo.collation', 'pymongo.bulk', 'pymongo.client_session', 'pymongo.monotonic', 'pymongo.command_cursor', 'pymongo.change_stream', 'pymongo.cursor', 'pymongo.operations', 'pymongo.results', 'pymongo.mongo_client', 'pymongo.database', 'pymongo.son_manipulator', 'pymongo.periodic_executor', 'pymongo.uri_parser', 'pymongo.srv_resolver', 'pymongo.client_options', 'pymongo.pool', 'pymongo.thread_util', 'pymongo.ismaster', '', 'pymongo.ssl_match_hostname', 'encodings.idna', 'pymongo.cursor_manager', 'pymongo.topology', 'pymongo.topology_description', 'pymongo.server_description', 'pymongo.monitor', 'pymongo.server', 'pymongo.response', 'pymongo.settings', 'pymongo.mongo_replica_set_client', 'pocs.utils.serializers', 'bson.json_util', 'pocs.utils.logger', 'logging.config', 'logging.handlers', 'socketserver', 'pocs.observatory', 'astroplan', 'astroplan._astropy_init', 'astroplan.version', 'astroplan.utils', 'astroplan.exceptions', '', 'astropy.extern.six', 'imp', 'IPython', 'IPython.core', 'IPython.core.getipython', 'IPython.core.release', 'IPython.core.application', 'traitlets', 'traitlets.traitlets', 'traitlets.utils', 'traitlets.utils.getargspec', 'traitlets.utils.importstring', 'ipython_genutils', 'ipython_genutils._version', 'ipython_genutils.py3compat', 'ipython_genutils.encoding', 'traitlets.utils.sentinel', 'traitlets.utils.bunch', 'traitlets._version', 'traitlets.config', 'traitlets.config.application', 'decorator', 'traitlets.config.configurable', 'traitlets.config.loader', 'ipython_genutils.path', 'ipython_genutils.text', 'ipython_genutils.importstring', 'IPython.core.crashhandler', 'IPython.core.ultratb', 'pydoc', 'pkgutil', 'IPython.core.debugger', 'bdb', 'IPython.utils', 'IPython.utils.PyColorize', 'IPython.utils.coloransi', 'IPython.utils.ipstruct', 'IPython.utils.colorable', 'pygments', 'pygments.util', 'IPython.utils.py3compat', 'IPython.utils.encoding', 'IPython.core.excolors', 'IPython.testing', 'IPython.testing.skipdoctest', 'pdb', 'code', 'codeop', 'IPython.core.display_trap', 'IPython.utils.path', 'IPython.utils.process', 'IPython.utils._process_posix', 'pexpect', 'pexpect.exceptions', 'pexpect.utils', 'pexpect.expect', 'pexpect.pty_spawn', 'pty', 'tty', 'ptyprocess', 'ptyprocess.ptyprocess', 'resource', 'ptyprocess.util', 'pexpect.spawnbase', '', 'IPython.utils._process_common', 'IPython.utils.decorators', '', 'IPython.utils.terminal', 'IPython.utils.sysinfo', 'IPython.utils._sysinfo', 'IPython.core.profiledir', 'IPython.paths', 'IPython.utils.importstring', 'IPython.terminal', 'IPython.terminal.embed', 'IPython.core.compilerop', 'IPython.core.magic_arguments', 'IPython.core.error', 'IPython.utils.text', 'IPython.core.magic', 'getopt', 'IPython.core.oinspect', '', 'IPython.core.display', 'mimetypes', 'IPython.lib', '', 'getpass', 'IPython.lib.pretty', 'IPython.utils.openpy', 'IPython.utils.dir2', 'IPython.utils.wildcard', 'pygments.lexers', 'pygments.lexers._mapping', 'pygments.modeline', 'pygments.plugin', 'pygments.lexers.python', 'pygments.lexer', 'pygments.filter', 'pygments.filters', 'pygments.token', 'pygments.regexopt', 'pygments.unistring', 'pygments.formatters', 'pygments.formatters._mapping', 'pygments.formatters.html', 'pygments.formatter', 'pygments.styles', 'IPython.core.inputtransformer2', 'typing', '', '', 'IPython.core.interactiveshell', 'asyncio', 'asyncio.base_events', 'concurrent', 'concurrent.futures', 'concurrent.futures._base', 'concurrent.futures.process', 'multiprocessing.connection', '_multiprocessing', 'multiprocessing.util', 'concurrent.futures.thread', 'asyncio.compat', 'asyncio.coroutines', 'asyncio.constants', '', 'asyncio.base_futures', 'asyncio.log', 'asyncio.futures', 'asyncio.base_tasks', '_asyncio', 'asyncio.tasks', 'asyncio.locks', 'asyncio.protocols', 'asyncio.queues', 'asyncio.streams', 'asyncio.subprocess', 'asyncio.transports', 'asyncio.unix_events', 'asyncio.base_subprocess', 'asyncio.selector_events', 'asyncio.sslproto', 'runpy', 'pickleshare', 'IPython.core.prefilter', 'IPython.core.autocall', 'IPython.core.macro', 'IPython.core.splitinput', 'IPython.core.alias', 'IPython.core.builtin_trap', '', 'backcall', 'backcall.backcall', 'IPython.core.displayhook', 'IPython.core.displaypub', 'IPython.core.extensions', 'IPython.core.formatters', 'IPython.utils.sentinel', 'IPython.core.history', 'sqlite3', 'sqlite3.dbapi2', '_sqlite3', 'IPython.core.logger', 'IPython.core.payload', 'IPython.core.usage', 'IPython.display', 'IPython.lib.display', 'html', 'html.entities', '', 'IPython.utils.capture', 'IPython.utils.strdispatch', 'IPython.core.hooks', 'IPython.utils.syspathcontext', 'IPython.utils.tempdir', 'IPython.utils.contexts', 'IPython.core.async_helpers', 'IPython.terminal.interactiveshell', 'prompt_toolkit', 'prompt_toolkit.application', 'prompt_toolkit.application.application', 'prompt_toolkit.buffer', 'six.moves', 'prompt_toolkit.application.current', 'prompt_toolkit.eventloop', 'prompt_toolkit.eventloop.async_generator', 'six.moves.queue', 'prompt_toolkit.eventloop.coroutine', 'prompt_toolkit.eventloop.defaults', 'prompt_toolkit.utils', 'wcwidth', 'wcwidth.wcwidth', 'wcwidth.table_wide', 'wcwidth.table_zero', 'prompt_toolkit.cache', 'prompt_toolkit.eventloop.base', 'prompt_toolkit.log', 'prompt_toolkit.eventloop.future', 'prompt_toolkit.eventloop.context', 'prompt_toolkit.eventloop.event', 'prompt_toolkit.application.run_in_terminal', 'prompt_toolkit.auto_suggest', 'prompt_toolkit.filters', '', 'prompt_toolkit.enums', 'prompt_toolkit.filters.base', 'prompt_toolkit.filters.cli', 'prompt_toolkit.filters.utils', 'prompt_toolkit.clipboard', 'prompt_toolkit.clipboard.base', 'prompt_toolkit.selection', 'prompt_toolkit.clipboard.in_memory', 'prompt_toolkit.completion', 'prompt_toolkit.completion.base', 'prompt_toolkit.completion.filesystem', 'prompt_toolkit.completion.fuzzy_completer', 'prompt_toolkit.document', 'prompt_toolkit.completion.word_completer', 'prompt_toolkit.history', '', 'prompt_toolkit.key_binding', 'prompt_toolkit.key_binding.key_bindings', 'prompt_toolkit.keys', 'prompt_toolkit.key_binding.vi_state', 'prompt_toolkit.validation', 'prompt_toolkit.input', 'prompt_toolkit.input.base', 'prompt_toolkit.input.defaults', 'prompt_toolkit.input.typeahead', 'prompt_toolkit.key_binding.bindings', 'prompt_toolkit.key_binding.bindings.page_navigation', 'prompt_toolkit.key_binding.bindings.scroll', 'prompt_toolkit.key_binding.defaults', 'prompt_toolkit.key_binding.bindings.basic', 'prompt_toolkit.key_binding.key_processor', 'prompt_toolkit.key_binding.bindings.named_commands', 'prompt_toolkit.key_binding.bindings.completion', 'prompt_toolkit.key_binding.bindings.cpr', 'prompt_toolkit.key_binding.bindings.emacs', 'prompt_toolkit.key_binding.bindings.mouse', 'prompt_toolkit.layout', 'prompt_toolkit.layout.containers', 'prompt_toolkit.formatted_text', 'prompt_toolkit.formatted_text.ansi', 'prompt_toolkit.output', 'prompt_toolkit.output.base', 'prompt_toolkit.layout.screen', 'prompt_toolkit.output.color_depth', 'prompt_toolkit.output.defaults', 'prompt_toolkit.output.vt100', 'prompt_toolkit.styles', 'prompt_toolkit.styles.base', 'prompt_toolkit.styles.defaults', 'prompt_toolkit.styles.named_colors', '', 'prompt_toolkit.styles.pygments', 'prompt_toolkit.styles.style_transformation', 'colorsys', 'prompt_toolkit.formatted_text.base', 'prompt_toolkit.formatted_text.html', 'xml', 'xml.dom', 'xml.dom.domreg', 'xml.dom.minidom', 'xml.dom.minicompat', 'xml.dom.xmlbuilder', 'xml.dom.NodeFilter', 'prompt_toolkit.formatted_text.pygments', 'prompt_toolkit.formatted_text.utils', 'prompt_toolkit.mouse_events', 'prompt_toolkit.layout.controls', 'prompt_toolkit.lexers', 'prompt_toolkit.lexers.base', 'prompt_toolkit.lexers.pygments', 'prompt_toolkit.layout.processors', 'prompt_toolkit.layout.utils', 'prompt_toolkit.layout.dimension', 'prompt_toolkit.layout.margins', 'prompt_toolkit.layout.layout', 'prompt_toolkit.layout.menus', 'prompt_toolkit.renderer', 'prompt_toolkit.layout.mouse_handlers', '', 'prompt_toolkit.input.vt100_parser', 'prompt_toolkit.input.ansi_escape_sequences', 'prompt_toolkit.key_binding.digraphs', 'prompt_toolkit.key_binding.emacs_state', 'prompt_toolkit.layout.dummy', 'prompt_toolkit.application.dummy', 'prompt_toolkit.shortcuts', 'prompt_toolkit.shortcuts.dialogs', 'prompt_toolkit.key_binding.bindings.focus', 'prompt_toolkit.widgets', 'prompt_toolkit.widgets.base', 'prompt_toolkit.widgets.toolbars', 'prompt_toolkit.widgets.dialogs', 'prompt_toolkit.widgets.menus', 'prompt_toolkit.shortcuts.progress_bar', 'prompt_toolkit.shortcuts.progress_bar.base', 'prompt_toolkit.shortcuts.progress_bar.formatters', 'prompt_toolkit.shortcuts.prompt', 'prompt_toolkit.key_binding.bindings.auto_suggest', 'prompt_toolkit.key_binding.bindings.open_in_editor', 'prompt_toolkit.shortcuts.utils', 'prompt_toolkit.patch_stdout', '', 'IPython.terminal.debugger', 'IPython.core.completer', 'IPython.core.latex_symbols', 'IPython.utils.generics', 'jedi', 'jedi.api', 'parso', 'parso.parser', 'parso.tree', 'parso._compatibility', 'parso.utils', 'parso.pgen2', 'parso.pgen2.generator', 'parso.pgen2.grammar_parser', 'parso.python', 'parso.python.tokenize', 'parso.python.token', 'parso.grammar', 'parso.python.diff', 'parso.python.parser', 'parso.python.tree', 'parso.python.prefix', 'parso.cache', 'parso.python.errors', 'parso.normalizer', 'parso.python.pep8', 'parso.file_io', 'jedi._compatibility', 'jedi.file_io', 'jedi.parser_utils', 'jedi.debug', 'jedi.settings', 'jedi.cache', 'jedi.api.classes', 'jedi.evaluate', 'jedi.evaluate.utils', 'jedi.evaluate.imports', 'jedi.evaluate.sys_path', 'jedi.evaluate.cache', 'jedi.evaluate.base_context', 'jedi.common', 'jedi.common.context', 'jedi.evaluate.helpers', 'jedi.common.utils', 'jedi.evaluate.compiled', 'jedi.evaluate.compiled.context', 'jedi.evaluate.filters', 'jedi.evaluate.flow_analysis', 'jedi.evaluate.recursion', 'jedi.evaluate.names', 'jedi.evaluate.lazy_context', 'jedi.evaluate.compiled.access', 'jedi.evaluate.compiled.getattr_static', 'jedi.evaluate.signature', 'jedi.evaluate.analysis', 'jedi.evaluate.gradual', 'jedi.evaluate.gradual.typeshed', 'jedi.evaluate.gradual.stub_context', 'jedi.evaluate.context', 'jedi.evaluate.context.module', 'jedi.evaluate.context.klass', 'jedi.evaluate.arguments', 'jedi.evaluate.context.iterable', 'jedi.evaluate.param', 'jedi.evaluate.docstrings', 'jedi.evaluate.context.function', 'jedi.evaluate.parser_cache', 'jedi.plugins', 'jedi.evaluate.context.instance', 'jedi.evaluate.gradual.typing', 'jedi.evaluate.syntax_tree', 'jedi.evaluate.finder', 'jedi.evaluate.gradual.conversion', 'jedi.evaluate.gradual.annotation', 'jedi.evaluate.context.decorator', 'jedi.api.keywords', 'pydoc_data', 'pydoc_data.topics', 'jedi.api.interpreter', 'jedi.evaluate.compiled.mixed', 'jedi.api.helpers', 'jedi.api.completion', 'jedi.api.file_name', 'jedi.api.environment', 'filecmp', 'jedi.evaluate.compiled.subprocess', 'jedi.evaluate.compiled.subprocess.functions', 'jedi.api.exceptions', 'jedi.api.project', 'jedi.evaluate.usages', 'jedi.evaluate.gradual.utils', 'jedi.plugins.registry', 'jedi.plugins.stdlib', 'jedi.plugins.flask', 'IPython.terminal.ptutils', 'IPython.terminal.shortcuts', 'IPython.terminal.magics', 'IPython.lib.clipboard', 'IPython.terminal.pt_inputhooks', 'IPython.terminal.prompts', 'IPython.terminal.ipapp', 'IPython.core.magics', '', 'IPython.core.magics.basic', 'IPython.core.magics.code', 'IPython.core.magics.config', 'IPython.core.magics.display', 'IPython.core.magics.execution', 'timeit', 'cProfile', '_lsprof', 'profile', 'optparse', 'pstats', 'IPython.utils.module_paths', 'IPython.utils.timing', 'IPython.core.magics.extension', 'IPython.core.magics.history', 'IPython.core.magics.logging', 'IPython.core.magics.namespace', 'IPython.core.magics.osm', 'IPython.core.magics.packaging', 'IPython.core.magics.pylab', 'IPython.core.pylabtools', 'IPython.core.magics.script', 'IPython.lib.backgroundjobs', 'IPython.core.shellapp', 'IPython.extensions', 'IPython.extensions.storemagic', 'IPython.utils.frame', 'ipykernel', 'ipykernel._version', 'ipykernel.connect', 'jupyter_client', 'jupyter_client._version', 'jupyter_client.connect', 'jupyter_client.localinterfaces', 'jupyter_core', 'jupyter_core.version', 'jupyter_core.paths', 'jupyter_client.launcher', 'traitlets.log', 'jupyter_client.client', 'jupyter_client.channels', 'jupyter_client.channelsabc', 'jupyter_client.clientabc', 'jupyter_client.manager', 'jupyter_client.kernelspec', 'jupyter_client.managerabc', 'jupyter_client.blocking', 'jupyter_client.blocking.client', 'jupyter_client.blocking.channels', 'jupyter_client.multikernelmanager', 'ipykernel.iostream', 'zmq.eventloop', 'zmq.eventloop.ioloop', 'tornado', 'tornado.log', 'tornado.escape', 'tornado.util', 'tornado.speedups', 'curses', '_curses', 'tornado.ioloop', 'tornado.concurrent', 'tornado.platform', 'tornado.platform.asyncio', 'tornado.gen', 'zmq.eventloop.zmqstream', 'jupyter_client.session', 'jupyter_client.jsonutil', 'dateutil', 'dateutil._version', 'dateutil.parser', 'dateutil.parser._parser', 'dateutil.relativedelta', 'dateutil._common', '', '', '', '', 'dateutil.parser.isoparser', 'jupyter_client.adapter', 'pytz', 'pytz.exceptions', 'pytz.lazy', 'pytz.tzinfo', 'pytz.tzfile', 'astroplan.moon', '', 'astroplan.constraints', 'astroplan.scheduling', 'astroplan.periodic', 'dbm', 'dbm.dumb', 'astropy.units.cds', '', 'pocs.utils.error', '', 'pocs.utils.images', 'matplotlib', 'matplotlib.cbook', 'matplotlib.cbook.deprecation', 'matplotlib.rcsetup', 'matplotlib.fontconfig_pattern', 'pyparsing', 'matplotlib.colors', 'matplotlib._color_data', 'cycler', 'matplotlib._version', 'matplotlib.ft2font', 'kiwisolver']

Log file continued:

D1012 03:47:14.037     ] CACHEDIR=/home/panoptes/.cache/matplotlib
D1012 03:47:14.042 ] Using fontManager instance from /home/panoptes/.cache/matplotlib/fontlist-v310.json
D1012 03:47:14.496     ] CONFIGDIR=/home/panoptes/.config/matplotlib
D1012 03:47:14.528       ] Loaded backend agg version unknown.
D1012 03:47:29.787     ] simulator_names = power
D1012 03:47:29.787     ] Camera config: {'auto_detect': True, 'primary': '14d3bd', 'devices': [{'model': 'canon_gphoto2'}, {'model': 'canon_gphoto2'}]}
D1012 03:47:29.787     ] Auto-detecting ports for cameras
D1012 03:47:29.888     ] Detected Ports: ['usb:001,062', 'usb:001,063']
D1012 03:47:29.889     ] Creating camera: canon_gphoto2
D1012 03:47:29.893     ] Camera module: <module '' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/camera/'>
D1012 03:47:30.007       ] Camera created: Cam00 (XXXXXX) on usb:001,063
D1012 03:47:30.007       ] GPhoto2 camera Cam00 created on usb:001,063
D1012 03:47:30.007] Connecting GPhoto2 camera
D1012 03:47:30.007] Connecting to camera
D1012 03:47:30.007       ] gphoto2 command: ['/usr/bin/gphoto2', '--port', 'usb:001,063', '--get-config', 'serialnumber']
D1012 03:47:30.020       ] Getting output from proc 26931
D1012 03:47:30.481       ] gphoto2 command: ['/usr/bin/gphoto2', '--port', 'usb:001,063', '--set-config-index', '/main/actions/viewfinder=1', '--set-config-index', '/main/capturesettings/autoexposuremode=3', '--set-config-index', '/main/capturesettings/continuousaf=0', '--set-config-index', '/main/capturesettings/drivemode=0', '--set-config-index', '/main/capturesettings/focusmode=0', '--set-config-index', '/main/capturesettings/shutterspeed=0', '--set-config-index', '/main/imgsettings/imageformat=9', '--set-config-index', '/main/imgsettings/imageformatcf=9', '--set-config-index', '/main/imgsettings/imageformatsd=9', '--set-config-index', '/main/imgsettings/iso=1', '--set-config-index', '/main/settings/autopoweroff=0', '--set-config-index', '/main/settings/capturetarget=0', '--set-config-index', '/main/settings/datetime=now', '--set-config-index', '/main/settings/datetimeutc=now', '--set-config-index', '/main/settings/reviewtime=0', '--set-config-value', '/main/settings/artist=Project PANOPTES', '--set-config-value', '/main/settings/copyright=owner_name Project PANOPTES', '--set-config-value', '/main/settings/ownername=Project PANOPTES']
D1012 03:47:30.494       ] Getting output from proc 26938
D1012 03:47:31.153] Cam00 connected
D1012 03:47:31.153     ] Camera created: Cam00 5205b8
D1012 03:47:31.153     ] Creating camera: canon_gphoto2
D1012 03:47:31.154     ] Camera module: <module '' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/camera/'>
D1012 03:47:31.177       ] Camera created: Cam01 (XXXXXX) on usb:001,062
D1012 03:47:31.178       ] GPhoto2 camera Cam01 created on usb:001,062
D1012 03:47:31.178] Connecting GPhoto2 camera
D1012 03:47:31.178] Connecting to camera
D1012 03:47:31.178       ] gphoto2 command: ['/usr/bin/gphoto2', '--port', 'usb:001,062', '--get-config', 'serialnumber']
D1012 03:47:31.191       ] Getting output from proc 26944
D1012 03:47:31.682       ] gphoto2 command: ['/usr/bin/gphoto2', '--port', 'usb:001,062', '--set-config-index', '/main/actions/viewfinder=1', '--set-config-index', '/main/capturesettings/autoexposuremode=3', '--set-config-index', '/main/capturesettings/continuousaf=0', '--set-config-index', '/main/capturesettings/drivemode=0', '--set-config-index', '/main/capturesettings/focusmode=0', '--set-config-index', '/main/capturesettings/shutterspeed=0', '--set-config-index', '/main/imgsettings/imageformat=9', '--set-config-index', '/main/imgsettings/imageformatcf=9', '--set-config-index', '/main/imgsettings/imageformatsd=9', '--set-config-index', '/main/imgsettings/iso=1', '--set-config-index', '/main/settings/autopoweroff=0', '--set-config-index', '/main/settings/capturetarget=0', '--set-config-index', '/main/settings/datetime=now', '--set-config-index', '/main/settings/datetimeutc=now', '--set-config-index', '/main/settings/reviewtime=0', '--set-config-value', '/main/settings/artist=Project PANOPTES', '--set-config-value', '/main/settings/copyright=owner_name Project PANOPTES', '--set-config-value', '/main/settings/ownername=Project PANOPTES']
D1012 03:47:31.697       ] Getting output from proc 26950
D1012 03:47:32.378] Cam01 connected
D1012 03:47:32.379     ] Camera created: Cam01 5f1e70
D1012 03:47:32.379     ] Primary camera: Cam00 [Primary] (5205b8) on usb:001,063
D1012 03:47:32.388     ] 2 cameras created
I1012 03:47:32.409  ] Initializing observatory
I1012 03:47:32.410  ] 	Setting up location
D1012 03:47:32.410  ] Setting up site details of observatory
D1012 03:47:32.413  ] Location: {'name': 'Wildwood School', 'latitude': <Quantity 34.033719 deg>, 'longitude': <Quantity -118.451393 deg>, 'elevation': <Quantity 48. m>, 'timezone': 'US/Pacific', 'pressure': <Quantity 0.68 bar>, 'horizon': <Quantity 30. deg>, 'flat_horizon': <Quantity -6. deg>, 'focus_horizon': <Quantity -12. deg>, 'observe_horizon': <Quantity -18. deg>}
I1012 03:47:32.460  ] Removing mount
I1012 03:47:32.461  ] Adding the cameras to the observatory: OrderedDict([('Cam00', < object at 0x7f3afc88d8d0>), ('Cam01', < object at 0x7f3afecdea58>)])
D1012 03:47:32.461  ] Adding Cam00: Cam00 [Primary] (5205b8) on usb:001,063
D1012 03:47:32.461  ] Adding Cam01: Cam01 (5f1e70) on usb:001,062
I1012 03:47:32.461  ] Removing dome.
I1012 03:47:32.461  ] Removing scheduler.
I1012 03:47:32.461  ] 	 Observatory initialized
I1012 03:47:32.482         ] Initializing PANOPTES unit - WISRD - Wildwood School
I1012 03:47:32.490    ] Creating forwarder sockets for 6500 -> 6501
D1012 03:47:32.491    ] Creating subscriber. Port: 6500 	Topic: 
D1012 03:47:32.492    ] Creating publisher. Binding to port 6501 
I1012 03:47:32.493    ] run_forwarder
I1012 03:47:32.494    ] run_forwarder calling zmq.device
I1012 03:47:32.496    ] Creating forwarder sockets for 6510 -> 6511
D1012 03:47:32.497    ] Creating subscriber. Port: 6510 	Topic: 
D1012 03:47:32.498    ] Creating publisher. Binding to port 6511 
I1012 03:47:32.500    ] run_forwarder
I1012 03:47:32.500    ] run_forwarder calling zmq.device
D1012 03:47:32.501    ] Creating publisher. Binding to port 6510 
D1012 03:47:32.502         ] Starting command message loop
D1012 03:47:32.509         ] Command message subscriber set up on port 6500
I1012 03:47:32.510         ] Making a POCS state machine from simple_state_table
I1012 03:47:32.510      ] Loading state table: simple_state_table
D1012 03:47:32.511    ] Creating subscriber. Port: 6501 	Topic: 
D1012 03:47:32.543      ] Loading transition: {'source': ['ready', 'scheduling', 'slewing', 'pointing', 'tracking', 'observing', 'analyzing'], 'dest': 'parking', 'trigger': 'park'}
D1012 03:47:32.543      ] Returning transition: {'source': ['ready', 'scheduling', 'slewing', 'pointing', 'tracking', 'observing', 'analyzing'], 'dest': 'parking', 'trigger': 'park', 'conditions': ['check_safety']}
D1012 03:47:32.543      ] Loading transition: {'source': 'parking', 'dest': 'parked', 'trigger': 'set_park'}
D1012 03:47:32.543      ] Returning transition: {'source': 'parking', 'dest': 'parked', 'trigger': 'set_park', 'conditions': ['check_safety']}
D1012 03:47:32.543      ] Loading transition: {'source': 'parked', 'dest': 'housekeeping', 'trigger': 'clean_up'}
D1012 03:47:32.543      ] Returning transition: {'source': 'parked', 'dest': 'housekeeping', 'trigger': 'clean_up', 'conditions': ['check_safety']}
D1012 03:47:32.544      ] Loading transition: {'source': 'housekeeping', 'dest': 'sleeping', 'trigger': 'goto_sleep'}
D1012 03:47:32.544      ] Returning transition: {'source': 'housekeeping', 'dest': 'sleeping', 'trigger': 'goto_sleep', 'conditions': ['check_safety']}
D1012 03:47:32.544      ] Loading transition: {'source': 'parked', 'dest': 'ready', 'trigger': 'get_ready', 'conditions': 'mount_is_initialized'}
D1012 03:47:32.544      ] Returning transition: {'source': 'parked', 'dest': 'ready', 'trigger': 'get_ready', 'conditions': ['check_safety', 'mount_is_initialized']}
D1012 03:47:32.544      ] Loading transition: {'source': 'sleeping', 'dest': 'ready', 'trigger': 'get_ready', 'conditions': 'mount_is_initialized'}
D1012 03:47:32.544      ] Returning transition: {'source': 'sleeping', 'dest': 'ready', 'trigger': 'get_ready', 'conditions': ['check_safety', 'mount_is_initialized']}
D1012 03:47:32.544      ] Loading transition: {'source': 'ready', 'dest': 'scheduling', 'trigger': 'schedule'}
D1012 03:47:32.544      ] Returning transition: {'source': 'ready', 'dest': 'scheduling', 'trigger': 'schedule', 'conditions': ['check_safety']}
D1012 03:47:32.544      ] Loading transition: {'source': 'analyzing', 'dest': 'scheduling', 'trigger': 'schedule'}
D1012 03:47:32.545      ] Returning transition: {'source': 'analyzing', 'dest': 'scheduling', 'trigger': 'schedule', 'conditions': ['check_safety']}
D1012 03:47:32.545      ] Loading transition: {'source': 'scheduling', 'dest': 'slewing', 'trigger': 'start_slewing'}
D1012 03:47:32.545      ] Returning transition: {'source': 'scheduling', 'dest': 'slewing', 'trigger': 'start_slewing', 'conditions': ['check_safety']}
D1012 03:47:32.545      ] Loading transition: {'source': 'scheduling', 'dest': 'tracking', 'trigger': 'adjust_tracking'}
D1012 03:47:32.545      ] Returning transition: {'source': 'scheduling', 'dest': 'tracking', 'trigger': 'adjust_tracking', 'conditions': ['check_safety']}
D1012 03:47:32.545      ] Loading transition: {'source': 'slewing', 'dest': 'pointing', 'trigger': 'adjust_pointing', 'conditions': 'mount_is_tracking'}
D1012 03:47:32.545      ] Returning transition: {'source': 'slewing', 'dest': 'pointing', 'trigger': 'adjust_pointing', 'conditions': ['check_safety', 'mount_is_tracking']}
D1012 03:47:32.545      ] Loading transition: {'source': 'pointing', 'dest': 'slewing', 'trigger': 'start_slewing', 'conditions': 'mount_is_tracking'}
D1012 03:47:32.545      ] Returning transition: {'source': 'pointing', 'dest': 'slewing', 'trigger': 'start_slewing', 'conditions': ['check_safety', 'mount_is_tracking']}
D1012 03:47:32.546      ] Loading transition: {'source': 'pointing', 'dest': 'tracking', 'trigger': 'track', 'conditions': 'mount_is_tracking'}
D1012 03:47:32.546      ] Returning transition: {'source': 'pointing', 'dest': 'tracking', 'trigger': 'track', 'conditions': ['check_safety', 'mount_is_tracking']}
D1012 03:47:32.546      ] Loading transition: {'source': 'tracking', 'dest': 'observing', 'trigger': 'observe', 'conditions': 'mount_is_tracking'}
D1012 03:47:32.546      ] Returning transition: {'source': 'tracking', 'dest': 'observing', 'trigger': 'observe', 'conditions': ['check_safety', 'mount_is_tracking']}
D1012 03:47:32.546      ] Loading transition: {'source': 'observing', 'dest': 'analyzing', 'trigger': 'analyze'}
D1012 03:47:32.546      ] Returning transition: {'source': 'observing', 'dest': 'analyzing', 'trigger': 'analyze', 'conditions': ['check_safety']}
D1012 03:47:32.546      ] Loading transition: {'source': 'observing', 'dest': 'observing', 'trigger': 'observe', 'conditions': 'mount_is_tracking'}
D1012 03:47:32.546      ] Returning transition: {'source': 'observing', 'dest': 'observing', 'trigger': 'observe', 'conditions': ['check_safety', 'mount_is_tracking']}
D1012 03:47:32.546      ] Loading transition: {'source': 'analyzing', 'dest': 'tracking', 'trigger': 'adjust_tracking', 'conditions': 'mount_is_tracking'}
D1012 03:47:32.547      ] Returning transition: {'source': 'analyzing', 'dest': 'tracking', 'trigger': 'adjust_tracking', 'conditions': ['check_safety', 'mount_is_tracking']}
D1012 03:47:32.547      ] Loading state: parked
D1012 03:47:32.549      ] Checking <module 'pocs.state.states.default.parked' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.550      ] Added `on_enter` method from <module 'pocs.state.states.default.parked' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b3730>
D1012 03:47:32.550      ] Created state
D1012 03:47:32.550      ] Loading state: sleeping
D1012 03:47:32.551      ] Checking <module 'pocs.state.states.default.sleeping' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.551      ] Added `on_enter` method from <module 'pocs.state.states.default.sleeping' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b36a8>
D1012 03:47:32.551      ] Created state
D1012 03:47:32.551      ] Loading state: housekeeping
D1012 03:47:32.552      ] Checking <module 'pocs.state.states.default.housekeeping' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.553      ] Added `on_enter` method from <module 'pocs.state.states.default.housekeeping' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b37b8>
D1012 03:47:32.553      ] Created state
D1012 03:47:32.553      ] Loading state: ready
D1012 03:47:32.554      ] Checking <module 'pocs.state.states.default.ready' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.554      ] Added `on_enter` method from <module 'pocs.state.states.default.ready' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b3840>
D1012 03:47:32.554      ] Created state
D1012 03:47:32.554      ] Loading state: scheduling
D1012 03:47:32.556      ] Checking <module 'pocs.state.states.default.scheduling' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.556      ] Added `on_enter` method from <module 'pocs.state.states.default.scheduling' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b38c8>
D1012 03:47:32.556      ] Created state
D1012 03:47:32.556      ] Loading state: slewing
D1012 03:47:32.557      ] Checking <module 'pocs.state.states.default.slewing' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.558      ] Added `on_enter` method from <module 'pocs.state.states.default.slewing' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b3950>
D1012 03:47:32.558      ] Created state
D1012 03:47:32.558      ] Loading state: pointing
D1012 03:47:32.561      ] Checking <module 'pocs.state.states.default.pointing' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.561      ] Added `on_enter` method from <module 'pocs.state.states.default.pointing' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b39d8>
D1012 03:47:32.561      ] Created state
D1012 03:47:32.561      ] Loading state: tracking
D1012 03:47:32.562      ] Checking <module 'pocs.state.states.default.tracking' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.562      ] Added `on_enter` method from <module 'pocs.state.states.default.tracking' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b3ae8>
D1012 03:47:32.563      ] Created state
D1012 03:47:32.563      ] Loading state: observing
D1012 03:47:32.564      ] Checking <module 'pocs.state.states.default.observing' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.564      ] Added `on_enter` method from <module 'pocs.state.states.default.observing' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b3bf8>
D1012 03:47:32.564      ] Created state
D1012 03:47:32.565      ] Loading state: analyzing
D1012 03:47:32.566      ] Checking <module 'pocs.state.states.default.analyzing' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.566      ] Added `on_enter` method from <module 'pocs.state.states.default.analyzing' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b3c80>
D1012 03:47:32.566      ] Created state
D1012 03:47:32.566      ] Loading state: parking
D1012 03:47:32.567      ] Checking <module 'pocs.state.states.default.parking' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'>
D1012 03:47:32.567      ] Added `on_enter` method from <module 'pocs.state.states.default.parking' from '/var/panoptes/miniconda/envs/panoptes-env/lib/python3.6/site-packages/pocs-0.6.2-py3.6.egg/pocs/state/states/default/'> <function on_enter at 0x7f3ae14b3d08>
D1012 03:47:32.567      ] Created state
D1012 03:47:32.589      ] State machine created
I1012 03:47:32.590  ] Scheduler not present, cannot observe.
W1012 03:47:32.592  ] Can't get observatory status: 'NoneType' object has no attribute 'is_initialized'
I1012 03:47:32.594    ] PANCHAT Hi there!
I1012 03:47:32.595         ] ********************************************************************************
I1012 03:47:32.596    ] PANCHAT Initializing the system! Woohoo!
D1012 03:47:32.596         ] Initializing observatory
D1012 03:47:32.596  ] Initializing mount
I1012 03:47:32.597    ] PANCHAT Oh wait. There was a problem initializing: 'NoneType' object has no attribute 'initialize'
I1012 03:47:32.599    ] PANCHAT Since we didn't initialize, I'm going to exit.
I1012 03:47:32.600    ] PANCHAT I'm powering down
I1012 03:47:32.600         ] Shutting down POCS State Machine: , please be patient and allow for exit.

While the log told me to be patient, I eventually keyboard interrupted the pocs_shell after 15 minutes of waiting. Note: I was working remotely over ssh, and had to connect to the unit in two separate windows to use grc tail -F $PANLOG/pocs_shell-all.log as the hanging stopped the logging for some reason. After investigating some of the python scripts, I am unsure of how to continue. Any ideas?

Update: Zach and the PAN015 team were able to download the latest version of POCS but are still having some issues getting it working (more details soon)

After talking with @tmcook about the previous errors, we have decided to switch to the docker version of POCS, and are using new docker documentation provided by @tmcook (Thanks!).
I successfully cloned POCS, PAWS, and panoptes-utils repos with $ bash -c "$(wget -qO-", and copy pasted our old pocs_local.yaml file into the $PANDIR/conf_files directory after checking to make sure that config files have identical formatting.

Then this happened when running the next step:

panoptes@pan015:/var/panoptes/POCS$ bin/pocs up -d
WARNING: The PANDIR variable is not set. Defaulting to a blank string.
ERROR: Couldn't find env file: /.env

I fixed this error based off this topic thread by duplicating the env_file and renaming it to .env:

cd $PANDIR #Defined in bash but not in docker
cp env_file .env

@wtgee, should I have to create the .env file or has the download already gone wrong? (@tmcook did you have to this on your unit?)

Running bin/pocs up -d from the $POCS directory for a second time gave this output:

aag-weather-reader is up-to-date
config-server is up-to-date
aag-weather-server is up-to-date
messaging-hub is up-to-date
Starting paws       ... done
Starting peas-shell ... done
Starting pocs-shell ... done

Which I assume means everything is working. I say assume because on the next step I was barraged with errors trying to enter the PEAS shell:

panoptes@pan015:/var/panoptes/POCS$ bin/peas-shell
/opt/conda/lib/python3.7/site-packages/astropy/config/ ConfigurationMissingWarning: Configuration defaults will be used due to PermissionError:13 on None
Traceback (most recent call last):
  File "/var/panoptes/POCS/scripts/", line 14, in <module>
    from peas.sensors import ArduinoSerialMonitor
  File "/var/panoptes/POCS/peas/", line 12, in <module>
    from panoptes.utils.messaging import PanMessaging
  File "/opt/conda/lib/python3.7/site-packages/panoptes/utils/", line 10, in <module>
    class PanMessaging(object):
  File "/opt/conda/lib/python3.7/site-packages/panoptes/utils/", line 72, in PanMessaging
    logger = get_root_logger()
  File "/opt/conda/lib/python3.7/site-packages/panoptes/utils/", line 251, in get_root_logger
    os.makedirs(per_run_dir, exist_ok=True)
  File "/opt/conda/lib/python3.7/", line 221, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/panoptes/logs/per-run/'

These failures were so bad that a file was not created. Looking in /opt there is no conda repository. I wasn’t able to fix these issues, and have put what I tried doing to fix them below. My main question is: How can something that doesn’t exist call an error and cause a permission error? Thanks in advance for the much needed support.

Also, it appears that you can’t reply to the forum if you respond three times in a row! Thanks to @tmcook who I had un-block the thread for me.

Troubleshooting attempts:
The missing conda direcotry made me believe I was missing the conda library, so I downloaded anaconda3 to the /opt directory following this guide, and found that by renaming the anaconda3 folder to conda the correct path would exist. Unfortunately, this had no effect on the error. I think I might be downloading the wrong library/python package but can’t find a library simply named “conda” that has this path.

The next thing I tried, after creating a backup of $PANDIR, was temporarily giving everything perms to see if that was the issue. In terminal I entered:

$ cd $PANDIR
$ sudo chmod -R 777 ./
$ cd $POCS
$ bin/peas-shell
/opt/conda/lib/python3.7/site-packages/astropy/config/ ConfigurationMissingWarning: Configuration defaults will be used due to PermissionError:13 on None
Welcome to PEAS Shell! Type ? for help

I know that this likely didn’t fix anything, but it hid the errors and was enough to create a log file. In the PEAS Shell I entered load_control_board and load_camera_board. Unfortunately, I don’t have the exact saved output in the PEAS Shell log file, but I remember it said it was unable to open the Arduino ports, and it tried to access the camera board arduino with the control box arduino’s port, and visa versa. This didn’t cause the PEAS Shell to exit, but it made it useless as I was unable to get data from either arduino due to the expected parsing errors the port switching caused.

Hmm, I thought the install script was properly doing that but it might not be. I’ll double-check. But ultimately yes, you do need a $PANDIR/.env file that contains some of the variables.

I usually don’t use the -d option and it keeps everything in the foreground. That will show some error messages from startup in a clean fashion. You would then need to open another terminal to do anything. It’s mostly personal preference for how you would want set up.

Docker works like a virtual machine, so all of the required software and files are loaded inside the docker image. When you run a copy of the docker image it is referred to as a docker container. In this way you can have many instances (i.e. docker containers) of the same virtual machine-like setup (i.e. the docker image).

So here what you are seeing are errors that are happening inside the virtual container. This container contains its own small version of linux and filesystem, so the /opt errors you are seeing are from inside the container. If using the docker setup you shouldn’t need to install any additional software such as anaconda.

Does the main computer have a panoptes user? The best would be to grant the panoptes user as the owner and group of $PANDIR, i.e.:

$ sudo chown -R panoptes:panoptes $PANDIR

There might be some misplaced owner/permissions from a previous setup so the above command should fix all of that.

The errors with connecting to the arduino might just be port issues (as listed in the config file). I think @tmcook had made some notes in the document about trying to determine the usb ports. It mostly has to do with the order in which they were plugged in but sometimes takes a bit of troubleshooting.

See if any of the above helps with the permission issues and then maybe try to unplug the usb ports and only plug in one at a time to test and verify things. i.e. just plug in the weather station, figure out what port it is, try to load it via peas_shell, etc.

Thanks for the help @wtgee . I made the panoptes user the owner and group of $PANDIR, and then tried bin/pocs up without -d and saw hundreds of error messages. The ones I looked at appeared to be caused by the AAG cloud sensor not being found, which makes sense since we don’t have one. (We were going to deploy on our school’s roof and use a weather station we built a few years ago, but we may be deploying at Mount Wilson along with PAN012) Is there a way to pass an argument to this command to have it load specific docker containers? Can the sensor be simulated the way it can be in the pocs shell or can its absence be stated in pocs_local.yaml?

We purchased the AAG Cloud sensor, connected it, and tested the docker software today. We are getting significantly less error messages when running bin/pocs up, and the AAG weather sensor errors are completely gone. Output of bin/pocs up is:

Starting config-server      ... done
Starting aag-weather-server ... done
Starting aag-weather-reader ... done
Starting messaging-hub      ... done
Starting paws               ... done
Starting peas-shell         ... done
Starting pocs-shell         ... done
Attaching to config-server, aag-weather-server, aag-weather-reader, messaging-hub, paws, peas-shell, pocs-shell
config-server         | Starting with UID : 9001
config-server         | useradd: user 'panoptes' already exists
aag-weather-server    |  * Serving Flask app "scripts/serve-aag"
aag-weather-server    |  * Environment: production
aag-weather-server    |    WARNING: This is a development server. Do not use it in a production deployment.
aag-weather-server    |    Use a production WSGI server instead.
aag-weather-server    |  * Debug mode: off
messaging-hub         | Starting with UID : 9001
messaging-hub         | useradd: user 'panoptes' already exists
paws                  | Starting with UID : 9001
paws                  | useradd: user 'panoptes' already exists
peas-shell            | Starting with UID : 9001
peas-shell            | useradd: user 'panoptes' already exists
pocs-shell            | Starting with UID : 9001
pocs-shell            | useradd: user 'panoptes' already exists
aag-weather-server    |  * Running on (Press CTRL+C to quit)
paws                  | ERROR: File "" not found. Directory cannot be installed in editable mode: /var/panoptes/PAWS
paws                  | Traceback (most recent call last):
paws                  |   File "", line 67, in <module>
paws                  |     http_server = tornado.httpserver.HTTPServer(WebAdmin(get_config(host='')))
paws                  |   File "", line 34, in __init__
paws                  |     name = config.setdefault('name', 'PAWS')
paws                  | AttributeError: 'NoneType' object has no attribute 'setdefault'
paws exited with code 1
messaging-hub         | waiting 15 seconds for localhost:6563
config-server         |  * Serving Flask app "panoptes.utils.config.server" (lazy loading)
config-server         |  * Environment: production
config-server         |    WARNING: This is a development server. Do not use it in a production deployment.
config-server         |    Use a production WSGI server instead.
config-server         |  * Debug mode: off
messaging-hub         | localhost:6563 is available after 2 seconds
peas-shell            | waiting 15 seconds for localhost:6563
peas-shell            | localhost:6563 is available after 0 seconds
pocs-shell            | waiting 15 seconds for localhost:6563
pocs-shell            | localhost:6563 is available after 0 seconds

The error:

paws                  | ERROR: File "" not found. Directory cannot be installed in editable mode: /var/panoptes/PAWS
paws                  | Traceback (most recent call last):
paws                  |   File "", line 67, in <module>
paws                  |     http_server = tornado.httpserver.HTTPServer(WebAdmin(get_config(host='')))
paws                  |   File "", line 34, in __init__
paws                  |     name = config.setdefault('name', 'PAWS')
paws                  | AttributeError: 'NoneType' object has no attribute 'setdefault'

After this, the script hangs. I believe that this script is supposed to hang and then the user can access peas and pocs shells from other terminal windows, but I would expect there to be some sort of message to the user saying that everything is complete. @wtgee Does pocs up have a message like this and is our script hanging to early?

I’m a little confused on why useradd: user 'panoptes' already exists is appearing. Is this adding panoptes as a user/owner/group to the docker containers or is it trying to create a new user outside of the containers?

Note: We learned the hard way that bin/pocs up does not check to see if docker containers are already started, and creates many errors.

Also, while waiting for the weather sensor to arrive I created a python script using the script that automatically finds the dev path of a usb port (/dev/tty*). If you would be interested in putting it on github let me know.

HI @zacharyt,

Yes this hangs by default. Some logs messages from the various containers will be output here as well. You can run bin/pocs up -d (for daemon) if you want it to not hang. The log messages it shows aren’t critical although the messages shown during startup can be helpful.

It looks like your /var/panoptes/.env is missing a LOCAL_USER_ID=1000, which might help with this issue. However it is most likely a result of restarting existing containers, which might be what you are referring to by saying you learned the hard way?

You are right that it should check to see if the containers are already running. I will make an issue on github to fix it (unless you want to? :grin:).

Yes, I’m curious about this. It automatically finds it how? Auto-assigning the usb ports in the config would be very helpful but has proven to be more difficult than one would imagine. Not a ton of time has been spent on this though but it’s starting to to be more necessary as more people bring their units online and have confusion about the ports. Please make a PR on github if you would like to share.

As an FYI, PAWS is still pretty broken with docker setup so I wouldn’t worry about those errors.

@wtgee I added LOCAL_USER_ID=1000 to .env. I created the .env by coping and pasting the contents of env_file into a new file and renaming it .env. What should be contained in .env? Our current .env only has:


The addition of LOCAL_USER_ID fixed the errors in the peas and pocs shells! :raised_hands:

We ran into some small errors that are likely not related to the software.
We cannot use setup_pocs, the command exits with something along the lines of cameras are not available. This makes sense since we have not killed the processes on the camera’s after we moved our unit downstairs for poster presentation night, where we unplugged USB’s and plugged them back in. We have been killing the processes by using kill [PID], and have been getting the PID from lsof | grep /dev/bus/usb/*** (where *** is our camera usb number given by gphoto auto detect). It appears that docker stops lsof from working, and we cant find the PID to kill to free up the camera USBs.

We cannot load the weather sensor. The error in the peas-shell is:

PEAS > load_weather
Loading weather reader endpoint
Traceback (most recent call last):
  File "/var/panoptes/POCS/scripts/", line 469, in <module>
  File "/opt/conda/lib/python3.7/", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/opt/conda/lib/python3.7/", line 217, in onecmd
    return func(arg)
  File "/var/panoptes/POCS/scripts/", line 224, in do_load_weather
    db_type=get_config('db.type', default='file')
  File "/var/panoptes/POCS/peas/", line 39, in __init__
    raise error.PanError(f'No endpoint_url for {sensor_name}')
panoptes.utils.error.PanError: PanError: No endpoint_url for weather

Note: I setup the weather sensor today, and was a little crunched for time, so if the AAG cloud sensor has a power button that explains the error…

We can load the camera and control boards although there seems to be parsing errors when trying to view their data using last_reading. I know we modified the camera board’s arduino script to work with an Uno instead of a Micro, and I’m guessing we might have commented out some print functions. Tomorrow I’ll update our fix for the Uno on the docker arduino file’s and see if the errors go away (and turn on the cloud sensor if it has a power switch).

I would be happy to put the usb port script up on github, but it simply parses information from the bash command: ls -lR /dev/serial/by-id. The script doesn’t properly name devices that are connected by an adapter to the NUC (Namely the mount and the camera). Also, it is my first time doing text parsing, and the script is based on a very brute force approach. I’ll try implement the checking for open containers on github, but like the usb script I’m mostly relying on my knowledge of bash commands and parsing their output, so it likely won’t be pretty or easy on the processor.

This is probably just a config issue. The PAN010 team (@jlumbres, @atrodack, @jmk1729) are having the same problem right now.

I think you just need this in the config:


It’s helpful to have httpie installed on the NUC (not in the containers). This can be installed via apt. You can then do simple calls on the command line to verify it’s working:

(panoptes-env) ➜  ~ http :5000/latest.json

HTTP/1.0 200 OK
Content-Length: 518
Content-Type: application/json
Date: Thu, 14 Nov 2019 02:37:26 GMT
Server: Werkzeug/0.16.0 Python/3.7.4

        "ambient_temp_C": 14.13,
        "date": "2019-11-14T02:36:44.576Z",
        "errors": "error_1=0 error_2=0 error_3=0 error_4=0",
        "gust_condition": "Calm",
        "internal_voltage_V": 5.9019230769,
        "ldr_resistance_Ohm": 0.4970414201,
        "pwm_value": 9.9706744868,
        "rain_condition": "Dry",
        "rain_frequency": 2652.0,
        "rain_sensor_temp_C": 19.0,
        "safe": false,
        "sky_condition": "Very Cloudy",
        "sky_temp_C": 6.43,
        "weather_sensor_firmware_version": 5.51,
        "weather_sensor_name": "CloudWatcher",
        "weather_sensor_serial_number": 821,
        "wind_condition": "Calm",
        "wind_speed_KPH": 0.0

Note that the weather url config does go under environment and not under the weather entry.

Basically the weather station has been decoupled from POCS and PEAS so can operate separately. For PAN008 (and a few others) the AAG is housed at the observatory site but is separate from PANOPTES so is not available on a USB port or anything. The AAG Weather Repo is in charge of the weather and sets up a little web server that serves JSON entries. PEAS then treats this as a RemoteSensor, which just listens on any url for some JSON text (we call those “endpoints” in the web development world). The aag-weather-server container just starts up that repo and serves the web page.

@wtgee is the end point url put before or after the camera_board and control_board information? Does that even matter? Is the config file parsed for info and put into a python dict? If I get it working I’ll push to github.

We have been having this issue as well at PAN010. I added that bit after the camera/control board lines, and ran the test that @wtgee suggested and it printed the same result as above. We will be testing with our AAG plugged in momentarily.

So our pocs_local.yaml environment section looks like this:

  auto_detect: false
    serial_port: /dev/ttyACM1
    serial_port: /dev/ttyACM0


I think he meant to put it under the environment section where the arduinos are. I’m testing as well and will let you know if I get it working. I would also expect that we have to download the AAG Weather Repo Wilfred mentioned.