CHORDS - Cloud.HOsted.Real-time.Data.Services

CHORDS is a masterpiece of the NSF EarthCube project. It is a real-time data service for the geosciences that provides an easy-to-use system to acquire, navigate, and distribute real-time data via cloud services and the Internet. CHORDS aims to lower the barrier to these services, especially for small instrument teams, and broaden access to real-time data for the geosciences community.

• CHORDS, Grafana, and AWS Sites
  • NCAR
    • 3D-PAWS CHORDS Site
    • 3D-PAWS Grafana Site
  • UND
    • UND CHORDS Site
    • UND Grafana Site
    • AWS Starting Point
• Documentation
  • Earthcube CORDS Documentation
  • GitHub Earthcube Wiki
  • Grafana Documentation
  • InfluxDB in Grafana
  • Create AWS Portal
  • AWS DNS Route 53

• Host address: http://3.89.34.59
• Username: admin@chordsrt.com
• Password: Standard + ???

• cd chords
• sudo python chords_control −−stop
• sudo python chords_control −−run

• Added Dashboard that summaries all available site Dashboards.
• Can not make public dashboards in Grafana because it opens up add data to the public. There is a workaround described at https://community.grafana.com/t/making-selected-dashboards-public/873.

• Added 1 Gbyte of Swap to increase memory from 2 Gbytes to 3 Gbytes to try to prevent server crashes.

• Issue - The sever does down so that nothing accessible and do uploaded data is saved.
• Cause - Has happen several times. Not sure why but will track times of resets.
  • Reset at 2022/02/26 13:35 UTC.
  • Reset at 2022/02/26 14:51 UTC. Ran for awhile; however, stopped before having CHORDS work.
  • Down at 4:11:40 UTC on 2022/03/04.
    • sudo docker exec -it chords_app tail -f log/production.rb gives that Container is not running.
    • sudo docker container ls gives that chords_nginx chords_grafana chords_influxdb and chords_mysql is running. Hence only chords_app died, which must be the issue.
    • sudo docker ps -a gives
      • 0607e0d72783 earthcubechords/chords:1.1.0-rc1 “bundle exec bash -c…” 5 days ago Exited (1) 7 hours ago chords_app
      • The above 7 hours is when the server clashed. Exited code 1 not very helpful.
    • sudo docker events –since 2022-02-25 gives
      • 2022-03-04T04:12:48.801690763Z container die 0607e0d727835ea59127e1c9afb4a18dcba61ad581db724ef2065cfbfbe35456 (com.docker.compose.config-hash=c9b5013f152568ae48b5612a390cb7da17b47a4d7bf096fe7dbeb53043e464a0, com.docker.compose.container-number=1, com.docker.compose.oneoff=False, com.docker.compose.project=chords, com.docker.compose.project.config_files=docker-compose.yml, com.docker.compose.project.working_dir=/var/lib/chords, com.docker.compose.service=app, com.docker.compose.version=1.26.2, exitCode=1, image=earthcubech ords/chords:1.1.0-rc1, name=chords_app)
      • Indicates the container died but not why.
    • sudo docker logs 0607e0d727835ea59127e1c9afb4a18dcba61ad581db724ef2065cfbfbe35456
      • Gives E, [2022-03-04T04:12:40.602405 #69] ERROR – : Cannot allocate memory - fork(2) (Errno::ENOMEM)
      • This is at the time of the crash.
    • Hence, the cause seems to be AWS instance that does not have enough memory.
  • Short Term Solution - Reset server using:
  • cd chorsds
  • sudo python chords_control –stop
  • sudo python chords_control –run
• Long Term Solution - Upgrade to larger AWS instance, with more memory. There does not seem to be any swap memory on AWS Linux, wondering why. Only 2 GB of real memory.
  • See https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-partition-hard-drive/ for how to add SWAP space.
• Note
  • Sometimes get a “502 Bad Gateway” error when accessing CHORDS for awhile after coming back up.
  • May be best not to try to access CHORDS right after restarting server.
  • After reset, had to clear the browser cache and cookie to be able to access the CHORDS site again.

• Issue - At ~00:08, the AWS server stopped accepting data. Error via “sudo docker logs chords_mysql” is “2022-01-12T00:08:48.988161Z 200 [Note] Aborted connection 200 to db: 'chords_demo_production' user: 'chords_demo_user' host: '192.168.112.5' (Got an error reading communication packets)”.
• Cause - Seems to be related to uploading CCN counter data, as have warnings about invalid strings.
• Solution - Added check to ensure that 'C' and 'H' strings have correct number of elements. Had to reboot the AWS system and not just restart the CHORDS/Grafana Docker containers.

• Had to get network port (80, 300, 8086) open on local network in addition to via VPN access.
• Kept getting “Something went Wrong” error on CHORDS. Only on certain CHORDS pages, http://docker1.atmos.und.edu/about would work. Seems to be a Influx access issue.

• Setup Issues (Comments incorporated from Charles Martin at UCAR)
  • AWS Setup - The configuration on AWS using a template to setup a system did not work. Error that could start CHORDS. Had to ssh into the cloud linux server manually and configure chords manually.
    • Documentation is in fact out of date, they plan to update. Charles Martin indicated that the AWS CloudFormation process is not very satisfactory and not surprised that it failed. Best to spin up a AWS Linux instance and configure/Start CHORDS.
  • Linux CHORDS Setup - The configuration of CHORDS would not work without using any configuration expect “Development” version.
    • Charles Martin indicated an issue with chords_control, which was fixed in the development version and has a new release candidate now.
  • Linux Firewall Setup - Had to manually change the firewall setup to allow port 25, 80, 3000 on the Linux server,in addition to the AWS cloud server restriction setup.
  • CHORDS login - Firefox login won't work for CHORDS login, but Chrome Browser works. Seems to be related to not enabling https yet.
  • Uploading Data - The information in the documentation about “particle” is confusing, The “particle” configuration is only if you have one of their devices. This should be make clear and the section about “particle” moved at the end, after the other data upload options.
    • Particle is for very specific devices and documentation should make this clear.
  • The wget/curl uploading of data does not work. Got data upload to work via a command to in Firefox Web browser. Moved on to using python code, made code to upload data from past data files, which was not too hard.
    • Could never get wget/curl uploading to work, Likely something to do with authentication.
  • Grafana Setup - The admin password would not work to log in. Re-configured the system with a different password and it did not work. Reset email does not seem to work, don't get emails. The password issue is that the password in the configuration does not work and have to use the default username and password as admin and admin. Hence, need to enter this and set to our standard password.
  • Influxdb Setup - When in CHORDS and setup the Influxdb Data source, I get a “Network Error: Bad Gateway(507), when saving and testing. Seems I had to use the URL of http://influxdb:8086/ instead of http://localhost:8086/
    • Confirm that this is an issue with the documentation, should use http://influxdb:8086.
  • Grafana Configuration - Had problems getting any data into the panel display. Had to use “from → default”, “data → tsdata”, “variable - 2”, selection (mean).
  • DNS Service - Need to setup some domain name service so we can get address that always points at the ASW server.
    • Need to use AWS Elastic IP addresses. Set one of these up and put it to the instance with CHORDS.
• 2021/01/22
  • When deleting instrument, the data is not deleted from the database at first. Still have same storage size in CHORDS. Hence, don't know how to remove non testing data. Maybe it just takes time to remove stuff from the data base.
  • Seems that there may be an issue with size of CHORDS/Influxdb. Got up to 150 MB with 1 Hz Windsonic data. Changed to 60 second data now. Maybe there is an issue with the number of measurements per instrument.
• 2021/03/20
  • When adding in first pushing new data to site, the CHORDS site crashes and need to do sudo python chords_control –stop and sudo python chords_control –run.
  • Default setup has no open (anonymous) viewing of Dashboards in Granafa. See https://stackoverflow.com/questions/33111835/how-to-set-up-grafana-so-that-no-password-is-necessary-to-view-dashboards on how to change. Really had to change the configuration to use UND-Atmos for the Organiation and ORG_ROLE to Viewer in docker-compose.yml. Had to add these variables.
• 2021/12/20
  • Need to open Firewall to allow access to IP. To find the public IP address on Linux, last line.
  • host myip.opendns.com resolver1.opendns.com