Differences

This shows you the differences between two versions of the page.

--- atmos:software:chords:home [2022/01/02 15:29] – delene
+++ atmos:software:chords:home [2023/04/26 14:17] (current) – [CHORDS Server Reset] delene
@@ Line 24: / Line 24: @@
   * Password: Standard + ???
-===== CHORDS/Grafana Issues (AWS 2021 Setup) =====
+===== CHORDS Server Reset =====
+  * cd chords
+  * sudo python chords_control −−stop
+  * sudo python chords_control −−run
+===== CHORDS/Grafana Issues Log (Newest to Oldest) =====
+==== 2022/03/26 - Added Swap Memory to AWS Server ====
+  * Added Dashboard that summaries all available site Dashboards.
+  * Can not make public dashboards in Grafana because it opens up add data to the public. There is a workaround described at [[https://community.grafana.com/t/making-selected-dashboards-public/873|https://community.grafana.com/t/making-selected-dashboards-public/873]].
+==== 2022/03/24 - Added Swap Memory to AWS Server ====
+  * Added 1 Gbyte of Swap to increase memory from 2 Gbytes to 3 Gbytes to try to prevent server crashes.
+==== 2022/02/26 - Sever Goes Down (Crashes) Routinely ====
+  * Issue - The sever does down so that nothing accessible and do uploaded data is saved.
+  * Cause - Has happen several times. Not sure why but will track times of resets.
+    * Reset at 2022/02/26 13:35 UTC.
+    * Reset at 2022/02/26 14:51 UTC.  Ran for awhile; however, stopped before having CHORDS work.
+    * Down at 4:11:40 UTC on 2022/03/04.
+      * sudo docker exec -it chords_app tail -f log/production.rb gives that Container is not running.
+      * sudo docker container ls gives that chords_nginx chords_grafana chords_influxdb and chords_mysql is running.  Hence only chords_app died, which must be the issue.
+      * sudo docker ps -a gives
+        * 0607e0d72783   earthcubechords/chords:1.1.0-rc1           "bundle exec bash -c…"   5 days ago   Exited (1) 7 hours ago                                                                                      chords_app
+        * The above 7 hours is when the server clashed.  Exited code 1 not very helpful.
+      * sudo docker events --since 2022-02-25 gives
+        * 2022-03-04T04:12:48.801690763Z container die 0607e0d727835ea59127e1c9afb4a18dcba61ad581db724ef2065cfbfbe35456 (com.docker.compose.config-hash=c9b5013f152568ae48b5612a390cb7da17b47a4d7bf096fe7dbeb53043e464a0, com.docker.compose.container-number=1, com.docker.compose.oneoff=False, com.docker.compose.project=chords, com.docker.compose.project.config_files=docker-compose.yml, com.docker.compose.project.working_dir=/var/lib/chords, com.docker.compose.service=app, com.docker.compose.version=1.26.2, exitCode=1, image=earthcubech ords/chords:1.1.0-rc1, name=chords_app)
+        * Indicates the container died but not why.
+      * sudo docker logs 0607e0d727835ea59127e1c9afb4a18dcba61ad581db724ef2065cfbfbe35456
+        * Gives E, [2022-03-04T04:12:40.602405 #69] ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM)
+        * This is at the time of the crash.
+      * Hence, the cause seems to be AWS instance that does not have enough memory.
+    * Short Term Solution - Reset server using:
+    * cd chorsds
+    * sudo python chords_control --stop
+    * sudo python chords_control --run
+  * Long Term Solution - Upgrade to larger AWS instance, with more memory.  There does not seem to be any swap memory on AWS Linux, wondering why.  Only 2 GB of real memory.
+    * See [[https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-partition-hard-drive/|https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-partition-hard-drive/]] for how to add SWAP space.
+  * Note
+    * Sometimes get a "502 Bad Gateway" error when accessing CHORDS for awhile after coming back up.
+    * May be best not to try to access CHORDS right after restarting server.
+    * After reset, had to clear the browser cache and cookie to be able to access the CHORDS site again.
+==== 2022/01/12 - Sever Data Acceptance Issues (AWS Server 3.89.34.59) ====
+  * Issue - At ~00:08, the AWS server stopped accepting data.  Error via "sudo docker logs chords_mysql" is "2022-01-12T00:08:48.988161Z 200 [Note] Aborted connection 200 to db: 'chords_demo_production' user: 'chords_demo_user' host: '192.168.112.5' (Got an error reading communication packets)".
+  * Cause - Seems to be related to uploading CCN counter data, as have warnings about invalid strings.
+  * Solution - Added check to ensure that 'C' and 'H' strings have correct number of elements.  Had to reboot the AWS system and not just restart the CHORDS/Grafana Docker containers.
+==== UND Docker1 2022 Setup ====
+  * Had to get network port (80, 300, 8086) open on local network in addition to via VPN access.
+  * Kept getting "Something went Wrong" error on CHORDS.  Only on certain CHORDS pages, http://docker1.atmos.und.edu/about would work.  Seems to be a Influx access issue.
+==== AWS 2021 Setup ====
   * Setup Issues (Comments incorporated from Charles Martin at UCAR)
     * AWS Setup - The configuration on AWS using a template to setup a system did not work.  Error that could start CHORDS.  Had to ssh into the cloud linux server manually and configure chords manually.
@@ Line 51: / Line 103: @@
     * Need to open Firewall to allow access to IP.  To find the public IP address on Linux, last line.
     * host myip.opendns.com resolver1.opendns.com
-===== CHORDS/Grafana Issues (UND Docker1 2022 Setup) =====
-  * Had to get network port (80, 300, 8086) open on local network in addition to via VPN access.
-  * Kept getting "Something went Wrong" error on CHORDS.  Only on certain CHORDS pages, http://docker1.atmos.und.edu/about would work.  Seems to be a Influx access issue.