Java Scala Mongo Installation

Getting Scala to work on EC2

Ref: https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04

# Java installation - installing Oracle JDK 8
 apt-get -y update
 add-apt-repository ppa:webupd8team/java
 apt-get -y update
 apt-get install -y oracle-java8-installer

# if there are multiple JDK installations on one server, then we can switch between versions using:
# update-alternatives --config java

# Setting JAVA_HOME
# Put the location of java from the config command above at the end of the file
# JAVA_HOME="/usr/lib/jvm/java-8-oracle"
# Then do source /etc/environment

# Scala installation
# First see which version of Scala we should install - We are using 2.11.8
# Got to Scala website and find the link to set.
# Make a directory like /tf_files/scala_bkend and get this tgz file
# wget https://github.com/sbt/sbt/releases/download/v1.0.2/sbt-1.0.2.tgz
# gunzip sbt-1.0.2.tgz
# tar xvf sbt-1.0.2.tar

 #Ref: https://gist.github.com/alexislucena/5947c925afc3a0f26b04c3785d482e9f
 # Scala Install
 mkdir -p /tf_files/DOWNLOAD/Scala_install
 cd /tf_files/DOWNLOAD/Scala_install
 apt-get remove scala-library scala
 wget www.scala-lang.org/files/archive/scala-2.11.8.deb
 dpkg -i scala-2.11.8.deb

 # SBT install
 # Ref: http://www.scala-sbt.org/0.13/docs/Installing-sbt-on-Linux.html
 echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
 apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
 apt-get update
 apt-get install sbt


 # Install MongoDB
 # Ref: https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-ubuntu-16-04
 # Ref: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
 apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6
 echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-3.4.list
 apt-get -y update
 apt-get install -y mongodb-org

 # Install MongoDB in ubuntu18.04
 ubuntu@ip-10-215-55-167:~$ cat mongo_install.sh 
#!/bin/bash
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
echo "mongodb-org hold" | sudo dpkg --set-selections
echo "mongodb-org-server hold" | sudo dpkg --set-selections
echo "mongodb-org-shell hold" | sudo dpkg --set-selections
echo "mongodb-org-mongos hold" | sudo dpkg --set-selections
echo "mongodb-org-tools hold" | sudo dpkg --set-selections

sudo service mongod start

./mongo_install.sh



# then follow to initiate systemd as given in the documentation - edit /etc/systemd/system/mongodb.service
# finally start the service - DID NOT WORK IN DOCKER
# systemctl start mongodb

# to start the mongo service manually:
# /usr/bin/mongod --quiet --config /etc/mongod.conf &

# Get AgShift mongoldb zip file from GitHub and then sep to the EC2 docker
# Start mongodb
# 1. To dump the entire database at some location, use the following command.
#   mongodump -d databasename -o /path/of/output/
# Then restore the database using the following command
# mongorestore --host localhost --port 27017 --db agshift /path/to/Downloaded unzipped file/agshift

# Ref image: https://s3-us-west-2.amazonaws.com/agskore/Users/IOS/devrelease/A23571BC-A7D4-4739-8778-C57FB36AE54A/2017-09-14/JPEGImages/pic_1505417603796.jpg
# Curl command: curl -X POST -d '{"to":"Support"}' -H "Content-Type: application/json" http://localhost:809
0/api/misc/contact-us -v

MongoDB connect

 mongo <DB Name> --host 127.0.0.1 --port 38270 -u <user id> -p <passwd>

 # to start the mongo service manually:
 /usr/bin/mongod --quiet --config /etc/mongod.conf &

 # to restore a DB
  mongorestore --host localhost --port 27017 --db <DB name> </path/to/Downloaded_unzipped_file>/<DB name>

MongoDB upgrade (community edition 4.0.7) and start service

Ref: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/

# Hope that you don't fear losing the DB directory
sudo apt list --installed | grep mongodb
sudo apt-get purge mongodb-org*

sudo rm -r /var/log/mongodb
sudo rm -r /var/lib/mongodb

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

# starting service might give an error
sudo systemctl start mongodb
Failed to start mongodb.service: Unknown unit: mongodb.service

# to rectify the above error and successfully restarting mongod service
sudo systemctl enable mongod
sudo service mongod start

# edit /etc/mongod.conf and change the port
port 39817

then start mongo shell as:
mongo --port=39817

MongoDB commands

 > db.<collection_name>.remove({})
 WriteResult({ "nRemoved" : 112 })


Data types reference:

Data Type    Number
Double            1
String            2
Object            3
Array             4
Binary Data       5
Undefined         6
Object Id         7
Boolean           8
Date              9
Null              10
Regular Expression    11
DBPointer       12
JavaScript        13
Symbol            14
JavaScript with scope    15
Integer 32 bit   16  
Integer 64 bit   18
Decimal          19
Timestamp        17
Min Key          -1
Max Key          127

Ref: https://examples.javacodegeeks.com/software-development/mongodb/mongodb-foreach-example/

 # to change the data type
 db._company.find({"subCompanyId":{$type:1}}).forEach(function(x) { \
     x.subCompanyId = new NumberInt(x.subCompanyId); \
     db._company.save(x); \
 });


 # to insert a new field "subCOmpanyId" in the collection for all the documents
 db._user.find().forEach( function (elem) {   \
    elem.subCompanyId = elem.company_code;   \
    db._user.save(elem); \
 } )

 # you may need to change the data type for "subCompanyId" after doing the above
 db._company.find({"subCompanyId":{$type:1}}).forEach(function(x) { \
    x.subCompanyId = new NumberInt(x.subCompanyId); \
    db._company.save(x); \
 });


 # to update nested array
 # Ref: https://docs.mongodb.com/manual/reference/operator/update/positional-all/
 db.students3.update({"_id":2}, {$set:{"grades.$[].questions.$[score]":100 }}, \
 { arrayFilters: [  { "score": { $gt: 10 } } ], multi: true})


 # update nested arrays - requires MongoDb 3.6
 Ref: https://docs.mongodb.com/manual/reference/operator/update/positional-all/#position-nested-arrays
 Ref: 
 db.acl_test.insert([
   { "companyCode" : 1,
      "ACL" : [
        { functionality: "new", roles: [ "birddog", "reviewer", "marketing"] },
        { functionality: "assign", roles: [ "birddog", "reviewer", "sales" ] },
        { functionality: "delete", roles: [ "birddog", "reviewer" ] },
        { functionality: "edit", roles: [ "reviewer" ] }
      ]
   },
   { "companyCode" : 2,
      "ACL" : [
        { functionality: "new", roles: [ "birddog", "reviewer", "marketing"] },
        { functionality: "assign", roles: [ "birddog", "reviewer", "sales" ] },
        { functionality: "delete", roles: [ "birddog", "reviewer" ] },
        { functionality: "edit", roles: [ "reviewer" ] }
      ]
   },
   { "companyCode" : 3,
      "ACL" : [
        { functionality: "new", roles: [ "birddog:desc1:new", "reviewer:desc1:new", "marketing:desc9:new"] },
        { functionality: "assign", roles: [ "birddog:desc2:assign", "reviewer:desc7:assign", "sales:desc2:assign" ] },
        { functionality: "delete", roles: [ "birddog:desc3:delete", "reviewer:desc6:delete" ] },
        { functionality: "edit", roles: [ "reviewer:desc4:edit" ] }
      ]
   }

])

db.acl_test.update(    {"companyCode":3},    \
{ $set: { "ACL.$[].roles.$[roleName]": "hawk:desc42:delete" } }, \
{ arrayFilters: [  { "roleName": { $in: ["birddog:desc3:delete"] } } ]} )


# for findAndModify
Ref: https://docs.mongodb.com/manual/reference/method/db.collection.findAndModify/#db.collection.findAndModify

# for $in[] operator
Ref: https://docs.mongodb.com/manual/reference/operator/query/in/


# multiple queries
db._input_data.find({inputDataId :{$gt :"2882",$lt :"2962"}, "annotatedFoodAwsUrl" : /INPUTEDITEDImages/}).

# querying with $gt for strings - javascript syntax
db._input_data.find( "this.inputDataId > 2888" ).count()

# querying a list using regex
db._input_data.find({"foodAwsUrl": { $regex: /pic_004_1548458142_good_top_0/}}, {"inputDataId":1}).pretty()

#removing using a query
 db._input_data.remove({"inputDataId":"3395"})

# removing a range of records/documents
 db._input_data.remove({"inputDataId":{$gte:4703}})

 # query using a "dateTime" string and getting out multiple projections (inputDataId, licenseKey, annotatedCoordinates)
 db._input_data.find({"dateTime" : "2019-03-26"}, {"inputDataId":1, "licenseKey":1, "annotatedCoordinates.top":1}).pretty()

 # multiple queries, multiple projections - for Olam (extracting the cashew_DT info)
 db._input_data.find({"subCompanyId":"6", "dataCollectionMode" : 3}, {"dateTime":1, "inputDataId" : 1, "commodity" : 1, "licenseKey":1, "foodAwsUrl" :1}).pretty()

 # querying a regex in an array object in a document
 db._input_data.find({"foodAwsUrl":{$elemMatch: {$regex: "pic_080_1560963470_generic_bot_1"}}}, {"inputDataId":1}).pretty()

MongoDB security

https://medium.com/@raj_adroit/mongodb-enable-authentication-enable-access-control-e8a75a26d332

Getting MongoDB secured in Kubernetes

http://pauldone.blogspot.com/2017/06/deploying-mongodb-on-kubernetes-gke25.html

MongoDB Kubernetes ec2_kops_shadow

 mongo -u <DB name> -p <passwd> -authenticationDatabase <authentication DB name>

root@mongod-0:/# mongo -u <user Id> -p <passwd> -authenticationDatabase <authentication DB name>
 use <DB name>
 show collections


 Dev machine
 To get the DB dump:
 ubuntu@ip-172-31-0-119:~/MongoDB_BACKUP$ sudo mongodump --host 127.0.0.1 --port 38270 --db <DB name> --username <user ID> --password <passwd> -o <output>

 scp the dump from dev to ec2_kops_shadow


 use kubectl command to copy from ec2_kops_shadow host to kubernetes mongod-o
 kubectl cp /tmp/<output_name>.tar.gz mongod-0:/tmp/<output_name>.tar.gz


 db.dropDatabase() => not able to drop

 So delete collections one by one
 use <DB name>
 db.<collection_name>.drop()

 Exhaustive list:
 db._access_token.drop();
db._acl.drop();
db._commodity.drop();
db._commodityDetails.drop();
db._company.drop();
db._company_client.drop();
db._counters.drop();
db._device_mapping.drop();
db._growing_region.drop();
db._input_data.drop();
db._iot_enclosure_config.drop();
db._item_master.drop();
db._market.drop();
db._market_group.drop();
db._menu.drop();
db._qt_counters.drop();
db._roi.drop();
db._temporary_user.drop();
db._user.drop();
db._variety.drop();
db._varietyDetails.drop();
db._version.drop();

# This programatic way will be even better, but could not make it work yet
var collectionNames = db.getCollectionNames();
for(var i = 0, len = collectionNames.length; i < len ; i++){
    var collectionName = collectionNames[i];
    if(collectionName.indexOf('cache_') == 0){
        db[collectionName].drop()
    }
}

# THIS PROGRAMMATIC WAY WORKS!!
db.getCollectionNames().forEach(function(collname) { print(collname);db[collname].drop();} );


 quit()


This appended the current database
mongorestore -d <DB name> -u <user ID> -p <passwd>  /tmp/<path>/<DB name>

Replicate MongoDB from Production DB to Staging DB

# Login to ec2_kops_staging machine
ssh ec2_kops_staging


# copy DB from prod to staging and to staging mongo pod
 cd ~/StagingMongoBackup/
 scp -P 39989 -i ~/.ssh/ubuntu1403_keypair.pem ubuntu@54.245.190.37:/volxvdf/HydraMogoDBbackup/mongoDB_DUMP_agshift_12_21_2018_03_00_01.tar.gz .
 tar xvzf mongoDB_DUMP_agshift_12_21_2018_03_00_01.tar.gz
 kubectl cp ~/StagingMongoBackup/tmp/mongoDB_DUMP_agshift_12_21_2018_03_00_01 mongod-0:/tmp/mongoDB_DUMP_agshift_12_21_2018_03_00_01


# Delete all existing collections in staging DB
 cd ~/Staging/Production_Access
 ./mongo_pod_access_new.sh mongod-0
 mongo -u agshift -p yrhy4Nqfuc9XEcdBqKyNCpeL -authenticationDatabase agshift
 use agshift
 db.getCollectionNames().forEach(function(collname) { print(collname);db[collname].drop();} );
 quit()


# Restore the prod DB, which has been copied to the mongod-o container
mongorestore -d agshift -u agshift -p yrhy4Nqfuc9XEcdBqKyNCpeL /tmp/mongoDB_DUMP_agshift_12_21_2018_03_00_01/agshift

MongoDB setting up replicate set on vanilla VM's

Ref: https://computingforgeeks.com/how-to-setup-mongodb-replication-on-ubuntu-18-04-lts/

Scala structure

src/main/scala/<project>
                       -> <project>Process.scala (templatizes the structure of DB collections for siphoning out the data)
                       -> <project>Model.scala   (inherits Process; this is the main bullwark of DB read/write happens here)
                       -> <project>Service.scala (inherits model, this defines /api endpoint) 
        /scala/utils
                       -> Database.scala (configures the DB, including userid, password, connectionOptions and Credentials, AuthenticationMechanism etc...)

        /scala/ServiceController.scala -> Entrypoint. This extends/inherits the "App", defines the routes and binds to HTTP
        /scala/CorsHandler.scala       -> All CORS related stuff goes here

Example:
Fit the above structure to the following example from GitHub
tooling/src/main/scala/databaseaccess

Scala Setup

Look at the Dockerfile. That will be a good start

Steps:
1. ssh -i ~/.ssh/ ubuntu1403_keypair.pem ubuntu@<domain-name>

2. mkdir -p GIT_STUFF
3. cd GIT_STUFF/
4. sudo apt update

5. sudo apt install -y openjdk-8-jdk
6. sudo apt install -y wget
7. mkdir $HOME/Downloads
8. cd $HOME/Downloads
9. wget www.scala-lang.org/files/archive/scala-2.11.8.deb
10. wget dl.bintray.com/sbt/debian/sbt-0.13.11.deb

11. sudo dpkg -i scala-2.11.8.deb
12. sudo dpkg -i sbt-0.13.11.deb
13. sbt (Revisit)
14. cd $HOME/GIT_STUFF
15. git clone https://github.com/agshift/cootbackend.git
16. git clone https://github.com/agshift/cootfendweb.git
17. sudo apt install -y mongodb
18. sudo apt install -y
       ○ build-essential
       ○ software-properties-common
       ○ apt-utils
       ○ gnupg 

○ curl
       ○ unzip
       ○ python
       ○ python-setuptools
       ○ vim
       ○ cron

       ○ openssh-client

○ xauth

○ git
       ○ xvfb
       ○ python-pip

○ s3cmd
       ○ python-arrow

○ iputils-ping
       ○ procps

○ dnsutils
 19. sudo apt install -y bsd-mailx mailutils
 20. curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
 21. sudo apt-get install -y nodejs
 22. sudo apt install -y npm //Gave Error -->Aman (don’t run, not is shell script)
 23. sudo npm install -g firebase-tools

24. sudo apt -y install nginx
 25. pip install --upgrade pip
 26. sudo apt install -y awscli
 27. wget 
https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_
linux-generic-amd64.tar.xz
 28. tar vxf wkhtmltox-0.12.4_linux-generic-amd64.tar.xz
 29. sudo cp wkhtmltox/bin/wk* /usr/local/bin/
 30. sudo apt -y install ssmtp

 # Configurations
 # MongoDB Backup
The location of the backup may vary
1. Fetch backup to local machine
     scp -i ~/.ssh/ubuntu1403_keypair.pem
ubuntu@ 52.38.137.129:/volxvdf/HydraMogoDBbackup/ mongoDB_DUMP_agshift_12_
31_2018_21_00_01.tar.gz ./
2. Extract directory from the tar file
     tar zxvf mongoDB_DUMP_agshift_12_31_2018_21_00_01.tar.gz
3. Restore data from that backup
     mongorestore -d agshift tmp/*/agshift



 #Deploy CootBackEnd (Without Kubernetes)
 1. cd
 2. cd $HOME/GIT_STUFF/cootbackend/agshift-rest-services/
3. git checkout develop (Branch is As Per Desire-->The most stable code is in Development
Branch)
4. Make Necessary Changes Required In application.conf file (Database Location, SMTP email
and password, cors domain name and domain url)
5. src/main/resources/application.conf (Make Necessary Changes and Save it.)
6. Change Database.scala file (Package Location --> com/agshift/utils/Database.scala)
(Changes required in password. If db instance requires password, we have to uncomment
password section)
7. Go to GIT_STUFF/cootbackend/agshift-rest-services/

8. sbt clean
9. export SBT_OPTS="-XX:+UseNUMA -XX:-UseBiasedLocking -Xms256M -Xmx1G
-Xss1500M -XX:MaxPermSize=128m -XX:+UseParallelGC -XX:MaxGCPauseMillis=100
-XX:GCTimeRatio=19"
10. sbt dist
11. cd target/universal/
12. rm -rf agshift-rest-services-1.0/ (agshift-rest-services-1.0 folder exists)
13. unzip agshift-rest-services-1.0.zip
14. cd agshift-rest-services-1.0/bin

15. sudo nohup ./agshift-rest-services &
16. ps -eaf | grep java (It should show java process running)