In this article, I will list the steps needed to deploy an Elasticsearch cluster on a private Google Cloud Platform (GCP) Kubernetes cluster using Helm, from creating a docker image with Elasticsearch, to the creation of a private Kubernetes cluster and more.<\/p>\n\n\n\n
Let’s begin.<\/p>\n\n\n\n\n\n\n\n
First, we need to create our own docker image for elasticsearch, based on the official image provided by elastico. This step is mandatory, because we do not have any connectivity to the official repository from inside a private GCP cluster.<\/p>\n\n\n\n
So, on my computer, I’m creating a Dockerfile with the following content:<\/p>\n\n\n\n
FROM docker.elastic.co\/elasticsearch\/elasticsearch:7.5.0<\/pre>\n\n\n\nNext step is to build the image and tag (replace PROJECT_ID).<\/p>\n\n\n\n
docker build --tag elasticsearch-gcs:7.5.0 .\ndocker tag elasticsearch-gcs:7.5.0 gcr.io\/PROJECT_ID\/elasticsearch-gcs:7.5.0<\/pre>\n\n\n\nThen, I push the newly created image to internal gcp repository of my project.<\/p>\n\n\n\n
docker push gcr.io\/PROJECT_ID\/elasticsearch-gcs:7.5.0<\/pre>\n\n\n\nIf needed, I authenticate to be able to push the image:<\/p>\n\n\n\n
gcloud auth configure-docker<\/pre>\n\n\n\nVPC network and client VM<\/h2>\n\n\n\n
We will create a dedicated VPC network for the cluster.<\/p>\n\n\n\n
gcloud compute networks create elasticsearch-network --subnet-mode=custom<\/pre>\n\n\n\nCreate a subnet in our VPC network.<\/p>\n\n\n\n
gcloud compute networks subnets create elasticsearch-subnet \\\n --network=elasticsearch-network --range=10.50.0.0\/16<\/pre>\n\n\n\nCreate VM which will be used to run kubectl commands.<\/p>\n\n\n\n
gcloud compute instances create --subnet=elasticsearch-subnet \\\n --scopes cloud-platform es-cluster-proxy<\/pre>\n\n\n\nMake note of the IP of the VM, or get it:<\/p>\n\n\n\n
export CLIENT_IP=`gcloud compute instances describe es-cluster-proxy \\
--format=\"value(networkInterfaces[0].networkIP)\"`<\/pre>\n\n\n\nCreate a firewall rule to allow SSH access to the VPC network.<\/p>\n\n\n\n
gcloud compute firewall-rules create elasticsearch-proxy-ssh --network elasticsearch-network \\\n --allow tcp:22<\/pre>\n\n\n\nConnectivity<\/h2>\n\n\n\n
Serverless VPC access<\/h4>\n\n\n\n
Create a serverless VPC access for cloud functions. Do not forget to change the region parameter (ex: europe-west2).<\/p>\n\n\n\n
gcloud compute networks vpc-access connectors create elasticsearch-cluster \\\n--network elasticsearch-network \\\n--region REGION \\\n--range 10.9.8.0\/28<\/pre>\n\n\n\nDon’t forget to update all cloud functions to use the newly created VPC connector using:<\/p>\n\n\n\n
--vpc-connector projects\/PROJECT_ID\/locations\/REGION\/connectors\/elasticsearch-cluster<\/pre>\n\n\n\nVPC network peering<\/h4>\n\n\n\n
In order for our backend API to be able to make request to the cluster, we need to peer networks together, that is default network with our newly created network elasticsearch-network.<\/p>\n\n\n\n
gcloud compute networks peerings create elasticsearch-default-peer \\\n --network=default \\\n --peer-project PROJECT_ID \\\n --peer-network elasticsearch-network \\\n --auto-create-routes<\/pre>\n\n\n\nPrivate cluster<\/h2>\n\n\n\n
Creation<\/h4>\n\n\n\n
Let’s continue by creating a private cluster with no public IP on compute engines used by the cluster. Do not forget to change the zone parameter (ex: europe-west2-a) and to configure the cluster so that it fits your needs.<\/p>\n\n\n\n
gcloud container clusters create \"elasticsearch-cluster\" \\\n --zone \"ZONE\" \\\n --no-enable-basic-auth \\\n --cluster-version \"1.14.10-gke.21\" \\\n --machine-type \"n1-standard-1\" \\\n --image-type=\"cos_containerd\" \\\n --disk-type \"pd-standard\" \\\n --disk-size \"50\" \\\n --metadata disable-legacy-endpoints=true \\\n --scopes \"https:\/\/www.googleapis.com\/auth\/devstorage.read_only\",\"https:\/\/www.googleapis.com\/auth\/logging.write\",\"https:\/\/www.googleapis.com\/auth\/monitoring\",\"https:\/\/www.googleapis.com\/auth\/servicecontrol\",\"https:\/\/www.googleapis.com\/auth\/service.management.readonly\",\"https:\/\/www.googleapis.com\/auth\/trace.append\" \\\n --num-nodes \"3\" \\\n --enable-stackdriver-kubernetes \\\n --enable-ip-alias \\\n --enable-private-nodes \\\n --enable-private-endpoint \\\n --master-ipv4-cidr 172.16.0.32\/28 \\\n --network elasticsearch-network \\\n --subnetwork=elasticsearch-subnet \\\n --no-issue-client-certificate \\\n --addons HorizontalPodAutoscaling,HttpLoadBalancing \\\n --enable-autoupgrade \\\n --enable-autorepair \\\n # --enable-master-authorized-networks \\\n # --master-authorized-networks <IP>\/32<\/pre>\n\n\n\nAllow access to VM<\/h4>\n\n\n\n
We update our cluster to authorize access from our previously created VM.<\/p>\n\n\n\n
gcloud container clusters update elasticsearch-cluster \\\n --enable-master-authorized-networks \\\n --master-authorized-networks <VM_IP>\/32<\/pre>\n\n\n\nNote that this can also be done at cluster creation.<\/p>\n\n\n\n
Elasticsearch cluster<\/h2>\n\n\n\n
Connect to VM<\/h4>\n\n\n\n
Connect to proxy VM either with GCP interface in browser, or using gcloud.<\/p>\n\n\n\n
gcloud compute ssh es-cluster-proxy<\/pre>\n\n\n\nInstall tools<\/h4>\n\n\n\n
Install
kubectl<\/code>.<\/p>\n\n\n\nsudo apt-get install kubectl<\/pre>\n\n\n\nInstall helm<\/a>.<\/p>\n\n\n\n
$ curl https:\/\/raw.githubusercontent.com\/helm\/helm\/master\/scripts\/get-helm-3 > get_helm.sh\n$ chmod 700 get_helm.sh\n$ .\/get_helm.sh\n$ helm init<\/pre>\n\n\n\nAdd elastic helm charts repo.<\/p>\n\n\n\n
helm repo add elastic https:\/\/helm.elastic.co<\/pre>\n\n\n\nConnect to cluster<\/h4>\n\n\n\n
Get cluster credentials.<\/p>\n\n\n\n
gcloud container clusters get-credentials elasticsearch-cluster \\\n--zone ZONE --internal-ip<\/pre>\n\n\n\nPrepare configuration<\/h4>\n\n\n\n
Create a YAML file to personalize elasticsearch installation (Change PROJECT_ID).<\/p>\n\n\n\n
---\nservice:\n type: LoadBalancer\n annotations:\n cloud.google.com\/load-balancer-type: Internal\nreplicas: 2\n\nimage: \"gcr.io\/PROJECT_ID\/elasticsearch-gcs\"\nimageTag: \"7.5.0\"\nresources:\n requests:\n cpu: \"100m\"<\/pre>\n\n\n\nHere, we add an annotation so that an internal load balancer will be deployed to expose our cluster on the internal network. We also specify the image to be used, that is our internal image.<\/p>\n\n\n\n
Install elasticsearch<\/h4>\n\n\n\n
Install elasticsearch with helm.<\/p>\n\n\n\n
helm install --values .\/es-config.yml elasticsearch elastic\/elasticsearch<\/pre>\n\n\n\nCheck<\/h4>\n\n\n\n
In order to assess deployment, make a request to the IP of the internal load balancer.<\/p>\n\n\n\n
$ curl http:\/\/10.50.0.xxx:9200\n$ curl http:\/\/10.50.0.xxx:9200\/_cluster\/state?pretty<\/pre>\n\n\n\nUninstall<\/h4>\n\n\n\n
If for any reason, you need to uninstall elasticsearch from cluster, connect back to the VM and connect
kubectl<\/code> to cluster, then :<\/p>\n\n\n\nhelm uninstall elasticsearch<\/pre>\n\n\n\nNotes<\/h2>\n\n\n\n
# Get cluster state.\nkubectl get all\n\n# Get pods\nkubectl get pods\n \n# Check a pod\nkubectl describe pod <pod-name>\n \n# List gcp images\ngcloud container images list\n \n# Configure password\nkubectl create secret generic elastic-credentials --from-literal=password=test --from-literal=username=elastic\n\n# Delete secrets\nkubectl delete secrets elastic-credentials<\/pre>\n\n\n\nSources<\/h2>\n\n\n\n
This is the result of aggregating multiple sources of documentation – listed right below – it is adviced to go through each link to get a better understanding of the whole process.<\/p>\n\n\n\n
- Creating GKE private clusters with network proxies for master access<\/a><\/li>
- Creating a private cluster with no client access to the public endpoint<\/a><\/li>
- Generating a kubeconfig entry using a private cluster’s internal IP address<\/a><\/li>
- Creating custom images<\/a><\/li>
- Running and Deploying Elasticsearch on Kubernetes<\/a><\/li>
- Elasticsearch Helm Chart<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
In this article, I will list the steps needed to deploy an Elasticsearch cluster on a private Google Cloud Platform (GCP) Kubernetes cluster using Helm, from creating a docker image with Elasticsearch, to the creation of a private Kubernetes cluster and more. Let’s begin.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[31],"tags":[37,500,499,502,501],"class_list":["post-4040","post","type-post","status-publish","format-standard","hentry","category-installation","tag-elasticsearch","tag-google-cloud-platform","tag-helm","tag-kubernetes","tag-private-cluster"],"_links":{"self":[{"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/posts\/4040","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4040"}],"version-history":[{"count":11,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/posts\/4040\/revisions"}],"predecessor-version":[{"id":4058,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/posts\/4040\/revisions\/4058"}],"wp:attachment":[{"href":"https:\/\/www.unicoda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4040"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4040"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4040"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}